Currently, Dr. Kingston teaches a course in Classical Test Theory in the fall of odd numbered years, Item Response Theory in the spring of even numbered years, a course on meta-analysis in the fall of even numbered years, and an advanced seminar in a topic to be announced in the spring of odd numbered years.

The field of education moves slowly. Theoretical improvements often take 30-50 years before wide-spread implementation. Often research-based practices are crowded out by the fad of the day, making the challenge of improving education even greater. Sub-disciplines within education far too often work in isolation. Simple solutions that do not address the complexity of individual students or the dynamics of a classroom at best have little impact and too often have a negative impact. Such has been the case in large-scale assessment where the use of assessment to drive curriculum and instruction has had numerous negative consequences.

Coming to the University of Kansas gave me the opportunity to consider the fundamental issues of education from a broader perspective. As have many others before me, I had long realized that thinking about curriculum, instruction, and assessment needs to be integrated. However, few researchers attempted to develop models or theories to do this. I was impressed by the efforts of some, particularly the research trajectories of Susan Embretson and Kikumi Tatsuoka, but I remained frustrated regarding how incomplete this work was and how little impact it was having on federally mandated state assessment programs. This led me to develop three conference presentations in 2009 that served to focus my thinking. The first was presented at the National Council on Measurement in Education and was entitled, "What Have We Learned about the Structure of Learning from 30 Years of Research on Integrated Cognitive-Psychometric Models? Not Much." The second was presented at the American Educational Research Association Conference – The Efficacy of Formative Assessment: A Meta-Analysis. The third, presented at the National Conference on Student Assessment, was entitled, "Large-Scale Formative Assessment: Panacea, Transitional Tool, or Oxymoron."

In 2010 an opportunity presented itself and allowed me to solidify my thinking. The US Department of Education issued a request for proposals to develop a large-scale assessment system for students with significant cognitive disabilities –the approximately one percent of students with the greatest learning challenges. It was clear to me that such an assessment system needed to do far more than measure learning – it needed to facilitate learning. I identified six features that needed to be present to do this. They are as follows:

  1. Comprehensive fine-grained learning maps that guide instruction and assessment
  2. A subset of particularly important nodes that serve as content standards to provide an organizational structure for teachers
  3. Instructionally embedded assessments that reinforce the primacy of instruction
  4. Instructionally relevant testlets that model good instruction and reinforce learning
  5. Accessibility by design
  6. Status and growth reporting that is readily actionable

No one had ever tried to develop a learning environment in this way. Comprehensive fine-grained learning maps did not exist. The concept of instructionally relevant assessment previously was unnamed and in its infancy. Clearly much research – both basic and applied – was necessary and this has become the focus of my research. A closely related second area of research is assessment that supports the needs of learners who face educational or assessment challenges. This includes issues of test development and universal design and which have close ties to features 3-5 in the list above. I separate it as a research focus because it is also applicable to traditional testing programs.

• Large-scale assessment
• Computer-based testing
• Diagnostic classification modelling
• Learning maps
• Test development
• Score reporting
• Assessment of students with significant cognitive disabilities
• Assessment in higher education

The following are a few categories of recent publications (publications may appear in more than one list):

Assessments that support learning

1. Heritage, M. & Kingston, N.M. (2019). Classroom assessment and large-scale psychometrics: shall the twain meet? (a conversation with Margaret Heritage and Neal Kingston). Journal of Educational Measurement, 56(4), 670-685.
2. Clark, A., Nash, B. Karvonen, M., & Kingston, N.M. (2017). Condensed Mastery Profile Method for Setting Standards for Diagnostic Assessment Systems. Educational Measurement: Issues and Practice. 36(4), 5–15.
3. Kingston, N.M., Karvonen, M., Thompson, J.R., Wehmeyer, M.L., & Shogren, K.A. (2017). Fostering Inclusion of Students with Significant Cognitive Disabilities through the use of Learning Maps and Learning Map Based Assessments. Inclusion, 5(2), 110-120.
4. Kingston, N.M. & Broaddus, A. (2017). The Use of Learning Map Systems to Support Formative Assessment in Mathematics. Education Sciences, 7 (41); doi:10.3390/educsci7010041.
5. Kingston, N.M., Karvonen, M., Bechard, S., & Erickson, K. (2016). The Philosophical Underpinnings and Key Features of the Dynamic Learning Maps Alternate Assessment. Teachers College Record (Yearbook), 118(14). Retrieved September 1, 2016, from http://www.tcrecord.org ID Number: 140311.
6. Popham, W. J., Berliner, D.C., Kingston, N., Fuhrman, S.H., Ladd, S.M., Charbonneau, J. & Chatterji, M. (2014). Can today's standardized tests yield instructionally useful data? Challenges, promises and the state of the art, Quality Assurance in Education, 22(4), 300-316.
7. Bechard, S., Clark, A. K., Swinburne Romine, R., Karvonen, M., Kingston, N.M., & Erickson, K. (2019). Use of evidence-centered design to develop learning maps-based assessments. International Journal of Testing, 19:2, 188-205.

Students who face education or assessment challenges

1. Davis, M. H., Wang, W., Kingston, N., Hock, M., Tonks, S. M., & Tiemann, G. (2020). Computer Adaptive Measure of Reading Motivation. Research in Reading, 43(4), 434-453.
2. Kingston, N.M., Karvonen, M., Thompson, J.R., Wehmeyer, M.L., & Shogren, K.A. (2017). Fostering Inclusion of Students with Significant Cognitive Disabilities through the use of Learning Maps and Learning Map Based Assessments. Inclusion, 5(2), 110-120.
3. Kingston, N.M., Karvonen, M., Bechard, S., & Erickson, K. (2016). The Philosophical Underpinnings and Key Features of the Dynamic Learning Maps Alternate Assessment. Teachers College Record (Yearbook), 118(14). Retrieved September 1, 2016, from http://www.tcrecord.org ID Number: 140311.
4. Cho, H. & Kingston, N.M. (2013). Why IEP Teams Assign Low Performers with Mild Disabilities to the Alternate Assessment Based on Alternate Achievement Standards. Journal of Special Education, 47, 162-174.
5. Cho, H., Wehmeyer, M. & Kingston, N.M. (2013). Factors that Predict Elementary Educators’ Perceptions and Practice in Teaching Self-Determination. Psychology in the Schools, 50: 770-780.

Psychometric methods

1. Wang, W., Chen, J., & Kingston, N. (2020). How well do simulation studies inform decisions about multistage testing? Journal of Applied Measurement, 21(3), 1-11.
2. Pan, Q., Qin, L., & Kingston, N. (2020). Growth Modeling in a Diagnostic Classification Model (DCM) Framework–A Multivariate Longitudinal Diagnostic Classification Model.
3. Wang, W. & Kingston, N.M. (2020). Using Bayesian Nonparametric Item Response Functions to Check Parametric Model Fit. Applied Psychological Measurement.
4. Wang, W, & Kingston, N.M. (2019). Adaptive testing with the Hierarchical Item Response Theory Model. Applied Psychological Measurement, 43(1), 51-67.
5. Embretson, S.E. & Kingston, N.M. (2018). Automatic Item Generation: A More Efficient Process for Developing Mathematics Achievement Items? Journal of Educational Measurement. 55(1), 112-131.
6. Adjei, S., Selent, D., Heffernan, N., Pardos, Z., Broaddus, A., Kingston, N. (2014). Refining Learning Maps with Data Fitting Techniques: Searching for Better Fitting Learning Maps. In Pardos & Stamper (Eds.) The 2014 Proceedings of International Educational Data Mining Society.
7. Gu, F., Little, T., & Kingston, N.M. (2013). Misestimation of Reliability Using Coefficient Alpha and Structural Equation Modeling when Assumptions of Tau-Equivalence and Uncorrelated Errors are Violated. Methodology, 9, 30-40.

Erkan Hasan ATALMIS
I am associate professsor in the Measurement and Evaluation in Education program at Kahramanmaras Sutcu Imam University (KSU) in Turkey. Prior to coming to KSU, I earned PhD degree in Research, Evaluation, Measurement, and Statistics Program from University of Kansas, USA. My research interests include item and test development, type of grading system in educational assessment in higher education, supplementary education, structural equation modeling, and meta-analysis. My current research interest focuses primarily on examination of the quality of multiple-choice items on classroom tests in higher education applying classical test theory and hierarchical linear modeling.

Jie Chen
I am a psychometrician with Center for Accessible Teaching, Learning, and Assessment Systems at the University of Kansas. Prior to this current position, I was a program development associate at ACT, Inc., supporting content teams in item development. I have over 10 years of experience working with large-scale assessments, analyzing student data, equating and scaling test forms, calibrating items, and providing support to content development and item pool construction. My current research interest focuses primarily on validity issues, particularly around predictive and concurrent sources of validity evidence and validating instruments measuring Montessori fidelity.

Amy Clark
I started my career as a classroom teacher and became interested in the use of assessment data to inform instruction, which led me to KU. I completed my MS and PhD in the Research, Evaluation, Measurement, and Statistics (REMS) track while working as a GRA on the Kansas Assessment Program and Dynamic Learning Maps projects. I am now Associate Director for Operational Research at ATLAS at KU. My research interests focus on assessment validation, score reporting, and the operational use of instructionally embedded and cognitive diagnostic assessments.

Fei Gu
After graduating from KU in 2013, and my first position was an assistant professor in quantitative psychology at McGill University. Currently, I am an assistant professor in the Educational Research and Evaluation (EDRE) program in the School of Education at Virginia Tech. The EDRE program is essentially a parallel counterpart of the REMS program at KU, and I feel very fortunate to be able to continue my work on the statistical issues of my interests. My research interest includes latent variable models, techniques of multivariate analysis, and statistical inference of rotated results. My scholarly work has been published in peer-reviewed journals, including Applied Psychological Measurement, Behavior Research Methods, Behaviormetrika, British Journal of Mathematical and Statistical Psychology, Journal of Educational and Behavioral Statistics, Methodology, Multivariate Behavioral Research, Psychometrika, and Structural Equation Modeling.

Hongling Lao
After getting a bachelor’s degree in psychology, I continued my MS and PhD study in the field of Research, Evaluation, Measurement, and Statistics (REMS) at the University of Kansas. After graduation in 2017, I expand my research focus from statistical modelling of large-scale assessment (macro) to formative assessment in everyday classroom (micro), leading to my current position as a postdoctoral fellow at The Education University of Hong Kong. Apart from (or as an alternative way of) doing research, I am passionate in school support to improve learning and teaching via in-depth collaboration.

Qianqian Pan
I am currently a Post-doc research fellow at the University of Hong Kong. I received training in Research, Evaluation, Measurement, Statistics under the direction of Dr. Neal Kingston at the University of Kansas and received my Ph.D. in 2018. Before attending the University of Kansas, I received my bachelor’s (2011) and master’s degree with Honors (2014) in Special Education from East China Normal University in Shanghai, China. At KU, I worked for the Dynamic Learning Maps® (DLM®) alternate assessment program as a graduate research assistant and conducted psychometric analyses. My current research interests include applying diagnostic classification models and conducting longitudinal data analysis and multilevel data analysis to better understand and improve student learning.

Jake Thompson
I completed my PhD in the Research, Evaluation, Measurement, and Statistics (REMS) program at KU in 2018. During my time in the program, I worked as a graduate research assistant on the Dynamic Learning Maps Alternate Assessment. I am now a senior psychometrician at Accessible Teaching, Learning, and Assessment Systems (ATLAS), a research center with the Achievement and Assessment Institute at KU, where I continue to work on the Dynamic Learning Maps assessment, and other grant-funded projects. My research interests focus on the theory and applications of diagnostic classification models, Bayesian inference, and R programming. I am also interested in data visualization techniques and have had multiple graphics featured on the cover of Educational Measurement: Issues and Practice. You can follow my work on my website at wjakethompson.com.

Gail Tiemann
Currently, I am a research project manager at Accessible Teaching, Learning, and Assessment Systems (ATLAS) at KU. I primarily work with research and development projects in computer-based testing, including serving as co-principal investigator of our Innovations in Science Map, Assessment, and Report Technologies (I-SMART) project. I've worked on many different education research projects in my 19 years of service to KU, including those in the areas of operational K-12 assessment, assessment research and development, large-scale computer-based testing systems, professional learning, educational technology, and special education. I earned a doctorate in educational psychology from the University of Kansas, specializing in research, evaluation, measurement, and statistics. Prior to coming to KU, I earned a master's degree in social work from the University of Wisconsin-Madison with a concentration in developmental disabilities, served as a service coordinator for an infant-toddler early intervention program, and as a state trainer for a child abuse prevention program.

Wenhao Wang
I am currently a senior psychometrician with Center for Accessible Teaching, Learning, and Assessment Systems (ATLAS) at the University of Kansas. After graduating in 2012, I remained at KU and worked as an operational psychometrician. I have extensive experience as a psychometrician working with state and national large-scale assessment programs. My research interests include item-model fit, computer adaptive testing, and advanced psychometric model fit comparison. My scholarly work has been published in peer-reviewed journals, including Applied Psychological Measurement, Journal of Applied Measurement, and Journal of Reading Research.

Chunmei (Rose) Zheng
I was a graduate student in the Research, Evaluation, and Statistics (REMS) program at KU between 2008 and 2013. After graduation, I worked at Pearson-Always Learning for 6 years and I am currently a state psychometrician at New York State Department of Education. Operational work and statistical research studies are my routines and I have enjoyed it very much. Besides psychometrics, I love traveling, watching movies, walking in a trail, and eating seafood.

Former Students Who Did Not Submit Pictures

Jessica Hess
I am a doctoral candidate in the Educational Psychology and Research program with an emphasis on Research, Evaluation, Measurement, and Statistics. Prior to coming to KU, I received a Bachelor of Arts degree in Psychology from Bethel University in Arden Hills, MN. Originally intending to use my degree in psychology to become a school counselor, I decided to pursue a PhD in measurement and statistics after realizing that I actually enjoyed statistics but still wanted to have an education focus. My current research interest focuses primarily on accounting for response styles in survey data through the development and use of polytomous IRT models.

Merve Akin Tas
After receiving my BA in Mathematics Education in Eskisehir, Turkey, I started my career as a math teacher in 2011. I received my master’s degree in Educational Psychology at Texas Tech University in 2017, where I also had more than 2 years of work experience in federally supported projects. I came to the University of Kansas in 2018 to pursue my PhD in the Research, Evaluation, Measurement and Statistics track of the Educational Psychology and Research Program. I have been working at KU Center for Public Partnerships and Research since 2018. I am responsible for quantitative data analysis and program evaluation in various projects to support collective efforts to optimize well-being of at-risk children, youth and families. My research interests are large-scale assessments, advanced quantitative methods, application of data sciences and evaluation methodologies in health care and early childhood education.

I am seeking one or two new doctoral students (previous master's degree not required) to start in Fall 2022. Students will receive a Graduate Research Assistant (GRA) position that covers tuition and fees and also provides a salary. An interest in learning and applying test theory to better support student learning is of critical importance. Prospective students can contact me at nkingsto@ku.edu for more information.

A link to the Achievement and Assessment Institute (AAI) graduate student handbook is included below for your reference:

2021 AAI GRA Handbook