Homepage | Dr. Neal Kingston

Neal Kingston, Director of the Achievement & Assessment Institute (AAI) and University Distinguished Professor of Educational Psychology
Curriculum Vitae (pdf)

University Distinguished Professor of Educational Psychology
Director, Achievement and Assessment Institute (AAI)

nkingsto@ku.edu
785-864-9705

The Achievement & Assessment Institute
1515 St. Andrews Rd
Lawrence, KS 66047

Neal Kingston, Ph.D., is a University Distinguished Professor in the department of Educational Psychology at the University of Kansas, in which he also serves as the Director of Graduate Studies and Director of the Achievement and Assessment Institute (AAI). His research focuses on large-scale assessment, with particular emphasis on how it can better support student learning through the use of learning maps and diagnostic classification models. He has served as principal investigator or co-principal investigator for over 180 research grants. Of particular note was the Dynamic Learning Maps Alternate Assessment grant from the US Department of Education, which was at that time was the largest grant in KU history and which currently serves 21 state departments of education. Other recent testing projects include the Kansas Assessment Program, Career Pathways Collaborative, and Adaptive Reading Motivation Measures.

Dr. Kingston is known internationally for his work on large-scale assessment, formative assessment, and learning maps. He has served as a consultant or advisor for organizations such as the AT&T, College Board, Department of Defense Advisory Committee on Military Personnel Testing, Edvantia, General Equivalency Diploma (GED), Kaplan, King Fahd University of Petroleum and Minerals, Merrill Lynch, National Council on Disability, Qeyas (Saudi Arabian National Center for Assessment in Higher Education), the state of New Hampshire, the state of Utah, the U.S. Department of Education, and Western Governors University.

As Director of AAI, Dr. Kingston is responsible for multiple research centers with about 300 year round staff and about 150 temporary employees.

Education

Ph.D. Educational Measurement, Teachers College, Columbia University, New York, NY, 1983
M.Phil. Educational Measurement, Teachers College, Columbia University, New York, NY, 1983
M.Ed. Educational Measurement, Teachers College, Columbia University, New York, NY, 1978
M.A. Psychology in Education, Teachers College, Columbia University, New York, NY, 1977
B.A. Liberal Studies (concentrations in Biology & Education), State University of New York, Stony Brook, NY, 1974

Currently, Dr. Kingston teaches a two semester sequence, Measurement Theory and Practice in years when the fall semester is an even numbered year. This sequence integrates classical and modern test theory and applications.

EPSY 921 Measurement Theory and Practice I

Historical overview of educational and psychological measurement
Score interpretation approaches (normative, standards-based, ipsative)
Validity theory
Evidence centered design
Classical test theory
Item response theory
Item analysis (classical and IRT)
Reliability and generalizability (classical and IRT)

EPSY 922 Measurement Theory and Practice II

Test development strategies
Scaling and reporting (total scores and subscores)
Standard setting
Equating
Item bias
Test bias
Accessibility and accommodations
Test administration approaches (linear, adaptive)
Test security

In the fall of odd numbered years Dr. Kingston teaches a seminar on a to-be-announced topic and in the spring of even numbered years he teaches meta-analysis.

The field of education moves slowly. Theoretical improvements often take 30-50 years before wide-spread implementation. Often research-based practices are crowded out by the fad of the day, making the challenge of improving education even greater. Sub-disciplines within education far too often work in isolation. Simple solutions that do not address the complexity of individual students or the dynamics of a classroom at best have little impact and too often have a negative impact. Such has been the case in large-scale assessment where the use of assessment to drive curriculum and instruction has had numerous negative consequences.

Coming to the University of Kansas gave me the opportunity to consider the fundamental issues of education from a broader perspective. As have many others before me, I had long realized that thinking about curriculum, instruction, and assessment needs to be integrated. However, few researchers attempted to develop models or theories to do this. I was impressed by the efforts of some, particularly the research trajectories of Susan Embretson and Kikumi Tatsuoka, but I remained frustrated regarding how incomplete this work was and how little impact it was having on federally mandated state assessment programs. This led me to develop three conference presentations in 2009 that served to focus my thinking. The first was presented at the National Council on Measurement in Education and was entitled, "What Have We Learned about the Structure of Learning from 30 Years of Research on Integrated Cognitive-Psychometric Models? Not Much." The second was presented at the American Educational Research Association Conference – The Efficacy of Formative Assessment: A Meta-Analysis. The third, presented at the National Conference on Student Assessment, was entitled, "Large-Scale Formative Assessment: Panacea, Transitional Tool, or Oxymoron."

In 2010 an opportunity presented itself and allowed me to solidify my thinking. The US Department of Education issued a request for proposals to develop a large-scale assessment system for students with significant cognitive disabilities –the approximately one percent of students with the greatest learning challenges. It was clear to me that such an assessment system needed to do far more than measure learning – it needed to facilitate learning. I identified six features that needed to be present to do this. They are as follows:

  1. Comprehensive fine-grained learning maps that guide instruction and assessment
  2. A subset of particularly important nodes that serve as content standards to provide an organizational structure for teachers
  3. Instructionally embedded assessments that reinforce the primacy of instruction
  4. Instructionally relevant testlets that model good instruction and reinforce learning
  5. Accessibility by design
  6. Status and growth reporting that is readily actionable

No one had ever tried to develop a learning environment in this way. Comprehensive fine-grained learning maps did not exist. The concept of instructionally relevant assessment previously was unnamed and in its infancy. Clearly much research – both basic and applied – was necessary and this has become the focus of my research. A closely related second area of research is assessment that supports the needs of learners who face educational or assessment challenges. This includes issues of test development and universal design and which have close ties to features 3-5 in the list above. I separate it as a research focus because it is also applicable to traditional testing programs.

Large-scale assessment
Computer-based testing
Diagnostic classification modelling
Learning maps
Test development
Score reporting
Assessment of students with significant cognitive disabilities
Assessment in higher education

The following are a few categories of recent publications (publications may appear in more than one list):

Assessments that support learning

Heritage, M. & Kingston, N.M. (2019). Classroom assessment and large-scale psychometrics: shall the twain meet? (a conversation with Margaret Heritage and Neal Kingston). Journal of Educational Measurement, 56(4), 670-685.
Clark, A., Nash, B. Karvonen, M., & Kingston, N.M. (2017). Condensed Mastery Profile Method for Setting Standards for Diagnostic Assessment Systems. Educational Measurement: Issues and Practice. 36(4), 5–15.
Kingston, N.M., Karvonen, M., Thompson, J.R., Wehmeyer, M.L., & Shogren, K.A. (2017). Fostering Inclusion of Students with Significant Cognitive Disabilities through the use of Learning Maps and Learning Map Based Assessments. Inclusion, 5(2), 110-120.
Kingston, N.M. & Broaddus, A. (2017). The Use of Learning Map Systems to Support Formative Assessment in Mathematics. Education Sciences, 7 (41); doi:10.3390/educsci7010041.
Kingston, N.M., Karvonen, M., Bechard, S., & Erickson, K. (2016). The Philosophical Underpinnings and Key Features of the Dynamic Learning Maps Alternate Assessment. Teachers College Record (Yearbook), 118(14). Retrieved September 1, 2016, from http://www.tcrecord.org ID Number: 140311.
Popham, W. J., Berliner, D.C., Kingston, N., Fuhrman, S.H., Ladd, S.M., Charbonneau, J. & Chatterji, M. (2014). Can today's standardized tests yield instructionally useful data? Challenges, promises and the state of the art, Quality Assurance in Education, 22(4), 300-316.
Bechard, S., Clark, A. K., Swinburne Romine, R., Karvonen, M., Kingston, N.M., & Erickson, K. (2019). Use of evidence-centered design to develop learning maps-based assessments. International Journal of Testing, 19:2, 188-205.

Students who face education or assessment challenges

Davis, M. H., Wang, W., Kingston, N., Hock, M., Tonks, S. M., & Tiemann, G. (2020). Computer Adaptive Measure of Reading Motivation. Research in Reading, 43(4), 434-453.
Kingston, N.M., Karvonen, M., Thompson, J.R., Wehmeyer, M.L., & Shogren, K.A. (2017). Fostering Inclusion of Students with Significant Cognitive Disabilities through the use of Learning Maps and Learning Map Based Assessments. Inclusion, 5(2), 110-120.
Kingston, N.M., Karvonen, M., Bechard, S., & Erickson, K. (2016). The Philosophical Underpinnings and Key Features of the Dynamic Learning Maps Alternate Assessment. Teachers College Record (Yearbook), 118(14). Retrieved September 1, 2016, from http://www.tcrecord.org ID Number: 140311.
Cho, H. & Kingston, N.M. (2013). Why IEP Teams Assign Low Performers with Mild Disabilities to the Alternate Assessment Based on Alternate Achievement Standards. Journal of Special Education, 47, 162-174.
Cho, H., Wehmeyer, M. & Kingston, N.M. (2013). Factors that Predict Elementary Educators’ Perceptions and Practice in Teaching Self-Determination. Psychology in the Schools, 50: 770-780.

Psychometric methods

Wang, W., Chen, J., & Kingston, N. (2020). How well do simulation studies inform decisions about multistage testing? Journal of Applied Measurement, 21(3), 1-11.
Pan, Q., Qin, L., & Kingston, N. (2020). Growth Modeling in a Diagnostic Classification Model (DCM) Framework–A Multivariate Longitudinal Diagnostic Classification Model.
Wang, W. & Kingston, N.M. (2020). Using Bayesian Nonparametric Item Response Functions to Check Parametric Model Fit. Applied Psychological Measurement.
Wang, W, & Kingston, N.M. (2019). Adaptive testing with the Hierarchical Item Response Theory Model. Applied Psychological Measurement, 43(1), 51-67.
Embretson, S.E. & Kingston, N.M. (2018). Automatic Item Generation: A More Efficient Process for Developing Mathematics Achievement Items? Journal of Educational Measurement. 55(1), 112-131.
Adjei, S., Selent, D., Heffernan, N., Pardos, Z., Broaddus, A., Kingston, N. (2014). Refining Learning Maps with Data Fitting Techniques: Searching for Better Fitting Learning Maps. In Pardos & Stamper (Eds.) The 2014 Proceedings of International Educational Data Mining Society.
Gu, F., Little, T., & Kingston, N.M. (2013). Misestimation of Reliability Using Coefficient Alpha and Structural Equation Modeling when Assumptions of Tau-Equivalence and Uncorrelated Errors are Violated. Methodology, 9, 30-40.

Erkan Hasan ATALMIS

I am associate professsor in the Measurement and Evaluation in Education program at Kahramanmaras Sutcu Imam University (KSU) in Turkey. Prior to coming to KSU, I earned PhD degree in Research, Evaluation, Measurement, and Statistics Program from University of Kansas, USA. My research interests include item and test development, type of grading system in educational assessment in higher education, supplementary education, structural equation modeling, and meta-analysis. My current research interest focuses primarily on examination of the quality of multiple-choice items on classroom tests in higher education applying classical test theory and hierarchical linear modeling.

Jie Chen

I am a psychometrician with Center for Accessible Teaching, Learning, and Assessment Systems at the University of Kansas. Prior to this current position, I was a program development associate at ACT, Inc., supporting content teams in item development. I have over 10 years of experience working with large-scale assessments, analyzing student data, equating and scaling test forms, calibrating items, and providing support to content development and item pool construction. My current research interest focuses primarily on validity issues, particularly around predictive and concurrent sources of validity evidence and validating instruments measuring Montessori fidelity.

Amy Clark

I started my career as a classroom teacher and became interested in the use of assessment data to inform instruction, which led me to KU. I completed my MS and PhD in the Research, Evaluation, Measurement, and Statistics (REMS) track while working as a GRA on the Kansas Assessment Program and Dynamic Learning Maps projects. I am now Associate Director for Operational Research at ATLAS at KU. My research interests focus on assessment validation, score reporting, and the operational use of instructionally embedded and cognitive diagnostic assessments.

Fei Gu

After graduating from KU in 2013, and my first position was an assistant professor in quantitative psychology at McGill University. Currently, I am an assistant professor in the Educational Research and Evaluation (EDRE) program in the School of Education at Virginia Tech. The EDRE program is essentially a parallel counterpart of the REMS program at KU, and I feel very fortunate to be able to continue my work on the statistical issues of my interests. My research interest includes latent variable models, techniques of multivariate analysis, and statistical inference of rotated results. My scholarly work has been published in peer-reviewed journals, including Applied Psychological Measurement, Behavior Research Methods, Behaviormetrika, British Journal of Mathematical and Statistical Psychology, Journal of Educational and Behavioral Statistics, Methodology, Multivariate Behavioral Research, Psychometrika, and Structural Equation Modeling.

Hongling Lao

After getting a bachelor’s degree in psychology, I continued my MS and PhD study in the field of Research, Evaluation, Measurement, and Statistics (REMS) at the University of Kansas. After graduation in 2017, I expand my research focus from statistical modelling of large-scale assessment (macro) to formative assessment in everyday classroom (micro), leading to my current position as a postdoctoral fellow at The Education University of Hong Kong. Apart from (or as an alternative way of) doing research, I am passionate in school support to improve learning and teaching via in-depth collaboration.

Qianqian Pan

I am currently a Post-doc research fellow at the University of Hong Kong. I received training in Research, Evaluation, Measurement, Statistics under the direction of Dr. Neal Kingston at the University of Kansas and received my Ph.D. in 2018. Before attending the University of Kansas, I received my bachelor’s (2011) and master’s degree with Honors (2014) in Special Education from East China Normal University in Shanghai, China. At KU, I worked for the Dynamic Learning Maps® (DLM®) alternate assessment program as a graduate research assistant and conducted psychometric analyses. My current research interests include applying diagnostic classification models and conducting longitudinal data analysis and multilevel data analysis to better understand and improve student learning.

Jake Thompson

I completed my PhD in the Research, Evaluation, Measurement, and Statistics (REMS) program at KU in 2018. During my time in the program, I worked as a graduate research assistant on the Dynamic Learning Maps Alternate Assessment. I am now a senior psychometrician at Accessible Teaching, Learning, and Assessment Systems (ATLAS), a research center with the Achievement and Assessment Institute at KU, where I continue to work on the Dynamic Learning Maps assessment, and other grant-funded projects. My research interests focus on the theory and applications of diagnostic classification models, Bayesian inference, and R programming. I am also interested in data visualization techniques and have had multiple graphics featured on the cover of Educational Measurement: Issues and Practice. You can follow my work on my website at wjakethompson.com.

Gail Tiemann

Currently, I am a research project manager at Accessible Teaching, Learning, and Assessment Systems (ATLAS) at KU. I primarily work with research and development projects in computer-based testing, including serving as co-principal investigator of our Innovations in Science Map, Assessment, and Report Technologies (I-SMART) project. I've worked on many different education research projects in my 19 years of service to KU, including those in the areas of operational K-12 assessment, assessment research and development, large-scale computer-based testing systems, professional learning, educational technology, and special education. I earned a doctorate in educational psychology from the University of Kansas, specializing in research, evaluation, measurement, and statistics. Prior to coming to KU, I earned a master's degree in social work from the University of Wisconsin-Madison with a concentration in developmental disabilities, served as a service coordinator for an infant-toddler early intervention program, and as a state trainer for a child abuse prevention program.

Wenhao Wang

I am currently a senior psychometrician with Center for Accessible Teaching, Learning, and Assessment Systems (ATLAS) at the University of Kansas. After graduating in 2012, I remained at KU and worked as an operational psychometrician. I have extensive experience as a psychometrician working with state and national large-scale assessment programs. My research interests include item-model fit, computer adaptive testing, and advanced psychometric model fit comparison. My scholarly work has been published in peer-reviewed journals, including Applied Psychological Measurement, Journal of Applied Measurement, and Journal of Reading Research.

Chunmei (Rose) Zheng

I was a graduate student in the Research, Evaluation, and Statistics (REMS) program at KU between 2008 and 2013. After graduation, I worked at Pearson-Always Learning for 6 years and I am currently a state psychometrician at New York State Department of Education. Operational work and statistical research studies are my routines and I have enjoyed it very much. Besides psychometrics, I love traveling, watching movies, walking in a trail, and eating seafood.

Former Students Who Did Not Submit Pictures

Jessica Loughran
Fei Zhao

Jessica Hess

I am a doctoral candidate in the Educational Psychology and Research program with an emphasis on Research, Evaluation, Measurement, and Statistics. Prior to coming to KU, I received a Bachelor of Arts degree in Psychology from Bethel University in Arden Hills, MN. Originally intending to use my degree in psychology to become a school counselor, I decided to pursue a PhD in measurement and statistics after realizing that I actually enjoyed statistics but still wanted to have an education focus. My current research interest focuses primarily on accounting for response styles in survey data through the development and use of polytomous IRT models.

Merve Akin Tas

After receiving my BA in Mathematics Education in Eskisehir, Turkey, I started my career as a math teacher in 2011. I received my master’s degree in Educational Psychology at Texas Tech University in 2017, where I also had more than 2 years of work experience in federally supported projects. I came to the University of Kansas in 2018 to pursue my PhD in the Research, Evaluation, Measurement and Statistics track of the Educational Psychology and Research Program. I have been working at KU Center for Public Partnerships and Research since 2018. I am responsible for quantitative data analysis and program evaluation in various projects to support collective efforts to optimize well-being of at-risk children, youth and families. My research interests are large-scale assessments, advanced quantitative methods, application of data sciences and evaluation methodologies in health care and early childhood education.

Haoyang Yu

Fascinated by liberal arts curriculum in the United States, I first came to Pittsburgh from China for undergraduate studies in 2011. Also a believer in the power of education, I received a master’s degree from Teachers College, Columbia University. After working for several years with students and schools in North Carolina, Massachusetts, and briefly back home in China, I chose to pursue the Ph.D. studies in Research, Evaluation, Measurement and Statistics (REMS) in the department of Educational Psychology and Research at KU in 2021. I am currently a Graduate Research Assistant for the Research and Psychometrics team at Accessible Teaching, Learning, and Assessment Systems (ATLAS), a research center within the Achievement and Assessment Institute (AAI) at KU. My current research interests are large-scale assessment, learning maps, and meta-analysis of educational practices.

I am seeking one or two new doctoral students (previous master's degree not required) to start in Fall 2022. Students will receive a Graduate Research Assistant (GRA) position that covers tuition and fees and also provides a salary. An interest in learning and applying test theory to better support student learning is of critical importance. Prospective students can contact me at nkingsto@ku.edu for more information.

A link to the Achievement and Assessment Institute (AAI) graduate student handbook is included below for your reference:

2021 AAI GRA Handbook