Kylie Anglin, Assistant Professor
Research Methods, Measurement, and Evaluation
Department of Educational Psychology
University of Connecticut
Improving the Alignment between Human and Large Language Model Classifications: An Empirical Assessment of Prompt Engineering for
Construct Identification
Abstract: Due to their vast pre-training data, large language models (LLMs) demonstrate strong text classification performance. However, LLM output—here, the category assigned to a text—depends heavily on the wording of the prompt. While literature on prompt engineering is expanding, few studies focus on classification tasks, and even fewer address domains such as psychology, where constructs have precise, theory-driven definitions that may not be well represented in pre-training data. This presentation introduces an empirical framework for optimizing LLM performance in classifying psychological constructs using prompt engineering. Five strategies (codebook-guided empirical prompt selection, automatic prompt engineering, persona prompting, chain-of-thought reasoning, and explanatory prompting) are systematically evaluated under zero- and few-shot settings across three constructs, four experimental samples, and two LLMs (GPT-4 and LLaMA 3.3). Analyses encompass thousands of performance evaluations, generating insights into robust patterns of performance and informing evidence-based recommendations for applied text classification with LLMs.
Bio: KYLIE ANGLIN is an assistant professor in research methods, measurement, and evaluation at the University of Connecticut. Her research improves the quality and depth of educational program evaluations, most often through the rigorous application of artificial intelligence (AI).
DATE: Friday, November 7, 2025, 1:00 PM, AUST 344
WebEx link:
https://uconn-cmr.webex.com/uconn-cmr/j.php?MTID=m378286d80716e4543fc1be52af72abde
Coffee will be available at 12:30 PM in the Noether Lounge (AUST 326)
For more information, contact: Tracy Burke at tracy.burke@uconn.edu