Test Development and Administration

The Utility of Chat GPT as an Item Writing Assistant

Thursday, October 26, 2023

2:30 PM – 3:30 PM MT

Location: Broadmoor Hall F

Main Contact/Presenter(s)

Paul Jones

Senior Research Scientist
Pearson VUE, United States

Presenter(s)

Kirk A. Becker, PhD

Senior Research Scientist
Pearson VUE
Chicago, Illinois, United States
Matt Ferris

Senior Director of Testing
National Board for Certification and Recertification for Nurse Anesthetists, United States

How good of an item writing assistant is ChatGPT? In this presentation we report on the characteristics of whole items generated in response to templated queries that include content area, cognitive level, item format, difficulty, and other characteristics. The AI output is assessed by panels of human judges in terms of item quality, cognitive level matching, target difficulty matching, and content relevance and redundancy as items are produced at scale. Blind comparison with human-authored questions written to the same objectives will investigated.

Thank you Excelsoft for sponsoring the Test Development and Administration track.

Learning Objectives:

Gain insight in the capacity of LLMs to generate items that are comparable in quality to human-created items that are awaiting pretesting.
Gain insight into the ability of LLMs to modulate the difficulty of items when they are produced in sequence.
Gain insight into the ability of LLMs to cover relevant parts of a domain when generating test items at scale, and the resulting levels of redundancy that might occur.