ChatGPT performed nearly on par with real students on university-level health policy exams, though not without stumbles, a recent study from George Mason University found.
Led by Sanja Avramovic of the College of Public Health and Ivan Avramovic of the College of Engineering and Computing, the research explored the potential of the AI tool as both a learning aid and a challenge to academic integrity within higher education.
“We wanted to understand what kinds of questions the tool can handle, and where it falls short,” said Sanja Avramovic, an associate professor of Health Administration and Policy. “That may help us design better teaching strategies and smarter assessments.”
The researchers put ChatGPT through undergraduate, master’s, and PhD-level healthcare database exams, grading it with the same rubrics applied to human students. ChatGPT passed all three exams, handling SQL-based coding questions with ease and earning an 83.3% on a doctoral-level test. But when questions demanded nuance or contextual reasoning, like interpreting patient records across multiple visits, the tool faltered.
The Avramovics’ study won the 2024 Best Paper Award from the Journal of Health Administration Education. They will accept the honor this week at the annual meeting of the Association of University Programs in Health Administration (AUPHA).
Their findings suggest that while ChatGPT can supplement learning by explaining code, checking logic, or generating alternative solutions, it often can’t yet replicate the critical thinking or contextual awareness students need to master more complex material.
“The current version of AI can answer many kinds of questions,” the researchers conclude, “but that doesn’t mean it understands them.”
Read the full study: Exploring the Potential Benefits and Limitations of Using an AI Text-Generation Tool in Education: An Examination of ChatGPT’s Performance on Assessments