Penn State’s ‘Cheat-a-Thon’ Highlights AI’s Strengths and Weaknesses
Understanding AI’s Impact on Higher Education
As artificial intelligence (AI) continues to advance, it poses significant questions for higher education: How are students utilizing these AI tools? What changes should instructors make to their courses regarding AI usage? How can educational institutions keep pace with these rapidly evolving technologies?
Penn State’s Innovative Approach
Penn State University recently launched an initiative through its Center for Socially Responsible AI to explore these questions. This initiative included the first-ever "Cheat-a-thon," where faculty were invited to submit challenging questions designed to be difficult for AI to answer. Faculty members were eager to discover how AI could influence academic integrity and student performance.
Ihab Ragai, an engineering professor and the competition’s winner, found that AI performed poorly on his exams, achieving only 20 to 40 percent accuracy. This inspired him to submit his exam questions to the contest.
Students participated by using their chosen AI tools to tackle the questions. They were rewarded for both correct answers and innovative queries from faculty members. The competition featured 66 questions across 17 subjects, generating 451 responses from students. Participants learned about the capabilities of AI tools and recognized the importance of being trained in how to use them ethically. In particular, certain subjects, like engineering, and visual question types posed greater challenges for AI.
How Students Engage with AI
The Cheat-a-thon awarded cash prizes to ten students, and insights from four participants were shared by Government Technology. Most students employed straightforward methods: they copied the prompts into large language models (LLMs) like ChatGPT to check for answers. Surprisingly, many of the AI’s initial responses were deemed reasonable, with students estimating around half being acceptable.
Kyle Ketterer, a senior in computer science who secured first place, relied on his foundational knowledge to verify AI-generated responses. For uncertainty, he compared outputs from multiple models as a sort of cross-checking method. David Zhu, a freshman studying AI, took a creative approach by inputting text and screenshots into two models and would refine answers based on their discrepancies.
Bryan Shabroski, another first-place winner, automated his process with a program that extracted questions from PDFs and solicited answers from the AI. He crafted his program in just five minutes, completing the questions within five hours, needing to answer only five manually. This competition highlighted the need for a balanced acknowledgement of AI in academia.
Key Factors Making Questions Difficult for AI
Among the five winning faculty members, three hailed from the engineering department. Nasibeh Zohrabi, one of the winners, shared her approach of incorporating open-ended and multistep problems, which encourage students to demonstrate their logical reasoning—a challenge for AI systems.
Students noted that questions requiring visualization and design are generally harder for AI to solve than straightforward conceptual questions. An example includes a well-known puzzle where AI struggled to find the solution because it could not "think outside the box."
The challenges AI faced often involved graphs, where it frequently misinterpreted numerical data unless the user provided explanations. This indicates AI excels primarily in straightforward, text-based tasks but lacks intuition in complex problem-solving scenarios.
The Importance of a Positive Approach
Students and faculty both emphasized the necessity of a supportive environment to discuss the responsible use of AI tools. The monetary rewards from competitions like the Cheat-a-thon motivate students while reinforcing the value of ethical conduct. Each of the students interviewed attempted all 66 questions, demonstrating their dedication and engagement with the subject matter.
Eric Hudson, a physics professor and contest winner, remarked that stimulating students’ enthusiasm for challenging assessments—even those aided by AI—can lead to meaningful learning experiences. An ongoing dialogue about AI’s role in education is crucial for preparing students for an AI-centric workforce while ensuring they retain critical thinking skills.
