• Tsekouras, D., Belo, R., and Cornelius, P., Generative AI and Student Performance: Evidence from a Large-Scale Intervention.

  • The emergence of generative AI (GenAI) has the potential to disrupt education, yet its impact on student productivity and learning is so far mixed. This study investigates the impact of GenAI on student performance in a European business school’s Digital Business course. Utilizing data from two course editions (2305 students), one before and one after the introduction of GenAI, the study assesses how the integration of GenAI impacts student performance across multiple dimensions. We find an initial negative impact on critical dimensions (relevance, argumentations, evidence) despite the increase in writing quality. These negative impacts decrease in subsequent assignments, especially for lower performance students, suggesting GenAI’s role in leveling the academic playing field. We show how GenAI experience and depth of interaction with GenAI further improves student performance. The findings highlight the value of integrating GenAI and the need for reassessing education, ensuring GenAI complements rather than replaces learning.

• Lehmann, M., Cornelius, P.B. and Sting, F.J., 2024. AI Meets the Classroom: When Does ChatGPT Harm Learning?. arXiv preprint arXiv:2409.09047.

  • In this paper, we study how generative AI and specifically large language models (LLMs) impact learning in coding classes. We show across three studies that LLM usage can have positive and negative effects on learning outcomes. Using observational data from universitylevel programming courses, we establish such effects in the field.We replicate these findings in subsequent experimental studies, which closely resemble typical learning scenarios, to show causality. We find evidence for two contrasting mechanisms that determine the overall effect of LLM usage on learning. Students who use LLMs as personal tutors by conversing about the topic and asking for explanations benefit from usage. However, learning is impaired for students who excessively rely on LLMs to solve practice exercises for them and thus do not invest sufficient own mental effort. Those who never used LLMs before are particularly prone to such adverse behavior. Students without prior domain knowledge gain more from having access to LLMs. Finally, we show that the self-perceived benefits of using LLMs for learning exceed the actual benefits, potentially resulting in an overestimation of one’s own abilities. Overall, our findings show promising potential of LLMs as learning support, however also that students have to be very cautious of possible pitfalls.

• Tsekouras, D., and Kucukyilmaz, T., GenAI for feedback support.

  • This project tackles workload and grading efficiency challenges in large courses. High assessment volumes cause capacity issues, delayed feedback, and inconsistent grading quality, often relying on TAs lacking time and experience. The goal is to use AI to improve feedback speed and quality, offer personalized insights, and reduce reliance on TAs, achieving cost efficiency. Integrating AI into grading tasks will ensure timely, consistent feedback and create a more efficient educational environment. We will train AI to match TA feedback quality across various assessments and use APIs for multiple AI graders. Initial training will use historical assessments, refining AI through TA comparisons. The project will emphasize transparency, explainability, student reception, and data privacy and security.

• Tsekouras, D., and Kucukyilmaz, T., GenAI for developing adaptive course material.

  • The recent advances in AI technology enabled unforeseen amount of automation possibilities and it is adapted by many industries. We believe that these advances also offer opportunities for education. In this project, we aim to explore the opportunities of AI for a more inclusive and personalized education. The aim of the project is to focus on material preparation and delivery stages of educational activities and use generative AI for automated creation/conversion of course material tailored to each student’s preferences. In this way, we aim to enhance delivery while fostering inclusivity by allowing students to interact with the material in their own terms. The project aims to assess whether such a delivery (i) is more motivational for the students (ii) achieves better learning outcomes.

• Priante, A. and Tsekouras, D. Integrating Technology in Physical Classrooms: The Impact of Game-Based Response Systems on Student Learning Experience

  • This study examines the impact of game-based student response systems (GSRSs) on students’ learning experiences in face-to-face education. Building on technology-mediated learning and active learning, we demonstrate the positive impact of GSRS use on learning outcomes and learning processes (student motivation, concentration, and enjoyment) in a field experiment in a Dutch secondary school. Our study expands information systems research by showing the educational and social impact of technology integration in physical classrooms and its equalizing role in bridging the performance disparity between underperforming and overperforming students while promoting an inclusive learning environment for all students.

• Tsekouras, D. and Priante, A. Educational Experience through the Lens of Blended Learning: A Conjoint Analysis of Student and Teacher Preferences

  • The adoption of blended learning in educational institutions has been driven by the growth of educational technology applications and evolving student learning preferences. While prior research highlights the positive impact of blended learning on student outcomes, its effects on student experience and the effectiveness of specific components (student-teacher, student-student, and student-material interactions) remain unclear. In two conjoint experiments, we assessed student and teacher preferences for diverse course configurations with blended learning elements. We show that the introduction of blended learning has a positive effect on the educational experience of students and teachers, yet the effect is non-linear. We test the heterogeneity of these effects across different course, student, and teacher characteristics. Finally, we provide qualitative insights on student and teacher evaluations of blended learning. Based on our findings, we offer recommendations for implementing blended learning, taking into account student learning preferences and teacher workload.

• Zhang, J.; Zhu, R.; Li, T.; and Yi, C. Breaking Barriers: Enhancing Learning Outcomes for Deaf and Hard of Hearing Students through Augmented Reality Captioning

  • Deaf and Hard of Hearing (DHH) students often face significant challenges in information-rich environments like classrooms. This research explores the potential of augmented reality (AR) smart glasses equipped with real-time speech-to-text captioning to address these challenges by overlaying captions directly into the user’s field of view. Unlike prior studies that primarily focused on multimedia presentation aspects of AR, this study investigates the impact of AR-rendered captions on two key learning outcomes: increased learning confidence and improved performance. Additionally, the research examines how nuanced verbal communication and the reading abilities of DHH students moderate the relationship between AR usage and these outcomes. The ultimate goal is to develop a classroom-ready AR captioning system and evaluate its effectiveness through a randomized field experiment involving approximately 100 high school students at a school for the deaf. This study aims to provide valuable insights into the transformative potential of AR technology to enhance educational experiences and outcomes for DHH students.

• Tsekouras D., and Priante A. Integrating Generative AI in Essay Writing: A Tutorial Based on a Digital Business Course

  • Based on a two-year intervention in a large-scale Digital Business course with 1200-1400 students yearly, we demonstrate how GenAI, such as ChatGPT, was integrated in the course and was used to support students’ essay writing. During the tutorial, we share our experience of implementing this GenAI intervention over two cohorts and showcase the impact of AI on student engagement and performance. Furthermore, we provide practical, step-by-step guidance into designing and using GenAI effectively while promoting critical thinking in students, and discuss ethical considerations of AI use. By addressing the need for a clear framework in adopting AI in academic settings, this tutorial offers practical strategies for assignment design, student support and academic integrity.

• Lee, C. The Harmonized Appraisal assessment methodology (HAPP): Grading and simultaneously providing high-information feedback

  • Feedback is important, but writing high-information feedback for students can be unrewarding for educators, given the limited time and resources that they have. To address this issue, we propose a new student assessment methodology for unstructured and semi-structured assignments, such as essays and presentations, that aims to provide an efficient and effective approach to the creation of high-information feedback and fair assessment so as to optimize the positive impact of assessment on students’ learning and development. The approach, which we termed Harmonized Appraisal (HAPP), uses the grader’s input on a relatively detailed grading form to compute the student’s scores on a conventional rubric and to aggregate a high- information feedback message by leveraging a comment bank. We test the key tenants of the approach in two undergraduate courses in Business Administration, and find that students consider feedback from HAPP to be of higher quality and fairer than feedback drawn from a conventional rubric, but we did not find a relationship with self-reported student learning. Looking at the effects for graders we find that HAPP provides greater consistency, while the time required to grade is comparable to grading with a rubric. Finally, we discuss how open questions can be addressed and how this study could inform future research on assessment.

If you’re an educator, learning innovation expert, researcher, or administrator interested in using HAPP, please reach out to us at happ@rsm.nl or clee@rsm.nl. For a demo, visit https://bit.ly/HAPP-Demo. For more information, go to https://happ.rsm.nl.

• Tsekouras D., Developing Cases using GenAI

• Mulder, A. Automated and personalised assessments & mastery path learning

• Tsekouras D., Adaptive quizzes and student performance

  • We investigate the effectiveness of adaptive quizzes on student performance. We set up an experiment on the learning management system of our institution where we randomly allocate quizzes questions to students. The questions are drawn from different item banks (one bank including repeated questions from previous quizzes in case a student did not perform excellent). We investigate the extent that repeating questions (hence making students revisit specific material) has an effect on the performance of students (quiz performance and thesis proposal quality)

• Kloc, A., Belo, R., and Li, T. Climbing the Ladder or Falling Behind: The Role of Leaderboard Composition in User Engagement

  • Leaderboards are designed to keep users engaged through competition. Despite their widespread use in educational environments, empirical findings on their effectiveness are mixed. While some research points toward enhancements in self-regulation, motivation, and performance through gamified competition, others highlight potential drawbacks such as reduced self-esteem and diminished intrinsic motivation. This study investigates the impact of leaderboard composition on user engagement through a randomized field experiment and a follow-up online experiment. We find that users exhibit higher engagement when competing against those with different scores rather than similar ones. The effect varies with individual competitiveness levels. In a follow-up experiment, we tested three competition mechanisms: social presence, performance feedback, and social comparison. Results indicated that performance feedback is the main factor driving leaderboards’ positive effects. Our study implies that a universal leaderboard approach is ineffective. Instead, leaderboards should be customized based on competitive conditions and the competitiveness of an average platform user.

• Kloc, A., Belo, R., and Li, T. Balancing Act: Determining the Optimal Free-to-Premium Content Distribution in Learning Platforms

  • Our study explores how different freemium content strategies affect user engagement and conversion rates in an online learning platform context. We analyze the impact of allowing users to select free content, the relevance and quantity of free content offered, and the redistribution of engagement between gifted and non-gifted sections. While prior literature suggests that freemium models act as advertisements for premium versions, our results show no significant effects of these strategies on overall engagement or conversion rates. Instead, we observe a redistribution of activity, with users spending more time in gifted sections and less in others. Allowing users to choose their free content increases time spent in the gifted sections’ initial modules, but providing additional sections appears to decrease engagement across the platform.These findings suggest that in the context of learning, gifting additional content influences engagement patterns without increasing overall study activity.

•  Kloc, A., and Gutt, D. Swipe and Study: Analyzing Study-Related Content on Social Media Platforms

  • This research investigates the nature of study-related content on social media platforms, focusing on the social functions that such content provides to the students who watch it. We scrape and analyze the content from TikTok, a popular video-based social media platform. Through qualitative content analysis, we uncover a variety of social functions present in the content, including social support, aestheticization, and social comparison. With our study, we advance the understanding of the indirect role social media platforms play in education, showing that such platforms can not only provide entertainment but also perform valuable social functions, such as emotional support. By exploring the unique characteristics of such content on social media, our research aims to bridge the gap between the literature on personal social media use and the emerging knowledge about social media’s potential for creating a positive societal impact in education. In the next research phase, we will conduct a randomized experiment to further evaluate how consuming study-related content impacts university students’ moods and motivations.