peer reviewed · Journal of Educational Computing Research (2025) · Nov 18, 2025
Do Generative AI-Powered Pedagogical Agents Improve Learners' Academic Performance Effectively? Evidence From Meta-Analysis
Liang Cheng, Hui Shi, Yuhan Wu, Feng Li
In plain English
Pooling 27 experiments, the authors find AI teaching agents produce a moderate gain in academic performance (g = 0.401) — real, but not transformative. Two design findings matter for practitioners: multimodal dialogue outperformed text-only, and agents worked better inside teacher-directed courses than as standalone self-study tools. Grade level, subject, gender and how long learners used the agent made no measurable difference, so the payoff appears to come from interaction design and instructional framing rather than from matching the agent to the audience.
Abstract
With the breakthrough advancements in generative artificial intelligence (GenAI) technology, GenAI-powered pedagogical agents (GenAI-PA) are emerging as a transformative paradigm shift in education. However, there is still debate about whether the use of GenAI-PA is beneficial for learners' academic performance. Therefore, this study conducted a meta-analysis of 27 experimental and quasi-experimental studies from 2015 to 2025. The results showed that (1) GenAI-PA had a significant effect on learners' academic performance (g = 0.401). (2) In a collectivist culture, GenAI-PA had a greater effect on learners' academic performance. (3) The effect of GenAI-PA in teacher-directed learning was significantly better than in self-directed learning. (4) The dialogue modality of GenAI-PA moderated the effect on learners' academic performance, with multimodal dialogue showing the highest pedagogical potential. (5) The predictive effect of GenAI-PA on learners' academic performance was not influenced by grade level, gender, learning domain, learning duration, or the instructional role of GenAI-PA.
Related research
- Do Generative AI-Powered Pedagogical Agents Improve Learners' Academic Performance Effectively? Evidence From Meta-Analysis
- How Effective Are Pedagogical Agents for Learning? A Meta-Analytic Review
- Effects of Artificial Intelligence-Powered Virtual Agents on Learning Outcomes in Computer-Based Simulations: A Meta-Analysis