Brock researchers Rahul Kumar and Michael Mindzak are exploring the impact of artificial intelligence (AI) technology on post-secondary education.

Although the launch of AI chatbot ChatGPT has had the world talking in recent months about the use of large language models in education, the topic has long been on the minds of the Assistant Professors in the Department of Educational Studies.

In a pilot study completed in spring 2022, Kumar and Mindzak examined whether GPT-3, an earlier generation of ChatGPT, could pass a simplified Turing test. They asked participants to not only identify whether a passage of text was written by a human or the AI tool but also to grade the text as if it were part of an assignment.

The empirical study involved 135 participants from Brock’s Faculty of Education, including undergraduate students, graduate students and faculty members.

Overall, participants were 66 per cent accurate in identifying the author of a passage when it was composed by a human.

However, when Kumar and Mindzak looked at just the AI-generated passages, the accuracy rate dropped to 40 per cent, meaning participants were more inclined to assume the text was written by a human than GPT. On average, passages written by GPT received a B-minus grade.

Factors like age and gender were not significant predictors of whether participants correctly identified the source of a passage, but higher levels of education slightly increased the probability of participants incorrectly attributing AI-generated text to a human author.

While the participant pool was too small to extrapolate conclusive results, Mindzak and Kumar have already begun research to expand their study in partnership with researchers at other universities.

“We are excited at the prospect of changes that will have to come within the education sector: post-secondary and K-12 alike,” says Kumar. “This is a great moment for faculties of education to take the lead and help other colleagues to navigate this disruptive force.”

Raising awareness about the capabilities and implications of ChatGPT and other AI tools is one of the primary goals of Kumar and Mindzak’s work.

“We can’t push this to the side,” Mindzak says. “It has to be a part of our teaching and learning strategies and how we do things in the University and the Faculty of Education, as well as how we prepare teacher candidates for what they’re going to be doing in their classrooms.”

Rather than banning the use of ChatGPT and similar tools, which is unlikely to work, Kumar and Mindzak advise educators and institutions to take a pedagogical approach to AI tools and reconsider teaching and assessment strategies.

Kumar plans to use ChatGPT as part of a graduate course he teaches in the Faculty.

“We can use it as a tool to promote higher order thinking and supplement students’ learning,” he says.

For example, students may examine an article or passage generated by ChatGPT and then be asked to identify errors or biases along with suggestions for improvement, Kumar says.

Developing these skills could help to prepare students for success in a fast-changing world, says Mindzak.

“We need to ensure that we are adequately preparing our students for their classrooms and ensuring the next generation of teachers are able to confront these big questions they will inevitably encounter.”