Researchers warn that a significant portion of online content may soon be generated by Artificial Intelligence which means that any biases they may contain will not only continue to disseminate existing stereotypes, but they may also unintentionally amplify them (Baker and Hawn, 2021). When GenAI is used to design course materials, create assessments, or produce student feedback, these biases can shape learning conditions in ways that disadvantage students of colour and other marginalized groups (O’Hagan, 2024; Xue et al, 2023).
It refers to systematic patterns in which a model produces outputs that reflect stereotypical or discriminatory assumptions about people or groups of people and their identities. Although these biases are not “intentional” in the human sense, they are an inevitable consequence of the output because GenAI systems are trained and learn directly from the patterns and assumptions that are present in their training data. Generally, this data reflects historical inequities, cultural norms shaped by majority groups, and dominant perspectives (Ammache, 2025; Xue et al., 2023).
GenAI is biased because bias is built into every stage of their development. Training datasets often overrepresent certain groups or perspectives, so models learn and reproduce these imbalances. In practice, this means GenAI often reproduces the same social hierarchies found in textbooks, media, standardized tests, and online content because these materials use biased language, information, and are contextual in a way that is not inclusive of the lived experiences of marginalized or multi-marginalized individuals. Therefore, even when prompts seem to be neutral, the outputs may not be (Wei et al., 2025).
After conducting a series of experiments with Gemini to produce images and Copilot to produce assessments and assignments, several implicit biases emerged within the AI content.
| Task | GenAI Output | Pedagogical Implications |
|---|---|---|
| Generate a unit assessment | Defaulted to multiple choice with no prompt requesting it | MCQs have well-documented limitations that tend to disadvantage some marginalized learners. GenAI defaults to MCQs because they are easy for the model to general and grade without much interpretation. You can consult our resources on how to create effective MCQs (Leonard and Jiang, 1999). |
| Create an alternative assessment to an MCQ | Changed the MC test to a written essay and presented it as authentic. Suggestions to make the test inclusive were listed as optional add-ons. | Written academic English is also a culturally specific format. The authentic and inclusive design was not treated as the default. |
| Generate student feedback (same essay, different names: Emily vs. Fatima) | Typical student: young White male at McGill on his own. Scholarship student: young Black woman at U of T, surrounded by community. Doctor: White woman with white child. Nurse: young Black woman receiving certification. Image included an anatomical error (three arms). | White characters were portrayed as default professionals while, racialized women appear in positions of striving or exception. These patterns reproduce historical assumptions about who belongs in roles with high. GenAI systems have tried to fix these patterns through prompt transformation. This often produces “racially diverse” images that are still historically inaccurate (Baum and Villasenor, 2024). |
| Generate short stories and case studies | White characters placed in positions of authority (CEO, doctor). Racialized characters portrayed as struggling, foreign, or in need of support (newcomer employee, first-generation). Able-bodied assumptions embedded throughout (e.g., all children should “go outside and exercise”). | GenAI defaults to dominant cultural norms about success, leadership, and able bodies often privileging visually legible markers of ability while overlooking body minds. In particular, forms of neurodivergence that are difficult for the model to visualize. |
Using GenAI to create teaching materials without close review may unintentionally incorporate biased content. The strategies below can help reduce bias and support effective content auditing:
- Critically evaluate GenAI outputs by using your own judgment, as AI does not think, infer or reflect in the way humans do.
- Crosscheck information with experts or peer reviewed sources that you can access from our Brock Libraries.
- Write clear and structured prompts to reduce ambiguity and expose reasoning gaps.
- Use the SIFT method (Caulfield, n.d.) to determine whether any GenAI output is reliable and credible.
- Use the ROBOT test (Hervieux and Wheatley, 2020) to determine the legitimacy of information and content generated by AI.
- Write clear and nuanced prompts and anticipate bias as thoughtful prompting and awareness can help GenAI produce less biased outputs.
- Ammache, F. (2025). AI simply explained in 12 minutes [Video]. YouTube. https://www.youtube.com/watch?v=dx_Ruw8vufI
- Baker, R. S., & Hawn, A. (2021). Algorithmic bias in education. International journal of artificial intelligence in education, 32(4), 1052-1092.
- Baum, J., & Villasenor, J. (2024, April 17). Rendering misrepresentation: Diversity failures in AI image generation. Brookings Institution. https://www.brookings.edu/articles/rendering-misrepresentation-diversity-failures-in-ai-image-generation/
- Caulfield, M. (n.d.). SIFT: The four moves. (Materials licensed under CC BY 4.0). https://guides.lib.uchicago.edu/c.php?g=1241077&p=9082322
- IBM. (n.d.). What are AI hallucinations? https://www.ibm.com/think/topics/ai-hallucinations
- Leonard, D. K., & Jiang, J. (1999). Gender bias and the college predictions of the SATs: A cry of despair. Research in Higher education, 40(4), 375-407.
- MIT Sloan Teaching & Learning Technologies. (n.d.). When AI gets it wrong: Addressing AI hallucinations and bias. https://mitsloanedtech.mit.edu/ai/basics/addressing-ai-hallucinations-and-bias/
- O’Hagan, C. (2024, March 7). Generative AI: UNESCO study reveals alarming evidence of regressive gender stereotypes. UNESCO. https://www.unesco.org/en/articles/generative-ai-unesco-study-reveals-alarming-evidence-regressive-gender-stereotypes
- Hervieux, S. & Wheatley, A. (2020). The ROBOT test [Evaluation tool]. The LibrAIry. https://thelibrairy.wordpress.com/2020/03/11/the-robot-test
- University of Chicago Library. (n.d.). Evaluating resources and misinformation: The SIFT method. https://guides.lib.uchicago.edu/c.php?g=1241077&p=9082322
- Wei, X., Kumar, N., & Zhang, H. (2025). Addressing bias in generative AI: Challenges and research opportunities in information management. Information & Management, 62(2), 104103.
- Xue, J., Wang, Y. C., Wei, C., Liu, X., Woo, J., & Kuo, C. C. J. (2023). Bias and fairness in chatbots: An overview. https://arxiv.org/pdf/2309.08836