Plagiarism-prevention tool Turnitin released an AI-detection feature in April that attempts to check for text generated by AI-writing tools, such as ChatGPT. For a number of reasons, UBC will be maintaining its April decision not to enable the Turnitin AI-detection feature, and the university will not centrally install any AI-detection tool at this time.
Why is UBC not enabling Turnitin’s AI-detection feature?
The LT Hub Leadership group—with the support of the Provosts at UBC Vancouver and UBC Okanagan—has affirmed the decision based on several factors.
Effectiveness of the feature is still unclear
- Testing for accuracy in the AI-detection feature remains in early stages: Turnitin claims that the feature is highly reliable, but this claim has not been independently evaluated. Additionally, Turnitin’s claim of accuracy that the feature can “detect the presence of AI writing with 98% confidence” was arrived at by checking their own training set of AI-written text against human-written text, and they have provided few details about that training set. Turnitin has indicated that their initial lab tests yielded different results than what is being seen in real-world settings. Even a small percentage of error can translate into significant numbers of false positives or negatives across a large sample of student work at an institution.
- Testing for potential bias in the feature also continues to be in early stages: Turnitin has stated that they have worked to address the concern of bias by including works in their training set from diverse contexts and authors. But without further information about Turnitin’s training set, process, or whether there has been any testing for bias, we cannot know the degree to which the feature may flag certain kinds of writing as AI-written more often than others.
- Ability of the feature to keep up with rapidly evolving AI is unknown: Turnitin’s feature has been trained to detect AI-written work from the GPT-3 and -3.5 language models. While Turnitin suggests that their detector will detect writing from GPT-4 “most of the time”, the development of new language models will likely outpace the ability for the detector to recognize them. Analogous to the race between anti-virus companies and hackers, there will be a race between AI writers and detectors. It is not yet clear the degree to which AI-detection tools such as Turnitin’s will be able to keep up.
It is not possible to double-check or review the results
- Instructors cannot double-check the feature results: Most plagiarism-prevention tools give instructors both the flagged passages of the student’s suspicious submission and the matching source material(s), to allow for intuitive comparison and help assess whether plagiarism occurred. However, in the case of AI-detection, the source material simply does not exist. Instructors are instead shown passages that are suspected of being AI-written, with nothing to check against. This limitation means over-reliance on such tools for academic integrity can be problematic.
- Results from the feature are not available for students to review: The report provided by the AI-detection feature in Turnitin is only accessible to instructors; students will not be able view the results. For other existing functionality in Turnitin, students may be able to see the outcomes, unless their instructor disables it. But with this new feature, there is no way for instructors to enable students to access results. That lack of functionality makes it more difficult for students to review information about suspected misconduct and to respond to any errors in flagged content.
What will happen next?
UBC is continuing to wait before deploying any AI-detection features, including the one by Turnitin. We will remain cautious until we can be confident about both a feature’s effectiveness, accuracy, and bias-mitigation, and its ability to employ an AI-detector as a robust component of talking to students about suspected academic misconduct. The use of other AI-detection tools is also not recommended, due to similar issues as those noted above, as well as privacy and security concerns; no AI-detection tool has undergone a UBC Privacy Impact Assessment yet.
The UBC Academic Integrity website has related information for faculty and students on how to teach and learn about academic integrity, including tips for assessment design and illustrative syllabus language that can be adapted for specific courses. On the website, there are also FAQs related to the use of generative AI in courses in the context of academic integrity.
If you have questions regarding the rationale for this decision, you can reach out to us in the LT Hub at LT.hub@ubc.ca.