Turnitin is a plagiarism-prevention tool that helps check the originality of student writing. Today Turnitin is releasing a new AI-detection feature that attempts to check for text generated by AI-writing tools, such as ChatGPT.
For a number of reasons, UBC has decided not to enable the new feature in Turnitin at this time.
Why is UBC not enabling Turnitin’s AI-detection feature?
The LT Hub Leadership group—with the support of the Provosts at UBC Vancouver and UBC Okanagan—has made the decision based on several important factors.
- UBC has not been able to review and validate the feature: Because of the speed of its release, we have not been able to go through our normal process of vetting a new feature. As such, we do not know much about this feature’s functional limitations, drawbacks, and risks of use. That lack of vetting is acutely felt in this case, since many generative AI-writing tools such as ChatGPT are relatively new, and AI-detection tools even more so.
- Timing of the feature’s release is not ideal: Turnitin’s plan to release today came with little advance notice. We did not have a chance to prepare the UBC community, nor did we have time to prepare guidelines and support resources. Having significantly new functionality appear in a tool at the end of an academic term—and when people are not expecting it and have not prepared themselves or their students—creates extra challenges at an already stressful time.
- Testing for accuracy in the feature is in early stages: Turnitin claims that the feature is highly reliable, with few false hits and few misses. But this claim has not been independently evaluated. Additionally, Turnitin’s claim of accuracy that the feature can “detect the presence of AI writing with 98% confidence” was arrived at by checking their own training set of AI-written text against human-written text, and they have provided few details about that training set. So far as we can tell, they have only tested in a lab setting; they have not done testing on a large set of real-world text yet.
- Instructors cannot double-check the feature results: Most plagiarism-prevention tools give instructors both the flagged passages of the student submission and the matching source material(s), to help determine whether plagiarism occurred. However, in the case of AI-detection, the source material simply does not exist. Instructors are instead shown passages that are suspected of being AI-written, with nothing to check against. This limitation means over-reliance on such tools for academic integrity can be problematic.
- Testing for potential bias in the feature is also in early stages: Turnitin has stated that they have worked to address the concern of bias by including works in their training set by “statistically under-represented groups like second-language learners, English users from non-majority-English countries, students at historically black colleges and universities, and less common subject areas such as anthropology, geology, sociology, and others.” But without further information about Turnitin’s training set, process, or whether there has been any testing for bias, we cannot know the degree to which the feature may flag certain kinds of writing as AI-written more often than others.
- Results from the feature are not available to students: The report provided by the AI-detection feature in Turnitin will only be accessible to instructors; students will not be able view the results. For other existing functionality in Turnitin, students may be able to see the outcomes, unless their instructor disables it. But with this new feature, there is no way for instructors to enable students to access results.
- Ability of the feature to keep up with rapidly evolving AI is unknown: Turnitin’s feature has been trained to detect AI-written work from the GPT-3 and -3.5 language models. But GPT-4 has already been released. Like the race between anti-virus companies and hackers, there will be a race between AI writers and detectors. It is not yet clear the degree to which AI-detection tools such as Turnitin’s will be able to keep up.
What will happen next?
UBC is taking a pause to review this new AI-detection feature from Turnitin. This pause also provides a chance for broader discussions at the institution around the capabilities, limits, and risks of AI-detection tools (this one and others), as well as their value to academic integrity. We aim to revisit our decision to enable the feature during the summer, in time for the start of fall term.
In the meantime, if you have questions, you can reach out to the LT Hub’s Stephen Michaud (Senior Manager, Learning Applications, Integrations, and Analytics) at stephen.michaud@ubc.ca.