AI Language Feedback Loop
I joined this UCISA webinar at almost the last minute, when I found out that they were going to be talking about Cadmus via the HeLF email list. Cadmus is something I need to learn a lot more about in connection with a project I’m involved with this year.
It opened with Julie Voce, of City St George’s, University of London, delving into some of the challenges the sector is facing in relation to generative AI. She talked about human-based detection of AI plagiarism by looking for hallmarks of LLM content, such as the use of words like ‘delve’ and the em dash—for an explanation of why this is problematic, watch Etymology Nerd’s short on the feedback loops which I’ve embedded above—Julie also talked about a practice she has observed in staff, finding that some people will dock students a few percentage points when marking if they suspect LLMs have been used, but can’t prove it.
This led on to Tom Hey’s case study on their use of Turnitin’s AI detection tool at Leeds Beckett. They have been a longtime user of Turnitin’s authorship tool which launched in 2019 to help detect contract cheating, and adopted the AI detection tool when that launched as it was already part of their license, academics wanted it, and they didn’t want staff using unauthorised tools at their own discretion. Tom reported good success with this, but noted that it had to be framed by their ‘Academic Honesty Policy’ for staff and students, which emphasises that these tools are a backstop to help academics, and are nor foolproof detectors of plagiarism. In the chat, someone posted a link to a Jisc paper on the validity of AI detection systems which makes for interesting / depressing reading (delete as appropriate).
Finally, Chie Adachi from Queen Mary University presented about their experience of using Camdus to support assessment during a pilot which ran over the previous year. Unfortunately I didn’t get to see the tool itself or learn much about it, but the results of the pilot were very positive, with 82% of students reporting a positive experience, a 7% increase in average grade, and a 38% decrease in first time failure rate.
Another couple of useful links from the discussion which I thought worth sharing: A Harvard Business Review article on how ‘workslop’ is harming office productivity, unfortunately behind a paywall, but you may have access through your Library, and a report from MIT (PDF), on the impact of Generative AI on business to date – “95% of organizations are getting zero return”.