Press "Enter" to skip to content

Tag: Feedback

AI-Augmented Marking

Published 26th June 2024
Chart showing correlation of human and KEATH.ai grading
Accuracy of KEATH.ai Grading vs. Human Markers

This was a HeLF webinar facilitated by Christopher Trace at the Surrey Institute of Education, to provide us with an introduction to KEATH.ai, a new generative AI powered feedback and marking service which Surrey have been piloting.

It looked very interesting. The service was described as a small language model, meaning that it is trained on very specific data which you – the academic end user – feeds into it. You provide some sample marked assignments, the rubric they were marked against, and the model can then grade new assignments with a high level of concurrence to human markers, as shown in the chart above of Surrey’s analysis of the pilot. Feedback and grading of a 3-5,000 word essay-style assignment takes less than a minute, and even with that being moderated by the academic for quality, which was highly recommended, it is easy to see how the system could save a great deal of time.

In our breakout rooms, questions arose around what the institution would do with this ‘extra time’, whether they would even be willing to pay the new upfront cost of such a service when the cost of marking and feedback work is already embedded into the contracts of academic and teaching staff, and how students would react to their work being AI graded? Someone in the chat shared this post by the University of Sydney discussing some of these questions.

Leave a Comment

Studiosity Partner Forum 2024

Published 24th April 2024

I attended my third Studiosity Partner Forum today, which kind of began last night with a dinner and discussion about generative artificial intelligence led by Henry Aider. Generative AI and Studiosity’s new GAI powered Writing Feedback+ service was of course the main topic of conversation throughout the event. Writing Feedback+ launched in February, and they have reported that uptake is around 40% of eligible students, which compares with 15-20% for the classic Writing Feedback service. The model has been built and trained internally, using only writing feedback provided by Studiosity’s subject specialists, no student data. The output of WF+ is being closely quality assured by those agents, and they estimate that quality is around 95-97% as good as human provided feedback.

David Pike, from the University of Bedfordshire presented on their experience with the service in the afternoon. They made it available to all of their students in February, around 20,000, and usage has already exceeded usage of the classic Writing Feedback service since September last year. The average return time from WF+ is around one and a half minutes, and student feedback on the service is very positive at 88.5%. However, he did also note that a number of students who have used both versions of the service stated that they preferred the human provided feedback.

On the flip side of AI, last year Studiosity were exploring a tool to detect submissions which had been written by generative AI. That’s gone. Nothing has come of it as they found that the reliability wasn’t good enough to roll out, especially so for students who have English as a second language. No surprises for me there, detection is a lie.

The keynote address was delivered by Nick Hillman from the Higher Education Policy Institute (HEPI), who talked about their most recent report on the benefits and costs associated with the graduate visa route. It’s overwhelmingly positive for us as a country, and it would be madness to limit this.

Other things which I picked up included learning more about Crossref, a service for checking the validity of academic references; a course on Generative AI in Higher Education from Future Learn was recommended; and Integrity Matters, a new course developed by the University of Greenwich and Bloom to teach new students about academic integrity.

Finally I was there presenting myself, doing my Studiosity talk about our implementation at Sunderland and the data we now have showing a strong positive correlation between engagement with Studiosity and student outcomes and continuation.

Leave a Comment

ALT NE User Group: June 2023

Published 9th June 2023

A photo of Durham's lightboard in action
Durham University’s Lightboard, a very cool (but smudgy) piece of tech

Hosted by my lovely colleagues at Durham, this ALT North East meeting began with a discussion of the practice of video assessment. I talked through what we do at Sunderland using Canvas and Panopto, covering our best practice advice and talking through the things which can go wrong. The problem of a VLE having multiple tools for recording / storing video was one such headache shared by all of us, no matter what systems we are using.

We then moved on to a discussion about Turnitin, ChatGPT and AI detection, pretty much a standing item now. Dan shared with us a new tool he has come across, which I’m not going to name or share, which uses AI to autocomplete MCQs. A new front has emerged. Some bravery from Northumbria who must be one of the few HEIs to have opted in to Turnitin’s beta checker, and New College Durham are going all in on the benefits of generative writing to help staff manage their workload by, for example, creating lesson plans for them. A couple of interesting experiments to keep an eye on there.

After lunch we had demonstrations of various tools and toys in Durham’s Digital Playground Lab. This included a Lightboard. This is a really cool and simple piece of tech that lets presenters write on a transparent board between them and the camera using UV pens. I came across this a few years ago, before the pandemic I think, but it’s a strange beast. It’s not a commercial system, but open hardware, so anyone can build one for themselves at little cost. Unfortunately at Sunderland, and I suspect at many bureaucracies, this actually makes it a lot harder to get one than just being able to go to a supplier. So it never happened, but at least today I got to see one live.

Another bespoke system demonstrated was a strip of LED lights around the whiteboard controlled through a web app which allows students to discretely indicate their level of comprehension. We had a short tour of the Playground’s media recording room, watched some video recordings of content created in VR to, for example, show the interaction of the magnetic fields of objects, a demonstration of Visual PDE which is an open source web tool for demonstrating differential equations, and Kaptivo, a system for capturing the content of a whiteboard but not the presenter. You can see the Kaptivo camera in the background of my photo, behind the Lightboard.

Leave a Comment

Studiosity Partner Forum 2022

Published 4th May 2022

My first in-person conference in two years at the University of Roehampton’s gorgeous campus was a chance to learn about Studiosity’s plans for the future, to network with colleagues at other UK HEIs using Studiosity and compare notes, and pretty randomly, I was able to get a tour of Roehampton’s new library building during lunchtime (it’s lovely).

On those future plans, we’re going to see an enhanced version of the student feedback view in the next couple of months which is going to allow their subject specialists to insert short videos and infographics explaining particular grammatical concepts, issues with spelling, and so on. They are also introducing a new ‘Student Connect’ tool which will help to facilitate peer-to-peer student support. This is currently in beta testing, and two UK universities are part of this evaluation.

The keynote address was by Sir Eric Thomas, who sits on Studiosity’s Academic Advisory Board, and he made a great point that, looking at historical precedents from past plagues, people at the time always think, “this is going to change everything, we can’t go back to how things used to be”, but invariably things do go back to exactly how they were once the threat is over. He speculated that this was because plagues and pandemics leave physical infrastructure unchanged, in contrast to wars, where the physical act of rebuilding allows for societal changes to be literally built in. However, what may be different as we ‘re-build’ after Covid, is that new communication technologies such as Teams and Zoom have come into their own and already effected change in how we live and work. The permanence of these changes is something that lingers in my mind as I contemplate my future.

Good opportunities for informal chats with colleagues at more advanced stages of Studiosity use, and no easy answers to be had in terms of managing use and expectations, and showing causal links between use of the service and student retention and attainment, something I’m in the midst of grappling with now as we approach the end of our pilot.

Leave a Comment

PG Cert AP: Day 14

Published 30th March 2017

Final day of the core module began with a session delivered by a guest lecturer who talked about workplace literacy and how the non-academic writing we do on a day-to-day basis is as valuable as academic writing and teaching in forming our professional identities. This was based on a paper by Mary and Barry Stierer – Lecturers’ everyday writing as professional practice in the university as workplace: new insights into academic identities.

In the afternoon there was a catch-up for a few people who missed the peer teaching session, followed by another run of the nominal group feedback exercise to get our feedback on the module now that it has completed.

Leave a Comment

PG Cert AP: Day 8

Published 9th February 2017

First day of my optional module, Assessment and Feedback for Learning, began with a discussion of how assessment can be used for learning, rather than as a tool to measure learning. The module has this concept at its core and, as such, the main assessment of this module is to critically analyse two assessments that you have used or written previously. There is also a second assessment, to write a personal reflective report on how you have found the problem based learning approach taken in this module, and how what you have learned impacts on your own academic practice. Very meta.

After setting out the learning objectives and the assessments of the modules, the remainder of the day was spent discussing the various factors and contexts which influence how assessments are set and marked. These included how student expectations have changed as a result of the marketisation of the sector, the university’s generic assessment criteria and how that relates to the learning outcomes on individual modules, and the cascading down of risk onto lecturers, e.g. pressures around graduate employability and how that influences the assessments which are set.

We also discussed the difference between formative and summative assessment, and how and why students often see formative assessments as options. There was a little about Foucault’s ‘regimes of truth’ (got to love a bit of Foucault!), and the concepts of the hidden curriculum and expectations – that everyone has a certain baseline IT literacy for example.

Leave a Comment

PG Cert AP: Day 6

Published 8th December 2016

The final day of the first semester was a little unusual. The morning was given over to a review of the assignments for this module which are to complete the UKPSF form, critique a learning session, analyse a learning theory, and write a report on the experience of peer observation, comparing the experience of being the observer and the observee. Drafts are due at the end of semester 2, with final versions by September. All well and good, and all covered in the module guide. This session didn’t add anything, and yet we did literally spend the entire morning debating it. Strange things happen when you have academics as students.

The afternoon session was more useful. First there was a short presentation on evaluation in general, why and how to do it, followed by an introduction to nominal group technique. A definition of evaluation was given as ‘assessing the process and practice of a prior learning strategy or event by feedback and trying to make objective summaries of an often subjective interpretation.’ This was followed by a discussion on the different types of evaluation – student, staff, data, and self – and the difference between quality assurance, which is backwards looking and tends to be about accountability, and quality enhancement, which is about how to improve and develop your programme or module.

With quality enhancement in mind, nominal group technique was then introduced followed by actually using it to evaluate this first semester. As a group, and with the programme leader absent, we drew up two lists of ten to twelve points of things that are going well, and things which we think need to be improved. These were written on a board in no particular order, then individually we had ten votes, or points, with which to rank what we thought were the most important points. So for example, if you thought that ‘over-assessment’ and ‘use of VLE’ were the two most important things that needed to be improved upon, then you could give each one five votes. The programme leader was then invited back in and the votes were added up to show what we collectively ranked as the most important things for improvement, and what we felt was going well. The outcome of this evaluation will be actively used in the development of the programme for the second semester.

Leave a Comment

Turnitin UK User Summit

Published 6th October 2016

student_feedback

Attended the afternoon sessions of Turnitin’s UK user summit which focused on customer experience, with talks from colleagues at the University of Edinburgh, the University of East London, Newcastle University and the University of Huddersfield. It’s always cathartic to hear your colleagues sharing their tales of woe and horror which are so familiar in your own work, like the academics who insist on treating the originality score as sacrosanct when making a plagiarism decision, but more productively there were some really good ideas and pieces of best practice shared. One colleague was using Blackboard’s adaptive release function to hide the Turnitin assignment submission link until students had completed a ‘quiz’ which was simply making them acknowledge in writing that they work they were about to submit was all their own. A couple of people presented their research findings on what students wanted from feedback, such as in the attached photo which shows a clear preference for electronic feedback. Someone made a product development suggestion, splitting the release of the grade and feedback in Turnitin so that students have to engage with their feedback before they get their grade. But I think my personal highlight from the day was the very diplomatic description of difficult customers as those who have ‘higher than average expectations’.

Though I missed out on the morning session due to another commitment, I was able to get the gist from networking with colleagues in-between sessions. Improvements to the Feedback Studio including the ability to embed links, multiple file upload, a new user portal which will show the most recent cases raised by people at your institution, and the development I found most interesting, the ability to identify ghost written assignments. This is still quite away from being ready, but it’s an increasing problem and one Turnitin has in their sights. They couldn’t reveal too much about how this will work for obvious reasons, but the gist is that they will attempt to build up a profile of the writing style of individuals so that they can flag up papers which seem to be written differently.

The Twitter conversation from the summit is available from the TurnitinUKSummit hashtag, where you will see I won the Top Tweet! Yay me, but alas there were no prizes.

Leave a Comment

Preparing to Teach

Published 28th April 2015

hounsell_guidance_and_feedback

This was an extremely dense two day course which “introduced key concepts, tools and issues which are important to teachers in higher education” (from the welcome document). The primary target audience for the course, which is delivered by our Academic Development Unit regularly, was PGR students who are starting to do some teaching as part of their work, but who may not have had any formal teacher training yet.

One of the course objectives was for itself to deploy some of the techniques under discussion, and the first such example of this was a ‘signature search’ icebreaker exercise, something which was completely new to me and most of the others there. This was followed by a reflection on the purposes on HE which employed the snowballing technique (more than learning, we concluded, also to develop students, at least potentially, into researchers and citizens who contribute to the advancement of society), how students learn in HE, techniques for reflective practice (including models of reflective practice from Schön, Gibbs and Rolfe, of which most of us preferred Rolfe’s model), key pedagogies (including scaffolding, repetition, cognitive load and chunking), planning your teaching (which included advice on how to plan, the difference between planning for a programme, session or individual learning activity, and a broadly applicable session template), and finally assessment and feedback strategies, e.g. how to select the appropriate assessment strategy for a given activity. The photograph is of the results of an exercise to complete a guidance and feedback loop based on Hounsell’s model. A joint exercise, my partner kept the original, hence my photo, which also explains why you can read the handwriting!

This was an extremely useful course for me which will help inform my development and, I hope, the quality of the sessions I deliver as I transition from a trainer, as I have previously characterised myself, to a teacher. An immediate impact will be on the fact that I will, from now, create proper sessions plans which go into a lot more detail than the notes I have previously prepared and which will include more thought on contingency measures and alternative activities which can be deployed depending on the nature of the group on the day. Some other things which will have an impact are that I now have an increased awareness of attention spans and the need to change activities at appropriate intervals to keep people awake, and the benefit of embedding informal assessment throughout a session to reinforce learning.

There were also some parts of the course which were not immediately applicable to my work and exercises which were harder for me to complete as they had in mind people delivering entire modules and programmes, not something I do at the moment, but I’m sure it will come in useful in the future as I look forwards to doing a PGCert in learning and teaching. That’s not going to be this year due to prior commitments, but possibly the presentation starting in September 2016.

I would have liked to have seen TEL being used to greater effect on the course. From a technological standpoint the course was a very tradition ‘PowerPoint plus handouts’ model which left plenty of room for improvement. I believe a small forest in South America must have been destroyed to furnish us with the enormous amounts of paperwork we were given; many of us, myself included, requested digital copies of the materials but there hasn’t been any follow-up on this yet, perhaps it is too soon. A lot of the paperwork was forms which we had to complete, the templates for the reflective feedback models for example, how much better to have delivered these via Mahara? Ideally in advance and that way we would also have had time to consider questions and points for discussion in order to get more out of the course (the flipped classroom approach). I work with Academic Development quite closely on a number of areas so I will have plenty of opportunity to feed this back to them.

Leave a Comment

How to Mark an Assignment – Storyline Presentation

Published 7th July 2014

mark_assignment

The presentation I put together for student submission was well received and it has led to others. This one is for markers where there is only a single marker. The tool that the team is developing has the ability to accommodate multiple independent markers. The player is looking a little better now and I added the video in a different way with better results, more like an actual video than a series of screenshots.

Making these has been a little difficult as Storyline kept crashing on me when I was using the precision timing editor. Upon investigation I found that this was because I was running Storyline on a Windows 7 virtual machine in Parallels and had all my files on the desktop of my computer. Parallels has a nice little feature whereby it links the desktop on the host Mac with that on the Windows client, but it does so by making the drives on the Mac a pseudo-network drive in Windows. I discovered on Articulate’s forums that working on, and saving files to a network drive can cause various performance problems, and when I moved my files to the actual C drive it solved all of my problems.

http://solar.sunderland.ac.uk/solar/file/eee5d915-96c4-41bd-95d1-e6e6695969a2/1/story.html

Leave a Comment