Tag: Marking

AI-Augmented Marking

Published 26th June 2024

Chart showing correlation of human and KEATH.ai grading — Accuracy of KEATH.ai Grading vs. Human Markers

This was a HeLF webinar facilitated by Christopher Trace at the Surrey Institute of Education, to provide us with an introduction to KEATH.ai, a new generative AI powered feedback and marking service which Surrey have been piloting.

It looked very interesting. The service was described as a small language model, meaning that it is trained on very specific data which you – the academic end user – feeds into it. You provide some sample marked assignments, the rubric they were marked against, and the model can then grade new assignments with a high level of concurrence to human markers, as shown in the chart above of Surrey’s analysis of the pilot. Feedback and grading of a 3-5,000 word essay-style assignment takes less than a minute, and even with that being moderated by the academic for quality, which was highly recommended, it is easy to see how the system could save a great deal of time.

In our breakout rooms, questions arose around what the institution would do with this ‘extra time’, whether they would even be willing to pay the new upfront cost of such a service when the cost of marking and feedback work is already embedded into the contracts of academic and teaching staff, and how students would react to their work being AI graded? Someone in the chat shared this post by the University of Sydney discussing some of these questions.

Leave a Comment

Moodle Munch: Jan. 2021

Published 12th January 2021

The January Moodle Munch recording

Welcome to 2021, folks! Let’s hope it’s going to be a better year for all.

Today marked the return of Moodle Munch, with two presentations as always. Mark Glynn from Dublin City University began by discussing some tools and techniques they are using to add some gamification to online modules to improve student engagement. First, the use of formative and summative quizzes, but not just quizzes, the pedagogy around their use, emphasising the opt-in nature of the summative component and giving students the ‘freedom to fail’. Mark presented some interesting research showing both strong positive student feedback and improved pass marks on the formative assessment component with the group of students who had engaged with the summative quizzes. As it should be, but it’s nice to see such strong evidence! Mark then showed us the Level Up Plus plugin for Moodle which can be used to add gamification elements to module spaces, such as progress bars and leaderboards.

The second presentation was from Nic Earle at the University of Gloucestershire who demonstrated the custom electronic marking and assessment system they have developed for managing student assessments, and how it integrates with their Moodle and student records system. They switched over to this system in 2017 wholesale, and again Nic was able to show very positive results demonstrating increased use of the VLE (even before the pandemic), and improved NSS scores.

As always, the presentations were recorded and I have embedded the YouTube above this post.

Leave a Comment

Canvas UK HE User Group

Published 8th October 2018

Isn’t it nice that the University are letting me get out and about again? In London for two days for the UK HE User Group today and CanvasCon Europe tomorrow.

Today was really useful. Around 40 of us from all over the country at St George’s Medical School in Tooting sharing our experience as Canvas users. In the morning we had a demonstration of anonymous and moderating marking from colleagues who are currently piloting it with positive results, though they noted that they have found a limited number of ways to circumvent the anonymisation. However, as they are all quite obscure and difficult they remain confident in the tool and are rolling it our further. It will be interesting to see how Instructure’s offering here compares with Turnitin’s pending anonymous and moderated marking tool.

Also in the morning we had some group discussions on different ways of using Canvas for assessment and feedback to stimulate discussion and share ideas and best practice.

In the afternoon we were joined by representatives from Instructure who gave us updates on their developments and allowed us to grill them quite freely. This is always an excellent opportunity to use our collective influence to nudge Canvas in a direction which helps to address the needs of the UK sector. The anonymous and moderated marking tool for example, is something that was proposed by, and has been driven by this group.

Instructure provided us with a progress report on our Top 10 priority development list from last year, as shown in the photo above, which shows ‘Non-Scoring Rubrics’ and ‘Analytics to include Mobile App Usage’ as complete, and most of the others in the design or development stages. Finally, we voted on the new Top 10 list for 2018-19. From a long list of suggestions collated prior to the User Group, each person at the group was allowed to vote for three issues, and I voted for QuickMark style functionality in SpeedGrader, improved Group functionality, and the ability to set Notifications by course. All things which I’m being pressed for by our academic community at Sunderland.

Leave a Comment

PG Cert AP: Day 5

Published 1st December 2016

A very interesting morning session for the technology module, EDPM08, covering uses of technology to support self and peer assessment. The great thing about the tutor on this module is that they don’t just know their stuff, they back everything up with research proving that what they’re talking about works. That’s definitely something to keep in mind and aspire to in my own teaching.

First there was a discussion about peer marking, and research that shows that it only takes a surprisingly small number of peer grades to be averaged for it to approximate the grade of a tutor. That’s something that could prove very useful in the assessment for the ArtWorks MOOC that I’ve been assisting to develop. Then we covered the value of real-time formative feedback assisted by quiz tools such as Socrative and Poll Everywhere. And finally, not strictly supported by technology, there was a discussion about comparative marking, giving tutors two papers and deciding which of the two should get a higher mark, but without actually grading them. An interesting idea that I would like to look into further to find out more about how it works.

There was also a nice, almost throwaway remark about the concept of ‘desirable difficulties’, and anecdotal evidence that students learn more from bad lecturers as it makes them have to work harder to make sense of what is being taught. A kind of unintended experiential learning!

The afternoon session was back to the core module, EDPM05, and the use of reflection on teaching and learning. This was facilitated through an iterative exercise where we discussed where and how reflection takes place, wrote down ideas on sticky card and then worked the cards round on a board to reach some conclusions as a group.

Leave a Comment

How to Mark an Assignment – Storyline Presentation

Published 7th July 2014

The presentation I put together for student submission was well received and it has led to others. This one is for markers where there is only a single marker. The tool that the team is developing has the ability to accommodate multiple independent markers. The player is looking a little better now and I added the video in a different way with better results, more like an actual video than a series of screenshots.

Making these has been a little difficult as Storyline kept crashing on me when I was using the precision timing editor. Upon investigation I found that this was because I was running Storyline on a Windows 7 virtual machine in Parallels and had all my files on the desktop of my computer. Parallels has a nice little feature whereby it links the desktop on the host Mac with that on the Windows client, but it does so by making the drives on the Mac a pseudo-network drive in Windows. I discovered on Articulate’s forums that working on, and saving files to a network drive can cause various performance problems, and when I moved my files to the actual C drive it solved all of my problems.

http://solar.sunderland.ac.uk/solar/file/eee5d915-96c4-41bd-95d1-e6e6695969a2/1/story.html

Leave a Comment

How to Submit an Assignment – Storyline Presentation

Published 3rd July 2014

We have been developing a new online submission, feedback and marking tool for one of our faculties and I was asked for advice on creating an interactive guide for students on how to submit to the system. I recommended Storyline for which we had just purchased a few licenses on my recommendation, and this is the result. I actually recommended Storyline for another project which is further down the line so I ended up having to put this together at very short notice, it is consequently a little rough around the edges. I will develop a proper template in due course.

http://solar.sunderland.ac.uk/solar/file/c3d36172-54b2-4456-bf4b-4c63bc05f4dd/1/story.html

1 Comment

e-Fest 14: An e-Ventful TALE

Published 5th June 2014

Organised by Jisc RSC Northern and held at the Stadium of Light in Sunderland, eFest 2014 was a conference bringing together staff from FE and HE institutions across the North East, with an emphasis on learning technologists and people from related fields, with service providers such as Turnitin, OneFile and MoodleRooms.

The whole day was fantastic, I got to meet lots of interesting new people, discovered some new services, many of which I went away and read up on, adding the best to my personal toolkit, but the highlight of the day was the presentation of Paula Kilburn from Stockton Riverside College who presented three case studies on the use of video marking. The first was the simplest, an academic using an iPad to record him as he annotated a student’s written work. In the second the academic used the screen and audio recording functions of QuickTime to record him as he worked through an audio file the student had created, demonstrating in real time the changes required which would have got the piece up a grade. In the final example an academic was watching a video while recording audio feedback, pausing or going back as required. In all three cases the resulting videos were uploaded to the College’s Planet eStream account with no, or minimal editing, the idea being to deliver better, faster feedback, not a polished video. In all three cases the academics reported that it was faster and easier for them to give better and more comprehensive feedback than would have been possible to write. The whole pilot was a huge success with students who received video feedback showing substantial improvement compared to the respective cohorts from previous years.

As always at these kinds of event, there was a open marketplace for tea, coffee, mingling and for various providers to demonstrate their wares, trying to attract people to them with the usual games and freebies. Turnitin, however, set the standard to beat with their Rubrics Cubes, very droll.

Finally, I would just like to say that with regards to the ‘Stadium of Light’ Metro station, I would humbly suggest to Nexus that to improve accuracy this station be renamed to the ‘Random Tesco car park over a kilometre away from the Stadium of Light, with no clear sign posting’ Metro station. My unexpected journey humbly reminded me to be grateful for smartphones, satellite navigation and the company of fellow wayward souls. In all seriousness, to anyone who needs to go to the Stadium of Light on the Metro, get off at St Peter’s station instead as it is actually closer.

http://www.jiscrsc.ac.uk/northern/e-fest-awards.aspx
http://www.planetestream.co.uk/
http://submit.ac.uk/en_gb/
http://www.onefile.co.uk/
http://uki.moodlerooms.com/
http://edina.ac.uk/

Leave a Comment