Turnitin’s Gradescope Workshop at ASCILITE 2020 Conference

Greetings from the ASCILITE 2020 Conference Online, where I am taking part in a Turnitin Gradescope Workshop. Two papers I helped with are being presented tomorrow, but first I am brushing up my grading skills. Anu had TurnItIn and I have used their GradeMark tool, but that is much the same as online assignment marking with Moodle (apart from integration with TurnItIn's plagiarism detector). Hopefully Gradescope does more than this.

I worry that online and semi-automated grading tools may make the fixation students and teachers have with grades worse. These tools look objective and analytical, which grading is not. I am reasonably confident that I can decide that a piece of work is below, at, or above the required standard. However, I am not sure I can really say where the work sits on a seven point scale, let alone a mark out of 100.

What would be useful would be a way to evaluate the quality of grading, especially where multiple graders are used.

Despite being a vendor provided workshop, there was a minimum of sales pitch. However, it would have been useful to compare Gradescope to commonly used online assessment tools, such as the Quiz, Assignment and Workshop tools in Moodle.

Gradescope has something called a "dynamic rubric". This allows the grader to create a rubric during the marking. I have some difficulties with this idea. First of all conventional wisdom, and natural justice, say that the student should be provided with the rubric before the assessment, so they know how they will be graded. If the rubric is made up during marking, the student can't have had it before. Next, this removes the discipline from the assessment designer to think carefully about what they are trying to assess and how it will be marked. There could also be a lack of consistency if the rubric changes during the grading. However, if the grader has not been provided with any rubric, this would be better than nothing. It would also be useful where a problem is found with a question, and the expected answer is wrong.

Gradescope provides statistics overall and on individual questions, much the same as other online marking tools. What I had hoped for were ways for evaluating the quality of marking, between multiple graders, but there doesn't seem anything for this. There was also mention of the use of AI, but no details in the workshop. Overall Gradescope seems to have the features I would expect in any grading tool.

Before creating a complex marking scheme, educators need to stop and consider the purpose of the assessment and the cost and suffering they are causing students. In most cases school and university assessment is intended to find if the student's skills and knowledge are to a required standard. We want citizens and workers who can function in a modern society. There is no need to rank all the students, or mark them to within 1%, for this: pass/fail is sufficient.

Universities might like an easy way to rank applicants, but is that really the job of schools and are school assessments really much use for that? Similarly, university academics may want to use assessment to select students for advanced research courses, but few students will be undertaking these advanced courses, so is it worth the cost of administering this assessment for all students? Outside university, no one much cares about university grades: as long as you passed, that is all that matters.

Arjun Singh on Gradescope

Greetings from the the IEEE 7th International Conference on Teaching, Assessment, and Learning for Engineering (TALE) in Wollongong where Arjun Singh, Co-Founder & CEO of Gradescope is talking on grading of large numbers of STEM exam papers. This product, recently acquired by TurnItIn, allows student exam papers to be scanned in, and then marked online by an examiner, using a rubric. 

One claim for the product is that the rubric can be customized by the examiner, as they do the marking. The product was demonstrated for engineering and computer science examinations, including for computer code. 

There appear to be two distinct uses for the product: one is for marking traditional paper based examinations. The other is for digitally input long-form input, such as computer code.

This product is new to Turnitin, and integration with their copy detection function is to be added in early 2019. However, even in its current form the interface looks similar to Turnitin's GradeMark.

Form or quantity of digital feedback has more effect on learners?

Starr (2016) reports on research into how the form of feedback students are given effects their engagement with it. In particular he looks at different ways to annotate when marking using the Turnitin product, and suggests individually composed text and audio comments are preferable to templates and rubrics. However, there may be an alternative explanation: bespoke comments take longer to compose, resulting in fewer, which students can better engage with.

Starr (2016)reports;

"It was found that GradeMark on-script ‘Bubble Comments’ and off-script ‘Voice Comments’ may both positively influence how valued learners feel and encourage and support their use of feedback for development by positively influencing their motivation and ability to engage. On-script ‘QuickMark Comments’ and off-script ‘General Comments’ and ‘Rubrics’ may negatively influence same." (Starr, 2016, p. 2)

The study (as the author admits) only looks at the form, not content of the feedback, nor the time required for different types of feedback. An assessor has only a fixed and limited amount of time to mark each assessment item. If providing feedback in a particular way takes longer, then the assessor will be able to provide less, or less considered feedback.

Turnitin's QuickMark provides a way to select from a pre-prepared set of comments. Similarly Rubrics make use of a structured template. Bubble and Voice comments are bespoke, composed each time (although there is no technical reason Turnitin could not provide a way to record a set of standard voice comments).

Starr's research suggests the bespoke comments are preferred by students, which is not surprising. However, does this take into account the additional time which it would take to prepare such comments? If it takes, for example, four times as long to compose a bespoke comment, the students will receive one quarter the quantity of comments. It may simply be that students are able to engage better when they receive fewer comments.

Correcting every error in a student paper is a waste to the marker's time and not helpful for student learning. I suggest more concise comments, in analogue or digital form, from templates or bespoke, text or audio.

Reference

Starr, S. (2016). An exploratory investigation into influences of form of digital feedback on learners’ engagement with their feedback (Doctoral dissertation, University of Edinburgh). Retrieved from https://create.canterbury.ac.uk/15171/1/Simon_Starr-UoE_MScDE-Dissertation-Aug-2016.pdf

On-line Grading to Improve Education

David Meacheam

Greetings from the Australian National University in Canberra, where I am talking part in "Education Development StudioTeaching & Learning Day: Evolving practice and expectations". David Meacheam, from UNSW Canberra is talking on Turnitin's GradeMark. He argues an online grading system not only increases the teacher's efficiency, it improve the quality of marking and student satisfaction. My experience agrees with this. However, I suggest the same results could be achieved without any proprietary software, simply by training teaching staff in better assessment techniques. I took the USQ course "Assessment, Evaluation and Learning" (EDU5713).

Moodle includes a feature for using rubrics, but even without this a teacher can create their own manual rubric and set of standard comments.

David also mentioned also this approach allows the institution to show an accreditation body that assessment is linked to course outcomes.