Don't Replace Long Paper Examinations with Electronic Ones

Last week I tried a couple of demo exams with one of the leading online exams products. While just a practice, these brought back a tinge of the terrors I suffered as a student with assessment. As an instructor, I would not use such a system for long examinations. Such examinations are not a good way to assess the knowledge, and even less so for assessing skills needed for real work tasks. In addition such examinations increase the risk to the health and safety of students. Students were already at risk of mental illness and self harm before COVID-19. They are now at higher risk, and there are better, safer, ways to carry out assessment. Online invigilated tests should have only a small part in assessing students.

There were some good points with the online exams. The compatibility test beforehand warned my CPU was too slow, but it worked okay. I have a relatively under-powered laptop by today's standards and slow Internet connection.  This is not just because I am a cheapskate, I am doing what Ken Mattingly did for Apollo 13: make sure that if it works on my equipment, it will work for those out in the void.

The business of panning the camera around the room, to make sure no one else was present was a bit of light relief.

I has some minor quibbles:

• The instructions said to click "I agree" but the button said "I accept". That might seem trivial, but it can unsettle an already anxious student.
• There was no timer I could see indicating how long I had left. I did not dare look at my watch in case this was seen as cheating. Apparently there was a timer, but when someone is under stress, their attention narrows. The ATSB recently reported that two pilots landed an aircraft without the wheels down, because they were distracted.
• I had difficulty getting the camera to accept my image. I had to move a lamp so it shone on my face, which was uncomfortable.
• After completing all questions I could not work out how to get out of the software. I ended up closing down the browser, which in a real exam would cause additional anxiety. It turned out the exit button was just under the timer which I had not seen either.

Having tried this process twice, I could image taking a short test this way.

Overall the process was stressful, like a regular exam. There is the comfort of a familiar environment, rather than an exam room, but help is further away. If it was for a small amount of my grade (perhaps 20%) and short (around 30 minutes), this might be tolerable, just. But if for the majority of the grade in a course and for hours, there is no way I would ever contemplate sitting such a test.

I have spent a lifetime avoiding formal examinations. The first strategy is, of course, to not enroll in any course or program of study which has a long formal high stakes examination. If confronted with an examination, my first thought is: "Do I have enough marks already to skip this and still pass?". If all else fails, my options are reduced to withdrawing from the course, or the program, and then complaining to accrediting and funding authorities about the poor assessment process.


In my view, any exam should be relatively short, more like a take-home examination, than end of semester one. One issue is when the students can do it. I worry that some course designers are assuming all the students take the test at precisely the same time. This will not work well at home. There should be a period of a day, or days, over which students can choose to start their exam.


Academics and university administrators may be using online examinations in the mistaken belief that some law, policy or rule requires examinations. Even before COVID-19, there was flexibility with how assessment could be done. Assessment rules generally have a lot of detail about how to do a traditional examination, but then include a get-out clause saying that some other form of assessment can be used. Other forms of assessment take skill to design and resources to administer. But if you have been trained in how to design assessment (as I have), this can be done reliably, at reasonable cost.


Online exams and quizzes have been used previously for low stakes assessment throughout a course. The products being used now add automated invigilation. But if the student can choose when to start the exam, other measures will still be needed to make cheating harder. It is usual to have a question bank from which questions are chosen at random for each student, and some form of plagiarism detection for essays. Otherwise a student could note the questions, and pass these on others doing the exam later, so they could memorize prepared answers.

ps: Last weekend I was a Lead Mentor for the Australian and New Zealand Defence Forces Logistics Hackerthon. The format is similar to ANU TechLauncher, but compressed into a few days. These are good examples of how teaching and assessment can be done in a more realistic way. Participants have to undertake a real world task, and are assessed on how well they do it. There are no formal examinations, instead participants show what they did.

Make Online Exams More Like the Real World

Universities are considering how to replace end of semester examinations. There is a technological solution with remote proctoring software such as ProctorU and Proctorio. These present the examination questions to the student on the home computer, and collect the answers. The software locks down the student's computer, so they can't look up answers on the Wikipedia. The software uses the student's web camera to see it is the student doing the test, and no one is helping. There are obvious limitations with such software, but what concerns me more is the stress it places on the student.

I have never had to take a test using online proctoring software. So I tried some practice exams. While just a practice, these brought back a tinge of the terrors I suffered as a student with conventional examinations. That could be lessened if taken with a human invigilator present (something CIT has used for years, in their libraries).

There were some good points: I was warned my CPU was too slow, but it still worked okay. I have a relatively under-powered laptop by today's standards and slow Internet connection.  This is not just because I am a cheapskate, I am doing what Ken Mattingly did for Apollo 13: make sure that if it works on my equipment, it will work for those out in the void.

Some minor quibbles:

1. The instructions said to click "I agree" but the button said "I accept". That sounds trivial, but such things can induce panic in an already stressed student.
2. The same questions in the quiz kept reappearing after I had answered them. This may have been a feature of the examination being used, not the proctoring software, but it was confusing.
3. There was no timer I could see indicating how long I had left. I did not dare look at my watch in case this was seen as cheating. Later I was assured there was a timer on screen, but I never noticed it. So it might be worth pointing this out to students.  When someone is under stress, their attention narrows. The ATSB recently reported that two pilots landed an aircraft without the wheels down, because they were worried about something else
4. I had difficulty getting the camera to accept my image. I had to move a lamp so it shone on my face, which was uncomfortable.
5. After completing all questions I could not work out how to get out of the exam. I ended up closing down the browser, which in a real exam would cause additional anxiety.

But I can't imagine being able to sit though more than 30 minutes of this mental pressure. If confronted with it for a small proportion of the course grade (up to about 20%), I would be tempted to skip the exam and hope to pass. Otherwise I might withdraw from the course, or the program (then complain to accrediting and funding authorities).

As an instructor, I would not use such a system for long, high stakes exams, as it would be an unacceptable risk to the health and safety of students. The students were already at risk of mental illness and self harm before COVID-19. They are now at higher risk, and there are better, safer, ways to carry out assessment.

 

Student well-being is more an issue of those setting the exams, than the technology. These exams could be relatively short, more like a take-home examination, than end of semester one. One issue is when the students can do it. I worry that some examiners are assuming this will be like a paper examination, where all the students take the test at precisely the same time, which will not work well at home. What I suggest is a period of a day, or days, over which students can choose to start their exam.

Take Home Exams? There is a general assumption by academics that assessment has to be in the form of an examination. In fact university rules, and the laws they comply with, allow any  form of assessment. The assessment has to be rigorous, but sitting people in a room all at the same time answering the same questions is not the only way, or the best way to do this.

Online invigilated exams differ from a traditional take home examination in that there is automated invigilation and an elapsed time limit. The student can still choose when to start the online exam, if the person setting the exam allows that.

However, if the student can choose their starting time, the usual measures would need to be in place for to make cheating harder. With an online test it is usual to have a question bank from which questions are chosen at random for each student, and some form of plagiarism detection for essays. Otherwise a student could note the questions, and pass these on others doing the exam later, so they could memorize prepared answers.

ps: Last weekend I was asked to help mentor the mentors in a Logistics Hackerthon for the Australian and New Zealand Defence Forces. The format is similar to ANU TechLauncher, but compressed into a few days. These provide good examples of how teaching and assessment can be done much better, without examinations.