Make Online Exams More Like the Real World

Universities are considering how to replace end of semester examinations. There is a technological solution with remote proctoring software such as ProctorU and Proctorio. These present the examination questions to the student on the home computer, and collect the answers. The software locks down the student's computer, so they can't look up answers on the Wikipedia. The software uses the student's web camera to see it is the student doing the test, and no one is helping. There are obvious limitations with such software, but what concerns me more is the stress it places on the student.

I have never had to take a test using online proctoring software. So I tried some practice exams. While just a practice, these brought back a tinge of the terrors I suffered as a student with conventional examinations. That could be lessened if taken with a human invigilator present (something CIT has used for years, in their libraries).

There were some good points: I was warned my CPU was too slow, but it still worked okay. I have a relatively under-powered laptop by today's standards and slow Internet connection.  This is not just because I am a cheapskate, I am doing what Ken Mattingly did for Apollo 13: make sure that if it works on my equipment, it will work for those out in the void.

Some minor quibbles:

1. The instructions said to click "I agree" but the button said "I accept". That sounds trivial, but such things can induce panic in an already stressed student.
2. The same questions in the quiz kept reappearing after I had answered them. This may have been a feature of the examination being used, not the proctoring software, but it was confusing.
3. There was no timer I could see indicating how long I had left. I did not dare look at my watch in case this was seen as cheating. Later I was assured there was a timer on screen, but I never noticed it. So it might be worth pointing this out to students.  When someone is under stress, their attention narrows. The ATSB recently reported that two pilots landed an aircraft without the wheels down, because they were worried about something else
4. I had difficulty getting the camera to accept my image. I had to move a lamp so it shone on my face, which was uncomfortable.
5. After completing all questions I could not work out how to get out of the exam. I ended up closing down the browser, which in a real exam would cause additional anxiety.

But I can't imagine being able to sit though more than 30 minutes of this mental pressure. If confronted with it for a small proportion of the course grade (up to about 20%), I would be tempted to skip the exam and hope to pass. Otherwise I might withdraw from the course, or the program (then complain to accrediting and funding authorities).

As an instructor, I would not use such a system for long, high stakes exams, as it would be an unacceptable risk to the health and safety of students. The students were already at risk of mental illness and self harm before COVID-19. They are now at higher risk, and there are better, safer, ways to carry out assessment.

 

Student well-being is more an issue of those setting the exams, than the technology. These exams could be relatively short, more like a take-home examination, than end of semester one. One issue is when the students can do it. I worry that some examiners are assuming this will be like a paper examination, where all the students take the test at precisely the same time, which will not work well at home. What I suggest is a period of a day, or days, over which students can choose to start their exam.

Take Home Exams? There is a general assumption by academics that assessment has to be in the form of an examination. In fact university rules, and the laws they comply with, allow any  form of assessment. The assessment has to be rigorous, but sitting people in a room all at the same time answering the same questions is not the only way, or the best way to do this.

Online invigilated exams differ from a traditional take home examination in that there is automated invigilation and an elapsed time limit. The student can still choose when to start the online exam, if the person setting the exam allows that.

However, if the student can choose their starting time, the usual measures would need to be in place for to make cheating harder. With an online test it is usual to have a question bank from which questions are chosen at random for each student, and some form of plagiarism detection for essays. Otherwise a student could note the questions, and pass these on others doing the exam later, so they could memorize prepared answers.

ps: Last weekend I was asked to help mentor the mentors in a Logistics Hackerthon for the Australian and New Zealand Defence Forces. The format is similar to ANU TechLauncher, but compressed into a few days. These provide good examples of how teaching and assessment can be done much better, without examinations.

Implementing Large Scale Electronic Examinations at ANU

Greetings form the famous room N101 at the Australian National University in Canberra, where Professor Steve Blackburn is speaking on "Electronic Exams" (video now available).

Steve pointed out that the ANU Research School of Computer Science have been conducting on-line examinations for a decade. This has focused on a Linux environment for testing computer programming, but Steve suggests the procedures could be applied more broadly for other disciplines.

Steve pointed out that Moodle had also been used for on-line examinations. I conducted such an examination in a RSCS lab in 2007 for a short course on Electronic Document Management. This was a small group and one invigilator was used to ensure the students only used the specified websites. As Steve pointed out, securing Moodle for a large scale examination is problematic.

UTas/UQ have a project for electronic examinations on BYOD devices "Transforming Exams". Steve mentioned some of the complexities in this approach.

Steve decided to use a browser based electronic examination which mimics the look of a paper based examination. The web pages use persistent storage, javascrpit, CSS and other standard web features.

A sample of the exam is available. The examination web page has the color and layout of a paper examination form at ANU. Interestingly, while intended for a specific desktop environment, because the web page uses standard and straightforward code, it displays reasonably well on a mobile device. Also an accessibility test reported only one known problem (title missing). This suggests the examination would be suitable for those with a disability.

One interesting point is that the 2015 electronic examination was designed based on the 2014  paper one. Steve commented that students complained that students found the electronic version too hard and too long, compared to the paper equivalent. Also students had lower marks for the electronic than paper examination. He suggests this may be because students would sketch on paper, whereas they are encouraged to immediately work on the detail in the electronic form. One of the audience commented that the expectations of examiners may be higher with electronic examinations, as the limitations of handwriting are not there. Steven suggests this needs some rethink of question design and practice examinations.

Steve commented that this was being presented as an example of practice not as something claiming to be groundbreaking research. Clearly the audience did not agree with this and suggested it was worthy to be shown at the Australasian Computer Science Week of Conferences (ACSW 2016), being hosted at ANU, 2 February 2016.


One issue electronic examinations raises is: why have examinations at all? If the students are using on-line systems day-to-day, then why not use this for assessment? One issue raised by the audience are the accreditation requirements of external agencies. However, I suggest there are ways to satisfy these requirements and in some cases the assumed impediments are not real. As an example, in 2009 when I set out to design a fully on-line course at ANU, some commented that this would not meet the university rules for accession, as every course had to have an "examination".  A check of the rules showed this was not the case: courses do not have to have examinations and where there are examinations these need not be on paper.

Large Scale Electronic Examinations at ANU

Professor Steve Blackburn will speak on "Electronic Exams" at the Australian National University in Canberra, 1pm 3 December 2015:

"This talk will discuss an experiment with a large scale electronic examination.

Pen and paper exams are the staple of examinations at institutions like ANU, however the pen-and-paper modality has a number of significant shortcomings:

1. It is pedagogically limiting; evaluating the students in a setting often unlike the environment in which the teaching and learning occurs, denying access to tools that may be fundamental to the subject matter;
2. It does not lend itself well to large classes and the possibility of auto-grading;
3. student handwriting can be very hard to read!  

Electronic exams address these concerns and create a number of opportunities in the design of the exam and in the marking process. The Research School of Computer Science has been using electronic exams for over a decade, using a flexible and robust system of locked-down computers. 

However, the systems used at the Research School of Computer Science were tailored towards computing courses and are did not include a generic, straightforward and accessible framework in which the exam itself could be conducted. Moodle also provides means for running exams, but has a number of open challenges relating to security and robustness in an exam setting.   This semester I built a browser-based electronic exam that is flexible, amenable to other disciplines and content, is designed to look familiar and easy to use, is client-side (not requiring connectivity to a server) and is usable in any operating system that supports a modern web browser.

In this talk I will discuss the various factors that motivated the design of this electronic exam, its applicability to other disciplines, challenges and opportunities ahead, and the outcome of using the system in a large (>200) first year class in the Semester 2 2015 examination period.

You can see a sample of the exam here (https://cs.anu.edu.au/courses/comp1110/sample-exam/1110-exam.html).  

This talk will be targeted to a broad audience and I will ensure that there is plenty of time for discussion among attendees."