Custom Questions for Each Student Based on Their Submitted Work?

Generative AI has the ability to create questions for students from the syllabus. This is very easy to do: you point the AI at the course documentation and tell it how many and what sort of questions you want. If multiple choice it also provides the answers. Could this also be done for each student, with custom questions based on their individual project work? 

Much has been written about oral examinations as the solution to students cheating. This is traditionally done for project based students, such as those undertaking a PhD. I had to go through a similar process at the end of my MEd, asked questions by a panel, based on what I submitted. However, this would be infeasible expensive for a program with hundreds, or thousands, of students. One option would be to have students question each other (this as been used in the ANU Techlaucher program). Also group projects reduce the workload, while introducing their own problems. 

Recently when attending training for Instructure's Canvas Learning Management System, being introduced at the Australian National University, I asked if each student could be given a separate file during an online test. The idea would be to provide them with a copy of their assignment, then ask them questions specifically about it. At the time I assumed the same set questions would be asked of every student. But it occurs to me now, why not ask them each a custom set of questions based on their work, and not provide a copy of the work to them?

High ceilings give low exam results

I suggest Bower, Broadbent, Coussens and Enticott receive an Ig Nobel Prize for their paper "Elevated ceiling heights reduce the cognitive performance of higher-education students during exams". The researchers found the higher the ceiling, the worse students do in exams. This is based on analysis of 15,400  exam results at three Australian campuses over eight years. At first I thought this an April fools day joke, but it seems to be real serious research. The work is deserving of an Ig Noble, recognizing research which has humor, but provokes thought. The key point of the research for me is not that high ceilings disadvantage some students, but that exams do. This can be corrected, I suggest, not by lowering ceilings, but by replacing exams with better assessment techniques.

The research comes at the time when exams should be on their way out. This is  like inventing a more efficient steam engine in the 21st century: an obsolete dangerous technology which no amount of technical improvement can save. Exams cause students stress (I have spent decades avoiding any course which had an exam). That a high ceiling might increase stress is interesting, but I doubt it could be lowered enough to make me comfortable with an exam. I stopped setting exams around the time I stopped giving lectures (2018).

The obvious factor which would cause the effect is the size of the room (bigger rooms having higher ceilings). However, the authors appear to have considered room size as a factor, and controlled for many other possible causes. One not mentioned might be that large rooms may have students from different classes taking tests at the same time, which could make students feel less comfortable.

If this is a real effect it could be easily corrected for by lowering the perceived height of the exam room ceiling. This could be done with lighting.

Reference

Bower, I. S., Broadbent, J., Coussens, S., & Enticott, P. G. (2024). Elevated ceiling heights reduce the cognitive performance of higher-education students during exams. Journal of Environmental Psychology, 102367. https://doi.org/10.1016/j.jenvp.2024.102367

Don't Replace Long Paper Examinations with Electronic Ones

Last week I tried a couple of demo exams with one of the leading online exams products. While just a practice, these brought back a tinge of the terrors I suffered as a student with assessment. As an instructor, I would not use such a system for long examinations. Such examinations are not a good way to assess the knowledge, and even less so for assessing skills needed for real work tasks. In addition such examinations increase the risk to the health and safety of students. Students were already at risk of mental illness and self harm before COVID-19. They are now at higher risk, and there are better, safer, ways to carry out assessment. Online invigilated tests should have only a small part in assessing students.

There were some good points with the online exams. The compatibility test beforehand warned my CPU was too slow, but it worked okay. I have a relatively under-powered laptop by today's standards and slow Internet connection.  This is not just because I am a cheapskate, I am doing what Ken Mattingly did for Apollo 13: make sure that if it works on my equipment, it will work for those out in the void.

The business of panning the camera around the room, to make sure no one else was present was a bit of light relief.

I has some minor quibbles:

• The instructions said to click "I agree" but the button said "I accept". That might seem trivial, but it can unsettle an already anxious student.
• There was no timer I could see indicating how long I had left. I did not dare look at my watch in case this was seen as cheating. Apparently there was a timer, but when someone is under stress, their attention narrows. The ATSB recently reported that two pilots landed an aircraft without the wheels down, because they were distracted.
• I had difficulty getting the camera to accept my image. I had to move a lamp so it shone on my face, which was uncomfortable.
• After completing all questions I could not work out how to get out of the software. I ended up closing down the browser, which in a real exam would cause additional anxiety. It turned out the exit button was just under the timer which I had not seen either.

Having tried this process twice, I could image taking a short test this way.

Overall the process was stressful, like a regular exam. There is the comfort of a familiar environment, rather than an exam room, but help is further away. If it was for a small amount of my grade (perhaps 20%) and short (around 30 minutes), this might be tolerable, just. But if for the majority of the grade in a course and for hours, there is no way I would ever contemplate sitting such a test.

I have spent a lifetime avoiding formal examinations. The first strategy is, of course, to not enroll in any course or program of study which has a long formal high stakes examination. If confronted with an examination, my first thought is: "Do I have enough marks already to skip this and still pass?". If all else fails, my options are reduced to withdrawing from the course, or the program, and then complaining to accrediting and funding authorities about the poor assessment process.


In my view, any exam should be relatively short, more like a take-home examination, than end of semester one. One issue is when the students can do it. I worry that some course designers are assuming all the students take the test at precisely the same time. This will not work well at home. There should be a period of a day, or days, over which students can choose to start their exam.


Academics and university administrators may be using online examinations in the mistaken belief that some law, policy or rule requires examinations. Even before COVID-19, there was flexibility with how assessment could be done. Assessment rules generally have a lot of detail about how to do a traditional examination, but then include a get-out clause saying that some other form of assessment can be used. Other forms of assessment take skill to design and resources to administer. But if you have been trained in how to design assessment (as I have), this can be done reliably, at reasonable cost.


Online exams and quizzes have been used previously for low stakes assessment throughout a course. The products being used now add automated invigilation. But if the student can choose when to start the exam, other measures will still be needed to make cheating harder. It is usual to have a question bank from which questions are chosen at random for each student, and some form of plagiarism detection for essays. Otherwise a student could note the questions, and pass these on others doing the exam later, so they could memorize prepared answers.

ps: Last weekend I was a Lead Mentor for the Australian and New Zealand Defence Forces Logistics Hackerthon. The format is similar to ANU TechLauncher, but compressed into a few days. These are good examples of how teaching and assessment can be done in a more realistic way. Participants have to undertake a real world task, and are assessed on how well they do it. There are no formal examinations, instead participants show what they did.

Make Online Exams More Like the Real World

Universities are considering how to replace end of semester examinations. There is a technological solution with remote proctoring software such as ProctorU and Proctorio. These present the examination questions to the student on the home computer, and collect the answers. The software locks down the student's computer, so they can't look up answers on the Wikipedia. The software uses the student's web camera to see it is the student doing the test, and no one is helping. There are obvious limitations with such software, but what concerns me more is the stress it places on the student.

I have never had to take a test using online proctoring software. So I tried some practice exams. While just a practice, these brought back a tinge of the terrors I suffered as a student with conventional examinations. That could be lessened if taken with a human invigilator present (something CIT has used for years, in their libraries).

There were some good points: I was warned my CPU was too slow, but it still worked okay. I have a relatively under-powered laptop by today's standards and slow Internet connection.  This is not just because I am a cheapskate, I am doing what Ken Mattingly did for Apollo 13: make sure that if it works on my equipment, it will work for those out in the void.

Some minor quibbles:

1. The instructions said to click "I agree" but the button said "I accept". That sounds trivial, but such things can induce panic in an already stressed student.
2. The same questions in the quiz kept reappearing after I had answered them. This may have been a feature of the examination being used, not the proctoring software, but it was confusing.
3. There was no timer I could see indicating how long I had left. I did not dare look at my watch in case this was seen as cheating. Later I was assured there was a timer on screen, but I never noticed it. So it might be worth pointing this out to students.  When someone is under stress, their attention narrows. The ATSB recently reported that two pilots landed an aircraft without the wheels down, because they were worried about something else
4. I had difficulty getting the camera to accept my image. I had to move a lamp so it shone on my face, which was uncomfortable.
5. After completing all questions I could not work out how to get out of the exam. I ended up closing down the browser, which in a real exam would cause additional anxiety.

But I can't imagine being able to sit though more than 30 minutes of this mental pressure. If confronted with it for a small proportion of the course grade (up to about 20%), I would be tempted to skip the exam and hope to pass. Otherwise I might withdraw from the course, or the program (then complain to accrediting and funding authorities).

As an instructor, I would not use such a system for long, high stakes exams, as it would be an unacceptable risk to the health and safety of students. The students were already at risk of mental illness and self harm before COVID-19. They are now at higher risk, and there are better, safer, ways to carry out assessment.

 

Student well-being is more an issue of those setting the exams, than the technology. These exams could be relatively short, more like a take-home examination, than end of semester one. One issue is when the students can do it. I worry that some examiners are assuming this will be like a paper examination, where all the students take the test at precisely the same time, which will not work well at home. What I suggest is a period of a day, or days, over which students can choose to start their exam.

Take Home Exams? There is a general assumption by academics that assessment has to be in the form of an examination. In fact university rules, and the laws they comply with, allow any  form of assessment. The assessment has to be rigorous, but sitting people in a room all at the same time answering the same questions is not the only way, or the best way to do this.

Online invigilated exams differ from a traditional take home examination in that there is automated invigilation and an elapsed time limit. The student can still choose when to start the online exam, if the person setting the exam allows that.

However, if the student can choose their starting time, the usual measures would need to be in place for to make cheating harder. With an online test it is usual to have a question bank from which questions are chosen at random for each student, and some form of plagiarism detection for essays. Otherwise a student could note the questions, and pass these on others doing the exam later, so they could memorize prepared answers.

ps: Last weekend I was asked to help mentor the mentors in a Logistics Hackerthon for the Australian and New Zealand Defence Forces. The format is similar to ANU TechLauncher, but compressed into a few days. These provide good examples of how teaching and assessment can be done much better, without examinations.

Implementing Large Scale Electronic Examinations at ANU

Greetings form the famous room N101 at the Australian National University in Canberra, where Professor Steve Blackburn is speaking on "Electronic Exams" (video now available).

Steve pointed out that the ANU Research School of Computer Science have been conducting on-line examinations for a decade. This has focused on a Linux environment for testing computer programming, but Steve suggests the procedures could be applied more broadly for other disciplines.

Steve pointed out that Moodle had also been used for on-line examinations. I conducted such an examination in a RSCS lab in 2007 for a short course on Electronic Document Management. This was a small group and one invigilator was used to ensure the students only used the specified websites. As Steve pointed out, securing Moodle for a large scale examination is problematic.

UTas/UQ have a project for electronic examinations on BYOD devices "Transforming Exams". Steve mentioned some of the complexities in this approach.

Steve decided to use a browser based electronic examination which mimics the look of a paper based examination. The web pages use persistent storage, javascrpit, CSS and other standard web features.

A sample of the exam is available. The examination web page has the color and layout of a paper examination form at ANU. Interestingly, while intended for a specific desktop environment, because the web page uses standard and straightforward code, it displays reasonably well on a mobile device. Also an accessibility test reported only one known problem (title missing). This suggests the examination would be suitable for those with a disability.

One interesting point is that the 2015 electronic examination was designed based on the 2014  paper one. Steve commented that students complained that students found the electronic version too hard and too long, compared to the paper equivalent. Also students had lower marks for the electronic than paper examination. He suggests this may be because students would sketch on paper, whereas they are encouraged to immediately work on the detail in the electronic form. One of the audience commented that the expectations of examiners may be higher with electronic examinations, as the limitations of handwriting are not there. Steven suggests this needs some rethink of question design and practice examinations.

Steve commented that this was being presented as an example of practice not as something claiming to be groundbreaking research. Clearly the audience did not agree with this and suggested it was worthy to be shown at the Australasian Computer Science Week of Conferences (ACSW 2016), being hosted at ANU, 2 February 2016.


One issue electronic examinations raises is: why have examinations at all? If the students are using on-line systems day-to-day, then why not use this for assessment? One issue raised by the audience are the accreditation requirements of external agencies. However, I suggest there are ways to satisfy these requirements and in some cases the assumed impediments are not real. As an example, in 2009 when I set out to design a fully on-line course at ANU, some commented that this would not meet the university rules for accession, as every course had to have an "examination".  A check of the rules showed this was not the case: courses do not have to have examinations and where there are examinations these need not be on paper.

Large Scale Electronic Examinations at ANU

Professor Steve Blackburn will speak on "Electronic Exams" at the Australian National University in Canberra, 1pm 3 December 2015:

"This talk will discuss an experiment with a large scale electronic examination.

Pen and paper exams are the staple of examinations at institutions like ANU, however the pen-and-paper modality has a number of significant shortcomings:

1. It is pedagogically limiting; evaluating the students in a setting often unlike the environment in which the teaching and learning occurs, denying access to tools that may be fundamental to the subject matter;
2. It does not lend itself well to large classes and the possibility of auto-grading;
3. student handwriting can be very hard to read!  

Electronic exams address these concerns and create a number of opportunities in the design of the exam and in the marking process. The Research School of Computer Science has been using electronic exams for over a decade, using a flexible and robust system of locked-down computers. 

However, the systems used at the Research School of Computer Science were tailored towards computing courses and are did not include a generic, straightforward and accessible framework in which the exam itself could be conducted. Moodle also provides means for running exams, but has a number of open challenges relating to security and robustness in an exam setting.   This semester I built a browser-based electronic exam that is flexible, amenable to other disciplines and content, is designed to look familiar and easy to use, is client-side (not requiring connectivity to a server) and is usable in any operating system that supports a modern web browser.

In this talk I will discuss the various factors that motivated the design of this electronic exam, its applicability to other disciplines, challenges and opportunities ahead, and the outcome of using the system in a large (>200) first year class in the Semester 2 2015 examination period.

You can see a sample of the exam here (https://cs.anu.edu.au/courses/comp1110/sample-exam/1110-exam.html).  

This talk will be targeted to a broad audience and I will ensure that there is plenty of time for discussion among attendees."

Thesis Examination at ANU

Greetings from the Australian National University where Margaret Kiley iS speaking on "Everything you ever wanted to know about thesis examination, but were too afraid to ask". She explained that the thesis is just the end of a process which starts with the entry process. Previously at least a first class honors, or equivalent, was required for a PHD. There are now students who are ready to learn, but may not have this background.

ANU has "Research Theses Submission and Examination: Information for Higher Degree Research Students". The engineering college of ANU is an example of a detailed step by step process for a PHD, with Progress Milestones.ANU also allows a "Thesis by Compilation".

Dr. Kiley described a European approach where the candidate has to publicly defend their work, with anyone able to ask a question. Under the UK system students are asked questions in a "viva", with external experts asking questions in a closed group. Canada has an external expert assessing the thesis, followed by a small group asking questions  (I am a Canadian distance education student, where the questions are asked on-line). In Australia only the written thesis is examined by two or three anonymous examiners. I asked Margaret f there were any universities which conduct a double blind process (the examiners do not know who the candidate is, as well as students not knowing the candidate), she didn't and argued that the examiners knowing who the candidate is a good thing.

One aspect which worries me is that the examiner is paid only a token few hundred dollars to examine a thesis, when the work required would cost thousands of dollars. Margaret explained that the examiner does the job not for the money, but as a reciprocal arrangement and because they then have access to detailed new research. It would seem to me the result will be that only salaried academics (and full time researchers) will therefore be able to be examiners. It would be prohibitively expensive for someone who is paid by the hour to spend a week examining a thesis for a few hundred dollars. Also this process of favors may not meet the requirements of ethical standards. Universities in several Australian states are subject investigation against commissions against corruption. The situation where academics trade flavors with supposedly independent examiners may be seen as corrupt.

ANU currently has no set period for examiners to return their comments (four to six weeks looks reasonable).  One enhancement which could be made to the ANU system would be for the degree to be conferred electronically as soon as the student has met all requirements. This would allow the student to receive their degree weeks, or months, earlier. The award ceremony with gowns and pieces of paper could still be held, but this should not hoold up awarding of the degree.

Event Details

The examination process is one of the least discussed aspects of research candidature. Candidates and supervisors are often so busy getting research done they spare little time for thinking about the examination process until the very end. There is a surprising amount of literature on how examiners approach and respond to thesis text which can be a rich source of information about how to write one in the first place. In this session Margaret Kiley, author of the seminal paper It's a PhD, not a Nobel Prize discusses her research on the issue and the subsequent research that has been conducted.
How does the Australian examination process work and how does the examination of ANU PhDs stand up to international best practice?
Why do people fail or asked to do major corrections?
How can supervisors better prepare candidates for thesis submission and the examination process and what can ANU do to help?

Bring-Your-Own-Device (BYOD) e-Examinations

Today I attended a fascinating webinar “Bring-Your-Own-Device (BYOD) e-Examinations” hosted by Ascilite. The first speaker described Transforming Exams platform, developed with an Office for Learning and Teaching grant by University of Queensland and University of Tasmania. The system provides an off-line electronic exam environment. The examination is prepared in a format compatible with Moodle and then loaded onto USB flash drives. The drives include the Ubuntu Linux operating system and when the computer is booted it restricts the user to the examination system.

This could provide a very useful transition for institutions which have paper based examinations. It allows an environment which emulates the paper system, but provides some benefits of electronic delivery.

While the Transforming Exams system is ingenious, I found the idea of having examinations a bit dated. I design online courses for students, who may be anywhere in the world and so use of an examination venue would be cumbersome, even without the paper. From the point of view of pedagogy, I don't think an examination is a realistic way to assess real skills. As a student I would not consider enrolling in a course which had a high stakes examination.

How, What and When of Improving Student Satisfaction

Recently I was asked for the how, what and when of improving the student's satisfaction with courses:

1. How: Focus on assessment, course materials and feedback. Student activities are important, lectures are not.
2. What: Small regular assessment items, at least two large assignments, minimal exams. On-line course materials, with optional lectures. Feedback with assessment results.
3. When: Assessment every week or two, all course notes and assessment tasks provided before the course starts, mark and feedback within a week of small assessment submission.

SUGGESTIONS FOR A TYPICAL COURSE

Here are some suggestions I provided for one course (some details omitted, to protect the course designer's feelings):

Better course materials and more progressive assessment would improve course completion rates. It does not matter if a Learning Management System (LMS), or something else, is used to deliver course materials, provided they are well designed. Ordinary web pages and Streams are a viable alternative to Wattle. Using Wattle for all courses would not be a significant improvement.

The assessment, course notes and staff contact details should be most prominent on course websites. Details of lecture times should not be so prominent, as students place a low value on lectures and if staff try to make students attend they sound desperate. All lecture notes and assessment items should be provided to student at the start of a course and no changed made, except where errors are found.

Just as there is an examiners meeting at the end of semester, I suggest a meeting at the beginning of the semester, where the person in charge of each course presents their course material and assessment design to their peers. There would then be peer pressure to produce clear, complete and consistent work.

Some points:

* TRIM COURSE DESCRIPTION:

The Course Description is a little long. I suggest cutting it by one quarter to just two paragraphs. The Learning Outcomes are far too long and I suggest cutting them by half to about four points. They should be derived from some accepted skills standard, such as (for ICT) the ACS Body of Knowledge, Skills Framework for the Information Age, or similar and be explicitly covered in parts of the course notes, workshops and assessment.

The Indicative Assessment description is misleading. It says there is almost an equal weighting between Continuous Assessment and Examinations. But the assessment page indicates a large end of semester examination a small mid semester exam and a couple of small assignments.

An alternative assessment strategy would be 20% for each of the mid and end of semester exams, mid and end of semester assignments and 20% total for a short lab exercise or quiz each week. However, consideration should be given to eliminating the conventional examinations entirely, as they have little value as an assessment tool.

I suggest having firm deadlines for all assessment: late submission should result in a mark of zero, unless special consideration is granted. Students should be encouraged to submit a draft of each assignment at least three days in advance, with failure of the student's computer system (or the institution's computer system) on the day the assignment is due, not being  grounds for special consideration (nor would an illness lasting less than three days).

Students who have a propensity to submit work late would quickly learn this was not going to be tolerated and change their ways (I find this very effective in my Green ICT course).

* MORE ON STAFF AND ASSESSMENT ON COURSE WEB PAGE:

The web page for the course has a photo which appears to have nothing to do with the topic of the course. Students like to know who their teachers are, so I suggest replacing this with ones of the teaching staff, with links to their bios. In preference to email, provide Moodle Dialogue or similar. This keeps all correspondence with the student in one place, so that all teaching staff can see it.
The web pages emphasize attending lectures, but these are the least useful part of a course (and if you have to beg students to attend lectures you sound desperate). I suggest providing more details of the assessment up front, which is what the students consider important.

Labs are a useful educational technique, so I suggest communicating this to the students by making all labs assessed. Students can submit material on-line and be marked for it.

* COURSE NOTES NOT LECTURE SCHEDULE:

The lecture schedule should be replaced with a course ebook. This would contain all the notes for the course, provided to the students at the start of the course. The subtitle "to be adapted and refined throughout the course" should be removed, as this says to the students that the course has not been properly designed.

Lectures could continue to be held, but it should not be assumed that students attend. Consideration could be given to using a workshop format with min-lectures, but this would require a TEAL type room, large enough to hold all students.

* COURSE BOOK FOR WEEKLY LABS:

I suggest the Weekly laboratories material be inserted in the course notes under the appropriate topic headings. Having many separate documents creates confusion.

* ONLINE PEER ASSISTED LEARNING:

Peer Assisted Learning (PAL) sounds like a good idea, but I could not understand from the web page how it was supposed to work. Also this appears to be only a face-to-face group. I suggest it be extended with on-line forums.

* CLARIFY DIFFERENCES WITH POSTGRADUATE COURSES:

I had difficulty understanding how the postgraduate version of the same course was supposed to work. It seems to add more work for postgraduates. That suggests that either the undergraduate version has less than the usual amount of work or postgraduate has too much.