Greetings from EduTech Asia in Singapore where a panel is discussing "Panel: Is it time for reform? The bid for authenticity in assessments". With Edwin Lim, Hwa Chong Institution, Yian Hoon Lim, Seng Kang Secondary School, Barbara Tajti, Saint John Mary International School and Kerry Moore, Stonyhurst International School Penang. There are many teachers from international schools in the audience. You may wonder why as a university lecturer I am at a school session. But there is a lot in common between international schools and universities. They have international students and have to cope with international needs.
The topic of reflection just came up. This is something I teach, mentor and assess students on. I have spent many years being trained on how to do it, but I am still not sure I understand what it is. One thing I would like to explore is how to automate asking the student about the work they submitted: "Here you wrote ... why?".
Problem based learning and design thinking also came up.
Generative AI has the ability to create questions for students from the syllabus. This is very easy to do: you point the AI at the course documentation and tell it how many and what sort of questions you want. If multiple choice it also provides the answers. Could this also be done for each student, with custom questions based on their individual project work?
Much has been written about oral examinations as the solution to students cheating. This is traditionally done for project based students, such as those undertaking a PhD. I had to go through a similar process at the end of my MEd, asked questions by a panel, based on what I submitted. However, this would be infeasible expensive for a program with hundreds, or thousands, of students. One option would be to have students question each other (this as been used in the ANU Techlaucher program). Also group projects reduce the workload, while introducing their own problems.
Recently when attending training for Instructure's Canvas Learning Management System, being introduced at the Australian National University, I asked if each student could be given a separate file during an online test. The idea would be to provide them with a copy of their assignment, then ask them questions specifically about it. At the time I assumed the same set questions would be asked of every student. But it occurs to me now, why not ask them each a custom set of questions based on their work, and not provide a copy of the work to them?
Greetings from the CRADLE Seminar "Assessment beyond the individual unit/module and AI". Associate Professor Jason Lodge is talking about how to see how students progress over time, rather than assessing in small packets of instruction. While he did not explicitly say it, I assume he envisions this makes it harder for the student to cheat, using AI, or otherwise, as they would not be showing consistent progress.
Margaret Bearman took us through the logic of current unit based assessment and asks about "big picture" outcomes. My reaction was "Inst that what capstones are for?". You have the student do a big project at the end of their study, where they have to demonstrate the skills needed.
Surprisingly, there was little mention of AI, which is refreshing. The approach is to get the assessment right and cheating will be harder, however it is done.
I asked the panel:
'Will technology help? Could we give the AI each student's CV and have it suggest what degree requirements they have already met? I help out with applications for course credit and there is a lot of stuff students have done they really don't have to do again. More than once I have thought we should have the student teach the course. ;-)
I suggest Bower, Broadbent, Coussens and Enticott receive an Ig Nobel Prize for their paper "Elevated ceiling heights reduce the cognitive performance of higher-education students during exams". The researchers found the higher the ceiling, the worse students do in exams. This is based on analysis of 15,400 exam results at three Australian campuses over eight years. At first I thought this an April fools day joke, but it seems to be real serious research. The work is deserving of an Ig Noble, recognizing research which has humor, but provokes thought. The key point of the research for me is not that high ceilings disadvantage some students, but that exams do. This can be corrected, I suggest, not by lowering ceilings, but by replacing exams with better assessment techniques.
The research comes at the time when exams should be on their way out. This is like inventing a more efficient steam engine in the 21st century: an obsolete dangerous technology which no amount of technical improvement can save. Exams cause students stress (I have spent decades avoiding any course which had an exam). That a high ceiling might increase stress is interesting, but I doubt it could be lowered enough to make me comfortable with an exam. I stopped setting exams around the time I stopped giving lectures (2018).
The obvious factor which would cause the effect is the size of the room (bigger rooms having higher ceilings). However, the authors appear to have considered room size as a factor, and controlled for many other possible causes. One not mentioned might be that large rooms may have students from different classes taking tests at the same time, which could make students feel less comfortable.
If this is a real effect it could be easily corrected for by lowering the perceived height of the exam room ceiling. This could be done with lighting.
Reference
Bower, I. S., Broadbent, J., Coussens, S., & Enticott, P. G. (2024). Elevated ceiling heights reduce the cognitive performance of higher-education students during exams. Journal of Environmental Psychology, 102367. https://doi.org/10.1016/j.jenvp.2024.102367
Change of plan, now speaking on "Evaluating Generative AI’s role in curriculum design", 11:45 am Wed, 6 November in the Higher Education – Teaching & Learning stream of EduTech 2024.
Here are the questions I have been asked to address. Any suggestions?
What specific strategies or tools did you use to teach students how to assess the reliability and accuracy of AI-generated outputs?
What challenges do students face when learning to critically analyze AI outputs, and how can educators support them in overcoming these?
Can you share examples of project activities or exercises that helped students understand both the benefits and limitations of AI in real-world scenarios?
Looking forward to speaking on a panel on "Safety check: maintaining academic integrity through digital proctoring in assessments" in Singapore, at EDUtech Asia 2024, 7 November, 11am, Stage 4, with Girija Veerappan, & Mohd Rozi Ismail. I have lost count how many EduTechs I have been to. ;-)
ps: Edutech Asia 2024 asked me to make a video to invite you to my talk.
Greetings from day two of EdutTech 2023 in Melbourne. The exhibition is buzzing, with plenty of free coffee,. I tried to go to a panel on assessment, but it was standing room only. So I am in the plenary session, in a room that can seat thousands. I thought it was a panel on training more tech workers, but AI is being discussed. I suspect, whatever the official topic, the discussion is going to stray onto AI.
There is some comfort in being an "honorary" lecturer at a university. As you are not getting paid a salary, you feel no obligation to go to administrative meetings. You just take on the teaching, or research, you want to do. Some prefer to avoid the stress of grading student work, and just do some mentoring. I have been formally trained in how to mark students so don't find it too stressful. One thing I have avoided, for decades, is an administrative role at a university. But recently I was asked to help process applicants for credit from studnts for study done elsewhere. In a way this is an extension of assessment. But in another way it is a very tedious administrative process, involving many rules, and use of precedents. As a former bureaucrat, I can cope with rules, but even so it is a tedious process.
Processing applications for credit for Australia courses is not so hard. First of all all Australian universities are real universities, federally accredited. In some countries anyone can call something a university, but in Australia that is illegal.
Next of all Australian universities professors borrow heavily from each other. Often courses at different institutions have the same titles, descriptions, learning objectives and assessment. Also degrees are accredited by the Australian Computer Society and Engineers Australia, and have much the same content.
Checking out an institution in another country is a little more difficult. First of all is it real? Australian universities employ staff to carry out forensic analysis of documents presented by students. Then there is the problem of the quantity of learning in courses, and the form of assessment. This becomes a particular issue for practically orientated skills: can a student learn to program in a few weeks, and be assessed with multiple choice questions on their coding ability? Can a student learn project management, without actually being a member of a project team? How much does workplace experience count, be it as part of a formal internship program, or concurrently with study, count? Ultimately these things come down to judgement.
I am doing the "Detecting Contract Cheating" course from TEQSA. In Module 2 "Supply and demand" under "who are cohorts likely to cheat" lists "Students who speak a language other than English". That is, if you speak a language other than English, the claim is you probably cheat. But isn't that most of the people in the world? I find it hard to believe that English speakers are inherently honest, or not speaking English makes you dishonest. The course cites Bretag et. al (2019), but my reading of it doesn't support that conclusion.
However, the course, so far, is mostly good. In particular I like the section on how to deter contract cheating, by providing the students with the help they need. I had to give this some thought, because it is easy to say there is no excuse for cheating. However, if your family, or your entire village, have invested everything they own in your education, if you are having difficulty with the language, you have limited time for study away from family or job (or both), if the universality does offer you any help getting up to speed with the topic, it is very easy to be tempted to cheat.
The course consists of some short videos, which like a typical student I play at high speed and read the subtitles. Curiously, the videos kept referring to "expectations" for what higher education intuitions would do about cheating. This is odd as TEQSA are overseeing legally mandated practices, which I would have thought were "requirements", not expectations.
The course contains so odd phrases, such as "collective competence", to describe a team of people who, between them, have the necessary skills to deal with academic misconduct. This appears designed for individuals to avoid responsibility. The example of a surgical team is used. But every member of such a team is tested for individual competence. No hospital would allow a team to operate where it was not clear that every member had the required skills.
The course also contained a weird video explaining the usefulness of subject matter experts. It was as if universities normally use random individuals to do assessment, and using experts is an innovation. This was followed by a video about the value of "investigators". This is a term I learned a few months ago. It is something unlikely to be familiar to most who teach at university, and needs more background, explanation, and justification. Also the full title of the role should be used, not just "investigator", as this is also a term used in academic research (I have been a chief investigator). While not mentioned in the course, I assume investigators are required to be qualified, to a similar level to those who carry on investigations in Commonwealth agencies, with a Certificate IV in Government Investigations (PSP40416).
The course has some views on the seriousness of plagiarism charges I don't agree with. As an example a finding of plagiarism resulting in a 10% grade penalty is classified as low severity. However, as a student I found the one and only time I was accused of plagiarism as very serious. Fortunately this was a mistake by my instructor, as they had accepted Turnitin's text matching report. Turnitin matched my assignment with the published paper I produced from the assignment (I hadn't expected the paper to be published so quickly, so hadn't mentioned it in the assignment).
Some of the legal advice in the course may need checking. As an example, it is suggested if an institution receives a report from a contract cheating service that one of their students is using it this may be blackmail, and so the institution should report this to TEQSA, but that cheating at an institution is not a crime. I am no legal expert, but thought blackmail is a crime, to be investigated not by TEQSA, but by the police (or in some states under the jurisdiction of a corruption commission). Also cheating at an institution may be a crime, where the student profits from it. Also the course argues that the saftey and well being of the students should be the priority. However, if cheating is resulting in many incompetent people being licenced to carry out dangerous procedures, the public interest may outweigh the student's interests.
Some of the approaches proposed by the course will throw suspicion on innocent studnts, and unfairly discriminate against specific groups. As an example, those with a disability who use accessibility tools will likely have the metadata stripped from their documents. If the investigator, as suggested here, looks for the name of the student and the time they took to edit to be consistent will suspect cheating when all the happened as when student prepared their answer in their accessibility tool, then pasted it in. I have this problem with some online systems, which do not allow me to spell check. I have to compose offline, then paste the answer in. That can look like I am getting someone else to write it.
Another approach which might cause unfair suspicion is bibliographic forensics. The course suggests checking that the references a student uses match their expected level of knowledge. However, I was a student in a topic I had two decades experience in, but the rules required me to do the introductory units. To avoid the tutors saying (as they occasionally did "Tom what are you doing in this class?"), I would hind my knowledge of the topic and pretend to be an ordinary student. But using the techniques in this course that could make me look like I was cheating.
The course suggests LMS logs as a source of evidence, but doesn't discuss under what conditions this, or other, information can be used against the student. Do investigators need a reasonable suspicion, or can they simply sift through the LMS logs looking for any wrongdoing? To use an extreme analogy, they could also use face recognition on security cameras in student dorms to look for collusion, but a court would likely find that an unreasonable invasion of privacy.
There are some power imbalances in the procedures suggested in the course. As an example, the student is required to provide proof of identity at an interview, but the interviewer is not. It is a well worn cliche of police procedurals that investigators show their identification to suspects. Another example is that it is suggested that a support person not be permitted to speak extensively to the student in a language the interviewer does not understand. Unless it is slowing down the interview, what right does the interviewer have to prevent the student and their support person communicating in whatever language they are most comfortable with?
The course suggests having studnts bring their device to the interview. I find it disturbing that investigators would conduct a search of a student's device, as this is likely to contain a lot of personal, private, and sensitive information, both their own, that of family and friends, and of their employer. If asked to do this, my first thought would be to wipe the device clean of all records and logs beforehand to protect myself, family, and clients. That act in itself might then be considered suspicious by the investigator.
It is a little annoying as when I stopped to fill out the workbook, I ended back at the start of the module, and had to fast forward through the video again. The modules seem far too large to each do at one sitting. But if I stop, I risk having to go back to the start.
The workbook is in PDF. To enter my work I had to import it into the word processor, where the formatting went askew. I have to create a text box for each answer, and hope it will be readable in the final result.
Also I received a warning I had an unreliable Internet connection, and my progress may not be saved. That seems odd, as I am sitting in my university office with a very high speed internet connection. Most of the time everything loaded quickly, although the final reflection video stuttered.
Overall the course was of some use for learning more about how to investigate cheating, but was based on an approach using formal investigators. I would have liked some background as to why and how a decision had been made to introduce this role, and if all universities are required to implement them (if so, when?). Also I signed up for the course so I could go to a workshop. But I now can't see any details of the workshop, and it took me so long to do the course (about 7 hours, whereas it was supposed to take 3), that I have forgotten where I saw the workshop details. In hunting around I found a Situational Judgement Test which was taken before the course (and I had forgotten about).
Curiously I was not prompted to take the test as part of the course, but had to stumble over it. Finding what I might have to do next is difficult as I have the text enlarged to make it readable, so all I see is "Modules Situational Situational", not what the modules are, or what "situational" is.
References
Tracey Bretag, Rowena Harper, Michael Burton, Cath Ellis, Philip Newton, Pearl Rozenberg, Sonia Saddiqui & Karen van Haeringen(2019)Contract cheating: a survey of Australian university students,Studies in Higher Education,44:11,1837-1856,DOI: 10.1080/03075079.2018.1462788
Does anyone else find it ironic that universities are using AI to make sure the students don't? ;-) More seriously, one of my students asked if they could use ChatGPT to write a report on their work experience.
I checked the institutional rules, and found I had discretion as to what to do. I said "no", on the basis they would not be the author of the work, just an editor of it. But then their supervisor gave me a call to say they had suggested ChatGPT. This puts it in a new light, if this is an accepted, and encouraged, work practice. So what I am now thinking is have the student prepare a report much like one used for a Systematic Review.
The student framed their request in terms of using ChatGPT as a grammar checker. The problem is that ChatGPT does much more than simply correct grammar, it selects facts and concepts, drawn from a large collection of work from numerous authors, not always correct, or attributed.
Universities provide advice and assistance with writing. Professor Inger Mewburn at ANU (the Thesis Whisperer), recommends Grammarly, and the free alternative Typely. I used Grammarly as a graduate student, and found it very useful (English is my first, and only, language, but I only barely passed the subject at school).
My experiences in using Chat GPT, and its predecessor GPT3, and other AI, have been mixed. Recently I reviewed a paper generated by Chat GPT. I gave the paper one out of five, and suggested the name of the author should be "Anonymous" with the person who submitted it listed as an editor, not author. But not everyone agreed, with an average review of 2.6 out of 5.
Using an approach similar to a systematic review, the student would set out what they are attempting to accomplish, what queries they tried, what the results they got were, how they selected the most appropriate result, what manual changes they made to the automated result, a critique of the result, and recommendations for improvements to the process. The idea would be the student would demonstrate skill in using a tool to produce a result, and value add to the automated output.
When universities are preparing a Learning and Teaching Strategy, it is important to make what is proposed, staff and prospective students know the details. But what should be in such a strategy? What has worked well during the COVID-19 pandemic, and should be kept?
For the last few years I have been helping teach ANU Techlauncher, which has no exams, progressive assessment, some peer feedback, authentic and oral assessment, Work Integrated Learning (WIL): the lot. This is challenging for staff and students, but that is the point: it is a capstone exercise to ensure graduates are ready for the real world.
One aspect which has worked well is a reflective portfolio disguised as a job application, and could be applied generally as a program capstone across a university. The idea is that a student has to think about what career they want, and what they have learned which help with that, in a very useful way. This is more relevant that a student having to prepare a portfolio which might be useful some time in the future.
Previously I had used small quizzes and assessed forum contributions to keep online students working. Also I used a “best of” assessment scheme for small assessment tasks, so students could have a couple of bad weeks, without penalty. This helps cut down on requests for special consideration, and extra marks, as students know one missing or bad result will not penalize them.
However, staff designing, and doing, assessment need to be trained in how to. The average academic knows about exams and assignments. There is also resistance from academics to spreading assessment throughout the courses to keep students working. There is the reasonable fear this will increase staff and student workload, but it can reduce the workload, if well designed. Academics are not familiar with vocational style assessment which looks for competency, not a 100 point scale. When they realise they do not have to treat a small test like a major exam, they can relax a bit.
More low stakes assessment can be used. This could be applied for small tasks, while large assignments are used to identify high achievers, to give the benefits of the ungraded approach, but with grades.
Obviously authentic assessment should be used: it is natural and easy to apply in vocational courses. However, the research staff may not be qualified in real-world skills, or how to teach them. Peer and self assessment are fine for low stakes tasks, but is problematic for high stakes ones.
Obviously assessment should be scaffolded. The whole course should be there to support the assessment. If something is not assessed it should not be in the course. High stakes exams should be abolished. Small tests are okay.
A little oral assessment is okay, as long as it is linked to the learning objectives. The approach used by innovation centers (such as Canberra's CBRIN) to teach giving compelling presentations could be adopted by universities, or this teaching handed off to associated centers. Orals can be high stress: ask me about the pile driver during my MEd presentation. ;-)
Hackerthons could be incorporated. These could be within a course, a program, university wide, or open. The hackerthon packages a group project into a few days, rather than weeks.
Assessment templates, tools, and marking tools would be of some use. However, this is not a substitute for training staff in assessment. Personalised automated feedback would be of some use. The Techlauncher students have access to the university's careers automated tools, and we will try to have them use these more next semester. Templates for e-portfolios would be useful. However, this also requires staff training. Also I suggest a GitLab type repository, and advanced group working tools, as used by Techlauncher.
An ungraded university first year would be disastrous, unless academics who were also trained, qualified educators were to run it. Otherwise this would cause great stress for students, particularly those from diverse backgrounds. I suggest instead an approach using progressive assessment, where only the best grades count. That is something the students, and especially the staff, will be better able to cope with.
The overriding constraint on changing a university's assessment is the lack of academics trained and qualified in education. The major challenge is how to get competent staff without compromising a university's’s research focus.
Along with many university staff, I have been invited to be part of developing a Learning and Teaching Strategy. This is an issue exercising the minds of people at many universities at present (or should be), as we recover from the COVID-19 pandemic. I have spent some of the last ten years studying, researching and presenting on these issues, so note that these are my views, not necessarily those of any particular institution.
An Approach
The approach I suggest is that universities design for a student who is remote, part time, and focused on practical vocational outcomes. Then add optional on-campus, and more academic activities. In terms of teacher development, require formal qualifications, but with training based on workplace experience.
It is much easier to start with a course designed for remote students, and add on-campus activities, than the reverse. Also, when another emergency forces some, or all, students online, this can be accomplished with no change in course design or delivery, the on campus components can be simply cancelled, leaving the remote component to continue. This approach was found to be effective in response to COVID-19 (Narayan, et al., p. 168, 2021).
Some issues:
1. Learning
Use of digital learning environments, on-campus learning, blended, flipped and flexible. As a result of the need to switch to online learning during the Pandemic, universities have well developed technical support for modern teaching approaches. The students are expecting these. The problem, up until 2020, was in convincing staff to do more than give the usual lectures. COVID-19 forced a crash program of online delivery. The problem is now how to make this more than just recorded lectures.
In part, the problem with learning is one of the self image of academics. When I joined the staff of a university decades ago, I assumed I would be researching, and teaching by giving an occasional lecture. As a computer professional, who had an award for helping getting the nation on the Internet, I assumed I could easily translate classroom teaching online. It took about ten years to realise I needed to swallow my pride and learn how to teach online without lectures. This is a process we need to take academic staff through, using dogfooding: teach them to teach by having them a student.
2 . Assessment
Also as a result of the pandemic, universities have technical support for flexible and advanced authentic forms of assessment (even if they don't use these). The ideal form of assessment is where the student does what they will need to do after graduation, in the workplace, and how they do it is checked. This can be in a real workplace, with an internship, or other Work Integrated Learning (WIL), or some form of simulation. Ideally the assessment is progressive, throughout the students courses, and accompanied by timely relevant feedback, so they pay attention. However, good assessment is much harder than end of semester examinations (which are bad assessment). This requires academics to be trained in how to assess, and time to set up. Once set up using the digital tools, the assessment takes more work. But academics will require the training and support to get to this point.
As a student of assessment I did not believe much of what I was being told, until I had to experience it first hand (more dogfooding). As an example, I did not believe students did not read detailed feedback on assignments, until I got back my assignment on assessment and did not read the feedback. Only after this did I set about delivering feedback in smaller, more frequent chunks. Only after having to do group-work online, and reflective portfolio, did I understand what these were about.
3. Teaching
How to ensure quality teaching is a dilemma for all university, but especially for research intensive ones. Whatever the marketing slogans might say, research is the priority, and researchers generally do not make good teachers. While vocationally focused, and not an elite researcher, I still did not volunteer to undertake teacher training, and had to be forced to do it.
Universities will need to require staff to undergo teacher training, making recruitment and promotion conditional on achieving the required standard. I suggest this be done with formal courses, and AQF aligned qualifications. Voluntary schemes and ad hoc training courses are not sufficient. Universities have the opportunity to set up nested programs which can be a showcase for future offerings across the institution. As an example, micro-credentials which nest into a graduate certificate, diploma, and masters degree in university education, with WIL, & recognition of prior learning (RPL). Components of these programs can be offered to students, who are plan to be trainers in their discipline, as well as to staff.
4. Job Ready Graduates
Internships, WIL, and career skills, as typically provided in computing, engineering and other closely vocationally linked disciplines, can be expanded to other fields. As an example, the ANU Computing School offers internships to individual students, and group projects for real clients (Awasthy, Flint, & Sankaranarayana, 2017). ANU Careers guides the group project students through the process of documenting the skills gained, considering careers, and applying for a job. Rather than this being extra-curricular, it is integrated into a course, with assessment (Worthington, 2019). Further digital support for these resource intensive programs can be developed. As with other forms of education and assessment, it would be valuable for teaching staff to have undergone such a program as a student.
References
Awasthy, R., Flint, S., & Sankaranarayana, R. (2017, April). Lifting the constraints—closing the skills gap with authentic student projects. In 2017 IEEE Global Engineering Education Conference (EDUCON) (pp. 955-960). IEEE. https://doi.org/10.1109/EDUCON.2017.7942964
Narayan, V., Cochrane, T., Aiello, S., Birt, J. R., Alizadeh, M., Cowie, N., Goldacre, P., Sinfield, D., Stretton, T., Worthington, T., Deneen, C., & Cowling, M. A. (2021). Mobile learning and socially constructed blended learning through the lens of Activity Theory. In S. Gregory, S. Warburton, & M. Schier (Eds.), Back to the Future – ASCILITE ‘21. Proceedings of the 38th International Conference of Innovation, Practice and Research in the Use of Educational Technologies in Tertiary Education (pp. 166-171). Australasian Society for Computers in Learning in Tertiary Education.https://2021conference.ascilite.org/wp-content/uploads/2021/11/ASCILITE-2021-Proceedings-Narayan-Cochrane-Cowie-Goldacre-Birt-Sinfield-Mehrasa-Worthington-Aiello.pdf
Worthington, T. (2019, December). Blend and flip for teaching communication skills to final year international computer science students. In 2019 IEEE International Conference on Engineering, Technology and Education (TALE) (pp. 1-5). IEEE. https://doi.org/10.1109/TALE48000.2019.9225921
"Learning to Reflect" is a module for the ANU TechLauncher program, where students reflect on what they have learned, by writing an application for a real job, as their last assessed task before graduating. This was developed in late 2018 and first run in semester 1, February 2019. It was designed for blended delivery, with theoption of easy conversion to full online delivery. That option was needed for Semesters 1 & 2 in 2020, and Semester 1 2021 due to COVID-19. This version 5 is designed for hybrid delivery, with students in a physical classroom linked to those online, using the MidFlex Minimal Hybrid Format. Unfortunately COVID-19 again required a return to pure online delivery, but without any changes required to the material.
Two additional workshops were added from Version 4, at student request, bringing the total to four. The first three have exercises for a 2% mark each (not for the last when a major assignment is due). The optional student logbook has been dropped due to lack of use. The middle two workshops are half the length of the others with the first hour used for the new ANU Computing Showcase, featuring prominent ANU alumni, speaking about their careers.
Please note that the course notes only supply the structure for the module and the assessment. The detailed content for the workshops is provided by Tempe Archer, Careers Consultant at ANU Careers, and the students undertake exercises using the ANU Careers Toolkit.
A paper on the design and blended delivery of the module is available:
After reading "Life After Truth", by Ceridwen Dovey (2020) I understand a little better why North American students appear obsessed by Grade Point Averages (GPA). The novel is about a reunion of students at Harvard University. While fictional, Dovey studied at Harvard after completing school in Australia. The work ranges over the issues someone fifteen years out of university faces, in terms of life choices made, when confronted by those who made different choices at the reunion. At one point a character who decided to take a socially worthy, but low paying job, reflects on how top NY firms would headhunt Harvard graduates, based solely on their high GPAs. The novel is at times wistful about college life, at times savage about human vanities.
My experience of university is not quite on par with Dovey. But I did complete school in Australia and eventually studied in North America. My study was at a much less prestigious institution, but even so there seemed an unhealthy obsession with grades and GPAs. I understood I needed a specific minimum grade in each course to be able to graduate, but beyond that, what did it matter? Out in the real world, no one looks at grades you got in specific courses, they just check you have the appropriate qualification for the job. There are other measures of how good you are at some task. Perhaps it is different for recruiting by NY headhunters, but none of them is ever going to look at me anyway, so why worry?
But even now, today, during a pandemic and a North American heat wave, I am reading heart wrenching social media posts from students worrying about getting high grades. Why worry about this, when viruses, and the planet itself, is trying to kill you? Perhaps educators are at fault for part of this, by focusing students on grades. Perhaps I am guilty of that myself.
I recommend Dovey's "Life After Truth". But it does get a bit slow in the middle, with a bit more on child rearing and the problems of couples than I really wanted to know. Also the death of a character briefly mentioned at the beginning is too hastily resolved at the end.
I never went to university with anyone rich, famous, or infamous, as Dovey apparently has. I did not live on campus and was not part of a student club. For that I am grateful, as it all sounds an awful experience.
Greetings from the project examiners meeting of the Computer School at the Australian National University in Canberra. Due to COVID-19 this is being conducted online, via video conference. The format is much the same as a face to face examiners meeting. The meeting works its way through a spreadsheet (displayed on screen), where each line shows the proposed grades for a student from reviewers, a proposed final grade and color code to indicate if the academic in charge thinks there is consensus. Most are coded green, indicating there is a consensus and these are generally accepted with little discussion. Those coded amber need more discussion, while the red ones have vigorous debate.
In past years the discussion has got quite heated for a few projects, but not today. There were quite a few comments about the difficulty students had due to COVID-19, with some overseas and unable to meet in person with their supervisor. This can also be difficult where students are working on a project with human subjects, but unable to see them in person. However, I was impressed how, despite the difficulties, staff and students were still able to undertake valuable work.
An observer might think there is not much to such a meeting. But behind this are years of work establishing standards. With that all done, the final decisions can be made smoothly.
I have been looking at which online tools could be used for students to create a logbook toShow-Your-Work. The Mahara Journal looked promising, especially as it is usually installed alongside Moodle. However, this would still be an additional tool students and staff would have to become familiar with. An alternative which looks promising is the Moodle Wiki. This can be set up so each student gets their own. I have created a logbook template which can be provided in the Wiki.
My template has a paragraph of explanatory text, then sections for the student to fill in. The student could create extra pages if they have a lot of content. But I expect one page will do for a typical student. They just click on "edit" the heading for a week, and put in some content.
The fill in the blanks sentences are adapted from James (p.43, 2005). The topics for each week are from the Australian National University's Techlauncher program (Awasthy, Flint, and Sankaranarayana, 2017). Questions for the Work Portfolio (Weeks 4 and 8) were suggested by Tempe Archer, ANU Careers. The idea here is to provide a prompt for the student each week to start writing and avoid presenting then with a confronting blank page. The students are asked to write about the activity set for that week and a specific aspect of it.
Jacques, Ouahabi and Lequeu (2020) refer to the use of logbooks in Google Drive for French first year first year engineering students learning online, but unfortunately give no more details. Kumar, Silva and Prelath, R. (2020) mention not having a project logbook as a problem for studio based learning in a Malaysian course, but again provide no more details.
Reference
Awasthy, R., Flint, S., & Sankaranarayana, R. (2017, April). Lifting the constraints—Closing the skills gap with authentic student projects. In 2017 IEEE Global Engineering Education Conference (EDUCON) (pp. 955-960). IEEE. URL https://doi.org/10.1109/EDUCON.2017.7942964
James, Alisa, "Journaling as an Assessment Option" (2005). Kinesiology, Sport Studies and Physical Education Faculty Publications. 78. https://digitalcommons.brockport.edu/pes_facpub/78
Jacques, S., Ouahabi, A., & Lequeu, T. (2020). Remote Knowledge Acquisition and Assessment During the COVID-19 Pandemic. International Journal of Engineering Pedagogy (iJEP), 10. URL https://www.thierry-lequeu.fr/data/JACQUES-04.pdf
Kumar, J. A., Silva, P. A., & Prelath, R. (2020). Implementing studio-based learning for design education: a study on the perception and challenges of Malaysian undergraduates. International Journal of Technology and Design Education, 1-21. URL https://link.springer.com/content/pdf/10.1007/s10798-020-09566-1.pdf
Chicago
Logbook
You can use this logbook to keep notes, plans, and drafts of your work. This logbook is not assessed and is not visible to other students. However, it can be used as supporting evidence that the work you submitted for assessment was your own. All entries are logged and timestamped by the system, so the examiner will be able to see when you made notes and prepared drafts of your work.
Logbook
Your Name:
Student ID:
Week 1: Team Formation
Date:
Entry: The project is to ___ . My role is to ___.
Week 2: Orientation
Date:
Entry: To successfully undertake my role I will need to learn to ___ . To do that I plan to ___.
Week 3: First Audit
Date:
Entry: The team managed to ___. To help with that I ___.
Week 4: First WPP Workshop
Date:
Entry: The key points from the WPP workshop were ___. The job I am considering is __ . So I will need to work on my ___ .
Week 5
Date:
Entry: One of the areas I need to work on with the teamwork is my ___ behavior. To address this I intend to ___ .
Week 6: Second Audit
Date:
Entry: The hardest part of the audit was ___ . This is because ___ .
Week 7
Date:
Entry: One of the skills I need to work on is ___ . To improve, outside the course I will ___ .
Week 8: Second Work Portfolio Workshop
Date:
Entry: Here is my unpacking of a job advertisement for the WPP workshop: ___
Week 9: Project Showcase Video
Date:
Entry: The main aspects to developing the video were to ___. My role was ___ . I found this ___.
Week 10: Third Audit
Date:
Entry: I was able to help the team with the audit by ___ . This could be useful in my career ___ .
Week 11
Date:
Entry: The team uses different forms of communication for different purposes. This includes ___ to ___ and ___ to ___. It could be improved by ___ .
I have been looking at how to use tools such as Microsoft Word in Office 365, Mahara Journals, or Git for students to have STEM students Show-Your-Work. There has been much written about the use of reflective e-portfolios for assessment. I have been trained to write, teach and assess reflectively. However this is something the average STEM student and teacher finds very difficult. So I propose a form of non-reflective, non-assessed student journal: the student is prompted to record what they did.
James (2005) sets out how to guide physical educations students through preparing an assessed journal. They are referring to paper journals, but the technique applies also to electronic ones. First the student should be encouraged to be prepared with a suitable journal personalized to their tastes. With an e-journal I suggest a preliminary exercise where the student ensures they have access to the e-journal system, and enter some information about themselves.
Ensuring students don't lose an e-journal is less of a problem than for the physical ones James discusses (p.42, 2005). An e-journal system used by an institution should obviously be protected from hacking and based up. However, it would nevertheless be useful to guide students through taking a copy of their journal, and reminding them not to edit old entries (as that then updates the time stamps ruining their value as evidence).
Writing prompts are more important for an e-journal used with an online course, than for the face to face classes James (2005) discusses. As the student will be mostly studying asynchronously, the teacher is not there and then saying "put a copy of that in your journal now". This has to be explicitly stated in course materials, ideally as part of a assessed task, so the student has an incentive. This can also be a good point. I suggest to mark a point in the course, an approach of synchronization of asynchronous learning (Worthington, 2013).
Writing prompt suggested by James (p.43, 2005) include:
"Write a paragraph about a ... goal you would like to reach. Explain why you want to reach that goal.
Write a paragraph about what you did today that helped you to be successful in today's activities.
Write a paragraph about your ... behavior for the day. What things might you do to demonstrate more sporting behavior in the future?
What was the hardest thing for you to do today? Why was it hard?
Write a paragraph that includes the cues of striking that we learned today. What will you do outside of school to practice these cues?
Write a paragraph that includes the main aspects one should consider when developing a fitness program. Hint: Remember the FIT principle
Write a paragraph that describes activities that you can do in your community that promote cardiovascular endurance .
Write a paragraph that explains the difference between verbal and nonverbal communication. Why is it important to use both in cooperative activities? Be sure to give specific examples of both verbal and nonverbal communication." From James p.43, 2005, numbering added.
These are ordered from more to less reflective. The first four are about the individual student goals, success, behavior, difficulties. The next three are about future plans of the student to learn skills. The last is a more traditional study question about the course material.
These questions are not that different to ones which might be asked as study aids in any course. One of my frustrations as a student was that my answers to such questions were never looked at by anyone, let alone count towards assessment. In theory they help with learning, but in practice, like any student, I would tend to focus on what got looked at and especially marked. The e-journal gets around this problem by having answers go somewhere, perhaps be looked at and help me at least pass the course.
James (p.44, 2005) points out It is important assess journals, but not overdo it. entries. The author suggests the use of rubrics and checklists. However, they are unrealistic in expecting students to be frank in their entries. If the e-journal is part of the assessment, then students will write what they want the examiner to read.
For STEM students it can be especially difficult to find a comfortable voice to write about themselves. They may not understand the reason for writing this way and have had years of training which emphasized a neutral, impersonal way of writing. Also asking students to write for their e-journal is an extra task, as is assessing this. So I suggest a less personal approach, where the student places copies of what they are doing for their assigned tasks in their journal and comments on these. In this way the student is commenting on the task and their work, not themselves.
James, Alisa, "Journaling as an Assessment Option" (2005). Kinesiology, Sport Studies and Physical Education Faculty Publications. 78. https://digitalcommons.brockport.edu/pes_facpub/78
Worthington, T. (2013). Synchronizing Asynchronous Learning: Combining Synchronous and Asynchronous Techniques. In Proceedings of 2013 8th International Conference on Computer Science & Education (ICCSE), 26 Apr - 28 Apr 2013 , Sri Lanka. URL: http://dx.doi.org/10.1109/ICCSE.2013.6553983 Preprint available at: http://hdl.handle.net/1885/9556
"It appears to me that students can "game" assignment deadlines by creating incomplete Journal entries before the assignment deadline, then editing them after the deadline, since the Journal entry time stamps don't update after editing..." From forum: https://mahara.org/interaction/forum/topic.php?id=6470&offset=0&limit=10
This was later fixed with each journal entry displayed showing both a "Posted on" and "Last updated" timestamp. I tried this out on Mahara and it seems to work. But is anyone using this? In practice, do students understand what to do and in particular not to make updates to journal entries (so it looks like they did all the work just before the deadline).
STEM professionals and in many other disciplines are expected to keep detailed records of their work. This is for intellectual property protection, accountability, and well as normal management of activities. Occasionally I am asked to be an expert witness for a court case when a computer project fails. I trawl through all these records to see what when wrong, and importantly, when. Some disciplines have specialized tools, such as Git for computing and Electronic lab notebooks, such as SciNote for medical research.
So it would be reasonable to require students to keep records, as part of projects and assignments. This should not require additional work for the students as they should be doing this as part of their normal work. If students are not doing this they are not doing their work professionally and so can expect a lower grade (all the way down to zero). This is apart from any penalty if it is shown the work submitted was not their own.
For students not using specialized tools, e-portfolios may be sufficient, such as Mahara Journals. It would still be possible for a student to contract cheat, by having someone else prepare the journal entries for them to post. However, this would require a level of forethought and planning. The student could also provide their password to the contractor to make the entries, but that would open the student and contractor to charges of criminal conspiracy, as well as misuse of a computer system. This would be in addition to legal penalties for the contract cheating.
Like many, it appears my students will be studying away from the face to face classroom again in 2021, at least for the first part of the year, due to COVID-19. So I have been considering how to improve online delivery, and in particular how to improve the experience of assessment for students while deterring misconduct. One way may be to require students to "Show Your Work" in assessments.
Some Literature
As Nguyen, Keuseman and Humston (2020) note online assessment was suddenly a major concern due to COVID-19. Some techniques they explored for chemistry students were short answers and multiple-choice tests to investigate higher-order thinking. They also looked at more frequent lower-stakes tests. This is possible as they are easier to administer online than face to face. The frequency, the authors suggest, may help with student learning and the lower stakes reduce the temptation to cheat. Interestingly the frequency is also claimed to reduce procrastination, as students are forced to study to pass the frequent small tests, rather than wait until a large test.
Nguyen, Keuseman and Humston (2020) also briefly looked at the use of an academic integrity pledge. However, they concluded this was only effective at an institution that has a culture of academic integrity. This seemed curiously circular reasoning: an institution that did not have a culture of academic integrity presumably would not have an integrity pledge.
To identify misconduct in online STEM courses, Sangalli, Martinez-Muñoz, and Cañabate (2020) analyzed the log from a learning management system of students undertaking exercises. They found co-occurrence of responses to exercises was an indicator of collusion, with pairs of students answering the same questions at the same time. Of course, this might give a false positive for students who are studying together.
Catalena (2020) was able to use the pattern of submission of student's coding assignments to identify plagiarism in a computer course. As students can submit repeatedly to a system that validates their code, it is possible to distinguish those who make incremental progress, from those who submit completed work. However, while submission of completed work without intermediate steps may indicate the student cheated by obtaining a correct answer from someone else it may be they used some other system for refining their answer, or they are exceptionally talented.
Pribela and Pracner (p. 99, 2017) propose building a system for students to create computer code, limiting the student's ability to copy from outside the system. This makes it possible to check the consistency of the student's work. It also prevents students from forgetting to include some element in their final assignment submission, as everything is in the system and accessible to the examiner. This seems a heavy-handed way to try to prevent misconduct and one which would stop students from using outside legitimate tools. The authors produced their temporal file system, based on Git, however, most of the benefit, I suggest, could be obtained by just using Git and allowing students import to it.
Dalziel (2008) suggests students prepare an ePortfolio, including the plans, notes, and comments to deter plagiarism. They point out that if entries are timestamped it is possible to see when entries are added. The author points out that the PebblePad ePortfolio tool has a link to the TurnItIn plagiarism detection tool. However, they don't report any results of using this approach.
Discussion
Students can be reluctant to start work on an assignment. Having a looming deadline, the student can be tempted to use old work of their own or someone else's, a form of poor academic practice or misconduct.
Students are urged to undertake their work methodically. However, they are only asked to submit the final product, not the steps taken to get there. This signals to the student that preparatory work is not of value. Students are therefore tempted to simply try to produce their final product, skipping steps in its production. Students may also become frustrated not understanding why they can't produce a polished result instantly, not realizing they are not the only ones who have to work through draft after draft.
An examiner competent in the discipline assisted by automated tools can look for signs of misconduct. However, this can be a time-consuming process which is stressful for staff as well as the accused student. Rather than look for poor practice, I suggest having students provide evidence of good practice.
A similar problem occurs with online examinations, where remote proctoring tools, such as ProctorU and Proctorio, have been employed. Those tools work reasonably well, but my colleagues at ANU Computer Science developed an alternative approach of self-invigilation for online examinations. Each student is encouraged to make recordings of themselves undertaking the exam, similar to those created by proctoring tools. The difference is the student makes the recording, rather than have software imposed on them. If the examiners raise concerns about who sat the exam, the student can present the video as evidence.
I suggest this self-invigilation process could be extended to assignments through a Show-Your-Work process, as Dalziel (2008) suggests. Rather than just submit their final completed assignment, the student would make available to the examiner all drafts, notes, and other work that was used in preparing the assignment. If there was doubt as to the author of the final work, the examiner could look to see if there was a consistent body of work by the student supporting it.
The student would be required to use a system that time stamped their work, and tracked any changes, not only of drafts of the final work but all notes used. It would then be possible to look for a consistent pattern of work over days or weeks.
This Show-Your-Work approach is already used in some group project-based courses in computer science at ANU. These require student teams to use project management software, such as GitLab. The examiner can see when work was uploaded to the system and who uploaded it, to see if this supports the team's claims.
Just as students can be videoed while undertaking an exam, a video record of all the time each student spent working on an assignment (or all the time they were studying) is technically feasible. However, this would be cumbersome to use, intrusive, and not necessary. An approach of logging all the work done on an assignment should be sufficient, as this is something the student should be doing anyway. Students in any field of study should be doing so in a methodical way, STEM, computer science, and engineering in particular.
While it would be possible for a student to fake the assignment preparation process, it would be cumbersome. The student would have to take the working notes of someone else (or commission these to be written) and submit these to the online system over several days or weeks. If the student tried to enter all the working notes just before the deadline, this will be evident from the timestamps.
The student could alternatively not only get the work from someone else, but have them submit it for them. This would require handing over their ID and password to upload the material. Using two-factor authentication might help deter this, where the student would be required to enter a code sent to the mobile phone. Also, the possibility of a jail term for conspiracy and computer crime may deter some students.
Asking students to show the notes and drafts of their assignment is much the same as telling students on an assignment question to "show their work". Where a student doesn't show how they derived the final result, the examiner can reduce the grade. This doesn't require accusing the student of plagiarism.
If students are asked to show their work, they will need to be guided as to what is a reasonable quantity of work over what period. During my MEd studies, I created a journal using the institution's Mahara e-portfolio system, in which I recorded my impressions of the program. Also, I created a journal for each course, and one for my capstone e-Portfolio (in place of a thesis). In these, I made notes on references found, answers to study questions, and appended drafts of assignments. This was mostly to aid me in my work, and to have material to draw on for my reflective e-portfolio. However, I also kept this as evidence, if I was ever to be charged with academic misconduct. Fortunately, that never occurred, and my journals remain read only by me. I made an average of 100 postings per course, a total of 100,000 words. This was in addition to the assignments and capstone e-portfolio.
Based on my experience as a student, a reasonable guide would be 50 words of supporting material per percent of assessment. So an assignment worth 20% of the course assessment undertaken over two weeks would require 1,000 words of supporting notes in 10 posts. This would be in addition to ten drafts of the assignments, over at least five days of the two weeks.
References
Catalena, K. A. (2020). Mining Student Submission Information to Refine Plagiarism Detection (Doctoral dissertation). https://hdl.handle.net/1969.1/191583
Nguyen, J. G., Keuseman, K. J., & Humston, J. J. (2020). Minimize Online Cheating for Online Assessments During COVID-19 Pandemic. Journal of Chemical Education, 97(9), 3429-3435. URL https://pubs.acs.org/doi/pdf/10.1021/acs.jchemed.0c00790
Dalziel, C. (2008). Using ePortfolios to combat plagiarism. Hello! Where are you in the landscape of educational technology? Proceedings ascilite Melbourne 2008. URL https://ascilite.org/conferences/melbourne08/procs/dalziel.pdf
Pribela, I., & Pracner, D. (2017, January). A Temporal File System for Student's Assignments in The System Svetovid. In SQAMIA. URL https://perun.pmf.uns.ac.rs/sqamia/2017/download/sqamia2017-proc.pdf#page=99
Sangalli, V. A., Martinez-Muñoz, G., & Cañabate, E. P. (2020, April). Identifying Cheating Users in Online Courses. In 2020 IEEE Global Engineering Education Conference (EDUCON) (pp. 1168-1175). IEEE. URL https://doi.org/10.1109/EDUCON45650.2020.9125252
