Higher Education Whisperer: Online Assessment with Portfolios in Response to the Coronavirus, 8am Wednesday

I will talk on "Online Assessment with Portfolios in Response to the Coronavirus" Wednesday 12 August, 8-9 am, AEST (Canberra Time) as part of the Microlearning Series curated by Manisha Khetarpal at Maskwacis Cultural College in Canada. This is the third of six weekly sessions. Register now.

Higher Education After COVID-19

Responding to the Coronavirus Emergency with e-Learning
Open Content for e-Learning in Response to the Coronavirus
Online Assessment with e-Portfolios in Response to the Coronavirus
Tools used to engage students in online delivery
Mentoring student group work onlin.
Higher education after COVID-19: Not business as usual

These are online, open to all and free, but please register now. Suggestions are welcome.

Part 3: Online Assessment with Portfolios in Response to the Coronavirus

Slides and notes (2 Mbytes PDF)

Due to the risk of COVID-19, universities are using online invigilated examinations in place of examination rooms. However, there are much better ways to assess students online. Assessment can be an afterthought when delivering a face to face course: after all, it comes after the learning, doesn't it? Perhaps not. Assessment can be used to provide the student and the teach information on what they already know, to help plan what to learn and how. Assessment can be used during learning to see what more needs to be done. Assessment can also be integrated into what the student works on, in real world tasks, or simulations of them. A portfolio can be used for students to collect evidence of the skills and knowledge they have gained in formal courses and co-curricular, to show they have the real world skills needed to graduate. All of these approaches to assessment require more skill of the educator, than a final examination, but may not take any extra time. There are tools which can be used online, such as Moodle (Quizzes, forums, and peer assessed assignments) and Mahara (e-portfolios), but these require a knowledge of pedagogy, as well as the technology. This third talk in the series and also I will cover some requests from the second talk.

Don't Replace Long Paper Examinations with Long Electronic Ones

Online invigilation allows paperless home exams
ProctorU at Athabasca University, Proctorio at ANU
Work okay, but do we need a long exam?

In the first half of the year I tried a couple of demo exams with one of the leading online exams products. There were some good points : the compatibility test beforehand warned my CPU was too slow, but it worked okay anyway. I have a relatively under-powered laptop by today's standards and slow Internet connection. This is not just because I am a cheapskate, I am doing what Ken Mattingly did for Apollo 13: make sure that if it works on my equipment, it will work for those out in the void.

The business of panning the camera around the room, to make sure no one else was present was a bit of light relief.

Having tried this process twice, I could image taking a short test this way. There is the comfort of a familiar environment, rather than an exam room, but help is further away. If it was for a small amount of my grade (perhaps 20%) and short (around 30 minutes), this might be tolerable, just. But if for the majority of the grade in a course and for hours, there is no way I would ever contemplate sitting such a test.

In my view, any exam should be relatively short, more like a take-home examination, than end of semester one. One issue is when the students can do it. I worry that some course designers are assuming all the students take the test at precisely the same time. This will not work well at home. There should be a period of a day, or days, over which students can choose to start their exam.

Academics and university administrators may be using online examinations in the mistaken belief that some law, policy or rule requires examinations. Even before COVID-19, there was flexibility with how assessment could be done. Assessment rules generally have a lot of detail about how to do a traditional examination, but then include a get-out clause saying that some other form of assessment can be used. Other forms of assessment take skill to design and resources to administer. But if you have been trained in how to design assessment (as I have), this can be done reliably, at reasonable cost.
While just a practice, these exams brought back a tinge of the terrors I suffered as a student with assessment. As an instructor, I would not use such a system for long examinations. Such examinations are not a good way to assess the knowledge, and even less so for assessing skills needed for real work tasks. In addition such examinations increase the risk to the health and safety of students. Students were already at risk of mental illness and self harm before COVID-19. They are now at higher risk, and there are better, safer, ways to carry out assessment. Online invigilated tests should have only a small part in assessing students.

Use Small Quizzes and Short Exercises For Building Confidence

Example quiz questions in Moodle

Multiple choice and short free form questions can be used to build the student's confidence and prompt them to study the material. A small number of marks can be allocated to encourage students to actually do the quiz, which can then prompt them to study the material.

Typically I will set a half dozen or fewer multiple choice questions per week and two discussion questions. These would normally make up about 20% of the final grade. To avoid this making it too easy for bright students, I set the grading scheme so these tests do not count for more than a mid range grade (a Credit, not a Distinction or High Distinction).

Multiple choice and some other types of quiz question can be marked automatically. They can also have questions drawn at random from a bank, to make cheating harder. But the object of the exercise is not to test the student's know under exam conditions, but to help them learn.

Free form text answers can't be automatically marked, but they can be peer assessed, also creating an additional learning opportunity. Students learn from having to read an answer from another student and make a useful comment about it. Of course some students will try using boilerplate phrase and give everyone a high mark, but that behavior is easily corrected by giving them zero for the exercise.

This type of assessment doesn't need fine grained marking, it can be a two, three or five point scale.

Use Projects for More Advanced Skills

"In the weekly forums and the two major assignments, you are asked to write about the ICT sustainability of an organization your are familiar with. It may be where you work, w
here you have worked, or an organization you can obtain information on from public sources. For this first assignment, describe the organization and how it uses ICT. Outline how you plan to obtain the information needed to report on the carbon footprint and materials use of the ICT operations of the organization."

From Assessment: ICT Sustainability

The way to provide students with real world skills is to have them working on projects as close to real world as possible. This is easiest to do for students who are already employed in a relevant job, where they can do their project in the workplace. This is common in engineering, computing and teaching. As an example, I encourage the ICT Sustainability students to approach their boss and get permission to estimate the carbon emissions of their organization. If students don't have a suitable job, I suggest they pick somewhere they would like to work and do the project based on open source information.

Some courses are built around an internship or a group project for a client (ANU has these for computing students). However, this approach requires a considerable investment of staff time and skills of dealing with industry.

Use an e-Portfolio for a Capstone

Hot Air Balloon Over Canberra,
Photo by Tom Worthington CC-by 3.0 1996

“Three years ago I set out to make the transition, from a computer professional, to a designer and teacher of computing courses. My original learning goal was to acquire the skills needed to undertake quantitative research in education. Instead, I have learned enough theory, balanced with practical skills, to help with the transition to e-learning in Australian universities. ...International tensions could disrupt the flow of students to Australia very quickly ... "

From: Capstone e-Portfolio for the Masters of Education, Athabasca University, Tom Worthington (2016)

In place of a conventional masters thesis, I completed a Capstone e-Portfolio using Mahara for my MEd at Athabasca University. This presented five artifacts, covering six competency areas, in a process described by Hoven (2015). With this the student prepares a collection of samples of their work, and explains how this is evidence that they have the skills and knowledge required. It can be used for a small part of a course or for an entire degree.

What is key is not having an e-portfolio software package (just about any document construction tool will do), but a process to help the student build the portfolio, piece by piece. Hoven describes how students were having difficult preparing a portfolio and this was delaying their degree completion. I was one of the students in the transition period, who could choose to do this as a course, with an instructor and with step by step feedback from peers. This process works well, but it takes the student a few attempts to understand what is happening. I had to undertake several of these portfolio experiences before I had the hang of it. This can be used for anything from a vocational short course to a postgraduate degree.

References:

Capstone e-Portfolio for the Masters of Education, Athabasca University, Tom Worthington (2016)

Hoven, D. (2014). ePortfolios in Post-Secondary Education: An Alternate Approach to Assessment. UAE Journal of Educational Technology and eLearning, 1, 11-24. URL https://web.archive.org/web/20160210160342/http://ejournal.hct.ac.ae/wp-content/uploads/2014_Article2_Debra.pdf

Disguise a Reflection Exercise as a Job Application

"Select a real position to apply for. Prepare a cover letter ... CV ... statement addressing the selection criteria ... Supplementary material ... and a copy of the advertisement you are responding to. ...
After submitting, each student will be allocated four assignments from other students to provide feedback on. ... The reviews are double-blind peer reviewed ... Twenty percent of your mark will be for the quality of your feedback."
From: Learning to Reflect for ANU COMP3500/3550/3710/4500/8715, Tom Worthington, 2019.

Educators are fond of asking students to "reflect on their learning" in the e-portfolio. However, as a STEM student I had difficulty understanding what was expected. The ANU TechLauncher students are equally confused by such a task. So several years ago the reflective exercise using Mahara was replaced with one where the student has to write a job application submitted as one PDF document. The student selects a real job to apply for, or application to undertake a PhD, apply for start-up funding or similar. In their application the student must present evidence of what they did, what they learned and how they learned it, in their course. This is, in effect, a capstone reflective e-portfolio exercise in disguise. It is much easier for the student to understand as they are (hopefully) about to graduate and be looking for a job (or a PhD program, or startup funding).

For the ANU TechLauncher program I wrote a blended module to assist the students. The essential point here is no so much the course content, but the exercises to help students and provide feedback. Last year I included an online contingency as envisaged in my MEd portfolio. This was activated for COVID-19, with relatively little difficulty.

References

Learning to Reflect for ANU COMP3500/3550/3710/4500/8715, Tom Worthington, 2019. URL http://www.tomw.net.au/technology/education/learning_to_reflect/learning_to_reflect_2_1.shtml

Worthington, T. (2020, June). Blend and Flip for Teaching Communication Skills to Final Year International Computer Science Students. Paper accepted for the IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE), 10-13 December 2019, Yogyakarta, Indonesia. URL https://openresearch-repository.anu.edu.au/handle/1885/204833

Designing in an On-line Learning Option, Tom Worthington, Higher Educaiton Whisperer, March 14, 2020 URL https://blog.highereducationwhisperer.com/2020/03/designing-in-on-line-learning-option.html

Use e-Portfolios Throughout a Program?

Mahara SmartEvidence Example

identify their study goals and licensing requirements
inventory existing skills and knowledge
obtain exemptions from courses
plan study
record evidence of achieving each goal

ePortfolios are something usually used at the end of a degree program as part of a capstone. The student may be told earlier they have to do this, but if it is only due at the end, then that is when the student will worry about it. I suggest e-portfolios could be used through a program of study, first to help the student identify their study goals (and specifically any program or industry licensing requirements they must meet), inventory their existing skills and knowledge, obtain exemptions from courses, plan their study, record evidence of the achievement of each goal.

What the student is seeking entry to a profession, the template of requirements could be loaded into the e-portfolio of them to complete. The student could plan courses to meet these requirements, but they could also check which could be met from prior experience, by co-curricular activities, work experience and projects. The aim should be for the average student to only undertake formal study for one third to one half their program.

For a university degree, plus multiple concurrent industry licensing requirements, there may be hundreds of requirements to meet, each requiring evidence to be kept and certification to the required standard recorded. Mahara SmartEvidence and Moodle Competencies have some of the features needed to be able to record student's achievements in a fine grained way needed for a whole of program e-Portfolio.

References

Assessing Competencies with Mahara and Moodle, Higher Educaiton Whisperer, August 5, 2016. URL https://blog.highereducationwhisperer.com/2016/08/mahara-annotation-function.html

Worthington, T. (2014, August). Chinese and Australian students learning to work together online proposal to expand the New Colombo Plan to the online environment. In 2014 9th International Conference on Computer Science & Education (pp. 164-168). IEEE. URL https://openresearch-repository.anu.edu.au/bitstream/1885/11724/1/Worthington%20Chinese%20and%20Australian%20students%20learning%202014.pdf

Virtual Hackathon on Fighting Pandemics

Remo Conference Example

Last weekend I talked at the Virtual Hackathon on Fighting Pandemics. This is organized by the Australian National University Humanitarian Innovation Society (ANU HISoc), with the Clinton Global Initiative University and IBM. We have had an introduction from the organizers at HISoc, and a design thinking workshop from IBM. I then talked about better logistics. The opening sessions used Remo Conference, and later ones Zoom video conferencing.

Remo Conference takes the "conference" part of video conferencing more literally than other products, with an interface which looks like a conference room floor plan. Participants can be pre-allocated to tables, with a tiny picture of them, labeled with their initials, around the table as if seated. Spare space at the table is indicated by empty chairs. Those at a table can see and hear each other with the video windows appearing at the top of the screen. Participants can move tables by double clicking on another. Some tables may be reserved for MCs and a help desk. The MC can call all participants to a presentation, at which point the table conversations shut down and a large window opens for the presenter. In contrast, when in presentation mode Remo Conference looks much like any other webinar product, with a presentation window, chat, Q&A and Raise-Hand functions.

While there are some technical problems with Remo, it shows potential for a way to provide a classroom-like experience.

Reference

Hacking the Virtual Hackerthon, Higher Education Whisperer, August 8, 2020 URL https://blog.highereducationwhisperer.com/2020/08/hacking-virtual-hackerthon.html

Discussion Questions

Who are you teaching?
How are they assessed?
How do you record the evidence?
Does this work online?

What worked well at the hackerthon was that I did not present too much or too complex information. I followed this up with four questions for the participants, which worked well in getting a discussion going. So I have prepared four questions for this week.

Reference

Better Logistics for Fighting Pandemics, Higher Education Whisperer, July 31, 2020. URL https://blog.highereducationwhisperer.com/2020/07/better-logistics-for-fighting-pandemics.html

Higher Education Whisperer

Thursday, August 6, 2020

Online Assessment with Portfolios in Response to the Coronavirus, 8am Wednesday