Thursday, August 6, 2020

Online Assessment with Portfolios in Response to the Coronavirus, 8am Wednesday

I will talk on "Online Assessment in Response to the Coronavirus" Wednesday 12 August, 8-9 am, AEST (Canberra Time) as part of the Microlearning Series curated by Manisha Khetarpal at Maskwacis Cultural College in Canada. This is the third of six weekly sessions.  Register now.

Higher Education After COVID-19

  1. Responding to the Coronavirus Emergency with e-Learning
  2. Open Content for e-Learning in Response to the Coronavirus
  3. Online Assessment with e-Portfolios in Response to the Coronavirus
  4. Tools used to engage students in online delivery
  5. Mentoring student group work onlin.
  6. Higher education after COVID-19: Not business as usual
These are online, open to all and free, but please  register now. Suggestions are welcome.

Part 3: Online Assessment with Portfolios in Response to the Coronavirus

Slides and notes (2 Mbytes PDF)

Due to the risk of COVID-19, universities are using online invigilated examinations in place of examination rooms. However, there are much better ways to assess students online. Assessment can be an afterthought when delivering a face to face course: after all, it comes after the learning, doesn't it? Perhaps not. Assessment can be used to provide the student and the teach information on what they already know, to help plan what to learn and how. Assessment can be used during learning to see what more needs to be done. Assessment can also be integrated into what the student works on, in real world tasks, or simulations of them. A portfolio can be used for students to collect evidence of the skills and knowledge they have gained in formal courses and co-curricular, to show they have the real world skills needed to graduate. All of these approaches to assessment require more skill of the educator, than a final examination, but may not take any extra time. There are tools which can be used online, such as Moodle (Quizzes, forums, and peer assessed assignments) and Mahara (e-portfolios), but these require a knowledge of pedagogy, as well as the technology.  This third talk in the series and also I will cover some requests from the second talk.

See also:
  1. Assessment for ICT Sustainability-Assessment and Strategies for a Low Carbon Future, (ANU COMP7310, Athabasca University COMP 635,  ACS Green Technology Strategies), Worthington, 2016
  2. Assessment for the Learning to Reflect module for the ANU TechLauncher program, (COMP3500, COMP3550, COMP3710, COMP4500, and COMP8715), Worthington, 2019
  3. Capstone e-Portfolio for the Masters of Education, Athabasca University, Tom Worthington (2016)

Don't Replace Long Paper Examinations with Long Electronic Ones

  • Online invigilation allows paperless home exams
  • ProctorU at Athabasca University,  Proctorio at ANU
  • Work okay, but do we need a long exam?

In the first half of the year I tried a couple of demo exams with one of the leading online exams products. There were some good points : the compatibility test beforehand warned my CPU was too slow, but it worked okay anyway. I have a relatively under-powered laptop by today's standards and slow Internet connection. This is not just because I am a cheapskate, I am doing what Ken Mattingly did for Apollo 13: make sure that if it works on my equipment, it will work for those out in the void.

The business of panning the camera around the room, to make sure no one else was present was a bit of light relief.

Having tried this process twice, I could image taking a short test this way. There is the comfort of a familiar environment, rather than an exam room, but help is further away. If it was for a small amount of my grade (perhaps 20%) and short (around 30 minutes), this might be tolerable, just. But if for the majority of the grade in a course and for hours, there is no way I would ever contemplate sitting such a test.

In my view, any exam should be relatively short, more like a take-home examination, than end of semester one. One issue is when the students can do it. I worry that some course designers are assuming all the students take the test at precisely the same time. This will not work well at home. There should be a period of a day, or days, over which students can choose to start their exam.

Academics and university administrators may be using online examinations in the mistaken belief that some law, policy or rule requires examinations. Even before COVID-19, there was flexibility with how assessment could be done. Assessment rules generally have a lot of detail about how to do a traditional examination, but then include a get-out clause saying that some other form of assessment can be used. Other forms of assessment take skill to design and resources to administer. But if you have been trained in how to design assessment (as I have), this can be done reliably, at reasonable cost.
While just a practice, these exams brought back a tinge of the terrors I suffered as a student with assessment. As an instructor, I would not use such a system for long examinations. Such examinations are not a good way to assess the knowledge, and even less so for assessing skills needed for real work tasks. In addition such examinations increase the risk to the health and safety of students. Students were already at risk of mental illness and self harm before COVID-19. They are now at higher risk, and there are better, safer, ways to carry out assessment. Online invigilated tests should have only a small part in assessing students.

See also:

Use Small Quizzes and Short Exercises For Building Confidence

Use Projects for More Advanced Skills

Use a Reflective e-Portfolio Disguised Job Application as a Capstone

Tuesday, August 4, 2020

Ask Zoom for a Low Bandwidth Option

Back in March I suggested to Zoom, they add a low bandwidth option which the participant can switch on (request #3204325, priority "high", 26 March). The documentation says screen sharing with video thumbnail is possible at 50-150kbps, but I could not get that to work, even when I shaped my network connection to that speed. As well as allowing to user to activate this manually, I asked Zoom to adjust their system so it came on automatically, sooner. Since then I have experimented with how to reduce the bandwidth Zoom uses, so it works better on slow links, particularity wireless ones. But there are many people who still have difficulties. On 4 August, Zoom replied to say my suggestion had been added to their feature request list. But it took Zoom four months to reply, so they don't seem to be in a hurry to implement this. If you think it is a good idea, please tell Zoom.

Monday, August 3, 2020

Open Content for e-Learning in Response to the Coronavirus, 8am Wednesday

I will talk on "Open Content for e-Learning in Response to the Coronavirus" 5 August, 8-9 am, AEST (Canberra Time) as part of the Microlearning Series curated by Manisha Khetarpal at Maskwacis Cultural College in Canada. This is the second of six weekly sessions.  Register now.

Higher Education After COVID-19

  1. Responding to the Coronavirus Emergency with e-Learning
  2. Open Content created
  3. Assessments in online delivery
  4. Tools used to engage students in online delivery
  5. Mentoring student group work onlin.
  6. Higher education after COVID-19: Not business as usual
These are online, open to all and free, but please  register now. Suggestions are welcome.

Part 2: Open Content for e-Learning in Response to the Coronavirus

Slides and notes (2 Mbytes PDF)

Openness in education can apply to the way education is provided  as well as the course materials. This is not just about using free stuff to save money. The Open Source movement in computing and the Wikipedia show a way of contributing, as well as using, content. This second talk in the series is based on the chapter "Use of Open Education Resources" in my free e-book "Digital Teaching In Higher Education". Also I will cover some requests from the first talk.

Openness in Education

  1. Open University
  2. Open Educational Resources
  3. Open Business Models
  4. OER Example: ICT Sustainability
  5. Questions from last week:
    1. Is elearning more time consuming?
    2. Hackerthons & Group Project Work Converging

Open University

  • UK Open University established in 1969
  • Provide access for disadvantaged students
  • Confronted today's issues decades before COVID-19
The UK Open University (OUUK) was established in 1969. Walter Perry, the first Vice Chancellor's 1976 account of the formation of the intuition is a must read for those interested in online university education in the face of COVID-19 today. Many of the issues now being addressed with learning during COVID-19 came up 50 years ago.
"Open" in OUUK terms is about providing access to a university education to those who do not have access to traditional university, due to a lack of formal education, money, location, gender or membership of a minority group. Open universities have used whatever communication technology is available: paper mail, radio and TV broadcast, video and audio tapes, now computers, the Internet, and smart phones.
The Open Universities Australia (OUA) consortium has been delivering on-line courses from some of Australia's most respected universities for decades. Australia's University of New England (UNE) was one of the models for OUUK and there are hundreds of open universities around the world.

I find it frustrating when I hear of concerns expressed about remote learning today as if these are new and novel: problems identified, and for which solutions were found, decades ago.
Perry, W. (1976). Open University: a personal account. Milton Keynes: Open University Press. 

Remove Barriers to Australian On-line Universities (Blog Post),  Tom Worthington, September 21, 2013, The Higher Education Whisperer

Open Educational Resources

"Think free as in free speech,
not free beer."
Richard Stallman
(quoted by
Lessig, Lawrence 2006).
Galuel / CC BY-SA
"digitised materials offered freely and openly for educators, students and self-learners to use and reuse for teaching, learning and research"

From OECD, Centre for Educational Research and Innovation, 2007 (in CETIS, 2008).

MIT's OpenCourseWare can be modified for reuse, unlike most MOOCs.

The Microlearning Series has had many presentations about the concept of Open Learning, so I will not go into much detail here. The idea is appealing, but the practice is difficult.

Two examples of OER are MIT's OpenCourseWare  and more recent Massive Open Online Courses (MOOCs). An important distinction is that MIT allowed their content to be modified and reused. Many other providers don't allow modification, and some limit reuse.


CETIS, J. (2008). Open educational resources-Opportunities and challenges for higher education.

Open Business Models

The major failing in many OER projects has been how to sustain them. Projects are usually funded by a government, an institution or a corporation in a burst of autistic enthusiasm, but the funding runs out and the project folds. There has to be a way for OER as to provide a reward to those undertaking it, so they can keep doing it.

The Open Source movement in computing provides one model. Individuals, corporations, governments and universities work on software which can be modified and used for free. They do this partly for the common good, but also so they can make use of the resulting software, including commercial companies for profit.

An alternative approach is where a not-for profit organization can solicit ongoing donations to cover the cost, while making use of mostly donated labor. While not working fr profit, The Wikimedia Foundation, which runs the Wikipedia, and the Apache Software Foundation, which provides much of the software for the Internet, are sophisticated organizations with significant revenue.

OER Example: ICT Sustainability

  1. Graduate online course
  2. Run by ACS and ANU in 2009
  3. Also offered through Open Universities Australia
  4. Adapted for Athabasca University by Brian Stewart
  5. Moodle file, e-book under a CC BY-SA open license.

The Australian Computer Society (ACS) me to design an e-learning course in “Green ICT Strategies” in 2008, to train computer professionals in how to estimate and reduce CO2 emissions. The ACS allowed me to retain the intellectual properly, so I released it under a Creative Commons license. It was run as part of the ACS graduate level certificate in February 2009, and the Australian National University (ANU) Masters of Computing from July 2009. ACS also offered the course through Open Universities Australia. A North American version of the course was developed by Brian Stewart for Athabasca University (Canada) in 2011 and is still running as "Green ICT Strategies". 
The course was developed in Moodle, with an e-book of notes, under a CC BY-SA open license. A traditional distance education asynchronous approach was used with quizzes, discussion questions and assignments.

Originally ACS created the course for public benefit. Each time I ran the course for ANU I would make improvements, which could be used by others.


Worthington, T., "A Green computing professional education course online: Designing and delivering a course in ICT sustainability using Internet and eBooks," Computer Science & Education (ICCSE), 2012 7th International Conference on , vol., no., pp.263,266, 14-17 July 2012 URL:  Preprint available at:

Is e-learning more time consuming?

  • Not if you know what you are doing, as a student or teacher.
  • Don't waste time talking at students while they passively listen.
  • Budget staff and student time on what is important: student activities and assessment.
  • OER and Open Access Software can help

Last week one of the questions asked in the forum was: "Is e-learning more time consuming". If you have learned how to learn, or teach, online and a disciplined in what you do, then the answer is "no". However, there can be much frustration, and time-wasting, where teachers try to replicate a classroom experience and students expect that.

A common complaint by students (one I read in a Facbook post today) is that the teachers are not teaching them online, but instead leaving them to work out what to do themselves.

The teachers job is to guide the student's learning, not tell them what to do in detail. Live "lectures" online, or in a classroom, are about the least useful form of instruction for this. A tutor talking to a small group of students is also of little educational value. Instead the teacher needs to focus on getting students to do things, and reflect on what they have done.

OER and Open Access software can help. These can not only save time, but also provide useful pointers to ways of working better. Rather than prepare materials yourself, alone, work with others. Rather than studying alone, work with others. Decide in advance to share your results, acknowledging everyone's contribution.

Hackerthons & Group Project Work Converging

A flat floor large classroom at ANU, with large mobile LCD screens used to relay presentation to the back of the room.
ANU TechLauncher,
Team Formation Exercise, 2018

Australian & NZ Defence Forces Logistics Hackathon: by ACS, 72 hours in May, 70 mentors and 500 participants online.

ANU TechLauncher team formation: 300 students and 40 clients for two hours online.

Same tools and techniques used: Slack, Zoom and GitLab.

One way to get students to engage with their studies is with projects. However, group projects can impose a large administrative overhead for teachers. For years I have been supervising formal university group projects, usually as part of a capstone: required to be completed before the student can graduate. These projects take 12 or 24 weeks, with students in class with a tutor for two hours a week.

At the same time I have been mentoring teams in hacker competitions, which typically take 72 hours, run at a venue 12 or 24 hours a day, usually over a weekend. The hackerthon participants are mostly students, but these are are extra-curricular with no course credit, but some prizes.

The photo shows how these large scale exercises used to be done: in a large room over many hours at a time (Worthington, 2020). Some large hackerthons took over whole university buildings, or floors of corporate headquarters. There is considerable excitement generated by the close physical proximity and fast pace.

With COVID-19 both the project courses and the hackerthons have moved online using the same project management and communication tools. There is the possibility to use hackerthons as a form of group teaching technique. This approach emphasizes the participants organizing their own work and those who participated previously running later events.

Steve Nouri organized a Secure Supply Chains COVVID-19 Hackerthon for the Australian and NZ Defence Forces, hosted by the Australian Computer Society in May 2020. I was one of the Lead Mentors, providing advice to 70 mentors, who in turn helped 500 participants, over 72 hours. 

Last week I helped with the two hour team formation exercise for ANU Techlauncher. There are about 300 students, who formed teams of about seven students each live online after listing to pitches from clients. The students will now spend one or two semesters working on projects for about forty clients. Each team has a tutor who they meet with weekly, but the emphasis is on students working togehter.

Both the hackerthon and Techlauncher used Slack, Zoom and other communication tools, plus project tools, such as GitLab. There is potential to make more use of OER templates for such group work and Open Source software.


Lessons Learned From Executing Successful Virtual Hackathons, Steve Nouri, Forbes, Jun 11, 2020. URL

Online real time student project team formation with Zoom and Slack,  Tom Worthington (Blog post), The Higher Educaiton Whisperer, 1 August 2020. URL

Worthington, T. (2020, June). Blend and Flip for Teaching Communication Skills to Final Year International Computer Science Students. Paper accepted for the IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE), 10-13 December 2019, Yogyakarta, Indonesia. URL

Saturday, August 1, 2020

Online real time student project team formation with Zoom and Slack

A flat floor large classroom at ANU, with large mobile LCD screens used to relay presentation to the back of the room.
ANU TechLauncher,
Team Formation Exercise, 2018

In previous years the ANU TechLauncher computer project team formation exercise was face to face in a large flat floor classroom for two hours with several hundred students. Due to COVID-19 restrictions, this semester the room was replaced with a Zoom video-conference, Slack text chat and web documents. The organizer gave an introduction on Zoom, after which each client pitched their project. Students could ask questions verbally via Zoom, or by text in Slack. Each student then selected a project and discussed the project with the client in the Slack channel for the project. They could also look up the video conference for that project in a web document. The Zoom conference remained open during this process for queries.

When a project had sufficient students, the newly formed teach would appoint two spokespersons who would register the team on a Slack channel. Those students were then free to leave. Instructors visited the forums answering questions. All unallocated students and clients then returned to Zoom for another round of pitches. The event ended when all students were allocated to a project (not all projects were successful in getting students).

This process was a reasonably direct translation from the face to face one used in previous years. Posters for projects had been placed on walls, with the client for each next to their poster pitching. Students would speak to the client and put their details on a post it note on the project poster.

One feature of Slack which might make the online process easier is the provision of video conferences in channels. This would allow students to click a link and open a video conference in Slack, rather than having to go to a web document to find the relevant conference.

However, slack and Zoom text chat have the limitation that information scrolls off the screen quickly. There was a need to keep posting some of the same information every ten minutes or so, as it was too difficult for participants to scroll back to find it (this also happened with the ACS Hackerthons using Zoom and Slack). One way around this would be to have a separate channel for very important information, or use a live web document.

Some form of status board might also be useful, with a traffic light indicator to show the progress of each project in obtaining a team.