Thursday, January 14, 2021

What is learning?

Mike Sosteric
Greetings from a meeting of the meeting for the IEEE Standard for Learning Metadata meeting (P2881) being held online. At this meeting we went back to fundamentals, starting with what is a "Learning Object". This then raises the question "What is learning?" and should we be discussing education, rather than learning.: if the student doesn't learn anything, is it still learning? At this level of abstraction it is difficult to avoid vague and somewhat circular definitions.

The Wikipedia entry for learning objects starts with the current definition from the standard: "any entity, digital or non-digital, that may be used for learning, education or training".[6] That is a vote of confidence in the standard, which is heartening, but doesn't help improve it. Other definitions given tend to be longer and more technology specific. 

Also I did a quick literature search. A paper which featured prominently was Sosteric and Hesemeier (2002) in IRRODL (my favorite educational journal, as it provides HTML as well as PDF versions. They also start with the IEEE, asserting '... a definition that includes “everything” is not a definition at all ...'. They also question if "digital" is useful. In my view qualifying leanring with "digital" is little more than window dressing.

Sosteric and Hesemeier (2002) describe as "counterproductive" the use of terminology from computing science. This is my own field, and it can be annoying when a computing term is taken and used, not quite correctly, in other disciplines. However, the authors have a more fundamental objection to learning objects, as it implies that learning can be chopped up into reusable, interchangeable "chunks". However, I suggest, to some extent, this is what learning designers and educational systems, do. An extreme example is the Australian vocational education system, where 1,300  qualifications are made up from 16,000 of "Units of competency", all listed in a database.

References

Sosteric, M., & Hesemeier, S. (2002). When is a Learning Object not an Object: A first step towards a theory of learning objects. The International Review of Research in Open and Distributed Learning3(2). https://doi.org/10.19173/irrodl.v3i2.106

Wednesday, January 13, 2021

Use Mahara Journal for Students to Show-Your-Work?

Is anyone using Mahara Journals for students to provide evidence of having work over days or weeks? Robert Lyon hinted at this, when he wrote: 

"It appears to me that students can "game" assignment deadlines by creating incomplete Journal entries before the assignment deadline, then editing them after the deadline, since the Journal entry time stamps don't update after editing..." From forum: https://mahara.org/interaction/forum/topic.php?id=6470&offset=0&limit=10

This was later fixed with each journal entry displayed showing both a "Posted on" and "Last updated" timestamp. I tried this out on Mahara and it seems to work. But is anyone using this? In practice, do students understand what to do and in particular not to make updates to journal entries (so it looks like they did all the work just before the deadline).

STEM professionals and in many other disciplines are expected to keep detailed records of their work. This is for intellectual property protection, accountability, and well as normal management of activities. Occasionally I am asked to be an expert witness for a court case when a computer project fails. I trawl through all these records to see what when wrong, and importantly, when. Some disciplines have specialized tools, such as Git for computing and Electronic lab notebooks, such as SciNote for medical research.

So it would be reasonable to require students to keep records, as part of projects and assignments. This should not require additional work for the students as they should be doing this as part of their normal work. If students are not doing this they are not doing their work professionally and so can expect a lower grade (all the way down to zero). This is apart from any penalty if it is shown the work submitted was not their own.

For students not using specialized tools, e-portfolios may be sufficient, such as Mahara Journals. It would still be possible for a student to contract cheat, by having someone else prepare the journal entries for them to post. However, this would require a level of forethought and planning. The student could also provide their password to the contractor to make the entries, but that would open the student and contractor to charges of criminal conspiracy, as well as misuse of a computer system. This would be in addition to legal penalties for the contract cheating.

Tuesday, January 12, 2021

Contract Cheating Service Promoted by Facebook

Recently I reported a Facebook post which was promoting a contract cheating service. Facebook replied that this did not go against their "Community Standards". I asked for a review and Facebook confirmed it met their standards. However, contract cheating is prohibited by Australian law, with a penalty of up to two years gaol and $100,000 fine. The service does not have to be provided from within Australia for the law to apply. The law applies to advertising the service, as well as providing it. It would be up to a court to determine if by knowingly profiting from the promotion of such services Facebook is in breech of Australia law and if its executives should serve gaol time.

Friday, January 8, 2021

Engaging Students in the Online Environment: Four Free Webinars


Manisha Khetarpal has asked me for a series of four 20 minute free webinars in the Microlearning Series at Maskwacis Cultural College in Canada. These will be on "Engaging students in the online environment". They are planned to be each Wednesday, from 20 January at 11:00 am AEDT Sydney time (Tuesday, January 19, 5 pm MST in Edmonton). This follows on my six part "Higher education after COVID-19" series in 2020. Please register now for the first webinar and send your suggestions.

The four webinars:

  1. What have we learned from teaching online in 2020 due to COVID-19? In early 2020, like many in higher education, Tom Worthington had to flip to teaching online. Come along and hear Tom Worthington's experience at the Australian National University. Come along and be ready to tell the webinar what worked, and what didn't work, for you in 2020. 
  2. Show-Your-Work to discourage students cheating online: How do we keep students engaged with their major assessment tasks all the way through a course? Tom Worthington is proposing to have students record their notes and work for assignments, both to keep them engaged and make cheating harder. Be ready to contribute your ideas and experience of having students show their work.
  3. Hackerthons for Learning: Hackathons came from the computer industry, where teams competed over a few days to collaborate intensively on a project. These have now expanded into other fields. Can we use this format to keep students engaged, solving real world problems? Tom Worthington will talk about his experience with hackerthons involving students and military personnel. Be ready to contribute your ideas.
  4. How do we get students to engage beyond the class? Education is supposed to be a social and cultural experience, not just learning stuff. With students online, how do we get them to engage outside formal coursework? Athabasca celebrated the end of year with their Athabasca University Cozy Mountain Lodge and the Australian National University is holding a hybrid multi-location virtual/real Grand Graduation: Class of 2020. Join Tom Worthington  and be ready to give your examples of informal student interaction.

The video presentation contains images that were used under a Creative Commons License. Click here to see the full list of images and attributions:

https://link.attribute.to/cc/1802022

Monday, January 4, 2021

New Australian Education Minister Needs to Put Online at Core of Strategy

Recently I was asked for comment by John Ross at Times Higher Education for an article on what might be Alan Tudge, the new Australian education minister's approach. In 2013 he chaired a committee investigating online learning particularly for International students. This did not appear to receive support from his colleagues, or the Australian higher education sector at the time. However, as I commented in THE article, the need to teach students during COVID-19 has shown the value of on-line learning. The problem for the new Minister  is now to recast the national strategy for higher education,  with online learning at its core. 

Even when COVID-19 is hopefully overcome, in a few years time, students will not necessarily flock back to campuses. In 2016 I suggested Australian providers should be ready with an online option if an international crisis kept students from campus. The crisis I had in mind was not a pandemic, but a military confrontation. That could still occur, again depleting our campuses of international students. Also we need a strategy to address increasing competition for students from campuses in their own country, as well as offshore and online. In particular China's Education Action Plan for the Belt and Road Program presents a challenge.  The Australian government should see value in helping with education in our region of the world, for strategic, humanitarian and economic reasons.

Australian educators have demonstrated that they could rapidly pivot from classroom to online teaching in an emergency. University and government leaders now have the difficult but important task of turning this from a short term unplanned expedient to the core of the new national strategy.

On 5 December 2019, Dan Tehan, the then Australian Minister for Education, announced a refresh of the National Strategy for International Education 2025 to be conducted in 2020. As I suggested in 2015, the strategy should have online learning at its core, to allow for short term crises which may keep students from campus (as later happened with COVID-19 ), and address the long term challenge of China's Education Action Plan for the Belt and Road Program.

I wrote of the strategy in 2015

'In my view the strategy does not place sufficient emphasis on the importance of on-line education, without which Australian education providers will not be viable ... international education needs to be "flipped" [with] on-line education as the central issue, not a peripheral one'. 

 I predicted most teaching would be online by 2020, but regrettably most institutions did not plan for this and had to implement crash programs due to COVID-19. However, even when COVID-19 has been overcome there may be regional geopolitical tensions which again keep students from Australian campuses without warning. Also there will be increasing competition for students from campuses in their own country, offshore and online, particularly from China's Education Action Plan for the Belt and Road Program.

References

Australian minister tipped to rethink building blocks of degrees by John Ross, Times Higher Education, January 3, 2021

The video contains images that were used under a Creative Commons License. Click here to see the full list of images and attributions: https://link.attribute.to/cc/1785196



Sunday, January 3, 2021

Show-Your-Work to Discourage Students Cheating Online


Like many, it appears my students will be studying away from the face to face classroom again in 2021, at least for the first part of the year, due to COVID-19. So I have been considering how to improve online delivery, and in particular how to improve the experience of assessment for students while deterring misconduct. One way may be to require students to "Show Your Work" in assessments. 

Some Literature

As Nguyen, Keuseman and Humston (2020) note online assessment was suddenly a major concern due to COVID-19. Some techniques they explored for chemistry students were short answers and multiple-choice tests to investigate higher-order thinking. They also looked at more frequent lower-stakes tests. This is possible as they are easier to administer online than face to face. The frequency, the authors suggest, may help with student learning and the lower stakes reduce the temptation to cheat. Interestingly the frequency is also claimed to reduce procrastination, as students are forced to study to pass the frequent small tests, rather than wait until a large test. 

 Nguyen, Keuseman and Humston (2020) also briefly looked at the use of an academic integrity pledge. However, they concluded this was only effective at an institution that has a culture of academic integrity. This seemed curiously circular reasoning: an institution that did not have a culture of academic integrity presumably would not have an integrity pledge.

To identify misconduct in online STEM courses, Sangalli, Martinez-Muñoz, and Cañabate (2020) analyzed the log from a learning management system of students undertaking exercises. They found co-occurrence of responses to exercises was an indicator of collusion, with pairs of students answering the same questions at the same time. Of course, this might give a false positive for students who are studying together.

Catalena (2020) was able to use the pattern of submission of student's coding assignments to identify plagiarism in a computer course. As students can submit repeatedly to a system that validates their code, it is possible to distinguish those who make incremental progress, from those who submit completed work. However, while submission of completed work without intermediate steps may indicate the student cheated by obtaining a correct answer from someone else it may be they used some other system for refining their answer, or they are exceptionally talented. 

Pribela and Pracner (p. 99, 2017) propose building a system for students to create computer code, limiting the student's ability to copy from outside the system. This makes it possible to check the consistency of the student's work. It also prevents students from forgetting to include some element in their final assignment submission, as everything is in the system and accessible to the examiner. This seems a heavy-handed way to try to prevent misconduct and one which would stop students from using outside legitimate tools. The authors produced their temporal file system, based on Git, however, most of the benefit, I suggest, could be obtained by just using Git and allowing students import to it.

Dalziel (2008) suggests students prepare an ePortfolio, including the plans, notes, and comments to deter plagiarism. They point out that if entries are timestamped it is possible to see when entries are added. The author points out that the PebblePad ePortfolio tool has a link to the TurnItIn plagiarism detection tool. However, they don't report any results of using this approach.

Discussion

Students can be reluctant to start work on an assignment. Having a looming deadline, the student can be tempted to use old work of their own or someone else's, a form of poor academic practice or misconduct

Students are urged to undertake their work methodically. However, they are only asked to submit the final product, not the steps taken to get there. This signals to the student that preparatory work is not of value. Students are therefore tempted to simply try to produce their final product, skipping steps in its production. Students may also become frustrated not understanding why they can't produce a polished result instantly, not realizing they are not the only ones who have to work through draft after draft. 

An examiner competent in the discipline assisted by automated tools can look for signs of misconduct. However, this can be a time-consuming process which is stressful for staff as well as the accused student. Rather than look for poor practice, I suggest having students provide evidence of good practice.

A similar problem occurs with online examinations, where remote proctoring tools, such as ProctorU and Proctorio, have been employed. Those tools work reasonably well, but my colleagues at ANU Computer Science developed an alternative  approach of self-invigilation for online examinations. Each student is encouraged to make recordings of themselves undertaking the exam, similar to those created by proctoring tools. The difference is the student makes the recording, rather than have software imposed on them. If the examiners raise concerns about who sat the exam, the student can present the video as evidence.

I suggest this self-invigilation process could be extended to assignments through a Show-Your-Work process, as Dalziel (2008) suggests. Rather than just submit their final completed assignment, the student would make available to the examiner all drafts, notes, and other work that was used in preparing the assignment. If there was doubt as to the author of the final work, the examiner could look to see if there was a consistent body of work by the student supporting it.

The student would be required to use a system that time stamped their work, and tracked any changes, not only of drafts of the final work but all notes used. It would then be possible to look for a consistent pattern of work over days or weeks. 

This Show-Your-Work approach is already used in some group project-based courses in computer science at ANU. These require student teams to use project management software, such as GitLab. The examiner can see when work was uploaded to the system and who uploaded it, to see if this supports the team's claims.

Just as students can be videoed while undertaking an exam, a video record of all the time each student spent working on an assignment (or all the time they were studying) is technically feasible. However, this would be cumbersome to use, intrusive, and not necessary. An approach of logging all the work done on an assignment should be sufficient, as this is something the student should be doing anyway. Students in any field of study should be doing so in a methodical way, STEM, computer science, and engineering in particular. 

While it would be possible for a student to fake the assignment preparation process, it would be cumbersome. The student would have to take the working notes of someone else (or commission these to be written) and submit these to the online system over several days or weeks. If the student tried to enter all the working notes just before the deadline, this will be evident from the timestamps. 

The student could alternatively not only get the work from someone else, but have them submit it for them. This would require handing over their ID and password to upload the material. Using two-factor authentication might help deter this, where the student would be required to enter a code sent to the mobile phone. Also, the possibility of a jail term for conspiracy and computer crime may deter some students.

Asking students to show the notes and drafts of their assignment is much the same as telling students on an assignment question to "show their work". Where a student doesn't show how they derived the final result, the examiner can reduce the grade. This doesn't require accusing the student of plagiarism.

If students are asked to show their work, they will need to be guided as to what is a reasonable quantity  of work over what period. During my MEd studies, I created a journal using the institution's Mahara e-portfolio system, in which I recorded my impressions of the program. Also, I created a journal for each course, and one for my capstone e-Portfolio (in place of a thesis). In these, I made notes on references found, answers to study questions, and appended drafts of assignments. This was mostly to aid me in my work, and to have material to draw on for my reflective e-portfolio. However, I also kept this as evidence, if I was ever to be charged with academic misconduct. Fortunately, that never occurred, and my journals remain read only by me. I made an average of 100 postings per course, a total of 100,000 words. This was in addition to the assignments and capstone e-portfolio

Based on my experience as a student, a reasonable guide would be 50 words of supporting material per percent of assessment. So an assignment worth 20% of the course assessment undertaken over two weeks would require 1,000 words of supporting notes in 10 posts. This would be in addition to ten drafts of the assignments, over at least five days of the two weeks.

References

Catalena, K. A. (2020). Mining Student Submission Information to Refine Plagiarism Detection (Doctoral dissertation). https://hdl.handle.net/1969.1/191583 

Nguyen, J. G., Keuseman, K. J., & Humston, J. J. (2020). Minimize Online Cheating for Online Assessments During COVID-19 Pandemic. Journal of Chemical Education97(9), 3429-3435. URL https://pubs.acs.org/doi/pdf/10.1021/acs.jchemed.0c00790

Dalziel, C. (2008). Using ePortfolios to combat plagiarism. Hello! Where are you in the landscape of educational technology? Proceedings ascilite Melbourne 2008. URL https://ascilite.org/conferences/melbourne08/procs/dalziel.pdf

Pribela, I., & Pracner, D. (2017, January). A Temporal File System for Student's Assignments in The System Svetovid. In SQAMIA. URL https://perun.pmf.uns.ac.rs/sqamia/2017/download/sqamia2017-proc.pdf#page=99

Sangalli, V. A., Martinez-Muñoz, G., & Cañabate, E. P. (2020, April). Identifying Cheating Users in Online Courses. In 
2020 IEEE Global Engineering Education Conference (EDUCON) (pp. 1168-1175). IEEE. URL 
https://doi.org/10.1109/EDUCON45650.2020.9125252