Assessing Competencies with Moodle

Last Friday, I attended a meeting in Canberra discussing how new features in Moodle and Mahara might be used for assessing the competencies of university students. As discussed in the previous post, Mahara SmartEvidence seems to be more suited to the reflective development of higher-order skills, while Moodle Competencies may be useful for low level skills.

Competencies were introduced in Moodle Version 3.1, 23 May 2016.  As with Mahara SmartEvidence,  Moodle Competencies allows a "framework" to be used to manage the assessment of a student against a large number of "competencies". As with Mahara only administrators can set up the frameworks in Moodle, not teachers. Unlike Mahara, Moodle has the concept of a "Learning plan", which can be designed and then applied to one student or a group of students. Teachers can also add competencies to whole and activities within courses activities.

What is unclear to me, at this stage, is the relationship between competences, learning plans, courses and frameworks. In particular what is a "learning plan": is it a collection, or sequence, of tasks with associated competencies? Is this the VET equivalent of a course?

Moodle is a course orientated system, but competencies will typically be at a finer level of detail, with perhaps six to twenty-four competencies per course (and hundreds for a program). Also students may only need a few of the competences from each course.

One interesting point is that students can also (if authorized) add evidence of prior learning, to provide evidence of competency. This practice is common in the Australian VET sector called "Recognition of Prior Learning" (RPL), but has not been so popular at universities. However, with the demand for university graduates to have more vocationally relevant skills, RPL is finding favor.

The competency tool might be used for Work Integrated Learning (WIL), where the student receives credit for practical work during their studies. However, with RPL and WIL, there is the problem of how this is assessed and verified. It is hard enough checking that a student really did a task at the institution, let alone one undertaken elsewhere.

There is provision in Moodle for manually assessing competencies, which is similar to the  process for process in Mahara SmartEvidence. One difference is that students in Moodle can be rated on a scale (so they are more than competent), rather than just partly or completely competent in Mahara.

In Mahara all the competencies are manually assessed: that is an assessor looks at the evidence the student provided on a page and decides if this partly or fully meets the requirement. In contrast, the emphasis in Moodle is on automated assessment of the competency.   Typically, the student will be considered competent when they gain a specific grade in a quiz or some other form of assessment. This follows the practice used by VET, where students do many small tests.

Assessing Competencies with Mahara

SmartEvidence overview page

Last week I attended a meeting in Canberra discussing how new features in Moodle and Mahara might be used for assessing the competencies of university students. Mahara is introducing SmartEvidence later in the year and Moodle already has "Competencies".

Mahara SmartEvidence seems to be more suited to the reflective development of higher-order skills, while Moodle Competencies may be useful for low level skills. SmartEvidence might be used across a program), while Competencies would be used within courses).

SmartEvidence is being introduced in Mahara version 16.10 (due for release 31 October 2016). There is a brief overview of Mahara SmartEvidence in the manual and detailed technical specification. The tool is intended to allow a "competency framework" with prescribed "competencies" (also called "skills" or "attributes") to be defined. A student can then nominate which pages in their e-portfolio they believe provides evidence of the competences. An assessor can then record if the evidence partly or fully meets the requirements. Without SmartEvidence a student can use Mahara to present their e-portfolio, but they and the assessor then has to manually tabulate the competencies.

SmartEvidence emulates the paper based process traditionally used for competency based evaluation. Competencies can be grouped into clusters of competences to make it easier to deal with a large assessment process. As an example, Athabasca University's MEd has six clusters, with a total of 47 competencies (Hoven, 2015).

SmartEvidence may not provide all the features needed to replace the on-line processes now used with Mahara and Moodle. As an example, SmartEvidence allows a student to nominate a page to support multiple competences. However, there is no way to point to a specific part of the page and the assessor is left to manually look though material seeing what may match.

SmartEvidence appears to have no provision for multiple levels of assessment. In particular there is no provision for peer review of e-portfolios by other students, which I have found to be a very useful part of a program Capstone (both as a reviewer and someone being reviewed).

In addition the competency frameworks for SmartEvidence are assumed to be centrally managed and fixed. The assessor cannot make up their own framework and cannot modify a provided framework. This makes sense where there is a centralized approach used, with a division between assessment designers and implementors, however, it limits flexibility.

Assessing Competencies with Mahara and Moodle

The Annotation tools in the Mahara ePortfolio is being developed to allow students to easily indicate which evidence supports particular competencies and then have an assessor verify this. There is a video “SmartEvidence: Why did it come about, and what is it?”  (and slides) by Shane Nuessler (University of Canberra) and Kristina Hoeppner (Catalyst IT) from 16 April 2015.

When I undertook the  Certificate IV in Training and Assessment (TAE40110) in 2013, the Canberra Institute of Technology (CIT) used a multi-page paper based form, with a long list of competences. For each competency there would be a page or two of evidence required. Where multiple competences were from the same evidence, there was a cross reference notation needed and much flipping back and forwards between pages. This resulted in a folder full of paper, which is presumably stored in a basement at CIT.

At the moment I am preparing my own e-Portfolio for my Master of Education in Distance Education, using Mahara. I was going to list the competences on the front page and then provide the assessor with a hypertext link to the page with the evidence. But this will require me to manually hypertext each competency to the relevant artifact page. This will be tedious to do and to maintain. The  annotation feature seems to be intended to do this automatically. But I am not sure there is enough implemented yet to be usable.

Mahara Version 15 implemented basic annotations. You add an annotation box to each artificat. This is just a text box, where you type in what competencies you are claiming. The assesor then comments and when it is approved the page is locked. There are also tags, which might be useful. More features are planned for release in October.

As it is the annotation tool might be used by tagging each annotation with the relevant competencies. The tag cloud generated by Mahara would then supply a cross reference of which pieces of evidence are relevant to which competences.

Moodle 3 Competencies

Moodle 3's Competencies may help. But appear more suited to Vocational Education and Training (VET) than university assessment. With the VET approach the student is presented with a long list of small tests: do a quiz, or submit some evidence. The assessor then gives each a tick, one after another in sequence (I had to do this for the Cert IV in Training and Assessment).

But what you will typically have with university is a small number of large pieces of evidence to support a large number of competences. As an example the capstone for the Master of Education I am doing requires 5 artifacts to support 47 competencies.

Taking the VET approach I would end up with 47 competences to fill in, submitting the same 5 artifacts on average 10 times each. The alternative is that I list the 5 artifacts, then each competency it covers. But then I have to manually provide a cross-reference table to show all competences in order and where they are covered in the e-portfolio.

What it would be useful for Mahara/Moodle to do that cross reference automatically. The student would put the e-portfolio in Mahara, with competences claimed for each artifact. A tool would then scan the eportfolio and place the cross-referenced list of competences in Moodle.

Learning reflective writing

As well as the mechanics of using an e-portfolio, there is the larger problem of learning what to write. Many courses and programs just seem to give the students a tool like Mahara, perhaps with some training on how to use the tool, but no training in reflective writing. This is something which students need to practice all the way through a course or program.

Zawacki-Richter, Baecker and Hanft (2010) divide a sample of e-portfolios into four competence classes:

1. SCC (socio-communicative competence),
2. MPC (methods and professional competence),
3. PLC (personal competence), and
4. AAC (activity and action competence).

I suspect I fall into the MPC class and was a little offended, on behalf of my class, by the way we are described.

MPCs are characterized as seeing group work as a “means to an end” and not crediting other group members in their e-portfolio. These comments appear to be meant as criticisms. However, group work is a means to an end, otherwise why would any rational person do it? Also I avoid mentioning others by name in e-portfolio entries (using *fellow student* and *instructor*), as otherwise it would be a breech of privacy.

Reference

Zawacki-Richter, O., Baecker, E., & Hanft, A. (2010). Validation of competencies in e-portfolios: A qualitative analysis. The International Review Of Research In Open And Distributed Learning, 12(1), 42-60. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/893/1671