What we know about student evaluations is reassuring

Katharine Gelber, 
University of Queensland

In "What we now know about student evaluations is much more depressing than you thought", Katharine Gelber (University of Queensland), reports that research confirms a gender bias in Students Evaluations of Teaching (SETs). Their research was of University of Queensland student surveys over three years from 2015 (Gelber, Brennan,  Duriesmith, & Fenton, 2022). The good news is that the numeric scores did not show gender bias, but the text comments did. Male and female identifying students evaluated teachers in some stereotypical gendered ways. This means that SETs may be rewarding female and male staff for behaviours that conform to gender stereotypes. It also may mean that female and male staff are rewarded for behaviours that have differentiated impacts on the amount of time and energy they have available for other activities, including of course research. 

The author's conclusion is that "These results on student evaluations are only depressing if you thought they were an objective measure of teaching quality". But who in any service delivery industry would think a consumer survey is an objective measure? Like any consumer survey, the results will reflect the biases of those surveyed. That the quantitative results are consistent is a positive result, and should be enough to use the surveys as a early warning indicator of a problem with a course or teachers.

If the numbers for one course or teacher is very low, then there is a reason to check why. But sifting through what students write on surveys to decide if one teacher is slightly better than another is nuts. Also such a survey should have nothing to do with the amount of time a teacher spends teaching. They should spend the time needed to meet learning outcomes for students, which have nothing to do with a popularity poll. 

Reference

1. Katharine Gelber, Katie Brennan, David Duriesmith & Ellyse Fenton (2022) Gendered mundanities: gender bias in student evaluations of teaching in political science, Australian Journal of Political Science, DOI: 10.1080/10361146.2022.2043241
2. 
   

Tuning Zoom Videoconferencing for a Slow computer and Low Speed Broadband

Greetings from the 2020 UQ Work Integrated Learning Symposium, currently on Zoom from Queensland University. This is free and on for four hours, so there may be time for you to join. The students are running the event, which shows commendable dogfooding by UQ.

To improve the performance of Zoom on my slow laptop, and low speed wireless broadband connection, I am trying a router with UDP shaping. 

Previously I noticed that with Zoom in full screen mode the sounds would break up and image freeze. If I reduced the window size it worked better, but this was fiddly. Also when a presenter shares their screen, Zoom tends to force the display back to full screen, after which I have to shrink it again.

An approach I tried in the past was to slow my Internet connection, which Zoom responded to by using lower resolution video, making everything more stable. However, this also slowed down all my other network use. There are software utilities which will slow just one application, but I found these did not work with Zoom.

The Internet uses two sorts of data transmission: TCP and UDP. Most applications, such as email and web browsing use TCP, as this provides a reliable connection. Video conferencing programs typically use UDP for the audio and video, as this has less overheads. The utilities I tried to slow an Internet connection only act on TCP, not UDP, so did not slow Zoom.

Not being able to find a software solution I turned to hardware. Many routers allow for slowing, or "shaping" of the speed of the connection. A few allow this to be targeted at TCP or UDP specifically. I tried a TP-Link TL-MR3020 V3 router (around $50 AU), and switched on the UDP shaping. So far it is working well.

Currently I have the data rates for UDP set to: 800 kbps transmitting and 512 kbps receiving. I want to provide a good quality standard definition image of myself when speaking, but a lower quality for the video I receive, so have sent the sending rate higher than receiving.

Zoom is currently providing an image of the speaker at 320 x 180 pixels, 12 frames per second and screen-sharing at 1440 x 900 pixels, 1 frame per second. The screen share looks very clear and readable, with a clear thumbnail of the speaker next to it. If a video is played in the screen sharing it looks jerky, but fortunately most presenters don't play videos.

A bigger test will come this evening, when I am speaking on "The Virtual University" (6pm, all welcome).

ps: One small glitch: after a short break when my computer went to standby mode the router was no longer connected: I had to reboot it.

pps: The video from some speakers was not appearing in Zoom gallery mode. So I increased the down speed from 512 kbps to 1024 kbps. The increased speed also allowed for screen sharing at 1920 x 1080 pixels at 8 frames per second, which is enough to follow a mouse around the screen and simple animation. In speaker's view the video is now at 640 x 360 pixels 26 frames per second. However, this is using too much of the limited CPU in my computer, so I will try setting up and down speeds to 800 kbps.

ppps: Had problems with the  TP-Link TL-MR3020 router, so changed it for the desktop version, a TL-MR3420, which is working well.

Role of the Tutor Collaborating on University Educaiton

A tutor’s guide to teaching
and learning at UQ,
Bath, Smith & Steel, 2019

In looking for materials to teach university tutors, I came across "A tutor’s guide to teaching and learning at UQ" (Bath, Smith & Steel, 2019). This is a detailed 68 page document from the University of Queensland's Institute for Teaching and Learning Innovation, intended to be read alongside the training offered to UQ tutors. It starts with responsibilities of a tutor, then addresses fears and concerns, preparation and planning. Then there is a chapter on how students learn, emphasizing that just telling students is not very effect, you have to get them to do something. The next chapter is "Effective small group teaching and learning". This is then followed by assessment, evaluation, and resources.

The emphasis in this guide is on small small group teaching (with mention made of classes of 16 students). The assumption is that the tutor works alone with a class, making up their own lessons. However, students are increasingly instructed in large groups, with a team of instructors, following a pre-designed lesson plan. Even where the tutor is on their own, they need to follow what the student is being asked to learn and do, in the course notes, lectures, videos and other instruction.

As a student myself, and as a tutor, a major frustration, and source of confusion has been when the tutor doesn't know what is going on in the rest of the course, so doesn't know how to support the students. As a tutor, on occasion, I have asked the students what was said in lectures. This can be a useful way to see what students have retained, and what misunderstanding there have been, but often it is because I really don't know what they were told.

I suggest tutors need to be trained to work collaboratively, with fellow tutors and the course supervisor. It is not be enough for the tutor to sit back and wait for this collaboration to happen, as many supervisors are too busy to organize it, don't understand the importance of this, or how to do it.

Higher Education Learning Framework

A Higher Education Learning Framework (HELF) has been produced by the Science of Learning Research Centre (SLRC), an ARC Special Research Initiative, based at the University of Queensland.

The HELF has seven themes:

1. Learning as becoming
2. Contextual learning
3. Emotions and learning
4. Interactive learning
5. Learning to learn and higher order thinking
6. Learning challenge and difficulty

Also provided are strategies for university teachers, students and assessment.

Unfortunately I did not find the HELF particularly useful. The document uses the term "university teachers" and discusses "designing courses". But the key problem with university education, not addressed by the HELF, is that university academics do not see themselves as "teachers", and are not routinely trained in course design, or delivery.

The HELF describes higher education as a "... transformational journey of students to become citizens of the world ...". I don't disagree with this, but it does not help me decide what to teach, or how to teach it. Someone learning to drive may be told how a car will take them on a transformational journey, but what they really need to know is the pedal on the right makes it go and the pedal in the middle makes it stop. Similarly, university teachers need practical training in how to teach: what works, and what does not. Those teachers will benefit from a little theory and philosophy of education, but only to help them master the practical skills.

Reference

Nugent A., Lodge, J. M., Carroll, A., Bagraith, R., MacMahon, S., Matthews, K. E. & Sah, P. (2019).
Higher Education Learning Framework: An evidence informed model for university learning.
Brisbane: The University of Queensland. URL https://itali.uq.edu.au/files/3110/Projects-HELF.pdf

Are today's universities failing society?

Greetings from the Australian National University in Canberra, where a panel is discussing "Are today's universities failing society?". This is topical as the federal government, having sorted out vocational education is turning its attention now to universities. Next week I will be starting my contribution to the policy discussions with "Learning to Teach On-line with an E80 Blend".

Panel:

• Moderator, Dr Nicholas Farrelly
• Associate Professor Julia Horne, University Historian at the University of Sydney.
• Julie Hare is Higher Education editor for The Australian.
• Associate Professor Kristen Lyons, University of Queensland

In the case of the ANU, is is to be much more specific in deciding if the unviersity is failing society. The Australian National University Act 1991, Section 5, sets out the Functions of the University to include:

(a)  advancing and transmitting knowledge, by undertaking research and teaching of the highest quality;

(b)  encouraging, and providing facilities for, research and postgraduate study, both generally and in relation to subjects of national importance to Australia;

(c)  providing facilities and courses for higher education generally, including education appropriate to professional and other occupations, for students from within Australia and overseas;

(d)  providing facilities and courses at higher education level and other levels in the visual and performing arts, and, in so doing, promoting the highest standards of practice in those fields;

(e)  awarding and conferring degrees, diplomas and certificates in its own right or jointly with other institutions, as determined by the Council;

(f)  providing opportunities for persons, including those who already have post-secondary qualifications, to obtain higher education qualifications;

(g)  engaging in extension activities.

(2)  In the performance of its functions, the University must pay attention to its national and international roles and to the needs of the Australian Capital Territory and the surrounding regions.

This can be summarized as research and teaching for Australia. The interesting part is "extension activities", which is normally taken to be a form of cut down university education for the general public (a moder example being the MOOC).

ANU, and other universities, also act as a reserve of independent expertise for the community, government and industry. This latter role only becomes controversial when someone in the community, government or industry does not like the independent advice provided (such as with climate change).

Australian Universities in EdX MicroMasters

Three Australian universities: ANU, Curtin and Queensland, are offering masters level blended courses as part of EdX MicroMasters Programs. Students can obtain credit towards up to one-quarter of a Masters (six months of the two years full-time study) for completing a series of EdX courses:

1. Australian National University: Evidence-Based Management
2. Curtin University: Human Rights
3. The University of Queensland: Leadership in Global Development

However, it should be noted that these course are not free and are not pure on-line courses, requiring attendance for a capstone examination. Also the completion rates for such on-line courses is lower than for conventional on-line distance education and campus-based courses.

The completion rate for the EdX MicroMasters could be expected to be less than 1%. Rather than a low cost university program, the MicroMasters could be seen as a form of extended for-fee university entrance examination. Rafael Reif, MIT President, is quoted as saying EdX MicroMasters is "an experiment in what I call inverted admissions. Anybody anywhere can try to take those courses online."

Under the Australian Qualifications Framework, the EdX MicroMasters is closest to a Graduate Certificate, as part of a nested qualification. The use of the term "Masters" is misleading and may not comply with Australian law.

University of Queensland Student Strategy

The University of Queensland (UQ) has issued a twenty page Student Strategy 2016-2020 White Paper (19 July 2016). The paper sets out five goals and twenty initiatives, mostly citing specific projects already underway. Some exceptions are Authentic assessment, Learning analytics and eAssessment, which have no specific projects listed to implement them.

It will be interesting to see how far UQ goes with online learning, offering short modules, competency-based credit, Learning analytics, Authentic and eAssessment. These require a high level of skill from course designers and instructors, otherwise they risk the reputation of the institution for high quality education. The old university approach of having someone with just a research degree design and teach courses will not work. UQ will need qualified educational specialists, working in teams. This approach has been routine at distance education teaching orientated institutions, but not research universities.

An interesting initiative is the  Employ101x: Unlocking your employability MOOC. One frustration with the white paper is the lack of hypertext links, or references, to initiatives cited. As an example, the VC’s flagship courses for ‘creating change’ sound interesting, but I could not find these on the UQ website. I have added links below, to some of the projects I could find.

Some of the initiatives do not appear very radial and perhaps do not go far enough. As an example, moving from two terms a year (Semesters) to three (trimesters), is not a very radical move. The universities in the Open Universities Australia consortium already offer four study periods a year. As a part-time postgraduate student I have found this a very useful option, allowing me to study at a steady pack all year and progress at a reasonable rate.

Goal 1: Game-changing graduates

1. Work integrated learning and student employability program expansion ... BEL Student Employability Team; UQ Advantage Office – Employ101x
2. UQ research fused with UQ teaching ... Undergraduate Research Program
3. The VC’s flagship courses for ‘creating change’ ... Schools of Law and Political Science – Undergraduate Working Groups
4. Idea acceleration ecosystem ... EAIT – Idea Hub
5. Global extension experiences ... Short-Term Mobility Project
6. Authentic assessment ... Develop a system of assessment that uses challenging and meaningful assessment tasks to engage and extend students across their programs.

Goal 2: Student-centred flexibility

1. Initiative 1: Extended online and on-campus active learning ... Prioritise the online delivery of courses ... HASS – Strengthening the First-year Gateways: Aligned, Literate and Flipped; EAIT – Flipping the classroom
2. Unbundled course options ... Offer key components of our courses in short modules that draw on UQ’s advanced digital learning resources and allow students to build competency-based credit. ... School of Psychology and UQx – Introduction to Psychology – Online Series: PSYC1030x
3. Program schedule alternatives ... Move to a trimester system across selected undergraduate and postgraduate programs where a need for accelerated and decelerated study options is identified. ... UQ Business School – Flexible MBA options
4. Learning analytics and eAssessment capability

Goal 3: Dynamic people and partnerships

1. Students as partners ... UQ Students as Partners Fellowships
2. Multi-point mentor program ... EAIT – MEET a Mentor Program
3. Teaching innovation support ... UQ’s Learning Designer Teams
4. Staff professional development and recognition provisions ... Teaching@UQ
5. A teaching engaged staff profile ... School of Chemistry and Molecular Biosciences – The Alure project

Goal 4: An integrated learning environment

1. Campus precinct development ... UQ student residential borough
2. The student learning and support hub ... Provide a central, integrated physical location on campus that enables innovative teaching practices and effective student enquiry/support services.
3. Enhanced multifunctional spaces ... UQ Library: Multifunctional, service-oriented spaces
4. IT infrastructure renewal ... Invest in IT infrastructure renewal to enhance UQ’s online user experience and support the increasing demand of eService requirements across the University.
5. Student support interfaces ... my.UQ

Credit for MOOCs from Australian, North American and European Universities

Grove (2016) reports that a consortium of the Australian National University, University of Queensland, University of British Columbia, Boston University and Delft University of Technology, are setting up a credit transfer system for MOOCs. All are members of the edX consortium and offer a small number of low cost on-line courses. Students who undertook a MOOC through one institution would be able to gain credit at any of the others.

What is interesting about this proposal is that the MOOCs are generally not integrated into the degree programs of the institutions offering them. Not only do the universities need to set up a system to recognize each others MOOCs, but their own. Also each institution only offers a small number of courses, which are usually a short part (typically 4 to 8 weeks) of an existing introductory course.

MOOCs are currently used as a marketing tool by universities to give potential students a taster of a topic, not provide them with a complete educational program. Some of these universities also offer full on-line courses as part of degree programs, but those courses are not part of the edX system, are charged at a higher rate, delivered using different software and part of the same quality control system as face-to-face courses. The University of Queensland has a UQx Design Methodology for Course Development.

If this scheme follows the pattern set by other MOOC recognition schemes, it will not be free and will cost about the same as existing e-learning. Coursera offer "Specialization Certificates", where the student does a series of MOOCs, then a capstone project and receives a university certificate. An example is the University of California, Irvine's Virtual Teacher Program. The student undertakes four courses and a final capstone. The courses are each 2 to 4 hours a week over five weeks. Each course and the capstone is US$66, for a total of US$333.

That may sound cheap, but it is about the same cost as an on-line Certificate IV in Training and Assessment (TAE40110) offered by more than seven hundred Australian Registered Training Organizations (RTO). The RTOs are part of the Australian Higher Education system and as well as offering vocational courses (for plumbers and electricians) they also offer introductory programs for those wanting to go on to university. The MOOC consortia seem to want to reproduce this system, without acknowledging its existence.

References

Grove, J. (2016, January 4). Moocs: international credit transfer system edges closer. Times Higher Education. Retrieved from https://www.timeshighereducation.com/news/moocs-international-credit-transfer-system-edges-closer

Bring-Your-Own-Device (BYOD) e-Examinations

Today I attended a fascinating webinar “Bring-Your-Own-Device (BYOD) e-Examinations” hosted by Ascilite. The first speaker described Transforming Exams platform, developed with an Office for Learning and Teaching grant by University of Queensland and University of Tasmania. The system provides an off-line electronic exam environment. The examination is prepared in a format compatible with Moodle and then loaded onto USB flash drives. The drives include the Ubuntu Linux operating system and when the computer is booted it restricts the user to the examination system.

This could provide a very useful transition for institutions which have paper based examinations. It allows an environment which emulates the paper system, but provides some benefits of electronic delivery.

While the Transforming Exams system is ingenious, I found the idea of having examinations a bit dated. I design online courses for students, who may be anywhere in the world and so use of an examination venue would be cumbersome, even without the paper. From the point of view of pedagogy, I don't think an examination is a realistic way to assess real skills. As a student I would not consider enrolling in a course which had a high stakes examination.

UQ Campus Sale the Start of Australian Education Rationalisation

The University of Queensland is reported to be selling its Ipswich campus to the University of Southern Queensland (I visited the very well equipped Ipswich Campus for a ARC Funded joint project with UQ in 2002). Rather than an isolated incident, I suggest this is the start of a rationalisation of Australian Higher Education, with the top tier research universities consolidating to their capital city campuses, leaving the rest of the country to teaching universities and on-line education. Many of the satellite campuses of the regional universities will also close.

At the next level down, assuming deregulation of federal funding takes place, we may see an expansion of the TAFE and on-government Registered Training Organisation (RTO) sector. These organisations use micro-campuses, to support their primarily on-line education techniques. An example is the  Canberra Institute of Technology (CIT) centre co-located with the public library and a secondary school college in Gungahlin, Canberra.

The typical higher education student will need to attend a campus for about 20% of their tuition. This would be one day a week for a full time student, although the typical student will be part time and might attend an intensive campus session for a few days each semester.

An Examination of Suspected Official Misconduct at an Australian University

The Queensland the Crime and Misconduct Commission (CMC) released "An examination of suspected official misconduct at the University of Queensland", 13 September 2013. The report concerns the University of Queensland (UQ) offering a place in its medical program to someone who had not met the requirements for admission, but was the daughter of the then Vice-Chancellor (VC). After the matter was reported to the CMC and became public, both the VC and previous acting-VC resigned. As the report points out, this incident is important in highlighting that conflicts of interest need to be dealt with transparently in public organizations, including universities.

While several Australia states have independent commissions to investigate misconduct, there is no such body covering  the Australian Capital Territory (ACT), nor federal agencies. As a result, universities in Canberra do not have an independent body to which staff and students can report misconduct. Internal procedures are not sufficient and I suggest such a body should be established to cover all ACT and Federal public bodies (including universities established under federal and ACT legislation).

The importance of what happened at the University of Queensland goes beyond the specific events of 2010. The originating issue — a conflict of interest involving persons of considerable power in the institution — could arise in any unit of public administration. The issues raised by this series of events at the University of Queensland are important ones, with wider application across the public sector.

Conflicts of interest can and will occur at all levels of the public sector. However, they may not always be immediately evident — they may arise out of a request for assistance, a casual conversation or an oblique suggestion, hence the difficulty in identifying them as a potentially improper exercise of influence.

To ensure that conflicts of interest are dealt with transparently, important principles or practices should be accepted and adopted by all publicly funded entities. In particular, it is essential to:

• have robust policies that help and encourage officers to identify, declare and report conflicts of interest
• ensure merit and equity is the basis for selection in publicly funded organisations
• keep documentation that details the matters in conflict and the parties involved, the process followed by the decision makers, and the manner in which the conflict was resolved
• develop a culture in which integrity issues can be raised openly with senior managers
• instil an attitude in staff that a perceived conflict between integrity and personal loyalty to senior colleagues should not prevent the reporting of suspected misconduct.

From: "An examination of suspected official misconduct at the University of Queensland", Queensland the Crime and Misconduct Commission, 13 September 2013.