Judging Universities by Papers Published and Grants Got is a Really, Really, Bad Idea

The Australian Tertiary Education Quality and Standards Agency (TEQSA) is considering how to measure the quality of research undertaken at a university. This is important, as Australian law does not allow for teaching only universities. If an institution doesn't have quality research, it can't be a university, regardless of the quality of the education provided. 

TEQSA's draft  proposed legislative instrument includes the volume of citations in peer-reviewed journal papers. Ironically, there is a considerable body of peer-reviewed literature to say citations are not a good measure of research quality. 

Australia's first university was established  to advance "... religion and morality and the promotion of useful knowledge ..." and  provide professionals with "... proficiency in literature, science and art ..." (Sydney University Act, October, 1850). The aims of universities to the present day are similar, for cultural and useful knowledge, plus the training of professionals. Publications in peer reviewed journals may go some way to do some of this, but I suggest measures of how useful the research carried out at universities is and how widely its is disseminated beyond the academic community.

The COVID-19 pandemic has shown the value of the professional training provided by university, with medical researchers and professionals key to combating the disease. However, this has also shown limitations. The paper review process has been unable to respond quickly enough to need, with un-reviewed papers being relied on. Also the dangers of scientific research being misunderstood, and in some cases deliberately misrepresented has been seen.

Global warming is another instance where scientific publications have been of limited value. Researchers have show a lack of ability, and willingness, to engage publicly, thus making their research of far less value.

I suggest that universities need to be judged on the value of their research output to the community and on the effectiveness of their researchers to communicate the results. The quality and quantity of research graduates should also be part of this measure.

ps: Any discussion of research quality quickly gets philosophical, putting me in mind of Pirsig. ;-)

Zen and the Art of Motorcycle
Maintenance, Robert M. Pirsig, 2006

"... the real university exists not as the physical campus, but as a body of reason within the minds of students and teachers ..." From Chapter 13, Zen and the Art of Motorcycle Maintenance, Robert M. Pirsig, 2006

What Makes for a Good Student Experience?

The Australian 2017 Student Experience Survey (QILT, 2018), shows three non-government universities with much higher undergraduate student satisfaction than other institutions. It would be easy to conclude that private institutions do better, but is that the case? It may be what they teach and who they teach, as much as how.

The good news is that while some Australian universities rate better than others on student satisfaction, they all rate well. Universities are tending to move more of the teaching online. This provides students with better access to learning, but may result in overall lower student satisfaction results.

Most Australian universities scored between 70% to just above 80% on educational experience for undergraduate students in 2017. Three exceptions were Bond University, The University of Notre Dame Australia, and the University of Divinity, all over 90%. However, not all non-government universities did well. Torrens University Australia, the only private for-profit university in Australia, had a score of only 77.6%, below many government universities.

Torrens is essentially an online university and these tend to rate poorly on student satisfaction. The University of Southern Queensland, which also has many online students also rated poorly at 73.6% (I have completed two online courses at USQ and thought they were good).

Absent from the QILT report is Open Universities Australia (OUA), a consortium of universities offering online courses. OUA would be likely to have a low student satisfaction score, similar to USQ and Torrens. It would be easy to conclude that these institutions are doing something wrong. But they accept students who would otherwise not be able to study at all. Given a choice, I am sure these students are more satisfied being able to study, than not.

The QILT report mention of online study in the Executive summary (page iv):

"The largest variation was that external/distance students were less likely to respond positively about their Learner Engagement, 22 per cent and 63 per cent respectively. Older students also rated Learner Engagement less positively than younger students, but this difference is most likely associated with the prevalence of external or internal study modes in these age groups.

Large differences in results by study mode for Learner Engagement continues to suggest that this scale may be performing differently for internal/mixed mode students and external mode students. The Department of Education and Training is currently considering a review
of the Learner Engagement scale. As an interim measure, the QILT website, which reports SES results at the institution by study area level, will continue to exclude external mode responses for the Learner Engagement focus area. This report, however, which reports SES results at national and aggregate levels, includes external mode responses in all Learner Engagement results. ..."

Also section 3.3 "The postgraduate coursework student experience by study area":

"The widest range in focus area results was for Learner Engagement, with 34 percentage points separating the study areas with the highest and lowest results, Rehabilitation at 72 per cent, and Nursing at 38 per cent, which may be associated with the relative proportion of online or distance learning associated with the various study areas."

Avoiding Innovation Changes Naive

Professor Glover,  head of Universities Australia, is reported to have said that research commercialization and impact should not be included in the criteria for research grants, as this will risk Australia's education export industry. On the contrary, I suggest that not incorporating innovation will result in an irrelevant university sector which produces papers which no one wants to read as that research has no impact on the real world and therefore no one wants to study at.

“It’s short sighted and naive to think we can change the direction of the academic world. Publications are critically important for driving citations and citations are critically important for driving rankings which are critically important for the health of the international education sector, which is worth $18bn this year alone,”  From Innovation changes “naive”: Glover, Julie Hare, The Australian, November 18, 2015

It may seem shocking to some academics to look on university research and publications as a business, but this is a significant export earner for Australia. However, I suggest a change is needed to the metrics used to quantify research output. International student enrollments are not going to increase if Australian universities produce research which is of no practical value. Students want to enroll in a course which makes a difference to the world. Universities should also produce research which gets used and is of practical value. However, measuring usefulness is difficult.

One simple change I suggest to make Australian universities more attuned to innovation and commercialization, is to teach this to students, particularly research students. Australian research students have been discouraged from considering commercialization of their work, by official government and university policies. As an example, if research student wanted to study how to commercialize their research, they had to suspend their research and stop receiving research scholarship while they did so. This should be reversed and research students instead be expected to undertake innovation and entrepreneurial courses as a routine part of their education. This will require a change to the thinking, and the procedures, at our leading universities, which regard formal courses for research students as an anathema.

In his book "Online Gravity: The Unseen Force Driving the Way You Live, Earn and Learn", Paul X. McCarthy points out that "A surprising number of the founders and leaders of many of today's technology giants share one little-known fact in common: they attended Montessori schools."* The Montessori approach emphasizes long blocks of time on one topic, a constructivist approach and trained teachers. While Montessori is thought of as a school teaching technique, a similar approach is applied to some university programs. McCarthy points out Montessori school students are over-represented in computer science university programs and it is perhaps no coincidence that there are similarities between the two in terms of teaching approaches.


World leading universities, not only produce  academic publications based on their research, but also encourage their researchers to apply their results, including by setting up companies. A few months ago I visited Cambridge University (UK). After talking at a roundtable discussion for library staff on how to teach graduates using the Internet, I dropped in on the Cambridge University Center for Entrepreneurial Learning, where students (and staff) learn to commercialize. Cambridge has a problem convincing its elite researchers to worry about commercialization, despite decades (in some cases centuries), of successful commercialization of research.

The Australian National University, along with other universities in Canberra, set up the Canberra Innovation Network (CBRIN) and run competitions, such as Innovation ACT and integrates with degree programs with ANU TechLauncher. With this students work in teams to build a computer application for a real client, or they can opt to do their own company start-up. The students build the computer software and then, as part of Innovation ACT, prepare a business plan and pitch to investors for a company to sell the product.  This model could be emulated by other Australian cities and universities. Perhaps only one in one hundred of the student start-ups will be become a successful business, but the students will learn how to speak to business people about their ideas.

* Note: Thanks to Suneeta Peres da Costa (Author of Homework), for pointing out the quote from McCarthy.

Ensuring Quality in University Programs

Yesterday, I commented on Professor Jim Barber's call for changes to Australian law governing the regulation of Australian universities to allow for lower cost on-line courses. Professor Barber proposed the regulations look at the quality of the educational outcomes, rather than measure the input. Today I attended a seminar on how to ensure quality of assessment for courses. This was putting the case for output based measures of quality, applying processes from manufacturing industry.

It may seem inappropriate to treat students as units of production, but if society demands consistent results within and between institutions, then there would seem to be no way to avoid some measurement and comparison of students. The concept of Quality is written into the "Tertiary Education Quality and Standards Agency Act 2011":

                     (a)  to provide for national consistency in the regulation of higher education; and

                     (b)  to regulate higher education using:

                              (i)  a standards‑based quality framework; and

                             (ii)  principles relating to regulatory necessity, risk and proportionality; and

                     (c)  to protect and enhance:

                              (i)  Australia’s reputation for quality higher education and training services; and

                             (ii)  Australia’s international competitiveness in the higher education sector; and

                            (iii)  excellence, diversity and innovation in higher education in Australia; and

                     (d)  to encourage and promote a higher education system that is appropriate to meet Australia’s social and economic needs for a highly educated and skilled population; and

                     (e)  to protect students undertaking, or proposing to undertake, higher education in Australia by requiring the provision of quality higher education; and

                      (f)  to ensure students undertaking, or proposing to undertake, higher education, have access to information relating to higher education in Australia ...

The Tertiary Education Quality and Standards Agency (TEQSA) oversees this quality process. The Act mentions "quality assurance practice", but does not define what this is. The Act defines a quality framework as a series of standards decided by the Minister on the advice of a Higher Education Standards Panel. Some institutions can self-accredit their courses, whereas others are accredited by TEQSA.

TEQSA's "Higher Education Standards Framework (Threshold Standards) 2011"  is very general and non-prescriptive. For example:

The higher education provider has sufficient appropriately qualified personnel to manage and to provide academic leadership for the higher education provider’s higher education operations.

 The framework doesn't define "sufficient" or "appropriately qualified". This would vary greatly between disciplines and methods of teaching. Strictly speaking, such an input measure has no place in a quality framework, unless it can be shown to directly relate to the product produced. As an example, there is little research to show that the research record of university staff improves the quality of teaching. What does improve teaching is training teachers in how to teach.

Another example of confusing input with output is class size. One university might limit tutorial sizes to fifteen students, while another has classes of 100 students. Research shows that class size has no effect on student outcomes.

The framework is also prescriptive in referring to "locations", which assumes that students attend a class on a campus. This can be easily interpreted for an on-line course, where students should also be safe. But, it would be better if the framework did not have an inbuilt bias.

Developing Nations Formally Adopting Quality Standards

The vague references to quality in
TEQSA's documents contrast with the approach of the proposed by Basir (2012) for  Malaysian universities to use of the ISO 9000 Quality Management Standards (ISO 2008) . It is suggested this could lift the quality of Malaysian coursework programs to western standards. While that may seem ambitious, the Japanese car industry used similar standards to overtake western car makers within a few decades. By use of on-line technology, the same may be possible in education in less than a decade. However, the Malaysian Qualifications Agency does not apprise to have mandated this strict us of standards.

Quality and Higher Education in India

In "Building the Links Between Funding and Quality in Higher Education: India's Challenge"(Lindsay Daugherty, Trey Miller, Rafiq Dossani, Megan Clifford, RAND Corporation, 2013) describe quality measures introduced by the Indian government. The authors suggest suggest seven policy actions to link funding to quality.

Many of the measures proposed for India are already in place in Australia, such as a student financial aid system tied to accreditation. However, others are yet to be implemented in Australia (or USA), such as: "... a quantitative data system to measure quality of higher education institutions...".