Petition Against Silly University Ranking Systems

Philip Moriarty has published an Open Letter "We reject the absurd QS/THE World University Rankings", which has so far been signed by 107 academics. While I agree with the sentiment, this list of distinguished academics have missed the point of university rankings: they aren't about scientifically measuring the scholarly & educational worth of institutions, they are a marketing tool. As a marketing tool, and a publishing business, rankings have been very successful. However flawed, the rankings are a response to a real need from students, to provide advice on the quality of institutions. 

When I wanted to study internationally, I turned to ranking systems to check the university I had selected was okay. I didn't care exactly where it rated, but more so it was actually a real university, not a scam. For this I used the Webometrics Ranking of World Universities, as it lists many more than the commercial ranking schemes, and is more open.

If academics want to do away with the current silly ranking systems, they have to come up with something better, which measures what they think should be measured. They then need a marketing strategy to have it adopted, and a business model to sustain it.

University Rankings Are a Marketing Tool Not a Student Guide

Julie Hare writes "Why university rankings don’t tell you what you need to know" (AFR, Oct 23, 2022). But ranking systems, such as that from  Times Higher Education (THE), were never intended as a guide for students in selecting a university. My favourite ranking system is the non-profit Webometrics, which includes things like openness. There are also some awards which explicitly look at teaching quality, such as the Good Universities Guide Awards, which show that institutions which rate poorly on THE rankings do well when it comes to education.

The ranking schemes from publishers are designed as marketing tools to help promote their publications, and help their advertisers (the universities), promote themselves. Like many industry awards, the university rankings are designed to appeal to the vanity of the established organisations and their executives. The reality is that research excellence has little to do with the quality of the education provided by a university. If anything there is a negative correlation, as researchers are not selected for their teaching ability. Also the quality of the teaching has little to do with the student's learning outcomes, as this mostly depends on the student, and their background, and the support they get externally. Universities in a particular system also tend to level outcomes. Australia has a strict government regulatory framework for universities, so there are no bad ones, and not that much difference between the top ones.

Australian Universities Rank Well for Impact Globally

Australian universities have come out well in The Times Higher Education Impact Rankings 2021, having 4 in the top 10 and 17 in the top 100.

Times Higher Education assesed universities against the UN Sustainable Development Goals (SDGs). Out of 1,115 universities, the University of Manchester (UK) comes first, followed by three Australians: University of Sydney, RMIT and La Trobe. University of Wollongong comes in sixth. 

On the SDG goal of "Quality of Education" (central to the mission of universities), Australia did not do so well, with only one in the top 10: University of Canberra at 5 and only 12 in the top 100.

The SDG goal of "Gender Equality" gave an interesting result, with a Saudi Arabian institution coming out on top: Princess Nourah bint Abdulrahman University, followed by three from Australia: La Trobe University, Western Sydney University and Charles Darwin University.

Australia had no universities in the top ten for "Industry Innovation and Infrastructure", with University of British Colombia (Canada) on top. This is well deserved, I visited UBC in 2014 to look at education and innovation.

Of course not too much should be read into these type of rankings. Those universities without the resources to devote to filling in the information requests from ranking organisations will tend to not do so well. They might instead want to devote resources to helping with development. In particular thousands of training focused institutions, which are providing most of the practical assistance with development, don't appear in these rankings. The "Webometrics Ranking of World Universities" includes about 9,000 more institutions than Times. 

Unfortunately some universities choose to release misleading statements based rankings. As an example CQ University previously claimed "CQUNI RANKS IN TOP 25 PER CENT OF UNIS WORLDWIDE FOR SUSTAINABILITY" (23 April 2020). However, only about 1,000 institutions are included in the rankings. So we don't know how CQU performs against the many more thousands not included. While it might be assumed these institutions would not rank well against leading universities in terms of research output, that may not be the case in terms of quality of education or sustainability. Australian universities should refrain from making such misleading claims as it could damage their reputation and that of the nation. In 2017 the UK Advertising Standards Authority issued guidelines to stop UK universities making similar misleading claims.  Perhaps similar guidelines are needed in Australia.

ps: I made a submission to the Australian Senate on the United Nations Sustainable Development Goals in 2018, on "Education for Sustainable Development".

UNSW Ranking Of Universities: a Result Looking for a Rationale

The UNSW Aggregate Ranking Of Top Universities combines the Times Higher Education (THE), Quacquarelli Symonds (QS) and Academic Ranking of World Universities (ARWU) to create the ARTU. This is claimed to provide a long term comparison of top universities. But it is not clear if this combination has any theoretical unpinning, or practical purpose, except to boost UNSW's rating. 

Such ranking systems are used by students in selecting an institution to study at. However, these rankings are heavily weighted to research performance, while most students are undertaking coursework. Also these ranking systems only include a few hundred universities, while there are thousands of other good educational institutions excluded. A reliable ranking of institutions would be useful for students, but ARTU is not that.

My favorite university ranking scheme is Webometrics, which relies on readily available information. As a result, about 30,000 institutions listed, including many vocational colleges, which are excluded other ranking systems. 

Here are institutions I have studied at most recently. All these institutions provided a useful quality education, but only the first (ANU), is in the ARTU list: 

National Ranking	World Rank	University	Presence Rank*	Impact Rank*	Openness Rank*	Excellence Rank*
6	75	Australian National University	159	77	70	140
29	764	University of Southern Queensland	1721	864	777	976
49	8269	Canberra Institute of Technology	13734	5420	5819	6626

Canada

 

National Ranking	World Rank	University	Presence Rank*	Impact Rank*	Openness Rank*	Excellence Rank*
44	1424	Athabasca University	2454	959	1884	2428

More Relivant Higher Educaiton Ranking Systems Needed

Australian education consultancy Studymove has pointed out that the higher the QS World University Ranking for an Australian university, the higher the international tuition fees charged. The correlation applies across fields and levels of study, but is least for education studies. The consultants speculate that high unemployment may result in students selecting institutions and programs with higher raking for employment outcomes. However, I expect many students don't look past the overall ranking of a university to examine individual measures. The current ranking schemes are weighted towards academic quality and research output, although these are not relevant to most students.

Like many such ranking systems, QS is heavily skewed to academic reputation. In the case of QS, the largest proportion of the overall measure is based on a survey of academics. Other raking systems use research papers published as a measure of academic quality. However, the research at a university has little effect on the quality of education a student receives. Most students are not planning to be researchers and few who complete research degrees end up in research jobs. Researchers don't make particularly good teachers anyway and this emphasis on research may be resulting in students making poor education choices.

Webometrics uses measures of openness, as well as quality, to provide a more relevant ranking of universities. This relies on readily available information, rather than surveys, which has the advantage of allowing inclusion of many more institution. In particular Webometrics includes thousands of vocational institutions which are excluded from most university ranking systems. Australian has only 43 universities, but Webometrics lists another 150 non-university institutions. 

Despite the different measures used, Australian universities outrank the non-university institutions on the Webometrics scoring and the raking of universities is not very different from QS. This suggests it should be possible to create new rankings of universities at low cost, using a similar approach to Webometrics.

The Webometrics methodology, like that of QS, is heavily weighted towards academic quality. Overall, 35% of the measure is based on papers cited in research publications which perhaps explains why universities are at the top of the list, outranking vocational institutions. 

The ordering of the universities is different to QS, but the same universities feature near the top of both lists. Near the bottom of the universities in the Webometrics ranking are a few large state government vocational intuitions: TAFE NSW, TAFE Queensland, Adelaide Institute of TAFE and Canberra Instituter of Technology. Just one private for-profit vocational provider also features: Open Colleges, which has a 100 year history in correspondence education.

I pasted the Webometrics list of Australian universities to a spreadsheet, and reweighed the scores by deleting the excellence measure. The leading universities remained at the top of the list, but a few vocational institutions crept up a little: Open Colleges from 41 to 39th place, TAFE NSW from 43 to 40, TAFE Queensland 46 to 44, Adelaide Institute of TAFE 45,  Kangan Institute 49. Canberra Instituter of Technology dropped one place from 49 to 50.

Webometrics Ranking of Ten Thousand Universities

The "Webometrics Ranking of World Universities" by Spanish researchers has advantages over the better known ranking systems and produces a few surprises. Webometrics ranks more than ten thousand universities around the world, which is many more than other measures provide. For Australia 199 institutions are listed, whereas other rankings mostly cover the 43 accredited as universities, not colleges which also issue Higher Education qualifications.

Webometrics uses four measures: Presence, Impact, Openness and Excellence. While the weighted score of these measures gives a similar result to other measures, there are some surprises. For example, many ranking system give a similar result to the the Times Higher Education University Rankings with the "Group of Eight" first:

1. University of Melbourne
2. Australian National University
3. University of Queensland
4. University of Sydney
5. Monash University
6. University of New South Wales
7. University of Western Australia
8. University of Adelaide

Webometrics lists the same top Australian universities , but in a different order:

1. University of Melbourne
2. University of Queensland
3. University of New South Wales
4. Australian National University
5. University of Sydney
6. University of Western Australia
7. University of Adelaide
8. Monash University

The ANU slips two places due to low Presence and Excellence scores. Monash drops three places to eight position, due to a very low Presence score.

The Presence measure is unusual in university ranking systems, in that it measures the quality of the university's website. Ranked by presence, the top eight Australian universities are:

1. University of Queensland
2. University of Melbourne
3. University of New South Wales
4. Australian National University
5. RMIT University
6. Edith Cowan University
7. University of Western Australia
8. Macquarie University

Sydney, Adelaide and Monash are replaced by RMIT, Edith Cowan and Macquarie. This effect of the Presence is more pronounced with world rankings, where the usual US and UK prestige institutions rank first in Webometrics, but for presence the University of São Paulo and  Fundação Getúlio Vargas (Brazil), at 6 and 7 in the world, outrank Clatech and the University of Oxford. 

Another interesting result is that universities with "open" in their names do not rate highly on the Webometrics openness scale (the Open University UK ranks highest at 403). Similarly those with "virtual" in their name do not rate highly for web presence (Tamil Virtual Academy, is highest at 1172).

The institutions I have studied at most recently rank as follows on Webometrics:

Rank	Institution	Presence	Impact	Openness	Excellence
78	ANU	300	70	69	131
738	USQ	1438	658	718	1200
1242	Athabasca University	1256	1169	1362	1879
8175	CIT	4961	6352	8635	5778

This is much as I would expect, with ANU being a leading research university, USQ and Athabasca teaching universities and CIT a vocational college. However both USQ and Athabasca score poorly compared to ANU on presence and openness, despite their emphasis on e-learning and access to education. USQ claims a "commitment to open education" and Athabasca describes itself as "Canada's Open University", but neither rates in the top one thousand.