My Last Review

I have decided to stop reviewing papers for academic journals (I will still do for conferences I am involved with). The reason is that the academic publishing system is exploitative. Reviewers don't get paid, or any other form of compensation, for reviewing. In theory this is something you do, as you will then have your papers reviewed. But in practice there are many free riders. My gesture is a tiny one, but then I remember when I decided to stop giving lectures and that gained traction.

Authors submit papers and expect others to review them, but are not required to review. The authors receive a benefit from published papers, and a financial benefit when this results in a job, or research grant. Some publications are for profit and the published receives revenue from subscribers. The ones missing out from this are reviewers, who get no credit, or payment. So I will stop doing this. 

The system could be easily fixed. Reviewers could receive a voucher for each four papers they review, entitling them to priority processing for one paper submitted. For for-profit publications, they could simply be paid.

Publications could also invest in automated tools to take some of the drudgery out of reviewing. This would check for plagiarism (especially self plagiarism, where autoes submit the same paper, or ones with just a few changes, to multiple publications). Systems could also check references in papers. 

Accelerating and Democratizing Scientific Research Lifecycle

Greetings from the ANU School of Computing, where Qingyun Wang from University of Illinois Urbana-Champaign is speaking on Accelerating and Democratizing Scientific Research Lifecycle. Much has been written about more research of lower quality being published. Rather than bemoaning the role of AI in making this worse, here the idea is to use "AI for Scientists" (AI4Scientist) to address it. The first part of this is to automatically analyze papers to provide structured data. This can be used to explain the paper to the reader and also fact check it.

This can be taken further to expand the generation of hypotheses, but so far in very restricted well defined fields, such as biomedical. AI could also be used to seek out "hidden treasures" in the scientific literature. This would be a boon, as there is a lot of papers which simply repeat previous research, with slight variations. 

What I find most interesting is the possibility to use this to teach students to write better papers. It could also be used to help working researchers write more readable papers, with fewer errors. It would be interesting to see how well this would work for non-STEM disciplines where the format of papers is not so constrained. A particular problem for human readers, and likely for AI, are interdisciplinary research. 

SIGPLAN Empirical Evaluation Checklist

The ACM Special Interest Group on Programming Languages (SIGPLAN) have provided a revised "SIGPLAN Empirical Evaluation Checklist". This provides criteria to help decides what is suitable in a research paper for publication. This might be of interest more widely, but emphasizes quantitative research, rather than qualitative.

Here are the criteria by Berger, Blackburn, Hauswirth, and Hicks (2018):

Clearly stated claims

• Explicit Claims
• Appropriately-Scoped Claims
• Acknowledges Limitations 

Suitable Comparison

• Appropriate Baseline for Comparison 
• Fair Comparison

Principled Benchmark Choice

•  Appropriate Suite
• Non-Standard Suite(s) Justified
• Applications, Not (Just) Kernels

Adequate Data Analysis

• Sufficient Number of Trials
• Appropriate Summary Statistics
• Report Data Distribution

Relevant Metrics

•  Direct or Appropriate Proxy Metric
• Measures All Important Effects

Appropriate and Clear Experimental Design

• Sufficient Information to Repeat
• Reasonable Platform
• Explores Key Design Parameters
• Open Loop in Workload Generator
• Cross-Validation Where Needed

Presentation of Results

• Comprehensive Summary Results
• Axes Include Zero
• Ratios Plotted Correctly
• Appropriate Level of Precision

Adapted from Berger, Blackburn, Hauswirth, and Hicks (2018).

References

 E. D. Berger, S. M. Blackburn, M. Hauswirth, and M. Hicks (June 2018). SIGPLAN Empirical Evaluation Checklist, ACM SIGPLAN. URL https://raw.githubusercontent.com/SIGPLAN/empirical-evaluation/master/checklist/checklist.pdf

Inquiry into Funding Australia's Research

The Australian Parliament is holding an inquiry into funding Australia’s research. Submissions are due by 30 June 2018. I suggest government funding should be conditional on Open Access publishing of the results and researchers receiving start-up training in how to apply their results. The era where researchers did the research and did nothing with it should end. Researchers need to be trained in how to take their work to the community.

"Terms of Reference

The House Standing Committee on Education, Employment and Training will inquire into and report on the efficiency, effectiveness and coherency of Australian Government funding for research, in the following terms:

• The diversity, fragmentation and efficiency of research investment across the Australian Government, including the range of programs, guidelines and methods of assessment of grants;

• The process and administrative role undertaken by research institutions, in particular universities, in developing and managing applications for research funding;

• The effectiveness and efficiency of operating a dual funding system for university research, namely competitive grants and performance-based block grants to cover systemic costs of research; and

• Opportunities to maximise the impact of funding by ensuring optimal simplicity and efficiency for researchers and research institutions while prioritising delivery of national priorities and public benefit.

This inquiry will be focused on federally funded research agencies, their funding mechanisms and university collaborative research. The inquiry will not consider the National Health and Medical Research Council, nor non-federal research funding."

The Scientific Paper Is Dead: LONG LIVE THE SCIENTIFIC PAPER!

James Somers argues that the scientific paper, developed as early as 1600, enabled our modern age but is now obsolete. He envisions the "paper" being replaced by something  more interactive ("The Scientific Paper Is Obsolete", The Atlantic, April 2018 ). However, interaction doesn't render the paper obsolete and is not new technology. In 1989 an interactive technology was developed specifically to enhance scientific communication, it came to be known as The World Wide Web. In "Information Management: A Proposal", Tim Berners-Lee wrote:

    "This proposal concerns the management of general information about accelerators and experiments at CERN. It discusses the problems of loss of information about complex evolving systems and derives a solution based on a distributed hypertext system."

Since then it seems the Web has been used for just about everything except scientific communication. But there have been some advances. There are social media-like forums for academia and in some fields data and working is published electronically, along with final results.

I expect we will continue to have scientific papers, but with hypertext links to the workings which lead to the paper and to where new work is being done. The resulting "papers" can be backward compatible, so if you print one out it looks like a "paper".

However, the purpose of a scientific paper needs to be kept in mind. Scientific papers are written by scientists for other scientists, they are not for the general public. Writing about science for the general public is, in itself, a specialist discipline and requires skills most scientists don't have and don't need.

However, we do need to train scientists, and other academics, in how to use the technology for communicating with their peers. I took part in a workshop recently on how to do the research to create a major new Australian export industry. There were to be several dozen researchers involved in multiple overlapping multidisciplinary projects spread across the country. The first issue was, inevitably, about funding. But the next question was which on-line tools were we to use for collaboration.

Software and engineering students now get trained to use tools to help with projects and which will, as a byproduct, produce documentation and reports. One of the first steps for a new student team is which tools to use. There is a lively discussion in tutorials between teams on the features of the different tools. But I doubt that many other STEM students do and few non-STEM.

Open Access Policy Draft from Australian Research Council

The Australian Research Council (ARC) have released a draft Open Access Policy for comment. The current policy was adopted in 1 January 2013. The policy requires researchers receiving a grant to make their research papers available in open access format within a year of publication, or tell the ARC why they can't. What is missing from the review are any statistics on how successful the current policy is. I suggest the new policy require institutions to publish a list of all publications on their web site, indicating which are open access and an annual count of the number open and closed, along with the amount of public funding received for those open and closed. The ARC should then use this information to produce statistics on openness of Australian research, in terms of numbers of publications and proportion of funding, along with the full list of institutions ranked by openness.

Open Researcher Identifier Launch in Canberra

ORCID are launching their Australian effort for globally standardized open IDs for tracking researcher's publications, at a free event in Canberra on 15 February 2016. See also "Breaking the myths of scholarly credit".

ORCID is an open, non-profit, community-driven effort to create and maintain a registry of unique researcher identifiers and a transparent method of linking research activities and outputs to these identifiers. ORCID is unique in its ability to reach across disciplines, research sectors and national boundaries. It is a hub that connects researchers and research through the embedding of ORCID identifiers in key workflows, such as research profile maintenance, manuscript submissions, grant applications, and patent applications.