SIGPLAN Empirical Evaluation Checklist

The ACM Special Interest Group on Programming Languages (SIGPLAN) have provided a revised "SIGPLAN Empirical Evaluation Checklist". This provides criteria to help decides what is suitable in a research paper for publication. This might be of interest more widely, but emphasizes quantitative research, rather than qualitative.

Here are the criteria by Berger, Blackburn, Hauswirth, and Hicks (2018):

Clearly stated claims

• Explicit Claims
• Appropriately-Scoped Claims
• Acknowledges Limitations 

Suitable Comparison

• Appropriate Baseline for Comparison 
• Fair Comparison

Principled Benchmark Choice

•  Appropriate Suite
• Non-Standard Suite(s) Justified
• Applications, Not (Just) Kernels

Adequate Data Analysis

• Sufficient Number of Trials
• Appropriate Summary Statistics
• Report Data Distribution

Relevant Metrics

•  Direct or Appropriate Proxy Metric
• Measures All Important Effects

Appropriate and Clear Experimental Design

• Sufficient Information to Repeat
• Reasonable Platform
• Explores Key Design Parameters
• Open Loop in Workload Generator
• Cross-Validation Where Needed

Presentation of Results

• Comprehensive Summary Results
• Axes Include Zero
• Ratios Plotted Correctly
• Appropriate Level of Precision

Adapted from Berger, Blackburn, Hauswirth, and Hicks (2018).

References

 E. D. Berger, S. M. Blackburn, M. Hauswirth, and M. Hicks (June 2018). SIGPLAN Empirical Evaluation Checklist, ACM SIGPLAN. URL https://raw.githubusercontent.com/SIGPLAN/empirical-evaluation/master/checklist/checklist.pdf