Wednesday, June 13, 2018

SIGPLAN Empirical Evaluation Checklist

The ACM Special Interest Group on Programming Languages (SIGPLAN) have provided a revised "SIGPLAN Empirical Evaluation Checklist". This provides criteria to help decides what is suitable in a research paper for publication. This might be of interest more widely, but emphasizes quantitative research, rather than qualitative.

Here are the criteria by Berger, Blackburn, Hauswirth, and Hicks (2018):
Clearly stated claims
  • Explicit Claims
  • Appropriately-Scoped Claims
  • Acknowledges Limitations 
Suitable Comparison
  • Appropriate Baseline for Comparison 
  • Fair Comparison
Principled Benchmark Choice
  •  Appropriate Suite
  • Non-Standard Suite(s) Justified
  • Applications, Not (Just) Kernels
Adequate Data Analysis
  • Sufficient Number of Trials
  • Appropriate Summary Statistics
  • Report Data Distribution
Relevant Metrics
  •  Direct or Appropriate Proxy Metric
  • Measures All Important Effects
Appropriate and Clear Experimental Design
  • Sufficient Information to Repeat
  • Reasonable Platform
  • Explores Key Design Parameters
  • Open Loop in Workload Generator
  • Cross-Validation Where Needed
Presentation of Results
  • Comprehensive Summary Results
  • Axes Include Zero
  • Ratios Plotted Correctly
  • Appropriate Level of Precision
Adapted from Berger, Blackburn, Hauswirth, and Hicks (2018).


 E. D. Berger, S. M. Blackburn, M. Hauswirth, and M. Hicks (June 2018). SIGPLAN Empirical Evaluation Checklist, ACM SIGPLAN. URL

No comments:

Post a Comment