SIGPLAN Empirical Evaluation Checklist

The ACM Special Interest Group on Programming Languages (SIGPLAN) have provided a revised "SIGPLAN Empirical Evaluation Checklist". This provides criteria to help decides what is suitable in a research paper for publication. This might be of interest more widely, but emphasizes quantitative research, rather than qualitative.

Here are the criteria by Berger, Blackburn, Hauswirth, and Hicks (2018):

Clearly stated claims

• Explicit Claims
• Appropriately-Scoped Claims
• Acknowledges Limitations 

Suitable Comparison

• Appropriate Baseline for Comparison 
• Fair Comparison

Principled Benchmark Choice

•  Appropriate Suite
• Non-Standard Suite(s) Justified
• Applications, Not (Just) Kernels

Adequate Data Analysis

• Sufficient Number of Trials
• Appropriate Summary Statistics
• Report Data Distribution

Relevant Metrics

•  Direct or Appropriate Proxy Metric
• Measures All Important Effects

Appropriate and Clear Experimental Design

• Sufficient Information to Repeat
• Reasonable Platform
• Explores Key Design Parameters
• Open Loop in Workload Generator
• Cross-Validation Where Needed

Presentation of Results

• Comprehensive Summary Results
• Axes Include Zero
• Ratios Plotted Correctly
• Appropriate Level of Precision

Adapted from Berger, Blackburn, Hauswirth, and Hicks (2018).

References

 E. D. Berger, S. M. Blackburn, M. Hauswirth, and M. Hicks (June 2018). SIGPLAN Empirical Evaluation Checklist, ACM SIGPLAN. URL https://raw.githubusercontent.com/SIGPLAN/empirical-evaluation/master/checklist/checklist.pdf

Draft Cybersecurity Curricula from IFIP, ACM, IEEE-CS AIS SIGSEC

A 74 page Draft Cybersecurity Curricula 2017, Version 0.75  is available (12 June 2017) from the Joint Task Force on Cybersecurity Education (JTF). The task force has representation from the IFIP Technical Committee on Information Security Education (IFIP WG 11.8), as well as ACM, IEEE Computer Society and AIS SIGSEC. A final curricula recommendation is due in December. It is not clear how the curricula relates to the cyber-security certifications recently announced by ACS and IFIP.

The 12 June draft of the task force divides the Curricular Content into six "Knowledge Areas":

1. Data Security
2. Software Security 
3. System Security 
4. Human Security 
5. Organizational Security 
6. Societal Security

Recommended study hours per knowledge area have not yet been specified.

The report contains a curious section 5.1 on "The Academic Myth" (page 58):

"Students who graduate from a four-year university program assume that the baccalaureate degree is a sufficient qualification to attain a position. This understanding may be true in some fields, but not necessarily in the computing disciplines nor specifically in cybersecurity. Belief in this myth has stymied many a job hunter worldwide. The degree credential is growing in importance, but it is not a sufficient condition for a position. A general understanding exists in cybersecurity and other fields that a successful professional must be a good communicator, a strong team player, and a person with passion to succeed. Hence, having a degree is not sufficient to secure employment."

The report goes on in the next section to detail Non-technical Skills (Section 5.2, Page 58):

"Non-technical (sometimes called “soft”) skills are vital to the success of cybersecurity professionals. The ability to work in a team, communicate technical topics to non-technical audiences, successfully argue for resource allocations, hone situational awareness, and operate within disparate organizational cultures are just a few of these skills. The US Chief Human Capital Officers Council (CHCO), among other bodies, has developed a list of non-technical competencies pertinent to the cybersecurity workforce. The list includes: accountability, attention to detail, resilience, conflict management, reasoning, verbal and written communication, and teamwork. The full list of competencies is available in the Competency Model for Cybersecurity. Professional associations such as (ISC) and ISACA also provide recommendations for non-technical skills required for cybersecurity professionals."

The report's authors seem to assume that that these soft skills have no place in a baccalaureate degree program. However, those are the skills I, and my colleagues, are teaching to computer science and engineering students at the Australian National University. As part of team projects and individual internships, the students have to learn to work together, communicate with a real client, negotiate for resources and present their work. Obviously, students with limited work-place experience can only learn so much and there is a continual discussion of the role of higher degrees for improving skills and smaller sub-degree courses. That approach fits with the ACS' approach to certification, which recognizes experience alongside formal qualifications.

Cybersecurity Degree Guidelines

The Association for Computing Machinery (ACM) have released a draft "Cybersecurity Curricula 2017: Curriculum Guidelines for Undergraduate Degree Programs in Cybersecurity" for comment by 14 February 2017. The security areas focused on are: Data, Software, System, Human, Organizational and Societal. Discipline areas ares: Computer Science (CS); Computer Engineering (CE); Software Engineering (SE); Information Technology (IT); Information Systems (IS); and Mixed Disciplinary majors (MD). This draft has not got to the point of setting hours for knowledge areas, but is a good start.

I have submitted this comment:

"The Cybersecurity Curricula is well thought out. The only surprise for me was section 5.1 "The Academic Myth" (p. 33). This polemic against the value of baccalaureate degrees and assessment standards is not appropriate. If the authors believe that a first degree does not provide the skills required for Cybersecurity, then they should be preparing a curriculum which includes a mandatory graduate component. If the authors truly believe that "... having a degree is not sufficient to secure employment.", then they should set down the curriculum for the additional non-degree training and education required.

Setting out to specify a baccalaureate curricula which does not meet the required need seems a pointless activity. In my view a baccalaureate degree is a vocationally useful qualification. However, no single qualification will provide everything everyone needs. The authors of the Cybersecurity Curricula should not set themselves an impossible task. Such a curricula will be useful when designing educational programs, at the sub-degree, degree and also graduate levels. I suggest deleting section  5.1."

Social Issues and Professional Practice in University Degree Programs

Previously I looked at the"Social Issues and Professional Practice (SP)" required by the Draft ACM/IEEE-CS Computer Science Curricula 2013. This works out to the equivalent of one course out of a 24 course degree program (but it could be covered in multiple courses). The ANU's two year Master of Engineering requires two courses in Professional communication (ENGN8150 and ENGN8160), with these normally being taken consecutively in the first two semesters. However, students may apply to be exempt from these courses. The ANU Research School of Management has Communication for Business (MGMT2100) as a compulsory course for the Bachelor of Business Administration. The Graduate Diploma in Legal Practice
 has the option of a Professional Practice Core (PPC) which is 15 units (2.5 conventional courses). This has the students working on-line in a simulated law first with three others.

ACM/IEEE-CS Computer Science Curricula 2013

The joint ACM/IEEE-CS Computer Science
Curricula 2013 Steering Committee (CS2013), have released a Draft Final Report (October 2013). The previous versions of the ACM/IEEE-CS curricular have been used by universities around the world in designing their degree programs. There is also a core learning outcomes spreadsheet and some Course and Curriculum Exemplars.

The ACM/IEE-CS Curricula specifies the proportion of course on each topic in terms of lecture hours, for delivery in a traditional face-to-face class. This does not include self-study time, lab sessions or student work on assessments. So confusingly it is less than the usual measure used by universities of  "contact hours" (which includes tutorials and labs sessions). The Total Core Hours are: Tier1 165 Hours, Tier1 143 Hours. Programs are required to have all of Tier 1, plus at least 80% of Tier 2, making a total of at least 279.4 Hours.

The ANU Bachelor of Information Technology
requires the completion of 144 units, with 36 compulsory computer science units, another 30 units from later years CS courses, 24 units of additional CS courses, 6 core maths courses, leaving 28 units of general electives. A typical 6 unit ANU Computer Science course has thirty one, one hour lectures and six two hour tutorials. The full time degree program is usually made up of three years of two semesters, with four courses per semester for a total of  24 courses (3 x 2 x 4). This equates to 744 hours of lectures.

While the ACM/IEE-CS have been influential with their curriculum internationally, it should be noted that this is a very US-centric document. The only mention of Australia in the document, is Judy Sheard's Human Computer Interaction (FIT3063), Monash University. The UK is represented by Paul Cairns' Human Aspects of Computer Science, University of York, Alan Blackwell's Human Computer Interaction and Software and Interface Design, University of Cambridge. Most of the other courses are from the USA, with a few from Europe and Asia.

ACM/IEEE-CS Computer Science Curricula 2013 

Table of Contents

Chapter 1: Introduction . 10

Overview of the CS2013 Process . 11
Survey Input  12
High-level Themes  13
Knowledge Areas  14
Professional Practice . 15
Exemplars of Curricula and Courses  16
Community Involvement and Website . 16
Acknowledgments  16
References . 19

Chapter 2: Principles . 20

Chapter 3: Characteristics of Graduates . 23

Chapter 4: Introduction to the Body of Knowledge 27

Knowledge Areas are Not Necessarily Courses (and Important Examples Thereof). 28
Core Tier-1, Core Tier-2, Elective: What These Terms Mean, What is Required . 29
Further Considerations in Designing a Curriculum  32
Organization of the Body of Knowledge  32
Curricular Hours . 32
Courses 33
Guidance on Learning Outcomes . 33
Overview of New Knowledge Areas  34

Chapter 5: Introductory Courses . 39

Design Dimensions . 39
Mapping to the Body of Knowledge. 45

Chapter 6: Institutional Challenges . 46

Localizing CS2013 . 46
Actively Promoting Computer Science  46
Broadening Participation  47
Computer Science Across Campus . 48
Computer Science Minors  48
Mathematics Requirements in Computer Science  49
Computing Resources . 51
Maintaining a Flexible and Healthy Faculty. 51
Teaching Faculty. 52
Undergraduate Teaching Assistants 53
Online Education  53
References . 54

Appendix A: The Body of Knowledge . 55

Algorithms and Complexity (AL). 55
Architecture and Organization (AR). 62
Computational Science (CN)  68
Discrete Structures (DS) . 76
Graphics and Visualization (GV) . 82
Human-Computer Interaction (HCI)  89
Information Assurance and Security (IAS)  97
-4-Information Management (IM) . 112
Intelligent Systems (IS)  121
Networking and Communication (NC). 131
Operating Systems (OS) . 136
Platform-Based Development (PBD) . 143
Parallel and Distributed Computing (PD) . 146
Programming Languages (PL)  156
Software Development Fundamentals (SDF) . 168
Software Engineering (SE) . 173
Systems Fundamentals (SF) 187
Social Issues and Professional Practice (SP)  193

Appendix B: Migrating to CS2013 . 205

Outcomes  205
Changes in Knowledge Area Structure . 206
Core Comparison  207
Conclusions . 211

Appendix C: Course Exemplars 228

Course Exemplar Template  232
CSCI 140: Algorithms, Pomona College  234
COS 226: Algorithms and Data Structures, Princeton University 237
CS 256 Algorithm Design and Analysis, Williams College . 240
CSE332: Data Abstractions, University of Washington . 243
CS/ECE 552: Introduction to Computer Architecture, University of Wisconsin . 246
CS150: Digital Components and Design, University of California, Berkeley . 249
-5-CC152: Computer Architecture and Engineering, University of California, Berkeley  251
eScience, University of North Carolina at Charlotte  253
COSC/MATH 201: Modeling and Simulation for the Sciences, Wofford College  258
MAT 267: Discrete Mathematics, Union County College . 262
CS103: Mathematical Foundations of Computer Science, Stanford University  265
CS109: Probability Theory for Computer Scientists, Stanford University  265
CS 250 - Discrete Structures I, Portland Community College . 268
CS 251 - Discrete Structures II, Portland Community College  271
CS 175 Computer Graphics, Harvard University . 274
CS371: Computer Graphics, Williams College  277
Human Aspects of Computer Science, University of York  280
FIT3063 Human Computer Interaction, Monash University 282
CO328: Human Computer Interaction, University of Kent  285
Human Computer Interaction, University of Cambridge . 287
Human-Computer Interaction, Stanford University . 289
Human Information Processing (HIP), Open University Netherlands . 291
Software and Interface Design, University of Cambridge  293
Computer Systems Security (CS-475), Lewis-Clark State College  295
CS430: Database Systems, Colorado State University. 298
Technology, Ethics, and Global Society (CSE 262), Miami University  301
CS 662; Artificial Intelligence Programming, University of San Francisco  304
Intelligenza Artificiale ( Artificial Intelligence), Politecnico di Milano  306
CMSC 471, Introduction to Artificial Intelligence, U. of Maryland, Baltimore, County  308
Introduction to Artificial Intelligence, Case Western Reserve University . 310
-6-CS188: Artificial Intelligence, University of California Berkeley . 313
Introduction to Artificial Intelligence, University of Hartford . 315
Computer Networks I, Case Western Reserve University  318
CS144: Introduction to Computer Networking, Stanford University . 320
Computer Networks, Williams College  323
CSCI 432 Operating Systems, Williams College . 327
CS 420, Operating Systems, Embry-Riddle Aeronautical University  330
CPSC 3380 Operating Systems, U. of Arkansas at Little Rock . 332
582219 Operating Systems, University of Helsinki . 334
RU STY1 Operating Systems, Reykjavik University. 336
Parallel Programming Principle and Practice, Huazhong U. of Science and Technology . 339
Introduction to Parallel Programming, Nizhni Novgorod State University . 342
CS in Parallel (course modules on parallel computing) . 344
CS453: Introduction to Compilers, Colorado State University  348
Csc 453: Translators and Systems Software, The University of Arizona  351
CSCI 434T: Compiler Design, Williams College  353
Compilers, Stanford University  356
Languages and Compilers, Utrecht University . 359
COMP 412: Topics in Compiler Construction, Rice University  361
CSC 131: Principles of Programming Languages, Pomona College 364
CSCI 1730: Introduction to Programming Languages, Brown University  367
CSC 2/454: Programming Language Design and Implementation, University of Rochester 369
CSE341: Programming Languages, University of Washington . 372
CSCI 334: Principles of Programming Languages, Williams College . 375
-7-Programming Languages and Techniques I, University of Pennsylvania  378
15-312 Principles of Programming Languages, Carnegie Mellon University 381
15-150: Functional Programming, Carnegie Mellon University  385
CIS 133J: Java Programming I, Portland Community College  389
Introduction to Computer Science, Harvey Mudd College  392
CpSc 215: Software Development Foundations, Clemson University . 395
CS1101: Introduction to Program Design, WPI . 398
Data Abstraction and Data Structures, Miami University  401
Software Engineering Practices, Embry Riddle Aeronautical University  403
CS169: Software Engineering, University of California, Berkeley 407
SE-2890 Software Engineering Practices, Milwaukee School of Engineering  410
Software Development, Quinnipiac University  412
CS2200: Introduction to Systems and Networking, Georgia Institute of Technology . 415
CS61C: Great Ideas in Computer Architecture, University of California, Berkeley 419
CSE333: Systems Programming, University of Washington . 421
Ethics in Technology (IFSM304), University of Maryland . 424
Technology Consulting in the Community, Carnegie Mellon University  427
Issues in Computing, Saint Xavier University 431
Ethics & the Information Age (CSI 194), Anne Arundel Community College . 433
Professional Development Seminar, Northwest Missouri State University . 436
The Digital Age, Grinnell College  439
COS 126: General Computer Science, Princeton University . 443
CSCI 0190: Accelerated Introduction to Computer Science, Brown University . 447
An Overview of the Two-Course Intro Sequence, Creighton University. 449
-8-CSC 221: Introduction to Programming, Creighton University . 450
CSC 222: Object-Oriented Programming, Creighton University . 452
An Overview of the Mulit-paradigm Three-course CS Introduction at Grinnell College  454
CSC 151: Functional problem solving, Grinnell College . 456
CSC 161: Imperative Problem Solving and Data Structures, Grinnell College . 458
CSC 207: Algorithms and Object-Oriented Design, Grinnell College . 460

Appendix D: Curricular Exemplars  463

Bluegrass Community and Technical College (A.S. Degree)  465
Bluegrass Community and Technical College (A.A.S. Degree)  472
Grinnell College  480
Stanford University . 492
Williams College  503 Brown University . 447
An Overview of the Two-Course Intro Sequence, Creighton University. 449
-8-CSC 221: Introduction to Programming, Creighton University . 450
CSC 222: Object-Oriented Programming, Creighton University . 452