Assessing Software Innovation

While in Vancouver for a computer education conference in 2014, I dropped in on Philippe Kuchen at UBC Electrical and Computer Engineering. He pointed out that they have engineering and business students work together in a "New Venture Design" course (APSC 486) and I suggested ANU do something similar in Canberra. The ANU Techlauncher program was set up by expanding the previous software group projects, to include the option of a start-up. This has now been running successfully for two years, however the great range of projects (startups, projects for established companies and for government agencies) can make supervision and assessment difficult.

Currently assessment is based on the student's group performance and on assessment of an e-portfolio. Students have difficulty writing a personal e-portfolio, having spent years writing in the third person and about group work.

The Simon Initiative at Carnegie Mellon University suggests addressing the problems of group work by assessing the individual, as well as the group, and assess process. They also suggest peer and self-evaluations.

The University of Sydney, Faculty of Education and Social Work, provide a Group Work Guide, which discusses evaluation and reflection. The University's course 6112 Entrepreneurship and Business Innovation, includes peer assessment of group contribution.

Andrew Cain suggests the use of constructive alignment with portfolio assessment, for software students, to provide deeper learning.

Aligning the Learning Outcomes

Techlauncher draws students from several courses and in effect has been retrofitted over these courses (in a similar way to UBC's New Ventures):

1. COMP3100 - Software Engineering Group Project
2. COMP3500 - Software Engineering Project
3. COMP3550 - Advanced Computing R&D Project
4. COMP4500 - Software Engineering Practice
5. COMP8715 - Computing Project
6. COMP3710 - 'Topics in Computer Science'  

Most of these courses have learning outcomes aligned with the accreditation requirements of Engineering Australia and the Australian Computer Society (COMP8715 and  COMP3710  are general purpose project courses).

Learning Outcomes

The courses COMP3100 and COMP3500 have the same learning outcomes set:

1. "Work as an effective member of a team to implement a software based solution that delivers measurable value to an industry or university client.
2. Develop life-long learning through reflection, as demonstrated through continual reflection on the software development lifecycle and team work processes experienced throughout the year.
3. Exhibit an awareness of- team formation strategies and stages leading to the development of high performing, self-managing teams;- sound meeting practice; and- how personality traits can impact upon team performance and how to use individual traits to achieve the most from team work.
4. Make and defend sound engineering decisions.
5. Communicate effectively, orally and in writing, with peers, supervisors and commercial clients/stakeholders.
6. Creatively identify and implement a solution to a complex problem that exists within the domain of ICT.
7. Participate effectively in project and artefact reviews with peers, supervisors and clients/stakeholders.
8. Explain the role and importance of project management, configuration and risk management processes when undertaking a software development project. Demonstrate experience in undertaking the activities associated with these.
9. Explain and understand the importance of the different stages of, and activities associated with each, the software development lifecycle (SDLC). Demonstrate experience in all stages of the SDLC.
10. Explain the role and importance of standards in software development. Demonstrate experience in tailoring those standards approriately [sic] according to the project they are currently undertaking.
11. Participate in a group presentation, including a demonstration, to an audience of peers, clients and supervisors."

COMP4500 is for students to gain more advanced experience working as the team manager:

1. "Work as an effective team manager of a small team of peers to implement a software based solution that delivers measurable value to an industry or university client."

COMP3550 has a slightly different emphasis, with the first learning objective, with development, rather than implementation and innovation added : 

1. Work as an effective member of a team to implement develop a software based solution that is innovative and delivers measurable value to an industry or university client.

UBC's  Interdisciplinary Capstone Project (APSC 496) has similar learning goals (outcomes) to ANU's:  

• "interacting with both users and clinicians to define the problem and to characterize the existing range of solutions
• identification of the value that the design team can offer (through understanding the relationship between the potential users’ needs and the constraints of those who will ultimately be asked to pay the costs of the proposed solutions, as well as by considering how a potential solution may ultimately be brought to market or otherwise to implementation).
• project planning – laying out a reasonable timeline, identifying and assigning responsibilities for different tasks to different team members.
• interface definition – explicitly identifying and documenting how each member’s or sub-team’s responsibilities connect with those of the other members or sub-teams.
• development and use of proper project documentation – students are acquainted with ISO 9000-type quality control regulations and  use similar (though likely simplified) versions of these regulations in their own projects.
• familiarization with the use of standards and associated regulatory and liability considerations – the Senior Chairholder has significant experience with safety-critical systems and has negotiated an accord with the Canadian Standards Association (Canada’s representative to ISO) to make important standards available to faculty members and students.  The Associate Chairholder is working with instructors in the graduate Biomedical Engineering program who are trained in medical regulatory issues to integrate these considerations into his current graduate courses.
• regular team communication – use of concurrent engineering approaches to ensure that issues relevant to and with implications for other team members are identified early and communicated clearly both amongst the team and to the faculty advisors and clients.
• familiarization with key prototyping technologies relevant to each discipline (e.g., for mechanical engineers, this involves rapid prototyping processes such as water-jet cutting, bending machines, spot welding, stereo-lithography machines, etc; for electrical engineers, this involves working with FPGAs or CAD-based circuit designers and simulators).
• quantitative reasoning – an emphasis on the use of prediction early in the design process, based on analytical models, simulations, experiments or prototype development and testing."

The difference here is that ANU includes life-long learning and reflection. UBC has more on the value process as well as technical standards and also has its own Strategic Design Method.

Life Long Learning

The description of ANU's Comp3500 has a  Mapping of Learning Outcomes to Assessment and Professional Competencies.  This maps the course Learning Outcomes and the assessment items to the compliances required by the ACS and Engineers Australia. Learning Outcome 2 "Develop life-long learning through reflection ...", is unusual, as it is assessed through just one assessment item (number 5), an individual reflective learning portfolio (30% of the total assessment). This is also unusual as the Learning Outcome does not map to any ACS Software Engineering Areas (whereas it maps to several Engineers Australia areas).

The SFIA framework used by ACS does mention education. However these are skills for educators who help others with their learning, not self directed learning. This is perhaps something which needs to be added to the framework, as being able to learn is an essential skill for any professional.