Wednesday, December 12, 2018

Computer Security is the Wild West

Vertigo film poster
PHOTO: Wikipedia (Theatrical
poster for the film Vertigo)
Greetings from the Australian National University in Canberra, where I am attending a reading group on cyber security. One of the attendees from "a government agency" asked a senior academic why some of the research at international conferences is so poor: the answer was "Computer Security is the Wild West".

It has been an interesting morning. I attended a seminar  on "Vertigo: Fake news/real theory" by the ANU College of Law. Appropriately for the government established university, there was discussion of  David Foster Wallace's unfinished novel The Pale King. This relates the horrors of being a tax inspector. There was also discussion of the difficulties of countering fake news.

The ANU cyber reading group primarily reviews papers on fuzzing.  With this random data is presented to a program to test its security. Millions of random variations can be input to see if the program does something it is not supposed to. This technique is now in routine use to the point where one of our resident experts commented we had reached "peak fuzzing".

It occurs to me the same technique might be used to text how the political system copes with fake news. This would be done by generating social media posts which are in grammatically correct language but containing random words. The program would then look to see which posts were liked, passed on and positively rated. My worry is that there are perhaps for-profit, and state based actors already doing this to attract clicks, and spread confusion. 


Also, I suggest looking at the ethics and legal issues with detecting bugs. What systems should you test, and when you find a vulnerability what can (and should) you do with that information? With a quick search I found a recent paper on the Pentagon's Vulnerability Reward Program (Chatfield & Reddick, 2017).

Reference


Chatfield, A. T., & Reddick, C. G. (2017, June). Cybersecurity Innovation in Government: A Case Study of US Pentagon's Vulnerability Reward Program. In Proceedings of the 18th Annual International Conference on Digital Government Research (pp. 64-73). ACM. URL https://doi.org/10.1145/3085228.3085233


Reference

Chatfield, A. T., & Reddick, C. G. (2017, June). Cybersecurity Innovation in Government: A Case Study of US Pentagon's Vulnerability Reward Program. In Proceedings of the 18th Annual International Conference on Digital Government Research (pp. 64-73). ACM. URL https://doi.org/10.1145/3085228.3085233

Learning to Reflect

This is the fourth of a series of posts on how to provide students with help when preparing a reflective portfolio. This is specifically for students of  ANU Tech Launcher and the ANU Computer Science Internship. Previously 

I looked at how they assess a portfolio. How should a a module be structured to teach this? How should the 18 to 20 hours of student study time be allocated?

Module title: Learning to Reflect


Overview:  This module will enable students to develop competencies expected of working professionals to plan what skills and knowledge they need to develop, acquire those skills and reflect on what they have learned.

Learning Outcomes:


Upon completion of this module, students will be able to:

  1. Determine the learning needed and possible sources, to grow individual skills for a project,  and career plans.
  2. Identify appropriate accreditation and qualification paths. 
  3. Manage the learning, evaluate outcomes through reflection.
Adapted from the skill "Learning and Development" (ETMG), Level 6, Skills Framework for the Information Age, Version 7, 2017

Indicative Assessment


Three online quizzes, 10% (5% per quiz, with best two counted). Contributions to three discussion forums, 20% (10% per forum, with best two out of three counted). Three assignments, 70% (35% each, best two out of three counted). Feedback from students in the forums, and on assignments, will be taken into account in assessment by the examiners.

Workload

Based on the credit points involved, from 18 to 20 hours of student learning time, consisting of participation in forums and assessment activities.

Friday, December 7, 2018

Tales from TALE 2018

On my way home from speaking at the IEEE 7th International Conference on Teaching, Assessment, and Learning for Engineering (TALE 2108) in Wollongong, so time to reflect. Some surprises were that chat-bots could be useful in education, and China is preparing to scale up its international university education offerings (placing Australia's third largest export industry at risk). One disappointment was that while the leading educators could detail what needed to be done to improve the quality of university teaching, none could offer a strategy to ensure this was actually done.

Why TALE?


I did not have high hopes for this conference. I decided to submit a paper because the conference I would usually attend was in a country I did not want to visit. TALE was two hours drive away, and was an IEEE event, so that was enough for me. Early in the year I started the grueling process of preparing a paper.

Come submission time I volunteered to also review. That turned out to be an unexpected pleasure, the management system worked well and the papers were of good quality. It was hard to find what to reject, and I worried my paper would never get accepted, with this competition. However, my paper was accepted, with lots of changes being required.

The last major frustration was the formatting, where there was something being rejected by the IEEE system, but no one could tell me exactly what. However, after many attempts, and days of work, laboriously reformatting the paper with different tools (and introducing new errors along the way), it was finally okay.

The Venue

An international conference from Australia usually involves a flight of at least half a day. So it was a strange feeling to just get in my car and drive an hour and a half to Wollongong. This is a beach-side city and the venue,was right on the beach.

Workshop Chatbot Tutors for Blended Learning

The workshops were held at University of Wollongong, a short free shuttle bus ride from the city.  I chose "Chatbot Tutors for Blended Learning" by Chi-Un Lei, Yuqian Chai, Xiangyu Hou, and Vincent Tam from TELI at University of Hong Kong. I had in mind using this for routine questions from students. The workshop is using the free version of the IBM Watson tool. In a few hours I was able to produce a credible Q&A. The process with the chatbot doesn't look any more time consuming that a quiz, with the AI system providing flexibility. What seems to be missing from this process is the intelligence to create the answers. For example, I would like to just give the system the rules for the course assessment and have it work out the possible questions and answers.

The Engineering of Learning

Keynote speaker Dr Bror Saxberg of the Chan Zuckerberg Initiative made the case for the engineering of education at the opening. Dr Saxberg's argument seems to be we need to teach students in ways which have been found to be effective. That may sound self-evident, but as he pointed out, with numerous examples, formal education is not necessarily using techniques show to be effective, and in some cases shown to not work. However, the problem I have found, is that just telling someone what works for education does not get them to do it, we need to also have people actually do things, to learn how to do them. Dr Saxberg did not provide any policy strategies to have academics learn to teach.

Learning Engineering

Professor Gregor Kennedy's keynote the next day was on "Learning Engineering: The Art of Applying Learning Science at Scale". He did not seem happy with the term “learning engineering”, coined by Herb Simon in the 1960's for the systematic design of learning, based on research. But I suspect there were only a handful of people in the room who had ever heard of the term (Dr Saxberg was obviously one), so why mention it at all? Professor Kennedy seemed to want evidence based teaching, but like Dr Saxberg did not seem to have any strategies to make this happen.

Later in the day Rebecca Shields (Central Queensland University) discussed the results of research on the "21st Century Skills" of Australian school students entering university. Rebecca suggested pre-teaching of students entering university, and in the longer term changes to school teacher training and education policy. Her proposed solution's would help answer Professor Kennedy's call for more systematic application of learning science. 

Smart Learning at the University of the South Pacific

Staff of the University of the South Pacific (UPS) discussed "Smart Learning in the Pacific: Design of New
Pedagogical Tools". USP is multi-national, with students who have studied under different school systems. They have an "early warning system" which extracts data from their Moodle Learning Management System to indicate which students are struggling. This was useful actionable advice.


Globalization of Chinese Education

Liang Zhao from Shenyang Aerospace University was talked on "How We Face Globalization of Chinese Education". International students in China are instructed in English, even though this is not the first language of the instructors or the students. Australian universities  will need to re-think their offerings, if China solves this problem, and takes most of the international student market in our region.

Arjun Singh on Gradescope

Arjun Singh, Co-founder & CEOArjun Singh, Co-Founder & CEO of Gradescope talked on grading of large numbers of STEM exam papers. This product, recently acquired by TurnItIn, allows student exam papers to be scanned in, and then marked online by an examiner, using a rubric. The product was demonstrated for engineering and computer science examinations, including for computer code.

Reference

 Worthington, Tom. (in press). Blended Learning for the Indo-Pacific. In Teaching, Assessment, and Learning for Engineering (TALE), 2018 IEEE 7th International Conference on. IEEE. url http://hdl.handle.net/1885/148733

Thursday, December 6, 2018

Blended Learning for the Indo-Pacific

Tom Worthington. Photo by Stuart Hay, ANU Senior Photographer, 2014Greetings from the the IEEE 7th International Conference on Teaching, Assessment, and Learning for Engineering (TALE) in Wollongong where I was speaking on "Blended Learning for the Indo-Pacific". This proposal has been entered in the the Solomon Islands Technology for Development Challenge to provide Micro-credentials by Mobile Phone for the Solomon Islands.
"Information technology disciplines make up a significant proportion of the degrees taken by international students at Australian universities. These programs are primarily delivered on-campus, but are increasingly using e-learning techniques and becoming, in effect, blended. This provides the opportunity to offer international students part of their program by distance education before, or instead of, traveling to Australia. This could complement the campus-based education provided and complement initiatives by China, Australia, Japan and the United States for regional development. However, Australian university academics have little background or training in e-learning and program designs have not made use of the flexibility this provides. In this paper, we discuss how computer professionals can be trained online to deliver online training to students of the Indo-Pacific. The application of learning theory to support distance learners is also discussed in this paper."
This followed papers from others at the conference on the globalization of Chinese education, Smart Learning at the University of the South Pacific and 21st Century Skills.

ps: TALE 2019 will be December 3-6,  in Yogyakarta, Indonesia. And, just announced, TALE 2020 will be in Takamatsu, Japan. 

Arjun Singh on Gradescope

Arjun Singh, Co-founder & CEOGreetings from the the IEEE 7th International Conference on Teaching, Assessment, and Learning for Engineering (TALE) in Wollongong where Arjun Singh, Co-Founder & CEO of Gradescope is talking on grading of large numbers of STEM exam papers. This product, recently acquired by TurnItIn, allows student exam papers to be scanned in, and then marked online by an examiner, using a rubric. 

One claim for the product is that the rubric can be customized by the examiner, as they do the marking. The product was demonstrated for engineering and computer science examinations, including for computer code. 

There appear to be two distinct uses for the product: one is for marking traditional paper based examinations. The other is for digitally input long-form input, such as computer code.

This product is new to Turnitin, and integration with their copy detection function is to be added in early 2019. However, even in its current form the interface looks similar to Turnitin's GradeMark.

Globalization of Chinese Education

Greetings from the the IEEE 7th International Conference on Teaching, Assessment, and Learning for Engineering (TALE) in Wollongong where Liang Zhao from Shenyang Aerospace University was speaking on "How We Face Globalization of Chinese Education". This, I suggest, is also a question for Australian universities in the next few years.

Just as China has been increasingly producing quality goods at a competitive price, it now aiming to export education services. One interesting point in the paper is that international students in China are instructed in English (although this is not the first language of the staff or students). Australian educational institutions, and companies, may be able to provide specialist services to their Chinese counterparts.
"In the next decade, China is expectedly becoming
the most significant education destination for foreigners. Since information technology is a leading industry in China, its related major computer science would surely attract a considerable number of international students. Therefore, our university and teaching group are going to face this big challenge. Due to the scarcity of teaching resources in ordinary Chinese universities, we have to seek a way to fulfil the vast demand of educating these international students while especially most of them require lecturers provide courses in English. In this paper, we discuss the current problems of international education in China and whether the mixed English teaching can be one possible solution or not. Then we present our teaching reform strategies by showing an example of a module called Mobile Programming with Android. Through applying these strategies, we also list the numerical improvements of students results and skills."
Australian universities need to consider how they will respond to expanded Chinese offerings. As with consumer goods, they might try to compete on price, quality, or convenience. I will be speaking on alternative delivery methods for education in "Blended Learning for the Indo-Pacific" later in the conference. In this I suggest, rather than trying to directly compete, a more flexible form of education could be offered.

Smart Learning at the University of the South Pacific

Greetings from the the IEEE 7th International Conference on Teaching, Assessment, and Learning for Engineering (TALE) in Wollongong where staff of the University of the South Pacific (UPS) are discussing "Smart Learning in the Pacific: Design of New
Pedagogical Tools". USP is multi-nationals with students who have studied under different school systems. They have ab "early warning system" which extracts data from their Moodle Learning Management System to indicate which students are struggling. They also use Big Blue Button (BBB), for synchronous webinars. Students are tested online for English skills and those who don't score well are automatically signed up of a support course.

Challenges at USP include limited bandwidth to some Pacific countries (something being addressed by the Australian Government funded fibre optic cable to the Solomon Islands). A major focus of USP is teacher training. I will be speaking on "Blended Learning for the Indo-Pacific" later in the conference.

"Smart learning ecosystems leverage on state-of-the-art tools and technologies to help students learn better with Information Communication Technologies (ICT). The ubiquity, innovations and advancements of ICT have transformed pedagogies and approaches to content facilitation and delivery in higher education worldwide, the Pacific region being no exception. The paper essays a number of learning and support tools designed in-house or adopted (or outsourced) recently by a higher education institution in the Pacific contributing to the smart learning ecosystem. The institution has integrated these ICT driven tools to its academic and support programmes, and more recently the in-country science programmes introduced in its member countries. The strengths and challenges from the implementation of these new adaptive tools are highlighted with recommendations to the wider academic populace."

Learning Engineering

Professor Gregor Kennedy's keynote presentation for TALE 2018 was on "Learning Engineering: The Art of Applying Learning Science at Scale". He did not seem happy with the term “learning engineering”, coined by Herb Simon in the 1960's for the systematic design of learning, based on research. But I suspect there were only a handful of people in the room who had ever heard of the term (Bror Saxberg was obviously one), so why mention it at all?

Later in the day Rebecca Shields (Central Queensland University) discussed the results of research on the "21st Century Skills" of Australian school students entering university. These students had grown up with digital technology, but can't apply it for educational purposes, and so need to have training in this. Rebecca suggested pre-teaching of students entering university, and in the longer term changes to school teacher training and education policy.

The problems Rebecca identified with school and teacher training, I suggest also apply at the university level. Her proposed solution's would help answer Professor Kennedy's call for more systematic application of learning science. We need to not only train academics who teach how to teach, but also teach their students how to learn. The academics who teach will then be better able to work with specialist educational designers, and their students take more responsibility for their own learning.

Tuesday, December 4, 2018

The Engineering of Learning

Greetings from the University of Wollongong where Dr Bror Saxberg of the Chan Zuckerberg Initiative is making the case for the engineering of education at the opening of the IEEE 7th International Conference on Teaching, Assessment, and Learning for Engineering (TALE).

Dr Saxberg's argument seems to be essentially the same as that for education based on research. That is, we need to teach students in ways which have been found to be effective. That may sound self-evident, but as he pointed out, with numerous examples, formal education is not necessarily using techniques show to be effective, and in some cases shown to not work. Dr Saxberg's described Maslow's hierarchy of needs, as like Aristotelian physics, in that it was not based on experimental research.

However, the problem I have found, is that just telling someone what works for education does not get them to do it. I even have difficulty convincing myself to do teaching differently, let alone others. Dr Saxberg might argue this requires putting the new techniques into long term memory. However, I suggest we need to also have people actually do things, to learn how to do them. It was only after I had used educational techniques, as a student of education, I was confident I could use them as a teacher.

Dr Saxberg pointed out that research shows learning requires persistence, through motivation. He said "you don't have to like it, you just have to do it". I suggest top-down educational design can help with this. As an example, I am designing a module to help STEM students reflect on learning. The first step is to align the learning objectives with external professional job requirements.

Chatbot Tutors for Blended Learning

Greetings from the University of Wollongong where I am taking part in a workshop on "Chatbot Tutors for Blended Learning" by Chi-Un Lei, Yuqian Chai, Xiangyu Hou, and Vincent Tam from TELI at University of Hong Kong. This is part of the IEEE 7th International Conference on Teaching, Assessment, and Learning for Engineering (TALE).
"Creating a pedagogical agent requires complex computer programming skills and it is usually built from scratch to fit the intended educational purpose. This makes it difficult for teachers to adapt existing systems or to attempt in creating a similar version. Recently, we have leveraged the IBM Watson Assistant chatbot engine to develop some chatbots. These chatbots have been adopted in a common core (general education) course for inquiry-based learning, with a promising outcome. In this workshop, we would like to share our experience gained from developing a set of chatbots for an online blended learning environment. Through the workshop, attendees will identify what and how chatbots can be designed for their own blended or fully online courses. They will also develop a simple chatbot system that can responses to some assessment inquiries."
I had in mind using this for routine questions from students. I provide tutors with a list of standard answers, but perhaps these could be provided by a chatbot.
The example given in the workshop is for  training forensics  scientists in crime scene analysis. The chatbot answers the easy questions and passes on the hard ones to the tutor.

The workshop is using the free version of the IBM Watson tool.  A JSON file was provided for the workshop to upload to Watson.We were cautioned that there were "25 steps" to do this from scratch.We were simply asked to enter the "intent", which we have to be specific about (such as the second assignment deadline). Then the "Entities", such as a specific assignment the student is asking about. The "Dialog" would be the answer.

While the example in workshop was to answer simple logistics questions (is the exam open book?), the idea is to use this for teaching. A simple example might be, rather than say "No, the exam ins not open book", but to guide the student through where they find the answer. This might address teascher's frustrations where students don't read the material given.

The prepared chatbot is not very bright and very single-minded. It reminds me of the talky toaster in Red Dwarf, which was obsessed with toast. One trick shown to make the dialogue more interesting was simply random versions of the same responses.A dialogue is added with answers for the questions. One of the features of using Watson is that it has a dictionary and lists of synonyms. As an example with the Q&A I prepared was about "books", Watson correctly interpreted "document" as being a sort of book (it even understood "Bring Dongle?").

Next we added an "entity".  These can be physical objects (the textbook for the course) or something more abstract.

This chatbot process reminds me of my first encounter with preparing a multiple choice online quiz. The process of setting up the questions and answers was very tedious. The reward only comes when you have many students using the system. The process with the chatbot doesn't look any more time consuming that a quiz, with the AI system providing flexibility.

What seems to be missing from this process is the intelligence to create the answers. For example, I would like to just give the system the rules for the course assessment and have it work out the possible questions and answers. As it is this form of AI seems to making the type of promises fourth generation programming language, did last century. Claims were made about programmers not being used, as it was easy for anyone to do. The reality was that it was easy for anyone to write a simple program, but it required a highly skilled expert for complex tasks. The result was a lot of time wasted by unskilled people writing bad code, and experts trying to untangle the resulting mess.

ps: The workshop was held in a UoW  lab. This had about 32 desktop PCs in four long rows of eight. There is a large display screen near the instructor's console at one end and on the opposite end. However, it would be useful f there were screens on the other two walls. The room is carpeted, but was still a bit noisy, with multiple tutors helping students. Perhaps the ceilings should be sound absorbing in these type of rooms(at the cost f requiring a microphone for the presenter).

Sunday, December 2, 2018

How is a Reflective Portfolio Assessed?

This is the third of a series of posts on how to provide students with help when preparing a reflective portfolio. This is specifically for students of  ANU Tech Launcher and the ANU Computer Science Internship. Previously 
I looked at what students are expected to learn through a reflective portfolio. But how is it assessed?




The details of the Work Portfolio Package (WPP) are available on the ANU TechLauncher website. The WPP is in the form of an application for a real position, selected by the student. This can be for employment, study, or similar.

The student prepares a document of about 8 pages:
  1. Cover letter (1 page)
  2. Statement addressing the selection criteria (200-250 words per criterion)
  3. CV (2 pages)
  4. Supplementary material, such as examples of work (2 pages).
The marking criteria are:
  1. evidence of decision-making
  2. maturity of reflection
  3. professional tone
  4. evidence of life-long learning
  5. acting on feedback
  6. professional attitude
 These can be placed under the previously developed headings:

Learning:
  • maturity of reflection
  • evidence of life-long learning 
Communication: 
  • professional tone
  • professional attitude 
Management:
  • evidence of decision-making
  • acting on feedback
Some of these have proven difficult for the students to understand. As an example, the concept of "reflection" on learning is a difficult one (which I still struggle with, despite some training). Also "life-long learning" difficult for a young student, yet to start their career, to discuss.