Monday, January 21, 2013

Cool research: happy sheets hopeless, training failed…..failure-led, spaced practice worked

This astonishing piece of research by Daniel Bilton & Charles Gluck sums up almost everything I believe about good and bad training.
Over nine months, 500 people in Booz Allen were initially given three types of training:
1.       Placebo
2.       Page-turning
3.       Interactive
All three groups were then given surprise:
Three simulated phishing emails with remedial help if they failed i.e. spaced practice, learn through failure exercises.
Results
Both trained groups seemed to know what to do if they received ‘phishing’ emails:
·         87.8% of static trained
·         95.6% of interactively trained
Happy sheet evaluations for both were through the roof.
However….
Based on actual simulated attacks, they discovered no significant difference between training and no training!
WTF!
They then implemented ‘Failure-Triggered Training’, like the “Secret Shoppers” used by the retailers. Phishing emails simply dropped into your in-box, three different phishing emails on spaced intervals. Each user’s response/action was tracked. What they found was that these simulated emails resulted in a huge learning effect as error rates plummeted. The authors concluded that two main factors were at work:
1. Spaced practice
The researchers attributes their success to the spaced practice approach where the simulated emails put them to the test and with each of the three iterations more and more trainees became competent at dealing with these dangerous, fiscal requests.
2. Learn by doing
The researchers also saw gains from learning at the point of realisation, by doing something relevant at that moment, not on some disassociated training course. It was the failure-triggered training, delivered on teh back of unannounced, blind exercises, combined with immediate tailored remedial training, provided only to the users that “fail” the exercises, that did the trick.
Conclusion
So happy sheets were hopeless, straight text based and interactive e-learning was not significantly better than a placebo BUT spaced practice delivered as learn by doing worked magnificently well. This testimony from a learner about sums it up:
I learned about the CIRT team through the phishing training email sent out a couple months back. It really stuck with me, since I ‘failed the test.’ 

 Subscribe to RSS

Sunday, January 20, 2013

Krug – don’t make online learners think (about structure & navigation)

Steve Krug has had a huge influence on web design through his best-selling book Don’t Make Me Think. He asks a simple question, ‘How do we really use the web?’ We glance, scan and muddle through. We don’t read pages, we scan them, choose the first reasonable option, and because we’re lazy, we meander through content. This is important and, if excesses in design are to be avoided, it has to be understood when designing web sites and online learning.
It was a welcome brake on the excesses of text-heavy, over designed, poorly navigable websites. His theory is based on real practice and positive results on real web sites. Krug’s first law of usability is to strive to make things self-evident or self-explanatory, hence the title ‘Don’t Make Me Think’.
Design Options
Sensitive to the needs of the internet as a medium in itself, he emphasises the importance of the Home page. This leads to reflection on the importance of the ‘Big Picture’, namely the essential purpose of the site or online learning programme. He loves tag lines that capture the essence of a site or web experience. Mission statements he hates, as they rarely tell you the real story and usually miss the Big Picture. He also hates badly designed rollovers, poorly designed pull down menus, unnecessary banner ads and the over promotion of other sites. Krug hates unnecessary noise. In online learning this can be translated into overlong animations at the start, boring learning objectives, merely illustrative graphics and animation, along with wallpaper video.
Structure and navigation
Taking his lead from newspapers, always an interesting source for screen design, he recommends carefully designed hierarchies. He hates navigation that breaks down when you get past the second level. The solution, he thinks, is persistent global navigation at the same position on every page with a home button and tracking. He loves fixed menus. He also makes the useful distinction between navigation, utilities (print, search and so on) and content. It is always a payoff between ‘wide and deep’ hierarchies.
Be conventional
Following on from Norman and Nielsen, he stresses conventions. Don’t play fast and loose, make things easy and consistent. Use conventions, such as shopping carts, standard video controls and icons. This is sound advice. Conventions are more than just objects of convenience, they are part of the grammar of interface design. Designers often refuse to use conventions as they crave creativity and innovation – this, in his view, is rarely useful. Pages should also be broken up into carefully defined areas, clickable areas should be obvious and every attempt made to minimise ‘noise’, again a Mayer and Clark principle in online learning.
Half the number of words and half again
True to his belief that screen readers are different from readers of print, he has strong view on writing for the screen. Less is more and so he exhorts designers and writers to omit needless words. In his own words, “Half the number of words and half again”. Mayer and Clark showed that this is especially applicable to online learning as it leads to significant gains in retention.
Usability testing
Krug, like Norman and Nielsen is a strong believer in usability testing. Following Nielsen and Landauer he takes the view that a few good, experienced testers and a few iterations are all you need. Forget the large-scale focus groups and massive testing, which suffer from the law of diminishing returns. His practical experience shows that just one, or a few testers early on are more effective than a large number at the end.
He recommends evidence gathering with a camcorder and facilitator who asks questions and gives tasks, especially ‘Get it’ tasks where you probe the user for their understanding of the point of the experience, how it works and how it is organised. The point of the facilitator is to probe and ask them not only what they’re looking at but what they’re thinking. Listen, keep an open mind and take lots of notes.
An underlying point, made many years before by Dewey and Heidegger is that technologies work best when they hide themselves in things and tasks. Technology is at its best when it is invisible. This is the consistent theme in all good usability theorists and practitioners. The task of the designer, to make the delivery mechanism as invisible as possible.
Krug understands the different roles of specialists in design teams and the tensions that arise between them. His solution is to objectify the debate through testing, not with the mythical average user, but with real users. His is a useful, practical and prescriptive approach to good usability through good design.
Rocket surgery made easy
His second book Rocket Surgery made easy, shows how to do modest, low budget testin. His starting point is that designers can’t see the bloomers as they get too close to the design and as the navigation has come from their own heads, they lack objectivity. You need other fingers and eyeballs but guided by experts, using voiced testimonies.
I started and ran a successful online learning test company for many years and couldn’t agree more. For technical testing, content testing, proof reading and functionality testing, you may need professional services. Krug recognises this but still recommends giving your work a good going over by some real users, under the eagle eye of people who know what to take from their voiced evidence.
Conclusion
Krug’s prescriptions are even more important in online learning than in web design, as learning’s great enemy is cognitive overload and dissonance. If learners have to work hard to understand, navigate and read online learning, they have less sustained attention for retentive learning. Most online learning, like most offline learning, is too long winded and needs to be seriously edited to avoid cognitive overload. Keep navigation simple and consistent, use de facto conventions, avoid deep hierarchies and write for the screen not the page. And don’t forget to test – a few iterations with experts.
Bibliography
Krug S. (2001) Don’t Make me Think

 Subscribe to RSS

Thursday, January 17, 2013

Mayer & Clark – 10 brilliant design rules for e-learning

Richard Mayer and Ruth Clark are among the foremost researchers in the empirical testing of media and media mix hypotheses in online learning. Their e-Learning and the Science of Instruction (2003) covers seven design principles; multimedia, contiguity, modality, redundancy, coherence, personalisation, and practice opportunities. Clear explanations are given about the risks of ignoring these principles - with support from worked examples and case study challenges. It should be a compulsory text for online learning designers.
Media mix is not mind rich
Their precise studies have confirmed that our media mix (text, graphics, audio, animation, video) in online learning is often flawed, resulting in cognitive overload and dissonance. Perhaps their greatest contribution has been in identifying redundancy as a serious problem in screen-based learning but they are known for research that produces clear practical recommendations that do not pander to those who think that media rich automatically means mind rich.
Less is more
If you were asked to sum up the psychology of learning in three words, it would be ‘less is more’, that’s also Mayer and Clark’s mantra. In one study, Mayer, et al (1996) presented 600 pieces of scientific learning and found that briefer versions, which were concise, coherent and co-ordinated, resulted in far more effective learning. They are precise in their recommendations, ‘There is a clear pattern in which the more words added to the core verbal explanation, the more poorly the student does in producing the core explanative idea units. These results are consistent with the idea that the additional words overload verbal working memory, drawing limited attentional and comprehension resources away from the core verbal explanation.’ The lesson with text is to cut it ‘til it bleeds! Bullet points, simple writing, highlighted keywords and short paragraphs are all useful screen writing techniques.
Avoid eye candy
They are critical of gratuitous graphics which are added to simply fill slots on pages of text. This is not uncommon in e-learning where designers simply take a noun within the text and slam in an associated image. This does nothing, according to Mayer and Clark, than add cognitive load and slow up learning.
Avoid ear candy
Background music and environmental sounds create unnecessary cognitive load and distract from, rather than increase, learning. Indeed, music, over longer periods of time can be incredibly annoying. Note that this also applies to sounds, such as beeps or applause, that reinforce right and wrong answers. This may be appropriate in a games, but not for most online learning. Ear candy is as bad as eye candy.
Text and graphics good
They argue that ‘text and simple relevant graphics’ can improve learning as they use separate cognitive channels. They are not absolutist on these rules, as text within graphics can be useful when explaining a process or in labeling.
Beware of text and animation
Text and animation’ which both use the visual channel, cause cognitive dissonance and often confuse rather than achieve learning. Animation, like video, should use audio narration, rather than accompanying text.
Beware of text and audio
They claim that words in both text and accompanying audio narration can hurt learning. This is interesting as it is often assumed that one needs both to cover accessibility issues. In other words, they argue for using ‘audio and graphics’ without screen text. According to Clark and Mayer (2003), ‘audio or text on their own’ are better than ‘text and audio together’. This is confirmed by another study by Kalyuga, Chandler and Sweller (1999) where the group with audio scored 64% better than the group with both text and audio. They claim that one or other is redundant and will overload the visual and aural channels.
Redundancy
A review of studies around this concept, known as the redundancy effect, by Sweller et al (1998) cites a list of research studies that all point to the damage done to learning when redundant material interferes with the efficacy of the learning. For example; they illustrate a point about leaving out extraneous or distracting graphics in media with an experiment, conducted by Harp and Mayer (1997), in which students were given a text to read on lightning strikes. Students who read the passage accompanied by elaborate colour photos with additional captions - as opposed to the text with simple graphics - showed 73% less retention of knowledge and 52% fewer solutions on a transfer test.
Keep it close
Mayer (1989), Mayer Steinhoff Bower (1995) and Moreno and Mayer (1999) in five separate studies compared graphics with text close to the graphics, and graphics with text below the graphics, at the foot of the screen. In all five studies, learners who used the co-located text and graphics improved their problem solving by between 43-89%. Similar results have been found by Chandler and Sweller (1991), Pass and Van Merrienboer, (1994). Making the learner’s eye jump from one part of the screen to another is disruptive and reduces the effectiveness of the learning. E-learning has also introduced heavy doses of rollover text which is displaced away from the item over which the cursor rolls so that the pop-up text appears elsewhere on the screen at a distance from the item in question. The research confirms that this is to be avoided in learning programmes.
Personalise
Backed up by the work of Nass and Reeves at Stanford (subject of my next post), they recommend a more conversational style, using first and second person language. This is not to say that it should be over-friendly or condescending. It should feel like a dialogue, not a lecture. They also recommend the use of an on-screen coach or agent. Note that they absolutely recommend self-paced user control, as well as frequent practice and context setting through interactions.
Lessons for production
Their research explains why broadcast TV and web design companies often fail to produce good online learning. They are drawn to techniques that entertain rather than educate, often adding media that unintentionally degrades the learning experience. This is why Nielsen and others were so critical of Flash, as it encouraged, the unnecessary addition of animation. On the other hand it confirms the use of short video lessons, with images and audio, as a form of instruction.
Conclusion
Clark and Mayer were among the first to seriously research the use of media in e-learning and have come up with empirically tested conclusions, often repeated by others, which suggest that many common practices in e-learning design are, in fact, wrong. They actually result in harming rather than helping the learning process. They call for simpler, less gimmicky use of media. Animation and audio do NOT necessarily lead to better learning and may, in fact, degrade the learning experience.
Bibliography
Clark, Ruth and Chopeta Lyons (2004).Graphics for Learning: Proven Guidelines for Planning, Designing and Evaluating Visuals in Training Materials. Jossey-Bass Pfeiffer
Mayer R E and Clark R, E-learning and the Science of Instruction (see p61 for multiple references), Pfeiffer, 2003
Richard Mayer (2001). Multi-Media Learning. Cambridge University Press
Clark, Ruth (1999). Developing Technical Training: A Structured Approach for Developing Classroom and Computer-based Instructional Materials. ISPI
Mayer R E, Systematic Thinking Fostered by Illustrations in Scientific Text. Journal of Educational Psychology, 81, 240-246.  1989
Mayer R E, and Gallini J K. When Is An Illustration Worth a Thousand Words? Journal of Educational Psychology, 88, 64-73. 1990
Mayer R E  and Anderson RB. Animations Need Narrations. Journal of Educational Psychology, 90, 312-320. 1991
Mayer R E  and Anderson RB. The Instructive Animation: Helping Students Build Connections Between Words and Pictures. Journal of Educational Psychology, 90, 312-320. 1992

 Subscribe to RSS

Wednesday, January 16, 2013

MOOCs: ‘dropout’ a category mistake, look at ‘uptake’?

I'm not a dropout, I'm a dropin!
I've dropped in to a number of MOOCs recently. Some I've liked and persevered with, others I've had my fill after a short-time, for most, life is too short and \i don't have the time, yet others have been awful, too slow and ponderous for words. But it's all good, that's what I expected. But I resent being universally classed as a 'dropout' and used as an excuse to dis' MOOCs.
Category mistake
Is it inappropriate to take the word ‘dropout’ from one context and stamp it upon another? With MOOCs I’d call it a category mistake, when a word is used to mean one thing (pejoratively) in the context of a long school, college or University course, then applied with the same pejorative force to a very different type of learning experience. Stopping during a MOOC is very different from dropping out school, an expensive long-term degree or a compulsory compliance course.
Dropout?
Lots of people dropout from MOOCS, so what? Lots of people stop doing lots of things.
Lot’s of people don’t finish books but we don’t see this as a sign of intellectual failure. In fact, it can be a sign of efficient learning and research. Read Pierre Bayard’s brilliant ‘How to talk about Books youhaven’t read’ to see how prevalent and positive this can be.
Lots of students don’t attend lectures. Imagine running a restaurant where all the meals have been paid for in advance, yet huge numbers of customers don’t turn up – that’s not unusual in Universities. Yet is there a demand to take a register in all lectures to collect the most basic form of teaching data? NO!
Lots of students attend lectures but drop out in terms of attention. In fact they nearly all do, as it’s a predictable function of the 1 hour lecture, whose length is based upon the fact that the Babylonians had a sexidecimal number system, not the psychology of learning. Yet we don’t demand any checks during lectures on psychological attention or insist on more action learning.
Lot’s of people drop out of college because the course, institution, teaching method, boredom, other opportunities, debt or academics are not for them. In fact, as any lecturer will tell you, colleges are full of people doing courses for reasons that have little to do with genuine interest in the subject. In fact, many degree courses simply lock in students to long-term 3/4 year courses.
MOOCs not failure factories
MOOCs must not be seen as failure factories. They must rise above the education models that filter and weed out learners through failure. Good MOOCs will allow you to truly go at your own pace, to stop and start, go off on an exploratory path and return again. This is what true adult learning is and should be. I always drop out of learning experiences as I never go on formal courses. I decide when I’ve had enough. They should not copy but complement or construct new models of learning.
Uptake not dropout
MOOCs encourage the ‘look see’ approach to learning, and as they are free or very cheap, the consequences are negligible. Do the people who don’t finish a MOOC rush back to college or Universities with cheques in their hand? Of course not. The decision to take or drop out of a MOOC is not a life changing decision in terms of money, time or commitment. Many MOOCs are, in fact, VOOCs (Vocational Open Onine Courses) where the aim is upskilling, not academic progress.
We need to look at uptake, not dropout. It’s astonishing that MOOCs exist at all, never mind the millions, and shortly many millions, who have given them a go. Dropout is a highly pejorative term that comes from ‘schooling’. The ‘high school dropout’. He’s ‘dropped out of ‘University’. It's this pathological view of education that has got us into this mess in the first place. MOOCs are NOT school, they eschew the lecture hall and are more about learning than teaching. MOOCs, like BOOKs, need to be seen as widely available opportunities, not compulsory attendance schooling. They need to be encouraged, not disparaged.
Examples
Take Sebastian Thrun’s famous AI course. He was teaching 200 students at 30k a year, suddenly he had 160,00o students who paid zilch. The fact that his own internal students opted for the online course, 26,000 students finished and that the top 400 students were all external and online is astonishing. Think on this. If we forget dropout and focus on the true comparison, 200 versus 160,000 means that it would take 800 years using traditional methods. Even with the 26,000, that’s 130 years!
The data for Duke’s first MOOC has come out, and at first it looks depressing but they don’t think so. Bioelectricity (Coursera) registered 12,461 (from 110 countries) with 7593 watching at least one video. 313 completed certification (basic+distinction) with 260 (distinctions). The real story here is that the number of students who completed the course is over ten times the campus enrollment.
PS
Note that I’m not saying that uptake is not an issue. It is in terms of investment and growth. The monetisation of MOOCs is important in terms of their sustainability but the monetisation models are evolving quickly to include recruitment, advertising, delivery fees and low cost certification.

 Subscribe to RSS

Sunday, January 13, 2013

Student cliff – 7 reasons for plummeting student numbers


Lots of angst has appeared around what is now being dubbed the ‘student cliff’. The Vice-Chancellor of the University of Worcester says there will be 10% drop in the proportion of UK students starting degrees this year, the steepest fall in 30 years. To be precise, he predicts a 70,000 drop in applications. We have already seen a 53,000 student shortfall. Then there's the catastrophic statistics on foreign students. UK universities get a third of their tuition fees from foreign students, yet Indian student numbers fell by 23.5% overall with a 28% drop in postgrads and Pakistani students by 13.4%, with a 19% drop in postgrads. Non-EU students coming to Britain for postgraduate courses has dropped for the first time in 16 years. The UK is now relying on Chinese students for growth but even these have dwindled and there’s every sign that China is doing everything it can to build their own capability. For the first time, no one is disagreeing about the falls. What we need to understand are the causes:
Changing demographics
Demographically, the number of UK 18-year-olds will decline over the next ten years by 11%. This will have a long-term effect on University applications. Projections released this week from the US Department of Education show enrolment figures for 2010-2021 falling miles below the 46% growth experienced between 1996-2010. This is already playing out in California, where there is a demographic shift to negative growth, combined with crippling state debt. Overall, a new report has shown that the projected ‘pupil cliff’ will result in the “death of the growth agenda in the US”.
Commoditisation
Scarcity creates value, commoditisation destroys value. Youth and graduate unemployment has gone through the roof in some countries and is still rising across Europe. The spectre of a high-cost degree with a low-salary future is starting to bite. A degree is no longer the goal but THE degree from better brand institutions. Because degrees have become commoditised, employers are also less interested in their value. Music, newspapers, retail in general, have all been commoditised through Napsterisation. The same thing is happening in learning.
Steeply rising costs
We have seen student costs soar in the US, UK and elsewhere, way beyond inflation and house price rises, yet the deliverable remains much the same. Remind you of the property bubble? Student loan costs have risen well above that of credit card debt in the US. When faced with student fees in the UK, many have chosen not to apply. Yet little has been done to lower the cost of HE which is still rooted in a high cost model based on low occupancy buildings, one intake a year, long vacations and inefficient teaching. Cost is pushing more young people to reject HE.
Debt
‘Debt’ is a dagger of a word that now strikes fear into people. It alone almost led to a global meltdown, is tearing apart the European Union, has led to massive youth unemployment and is NOT going away any time soon. To take on debt now, is to take on a massive risk. It will affect your ability to buy a flat or house, have children, sustain a credit rating, with no guarantee of a job with golden prospects. Potential students have wised up to this fact.
Relevance
The medieval model of the University as somewhere that provides a huge breadth of courses, with a focus on research rather than teaching, has led to parking the relevance argument.  We have seen dramatic drops in students applying for Universities in the UK, when they have been asked to pay £9000 a year for that privilege. Foreign student income (one third of all tuition in UK) is not geared towards this breadth but towards business and STEM subjects. Economic relevance is not the only aim of higher education but neither is the abandonment of relevance. We have to face up to the fact that relevance has become a greater factor in student choice,
Groupthink
Universities struggle with rapid, innovative change and governments are still stuck in the mindset of more degrees as an intrinsic good. Peter Thiel identified this as one of the fundamental symptoms of a bubble – groupthink. ‘The nth degree’ problem is the simplistic idea that the more degrees we fund the better. Forget the fact that the world has been brought to its fiscal knees by graduate bankers or that many of the skills we require are not taught, or taught badly, at our Universities, we need to reduce costs through the mass adoption of cheaper solutions, such as online learning. This, especially in the UK, has hindered innovation.
More options
The race is now on. Different models are emerging that lower costs and increase reach and access. This draws students away from traditional HE. MOOCs, with accreditation (Signature Course from Coursera), are now available. Other models are emerging, such as separate online departments within Universities (UDOL – University of Derby Online), outsourcing to online delivery companies that have multiple yearly intakes, low costs and no VISA problems (Interactive Design Institute). These are, at present, modifications to the existing model. But there are other more radical, tectonic shifts at work here. There’s a genuine thirst for shorter, faster courses, that are available when you want them, more relevant apprenticeships and high quality workplace learning, and not just the 18 year old undergraduate meander through a 3 or 4 year degree course.
Conclusion
Unlike the fiscal cliff, there is no sign of any immediate solution to this problem, other than taking the pain. There’s no way politicians can do a 180 degree (sic) turn on this but that’s what’s needed. After decades of expansion, the whole system has ballooned out of control with quality, and now quantity, falling. The danger is in behaving like lemmings heading towards the student cliff without adequate planning.

 Subscribe to RSS