Saturday, September 19, 2015

10 uncomfortable truths about the con that is University rankings

Universities claim to be above the lowly business of commerce but still willingly contribute to the petty hit parades in the University league table season. It demeans the sector. They search for whatever scraps they can find by selecting data from one ranking table or another. Why? For money. They love to claim they are above the competitive, capitalist, corporate game but they are by far the worst when it comes to the dog-eat-dog, institutional competition that are the rankings.
Nozick wrote an incisive essay called "Why do Intellectuals hate Capitalism'. It explains the causality behind the duplicitous thinking of academic distaste for commerce but the utter love of commerce when it suits their institutions. Worst of all, for the people that pay, whether its taxpayers, parents, national or international students, the University Rankings are largely a con.
1. Bait and switch
The sector loves to take the high moral ground on keeping managerialism out of education, then use the slimiest form of managerial marketing, ranking tables, to promote their wares. Aimed firmly at parents and students, they bait and switch. The hook is baited with data on research and facilities, then the message switched to make it look like the teaching experience you’ll pay for, when in fact, the rankings are about measures that have little to do with teaching. That is a classic 'bait and switch' con.
2. Teaching ignored
They may SAY they take teaching into account but they don’t. They often claim to have ‘measures’ on teaching, but actually draw their data from proxies, such as staff qualifications and research activity and use nothing but indirect measures to measure teaching. The Times rankings are a case in point. They claim that their ranking scores include teaching. In fact, only 30% is based on teaching but they use NO direct metrics. The proxies include student/staff ratios (which is skewed by how much research is done) and, even more absurdly, the ratio of PhDs to BAs. It is therefore, a self-fulfilling table, where the elite Universities are bound to rise to the top. There is little direct measurement of face-to face time, lecture attendance or student satisfaction. In some cases it’s laughable, as Malcolm Gladwell pointed out, with Faculty salary, levels of degree in Faculty and proportion of faculty who are full time, being taken as proxies for quality of teaching. It’s like having a Premier League table based on the performance of the backroom staff and not the real games and players.
3. False precision
Up one place in the rankings – yippee! Down two places – time to worry. Yet the idea that these rankings are in any way precise is silly. They’re a mish-mash of misleading data, under vague (even misleading) categories and often watered with a heavy dose of opinion (expert panels drwn from top Universities). In any case, they’re always changing the criteria for ranking, so year-on-year comparisons are useless. This shows itself in the huge disparities between the different ranking systems. The LSE is 3rd in The Sunday Times rankings but 328th in the US News & World Report Rankings, 71st in the QS Rankings and 34th in the THE Rankings). Other universities like Manchester and KCL do badly in British rankings but well in international tables. This gives ample room for cherry picking but is poof enough that the way the rankings are calculated is seriously flawed. If the rankings were research they'd be rejected by even the lowliest of Journals.
4. Apples and oranges
They don’t compare like with like. In Edinburgh, where I come from, we have four Universities; Napier, Heriot-Watt, Edinburgh and Queen Margaret. You couldn’t get four more diverse institutions in terms of what they teach and their history. In 2012 Edinburgh were in top five for research but came stone-cold last in the teaching survey. That same year, Heriot Watt came top in Scotland and 4th in UK on Student experience but way, way down in the rankings. In that same year, more than a third of the Russell Group Universities found themselves in the bottom 40 of 125 institutions (2012) on teaching. These comparisons are truly odious.
5. Skews spending
What is sad, even morally wrong, is they they really do influence strategy and spending. Ranking status is often stated explicitly in their goals. In effect, as teaching doesn’t really get measured, except through false proxies, it leads to spending on everything but good teaching – physical facilities, research and so on. This direct causal effect on behaviour also leads to overspending, as it’s a runaway train, where everyone tries to outdo everyone else. There is no incentive to save money and become more efficient, only to spend more. Weirdly, there’s rarely any accounting for students costs in calculating the rankings. Shouldn’t a University that costs a lot less get ranked higher than one that does not? It would appear that prejudice trumps economics. This is a topsy-turvy world, where being more expensive is an intrinsic good.
6. Gaming the system
It’s not just spending that’s skewed by rankings, they also skew behaviour and priorities. Universities are far from being free from the rat race, they just have some very smart rats. In practice, this means that they are good at gaming the system. What are the criteria and weightings for ranking? OK, those are this year’s targets. More facilities, let’s get them built.
7. Self-fulfilling prophecy
The more you spend, the higher your ranking. So the rich get richer, the poor get poorer. The separation, in terms of research grants between the handful at the top and the rest is huge. Naturally, this leads to a separation of the so-called cream from the so-called milk. In that sense it’s a deterministic system, where the top remain at the top and the rest scrabble around for the scraps.
8. Agendas
What’s more, the different tables often have uncomfortable relationships with newspapers. And let’s not imagine that, given the nature of newspaper ownership in this country, they don’t have agendas. The Complete University Guide has had relationships with The Telegraph, Times and Independent. They keep falling out. The Sunday Times has its Good University Guide. The Guardian has yet another. These tables sell newspapers to middle class parents, that’s the real driver.
10. Old boys club
Reputation scores feature in lots of the rankings. You go out and ask people what they think; academics, publishers, employers etc. Of course, given that most of the people asked are from the highly ranked Universities, there’s an obvious  skew in the data. That's shameful, qualitative nonsense.
10. Status anxiety
What is their real effect on parents and students? Nothing but an irrational race. They induce ridiculous amounts of status anxiety. Parents and kids are being encouraged to play a game which is already gamed and get stressed over data that encourages distasteful behaviour.

I haven’t even begun to tackle the issue of cheating, being economical with the truth or fiddling around with the submissions. There are examples of straight up cheating, and as there’s no real quality control, it’s likely to be far more common than reported. In truth, no one really knows what the ideal criteria for ranking should be, as it’s a set of competing ideological choices – accessibility, teaching, research, graduation rates? And with what weightings? That’s why the different rankings have these huge disparities. We need, like Reed University in the US, to refuse to hand in the assessments. If the game is being gamed, don’t play the game.

 Subscribe to RSS

Friday, September 18, 2015

10 x 10 lists on common mistakes in online learning

To ten tips in top ten topics in online learning:
10 ways to make badass INTROs in online learning 
10 bloody good reasons for using much-maligned TEXT in online learning 
10 essential online learning WRITING TIPS in online learning 
10 stupid mistakes in design of MULTIPLE CHOICE questions
10 essential points on use of (recall not recognition) OPEN RESPONSE questions
10 rules on how to create great GRAPHICS in online learning 
10 sound pieces of advice on use of AUDIO in onlinelearning 
10 ways based on research to use VIDEO in online learning
10 ideas on use of much maligned TALKING HEAD videos in online learning

This started with a simple observation that I'm seeing, over and over again, the same mistakes being make on screen, with online learning. I hope you find them useful.

 Subscribe to RSS

Monday, September 14, 2015

10 essential rules on the use of ANIMATION in online learning

Animation, don’t we just love it – The Simpsons, Family Guy, Southpark, The Lion King, Wallace and Grommet. Unfortunately in many online learning programmes, I’ve learned to loathe it, not because it’s bad but because it’s often inappropriate and sometimes unnecessary.
By animation, I mean graphics that move. This ranges from full character animation, to crude figure animation, lip-synched but static faces, objects that move, lines that are drawn upon the screen, builds and animated transitions.
1. Be wary of extraneous animation
In a now famous experiment, where one group who only saw a relevant three minute animation of a lighting strike was compared to a group who viewed the same animation of a lightning strike with six additional, ten second animation clips, of bending trees, lightning striking into the trees, an ambulance arriving and a victim being stretched off with a crowd of onlookers. The first group outperformed the second by coming up with 30% more solutions, Mayer, Heiser and Lonn (2001). You see a lot of this, as animators go into movie mode and extend simple, meaningful, explanatory, instructional animation into complex, extraneous, non-explanatory moving wallpaper.
2. Rinse the animation
In a car brake animation, Mayer found that stripping out some of the complex concepts and getting these over, before the animation was shown, resulted in an increased efficacy of the animation. The lesson here, is to examine the animation script and rinse out things that you think could be better explained using text and static graphics. This also makes budgetary sense.

3. Avoid animating text
Text is meant to be read not admired for its dance moves. Whenever I see a tile or text flip, rotate or bend without good reason, I’m annoyed. Maybe, just maybe, if it’s meaningful. I once animated  the word University to make it turn upside down, as I was talking about the ‘Flipped University’. That’s fair.
4. Do not use text and animation together
‘Text and animation’ which both use the visual channel, cause cognitive dissonance and often confuse rather than achieve learning. Animation, like video, should use audio narration, rather than accompanying text, Moreno & Mayer (1999).
5. Do not separate animations from explanatory audio
In an experiment where the workings of a bicycle tyre were explained, to a group with an integrated animation and audio and another group with separate ( one after the other) animation and audio, when tested for transfer, the first group generated 50% more solutions. This large effect was repeated over eight different experiments, Mayer & Anderson (1991) (1992), Mayer & Sims (1994), Mayer et al (1999).
6. Animate for flow
For processes, procedures and ordered steps, animation can be used to build and explain the flow. In particular, animated invisible processes, such as fluid (blood flow, hydraulics etc.) and air-flow can be used to good effect. Flow charts, mindmaps and other complex diagrams can benefit from meaningful and useful animation. The 2D and 3D benefits of animation can do what text and audio can never do, show things they can never show.
7. Graphics and diagrams
Simple wipes that show lines on a line graph appear as if being drawn from left to right, or histogram bars being wiped on, can be explanatory and therefore useful. Indeed, animation can be used to bring data to life, with just a few simple moves.
8. Avoid animation in navigation
Animated icons, menus that have things rotating, spinning logos – you name it, I’ve seen it. There are things that just need to be stopped, superfluous distractions that damage learning. Don’t turn your learning programme into a fruit machine – unless it’s meant to be a fruit machine. Just don’t do it.
9. Lipsynched agents
The jury is out on animated tutors and agents but crudely animated agents or crudely animated lipsynch is a no, no.
10. Don’t get seduced
There is a bottom line here – don’t use animation until you have exhausted all other possibilities. It’s expensive, can be distracting and is difficult to change and update. Don’t get seduced by animation, move on.
Once again, rules are never absolute. There are times, in children’s learning, for people with specific learning difficulties and so on, where these rules should be broken. But there’s nothing like animation for producing cognitive overload which inhibits learning. Note also that I’m not referring to animated graphics as used in high level games and virtual reality.
More in the series....
10 ways to make badass INTROs in online learning 
10 bloody good reasons for using much-maligned TEXT in online learning 
10 essential online learning WRITING TIPS in online learning 
10 stupid mistakes in design of MULTIPLE CHOICE questions
10 essential points on use of (recall not recognition) OPEN RESPONSE questions
10 rules on how to create great GRAPHICS in online learning 
10 sound pieces of advice on use of AUDIO in onlinelearning 
10 ways based on research to use VIDEO in online learning


 Subscribe to RSS