Being an air traffic controller is hard. Really hard. The job entails having to remember vast amounts of fluid information often within a context of enormous pressure. Essentially the job is about ‘situational awareness’ which involves “the continuous extraction of environmental information, the integration of this information with prior knowledge to form a coherent understanding of the present situation.” The job is sometimes done under extreme duress, where they have to make life or death decisions often with a lack of sleep leading in some cases to long-term fatigue and burnout. So stressful is the job that they are eligible for retirement at age 50 or after 25 years of service.
In the 1960s, a series of interesting experiments was done on air traffic controllers. Researchers wanted to explore if they had a general enhanced ability to “keep track of a number of things at once”  and whether that skill could be applied to other situations. After observing their sophisticated abilities in air traffic control, they then gave them a set of generic memory based tasks with shapes and colours. The extraordinary thing was that when tested on these skills outside their own area of expertise the air traffic controllers did no better than anyone else.
These findings challenged contemporary thinking on generic skills. Surely they had developed a set of general cognitive capacities that could be used in other areas or ‘domains’? The evidence suggested the opposite. In order to be good in a specific domain you need to know a lot about that specific domain and moreover, “the more complex the domain, the more important is domain-specific knowledge.” This phenomenon is now well established and has been replicated many times. Other research for example has shown that the ability to remember long strings of digits does not transfer to the ability to remember long strings of letters. Indeed, we all know very ‘clever’ people in their professional lives who seem to often make very stupid decisions in their personal lives:
“A person who is able to reason logically in science may show no such ability in his or her personal life or in any areas outside of his or her areas of science. Knowing that we should only test one variable at a time when conducting a scientific experiment is critical. Outside of hypothesis testing, it may be irrelevant, with other knowledge being pre-eminent.” 
Take another example, sport. Within a football team you have many different types of positions or ‘domains’ such as goalkeepers, defenders and attackers. Within those domains you have further categories such as centre backs, full backs, attacking midfielders, holding midfielders and attacking players. Now the ‘general skill’ that all these players have is the ability to play football, however if you put a left back in a striker’s position or put a central midfielder in goal they would be lost.
A footballer’s ability to be effective in a particular position or domain is based on years of experience where they have built up thousands of mental models from playing the game in that particular position so that when they have to perform at a high level they can do so with faster reaction times and their full concentration can go on anticipating the complexities of the game faster than their opponent. Of course there are elements that are consistent with each position such as touch and technical ability but they look very different in each position and are heavily context specific. For example, a central defender heading a ball away to safety is very different to a striker heading a goal and the types of positioning and runs an attacking player needs to make are radically different to those of a defender. In other words, elite footballers are not “good at football” as such, they’re good at being a left back, defensive midfielder or attacker.
Despite the growing body of evidence questioning the efficacy of teaching general skills in recent years, there is still a near constant refrain for them to be prioritised in schools. This usually takes the form of generic “critical thinking skills” often taught in some form for an hour or two a week and decontextualised from any specific subject. This is a problem as Dan Willingham reminds us
Critical thinking (as well as scientific thinking and other domain-based thinking) is not a skill. There is not a set of critical thinking skills that can be acquired and deployed regardless of context.”
Thus, if you remind a student to “look at an issue from multiple perspectives” often enough, he will learn that he ought to do so, but if he doesn’t know much about an issue, he can’t think about it from multiple perspectives. 
Another problematic area is the diaphanous world of “21st century learning skills” which some schools have made a central part of their mission. It’s even been suggested that some of these nebulous skills are now as important as literacy and should be afforded the same status. An example of this is brain training games whose proponents claim can help kids become smarter, more alert, and able to learn faster. However recent research has shown that brain training games are really only good for one thing – getting good a brain training games. The claim that they offer students a general set of critical or problem solving skills was recently debunked by a new study reviewing over 130 papers :
We know of no evidence for broad-based improvement in cognition, academic achievement, professional performance, and/or social competencies that derives from decontextualized practice of cognitive skills devoid of domain-specific content.
Instead of teaching generic critical thinking skills, an alternative strategy would be to focus instead on subject specific critical thinking skills that seek to broaden student’s individual subject knowledge and unlock the unique, intricate mysteries of each subject. This goes for other dispositions and faculties taught generically such as Growth Mindset and Grit – students may well have a Growth Mindset in English but not in Maths, and yet the concept is often portrayed to students as a general capacity that can supposedly function in a transversal way across all subjects.(Despite the fact that the jury is still out on whether these can be taught at all.)
In the same way that teaching knowledge devoid of any platform for students to discuss, explore and develop that knowledge makes no sense, the teaching of standalone, decontextualised general skills is a questionable practice at best. It’s enduring appeal is probably in the fact that the concept seems so intuitively right yet when the evidence is appraised we find their justification weak. To those advocates of the ubiquitous critical thinking skills we might risk the question: “but what are they going to think with?” As Dan Willingham reminds us “thought processes are intertwined with what is being thought about.”
The Role of Memory in Air Traffic Control (Gronlund, Dougherty)
Domain-Specific Knowledge and Why Teaching Generic Skills Does not Work (Tricot, Sweller 2014)
Critical Thinking: Why Is It So Hard to Teach? (Willingham)
Do “Brain-Training” Programs Work? (Simons et al 2016)
1 (Dominquez, 1994)
2 (Yntema & Mueser, 1960)
3 (Ericsson & Charness, 1994)
4 (Tricot, Sweller 2014)
5 (Tricot, Sweller 2014)
6 (Willingham 2007)
In the 1930s endocrinologist Hans Selye differentiated between two types of stress, distress and eustress. We are all familiar with the first term but perhaps less with the second term which refers to a positive response to external stressors leading to a state of optimism, confidence and agency, in other words ‘good stress.’ The origins of this model has its roots in 1908 when psychologists Robert M. Yerkes and John Dillingham Dodson posited that productivity is directly correlated with an optimal state of stress. Too little of it and you get nothing done, too much of it and you get nothing done either.
A key concern of anyone working in education is monitoring the stress levels of staff and students. Of course we don’t want anyone to be in a state of distress but we now live in an age that often views all stress as distress without acknowledging the benefits of eustress. Is it possible to imagine a more ‘stress-tolerant’ culture where students embrace a ‘sweet spot’ or optimal level of stress, one where we could engender a atmosphere of positive challenge and agency? As Ben Martynoga points out:
This is where good teachers and managers should push their charges: to the sweet spot that separates predictable tedium from chaotic overload. Where stress gets more persistent, unmanageable and damaging, Selye calls it “distress”. Eustress and distress have identical biological bases; they are simply found at different points on the same curve.
The key point here is that both of these states are responses to external stressors as opposed to being caused by events themselves, in other words, perception is everything. A key question here is in what way do educators shape the perception that all stress is distress?
Broadly there are two responses to stress, an initial avoidance and then subsequent coping strategies. For a group of Yale researchers, both of these approaches deny the benefits of eustress because they perpetuate the idea that all stress is bad:
These approaches advocate and perpetuate the mindset that stress-is-debilitating, a mindset that not only is partly inaccurate but may also be counter-effective. Even hardiness and resilience approaches to stress, while acknowledging the enhancing outcomes, still ultimately affirm the mindset that the debilitating effects of stress must be managed or avoided.
In contrast to the “stress-is-debilitating” mindset, these researchers discovered that students could be primed to adopt a “stress-is-enhancing” mindset in which they embraced a certain level of stress and which resulted in them being more open to seeking help, more open to feedback, which led to lower levels of distress overall and which had “positive consequences relating to improved health and work performance.” This “stress-is-enhancing” mindset has many resonances with Robert Bjork’s notion of desirable difficulties.
We are all familiar with the”stress-is-debilitating” mindset. When we have open ended large tasks, we are often are on the left of the Yerkes-Dodson curve, with little or no stress and thus no stimulation to act, but when the deadline is looming, we find ourselves often on the right of that curve, in a state of paralysis, unable to act and making poor decisions in an effort to alleviate the distress. Clearly then the ‘sweet spot’ is to be in a state of eustress, characterised by hope, excitement, active engagement, (O Sullivan, 2010) and that feeling that you are in control of the task you are faced with.
While there are some serious external stressors that are debilitating no matter what your response to them, two questions worth asking are:
- Are the kinds of tasks we are asking students to do genuinely placing them in a state of distress or could they be seen more positively as a potential state of eustress?
- Are we focusing on teaching methods that actually increase distress such as a focus on the storing of information as opposed to the retrieval of it?
In education research there is often very little consensus, but one area in which there is almost unanimous agreement is in the testing effect. We now know that the worst thing we can advise students to do in terms of revision is to re-read material and highlight key points, and that the most effective thing we can advise them to do is to practice retrieving information through testing, preferable through self testing, low stakes quizzing and flash cards. This distinction between storage and retrieval processes is well researched as Roediger and Butler explain:
“The testing effect is a robust phenomenon: The basic finding has been replicated over a hundred times and its generalizability is well established.”
So we know that testing is beneficial for learning but yet the general perception of testing seems to be altogether negative. Is the problem not just the high stakes nature of them but also how students are prepared for them? If students are using poor study techniques like re-reading and highlighting material for most of the school year within a curriculum that is not interleaved but focuses on mass practice, is it any wonder that they enter a state of distress when they enter exam season?
Stress experienced early in life can be debilitating and potentially devastating if compounded throughout life. Where children experience prolonged periods of distress they need the proper help and support to enable them to cope and we clearly have some way to go in this area. But are the kinds of tasks that we are asking them to do in schools genuinely creating a state of distress? If stress is a often a question of perception as Selye claimed then to what extent is it helpful to portray testing and exams for example as a key contributor to a “mental health crisis spiralling out of control?”
Stress is a very difficult area because it is highly subjective and often results in emotional and sometimes irrational reactions to it. We all want to create a healthy, productive atmosphere for staff and students in which they feel they have agency over their future and in which they don’t feel overwhelmed by external stressors but by viewing all stress as distress without harnessing the hidden benefits of eustress, we might just be missing a trick.
Diamond DM, et al. (2007). “The Temporal Dynamics Model of Emotional Memory Processing: A Synthesis on the Neurobiological Basis of Stress-Induced Amnesia, Flashbulb and Traumatic Memories, and the Yerkes-Dodson Law”. Neural Plasticity: 33. doi:10.1155/2007/60803. PMID 17641736.–
O’Sullivan, Geraldine (18 July 2010). “The Relationship Between Hope, Eustress, Self-Efficacy, and Life Satisfaction Among Undergraduates”. Social Indicators Research 101 (1): 155–172. doi:10.1007/s11205-010-9662-z.
On the 3rd May 2015, Chelsea won the Premier League title with three games to spare. For manager Jose Mourinho, it was his 21st trophy, marking him out as the most decorated manager in recent club football history. In August he was rewarded with a multi-million pound contract that would see him at the club until 2019. By December he was sacked.
The club had inexplicably nosedived in the new season with reports of “palpable discord” in the dressing room exacerbated by his public admonishment and subsequent ostracism of team doctor and well respected member of the group, Eva Carnerio. One of the major questions that emerged from Mourinho’s “annus horribilis” is how did a group of players who won the league at a canter a matter of months ago, capitulate in such a dramatic fashion?
In direct contrast to this, in 2016 we witnessed possibly the greatest sporting phenomenon in English football history with Leicester City winning the Premier League, a feat made all the more remarkable by the fact that at the same time the previous year they were bottom of the league and fighting for their lives to even stay in it. The difference appears to be their new manager, the genial Claudio Ranieri who has elevated a disparate band of largely unknown players to the pinnacle of football history within less than a year by fostering an indomitable team spirit that has seen each and every player performing well beyond their limits.
Great teachers have much in common with great coaches. They have a vertiginous knowledge of their field with an infectious passion for it, and they can communicate that passion clearly and in ways that inspire. They have an unquestionable authority, the total respect of the players in their charge and crucially, they can engender trust and belief in their team to the extent that they will walk through walls for them. Conversely, if that relationship breaks down, players can be shadows of their former selves; aimless, lacking in confidence and self-belief and playing ‘within themselves.’
Anyone working in education has experienced this same dynamic, either themselves as students through a teacher who ignited a passion for the subject they went on to study, or through witnessing that colleague who unfailingly seems to get the best out of even the most resistant of students. I’m not talking about teachers who are well liked and who allow students to “define their own learning” (often described as a “legend”) but teachers who set high expectations, command genuine respect and trust, who model the kind of behaviour they expect, who have the authority to create a scholarly space that allows kids to really achieve, who have the ability to make students believe they are better than they ever thought they could be, and who can transmit their own obsession for their subject into a lifelong appreciation of it from their students.
Yet this element is rarely mentioned in education research. Possibly because there is no effective way of measuring such a thing and possibly because it is the elephant in the room that many simply don’t want to face. The uncomfortable truth is that without the respect and trust of their students, a teacher will often be ineffective no matter how many interventions they try or how many leadership training courses they are sent on.
One of the many blind spots in education research is that it often doesn’t take into account the context in which an particular approach occurs, which makes comparing ‘like for like’ extremely problematic and which has resulted in widely differing interpretations of what works.
However, despite the fact there is often very little consensus in education research, one area in which there is almost unanimous agreement is in feedback as the most effective agent of learning. For Dylan Wiliam, in order for this approach to function properly, context is all important and a healthy relationship between teacher and pupil is paramount:
“In the end, it all comes down to the relationship between the teacher and the student. To give effective feedback, the teacher needs to know the student—to understand what feedback the student needs right now. And to receive feedback in a meaningful way, the student needs to trust the teacher—to believe that the teacher knows what he or she is talking about and has the student’s best interests at heart. Without this trust, the student is unlikely to invest the time and effort needed to absorb and use the feedback.”
There isn’t a lot of research in this area but one interesting (yet somewhat disturbing) study conducted by Hunter Gehlbach from Harvard in which researchers tried to improve teacher/student relationships by showing areas where they had something in common, has yielded some intriguing results:
“For the experiment he had in mind, Hunter and his team created a survey for students and teachers of a ninth-grade class. The researchers then selectively shared examples from the survey results with teachers and students to show them that they had things in common. When Hunter examined the test scores of students who had been induced to see that they had things in common with their teachers, he found something astonishing: students — especially minorities — suddenly started to perform better in class.”
This study is yet to be replicated and there are many problematic aspects of it, but it does perhaps signal a new avenue of enquiry that moves away from focussing on disembodied education interventions and instead focuses on the context in which those interventions take place. If a teacher doesn’t have authority and the respect of his students, does it matter what approach they take?
It’s not a teacher’s job to be liked or popular, but it is their job to ensure students can achieve their potential and ideally, open their minds up to wonders of Shakespeare, Newtonian Physics or Minoan civilisation. If the strength of the relationship between teacher and pupil is the determining factor in how well students engage with their subject then maybe we need to talk about this rather than focussing on a set of ‘what works’ interventions that no matter how well evidenced, won’t work if the teacher has ‘lost the dressing room.’
An incredibly useful and informative talk on the research around how students learn best from one of the Deans for Impact who are “a group of deans from schools of education around the country, that have united to make sure future teachers are armed with information about what works in the classroom as they begin their careers.”
Podcast from the Harvard Graduate School of Education asking “How do you effectively measure teacher effectiveness?” Interesting conversation on triangulating data to create a broader, more in-depth picture of the impact a teacher is having beyond test scores.
One of the best podcasts around. This episode features stories of adults taking very different approaches to communicating with children with some very funny kids saying what particularly annoys them about how they are spoken to, specifically age appropriate talk. (TAL spawned the outstanding Serial podcast.)
If you work in education and you haven’t read Dan Willingham’s work then you’re doing it wrong. In this podcast, he elegantly dispatches the fallacious notion of learning styles and also looks at multiple intelligences.
Why can’t we replicate so much of the research on psychological studies? An interesting listen for anyone interested in education research and the problems of social science in general.
Intelligent podcast that asks some very controversial, almost taboo questions such as “is the issue with America’s failing education system simply that teachers just aren’t very bright?”
Laurie Taylor talks to Will Davies, Senior Lecturer in Politics at Goldsmiths, University of London, who asks why policy makers have become increasingly focused on measuring happiness. Interesting contribution to the conversation around the emergence of positive psychology in education.
Engaging podcast which essentially explores self delusion. This episode asks what keeps people in bad jobs, poor health, terrible relationships, and awful circumstances despite how easy it might be to escape any one of those scenarios with just one more effort. Useful discussion for anyone dealing with young people and reluctant learners.
Radiolab make unique audio documentaries in the style of This American Life. This episode features Oliver Sacks telling the story of an amnesiac whose love for his wife and music transcend his 7-second memory.
“Historically the philosophy of education has been at the core of the subject. Today there are relatively few philosophers working in this area. Meira Levinson, a philosopher with experience of teaching in US public schools, is one of them. Here she discusses fundamental questions about what we are trying to do when we educate our children.”
The ‘In Our Time’ podcasts are an indispensable resource for navigating an incredibly diverse range of areas of knowledge featuring experts in the field. Melvyn Bragg and guests discuss the history and purpose of education.
NPR podcast on schools in the U.S. using traditional approaches to behaviour management that are challenging for many but yielding impressive results. Contributions from teachers in the frontline applying these methods and education academics who are less than convinced.
Robert McNamara was by any standards, a wildly successful man. Harvard graduate, president of Ford motors then rising to the heights of U.S. Secretary of Defense in the 1960s, McNamara epitomised American élan and brio. But he had one major flaw – he saw the world in numbers.
During the Vietnam War, McNamara employed a strategic method he had successfully used during his days at Ford where he created data points for every element of production and quantified everything in a ruthless fashion to improve efficiency and production. One of the main metrics he used to evaluate progress and inform strategy was body counts. “Things you can count, you ought to count,” claimed McNamara, “loss of life is one.”
The problem with this method was that the Vietnam war was characterised by the unmeasurable chaos of human conflict not the definable production of parts on a factory assembly line. Things spun out of control as McNamara’s statistical method failed to take into account numerous unseen variables and the public turned against US involvement in the war through a cultural outcry that would change the country. Although on paper America was ‘winning’ the war, ultimately they lost it.
As the war became more and more untenable, McNamara had to increasingly justify his methods. Far from providing an objective clarity, his algorithmic approach gave a misleading picture of what was becoming an unfathomably complex situation. In a 1967 speech he said that:
“It is true enough that not every conceivable complex human situation can be fully reduced to the lines on a graph, or to percentage points on a chart, or to figures on a balance sheet, but all reality can be reasoned about. And not to quantify what can be quantified is only to be content with something less than the full range of reason.”
While there is some merit to this approach in certain situations, there is a deeply hubristic arrogance in the reduction of complex human processes to statistics, an aberration which led the sociologist Daniel Yankelovitch coining the term the “McNamara fallacy”:
1. Measure whatever can be easily measured.
2. Disregard that which cannot be measured easily.
3. Presume that which cannot be measured easily is not important.
4. Presume that which cannot be measured easily does not exist.
Sadly, some of these tenets will be recognisable to many of us in education – certainly the first two are consistent with many aspects of standardised testing, inspections and graded lesson observations. This fiscal approach been allowed to embed itself in education with the justification given often to ‘use data to drive up standards.’ What we should be doing is using “standards to drive up data” as Keven Bartle reminds us.
The fallacy is based on the misguided notion that you can improve something by consistently measuring it. In the classroom, this is best illustrated by the conflation between learning and performance, which For Robert Bjork, are two very different things – the former is almost impossible to measure, the latter much simpler. It is very easy to transpose observable performance onto a spreadsheet and so that has become the metric used to measure pupil achievement and concomitantly, teacher performance. In tandem with that you’ve had the hugely problematic grading of lesson observations on a linear scale against often erroneous criteria as Greg Ashman has written about here.
Two years after the Vietnam war ended, Douglas Kinnard, published a significant study called The War Managers in which almost every US general interviewed said that the metric of body counts were a totally misguided way of measuring progress. One noted that they were “grossly exaggerated by many units primarily because of the incredible interest shown by people like McNamara.”
In education, the ‘incredible interest’ of the few over the many is having a disastrous impact in many areas. One inevitable endpoint of a system that audits itself in terms of numbers and then makes high-stakes decisions based on that narrow measurement is the wilful manipulation of those numbers. A culture that sees pupils as numbers and reduces the complex relational process of teaching to data points on a spreadsheet will ultimately become untethered from the moral and ethical principles that are at the heart of the profession, as the recent Atlanta cheating scandal suggests.
Even in the field of education research, there is a dangerous view in some quarters that the only game in town is a randomised controlled trial (its inherent problems have been flagged up by people like Dylan Wiliam.) If the only ‘evidence’ in evidence based practice is that which can be measured through this dollars and cents approach then we are again risking the kind of blind spots associated with the McNamara fallacy.
Teaching is often an unfathomable enterprise that is relational in essence, and resists the crude measures often imposed upon it. There should be more emphasis on phronesis or discretionary practitioner judgement that is informed by a deep subject knowledge, a set of ethical and philosophical principles and quality research/sustained inquiry into complex problems.
In my experience, the most important factors in great teaching are almost unmeasurable in numbers. The best teachers I know have a set of common characteristics:
1. They are not only very knowledgable about their subject but they are almost unreasonably passionate about it – something which is infectious for kids.
2. They create healthy relationships with those students in a million subtle ways, which are not only unmeasurable but often invisible to those involved.
3. They view teaching as an emancipatory enterprise which informs/guides everything they do. They see it as the most important job in the world and feel it’s a privilege to stand in a room with kids talking about their passion.
Are these things measurable in numbers and is it even appropriate to do so? Are these things helped or hindered by the current league table culture?
Robert McNamara died an old man and had opportunity to reflect on his long life, most notably in the Academy award winning documentary The Fog of War. His obituary in the Economist records that:
“He was haunted by the thought that amid all the objective-setting and evaluating, the careful counting and the cost-benefit analysis, stood ordinary human beings. They behaved unpredictably.”
Measuring progress is important. We need to know what we are doing is having impact against another approach that might yield better outcomes, but the current fetish of crude numerical quantification in education is misleading and fundamentally inappropriate for the unpredictable nature of the classroom. We need better ways of recording the phenomenon of the classroom that captures more than simply test scores and arbitrary judgements on teachers, and seeks to impose an order where often there is none.
In October I went to visit Glen Whitman at St. Andrew’s School near Washington. He is the Dean of Studies there but is also a force of nature who runs an in-house research centre that has completely transformed the culture of the school. I asked him how he did it.
For more on Glenn see him speak here at the Center for American Progress.
It’s very interesting that when you you talk to people about Hattie’s work, a lot of people seem unsure about what it is or what “Visible Learning” actually means. I thought it might be good to ask him.
A VERY slow news day: