Towards the end of 2016 I was explaining to a friend that I was launching The Education Research Reading Room podcast because I couldn't find an edu podcast that delved into teaching and learning to the nerdy level that I desired. His response? ‘Have you heard of the Mr. Barton Maths podcast? Once you get over the number of times the host says ‘Flippin', it's pretty good. I suggest you start with the Dylan Wiliam episode‘.
This recommendation seriously changed my life! Engaging with the work of Craig Barton sent me hurtling into a vortex of inspiring teacher bloggers (mostly based in the UK) that I never knew existed, and it's fundamentally changed the way I teach, as well as the way I engage with the discussion around education.
I recently listened to Craig Barton's second interview with Dylan William, and, as I was frantically pulling over my bicycle on the verge of busy roads to jot down key takeaways, I had the thought that it might be nice to share my notes from that first episode, along with my current ones as a way to consolidate my learnings from these two discussions.
And so, here they are below. Interview 1 are my notes (pretty unchanged, aside from where I've noted) from the first interview. Interview 2 details my takeaways from the most recent podcast. Looking back, it's really incredible to see just how much of the content from that first interview I now feel like I've always known! It's amazing to think that many of these ideas were totally new to me just 18 months ago!
Interview 1 notes begin here
Link to the interview.
Reciprocal Teaching
Robert Slavin: When we encourage students to help each other, whilst there are great benefits to both students, the students who learn the most are the ones who do the most explaining.
The Relevance of Problem Contexts
Jo Boaler:
Q: ‘When do girls prefer football to fashion?’
A: When it’s the context of a maths question. Presented with a structurally identical maths question in two different contexts, girls do better than boys when the context is that of football (soccer). This is because they bring less irrelevant and confounding background knowledge into the solving process.
What is learning?
Paul Kirschner: Learning is a change in long term memory, Aka: if they don’t remember it in 6 weeks, they haven’t really learnt it.
Relatedly… John Mason: ‘Teaching takes place in time, but learning takes place over time.’
We don’t actually Know what Good Teaching Looks Like!
1. From the work of Heather Hill: We need to stop kidding ourselves by thinking that we can pick a good or a bad teacher by observing them teach a class. Hill suggests they would need to be observed in 6 different classes by 5 different observers (a total of 30 observations) to obtain a reliable rating.
2018 Addition: I emailed Heather Hill about this, and this is what she said: “Thanks for your question. For my own instrument, it originally looked like we needed 4 observations each scored by 2 raters (see attached paper). However, Andrew Ho and colleagues came up with the 6 observations/5 observer estimates from MET data:” Ho’s paper. (More of Heather's response here)
2. From the work of Dan Goldhaber: Comparing two models of ‘good teaching’ (a fixed effect and a random effect model) based upon ‘value added’ metrics, the best 9% of teachers as rated by one model were classified as the worst teachers in the other!
Dylan concludes that we can only really comment in the extremes, i.e., ‘We can be pretty sure that a teacher who appears to be very very good is in fact not very very bad, and we can be pretty sure that a teacher who appears very very bad is in fact not very very good.’, but that’s about the extent of it.
So… where to?
Dylan says that team leaders should focus on one question: ‘What do you want to get better at and how can we do it?’.
2018 Addition: I'm not totally convinced about this, especially for novice teachers who might not know what they should be focussing on. I reckon a more structured approach for novices would probably be more beneficial. For example, building the process around Rosenshine's principles of instruction, as Adam Boxer suggests here, then helping teachers to deepen their practice through Jenny Gore's framework, as discussed on the ERRR Podcast here (RCT currently underway to check if Gore's framework positively impacts student outcomes, and I'll be blogging about my experience with Gore's framework shortly).
Thinking Hard and Distributed Practice
Robert Bjork: The harder you think about something the better you remember it. Relatedly, the best time to study something is at the point just before you’ve completely forgotten it!
Simple Hacks to improve Assessment
The hypercorrection effect: You get two benefits of assessment, the first is when the testee is forced to recall the information in the first place, this strengthens the synaptic connections. The second benefit is when they see the answer. Thus, in order to maximise learning, the best person to mark a test is the person who just took it.
Synoptic testing: Testing all the content that's bean learnt to date, in an accumulative way.
Interview 2 notes begin here
Link to interview here. This interview focussed on Wiliam's recent book ‘Creating the Schools Our Children Need‘, but covered heaps more territory too. You can see some poignant snippets from this book here.
One of Wiliam's favourite Foulcault quotes, on writing
“I don't feel that it is necessary to know exactly what I am. The main interest in life and work is to become someone else that you were not in the beginning. If you knew when you began a book what you would say at the end, do you think that you would have the courage to write it? What is true for writing and for a love relationship is true also for life. The game is worthwhile insofar as we don't know what will be the end. My field is the history of thought. Man is a thinking being.” Truth, Power, Self : An Interview with Michel Foucault (25 October 1982)
How different the U.S educational system is to the UK and Australia
In the US, schools are run by district school boards. There are 13491 district school boards in the US and they vary greatly in size. Small boards could govern as few as one school and 19 students (source), and the currently biggest district governs almost 1 million students (source). That said, the modal district school board governs 6 schools.
Most surprisingly to me was that these district school boards control not only curriculum, but also, teacher pay and pensions. That's right, there's no national policy on how much teachers get paid, it's all down to school boards!
Dylan Wiliam's three criteria for hiring a teacher
- They like kids
- They think they should be continuously improving
- Solid content knowledge
- But. Top 6th of teachers (based upon content knowledge of the content they're to teach) only get 2 to 3 more weeks of growth per year.
A teacher is fully qualified when…
Teachers should be considered fully qualified when they're able to evaluate their own practice as well as someone else is able to evaluate their practice.
The rating of your lesson is likely to relate to the set that you're teaching
IIf you're teaching the top of 5 seats your 6 times more likely to get a top rating on a 4 point scale than if you're teaching teaching the bottom set
Why we shouldn't be copying what Finland is doing today
The things that contributed to Finland good PISA results in 2003 and 2006 are from policy decisions made in the late 80's and early 90's. Anyone going to Finland in 2006 or later, are actually looking at what they do were looking at what contributed to their PISA decline, not their PISA success. 
Here's a helpful illustration of Finland's PISA results over time, produced by Greg Ashman.
Don't select on the dependent variable!
Wiliam old the story of a WWII statistician, Abraham Wald, who was tasked with analysing damage to returning fighter planes, and making recommendations about how to improve their resilience. Counter to common approaches at the time, Wald pointed out that bullet holes in returning planes didn't represent where planes should be reinforced, but rather where they can take a hit and still return safely.
Thus, taking data from returning planes represented ‘survivorship bias' and, planes should be reinforced in areas where returning planes never appeared to have damages.
Wiliam uses this example to demonstrate the danger of ‘selecting on the dependent variable'. What this means is that we can't explain what causes an outcome by looking only at instances in which that outcome comes to fruition. This is a pretty tricky concept, here are a couple of examples.
- Imagine you're trying to work out what causes students to finish their homework. You select all of the students in your school who regularly complete their homework and you work out what they do every night prior to completing their homework. You find that the one thing that they all have in common is that they all eat dinner, thus, you conclude that eating dinner helps students complete their homework. By not considering cases of students who don't regularly complete their homework, you're selecting on the dependent variable, and coming to an unwarranted conclusion.
- You want to run an inspirational assembly at your school to explain to all of your students how to be a success in life. You get three of your ex-students who have become quite successful (a debatable concept but let's roll with it anyway) to come on and share their stories. The first talks about how she got really good at makeup, started to make youtube videos about it, and now makes $15k per month through sponsoring. The second talks about how she volunteered at an environmental organisation after she finished school, and now acts as a corporate social responsibility consultant in large organisations, also making a financial killing. The third runs a blog and podcast about stay at home parenting for dads (his blog and podcast don't make any money, but his partner earns enough for the both of them). One student astutely points out at the end of the presentations that none of the three presenters went to uni and asks innocently ‘Does that mean I don't have to go to uni to be successful? The school would then likely have to run a follow up assembly explaining what it means to select on the dependent variable (good luck with that one!).
This section of the discussion between Wiliam and Barton gave me some helpful language to better identify and name cases of this when I see them.
The 10,000 hour rule isn't as simple as it appears
Relatedly, Wiliam suggests that we should consider survivorship bias/selecting on the dependent variable when thinking about the work of Anders Ericsson (10,000 hour rule). It may be true that those who make it through 10,000 hours of practice all end up as experts in their field (or at least very good), but we must consider the filtering effect that talent could be having along the way. It may be that only the people who manage to last through 1000p hrs of practice are those who can see themselves tangibly improving along along the way. The people who improve quicker are also perhaps those with more talent in the first place! See the work of David Hambrick as a balanced response to Ericsson's work.
Hugh Burkhards on Mathematical Modelling
Students can't use anything that they've learnt in the last 2 years for mathematical modelling.
Craig isn't completely convinced by this one. You'll hear his take on it in the Education Research Reading Room Podcast episode with Craig, to be released August 1st, 2018.
Timing student responses is the only way to know if they're fluent
For diagnostic purpose, the only way to know if your students know their number facts, as opposed to deriving them, is to test them under timed conditions.
GREAT POINT!!!
What's the source of the power of the hypercorrection effect?
Janet Metcalf's work suggest that it's not clear if it the power of the hypercorrection effect comes from the certainty of a students judgment, or the act of the student rating that certainty. In all the hypercorrection research, the participant is asked to rate their confidence, but this is just because that's the only way to work out their level of confidence. Thus, getting students to rate the confidence of their answers (according to Wiliam) won't necessarily increase the efficacy of the hypercorrection effect, it could just occur by students being surprised that they were wrong.
Wiliam also warns to be careful of differential impacts on boys and girls of getting students to rate their confidence. Boys, on average, tend to be more brazen in their confidence assessments, so if we link confidence ratings to marks, this could have differential consequences by sex.
(Hypothesis) Hint as halfway house to retrieval
A hypothesis from Wiliam: If a student is struggling to retrieve some info, a hint that prompts the retrieval may be as effective, in terms of additional encoding, as if the student was able to retrieve the info outright.
You don't have to record marks for a test to be useful
The two main benefits of testing occur before the teacher enters a mark in their mark book. That is, the retrieval effect, and the hypercorrection effect.
Everything in education research entails tradeoffs (The most powerful thing that Wiliam has learnt)
The most powerful thing that Dylan has learnt about education research is that everything entails trade offs.
I'll be remembering this one!
Quote: Jean Paul-Junker on knowing the right thing to do
We all know what to do, we just don’t know how to get re-elected after we’ve done it.
- When talking about economic reforms.[1] (page 4 of the report). (from wikiquote)