ERRR #018. John Hattie defending the meta-analysis

The ERRR podcast can also be listened to on Spotify, apple podcasts, and all other podcasting apps. 

Edit: You can now read my conclusions following my interviews with both Adrian Simpson, and John Hattie, in this blog post: Effect Sizes, Robust or Bogus? Reflections from my discussions with Hattie and Simpson

ERRR listeners, today is an exciting day, because it’s the first time that we’ve had a guest onto the ERRR in response to another ERRR episode. As many of you will be aware, last episode we spoke to Professor Adrian Simpson as he took a no-holds-barred critique to the meta-analysis, suggesting that the oft-quoted effect size is maybe not all it’s cracked up to be.

As a quick Summary, we discussed with Adrian three factors that he suggested can skew effect sizes to such an extent that we should be very, very careful when comparing them across studies. These three factors were: unequal comparison groups, range restriction, and measurement design. In discussing the cumulative effects of these factors Adrian proposed that using effect sizes to rank the effectiveness of educational interventions is a category error. This is because the factors that play into creating an effect size often have just as much, if not more, to do with the design of a study, than they do with the efficacy of the intervention in question. In Sum, Adrian suggested that effect sizes are a better measure of a study’s clarity, than the impact of an educational initiative. (If you haven't listened to that ep as yet, I suggest you do so prior to listening to this episode. Check it out here)

In this podcast,  the most famous advocate of the meta-analysis in education research, John Hattie, defends the meta-analytic, and meta-meta-analytic approach!

Laureate Professor John Hattie‘s work is internationally acclaimed. His influential 2008 book Visible Learning, is believed by many to be the world’s largest evidence-based study into the factors that improve student learning. Hailed by the Times Education Supplement as “teaching’s Holy Grail”, Hattie’s book involved more than 80 million students from around the world and brought together 50,000 smaller studies. Visible Learning found that positive teacher-student interaction is the most important factor in effective teaching.

Since 2011, Professor Hattie has been Director of the Melbourne Education Research Institute at the University of Melbourne. He is also the Chair of the Australian Institute for Teaching and School Leadership (AITSL), through which he provides national leadership in promoting excellence so that teachers and school leaders have maximum impact on learning.

Professor Hattie has been awarded a New Zealand Order of Merit, has published over 500 papers, and supervised 200 PhD students.

Links mentioned during the interview

• • Article by EEF and Evidence for Learning in response to the Adrian Simpson ERRR episode here.
  • Danish Philosopher, Steen Nepper Larsen's critique of Hattie's work here.
  • Visible Learning Plus website. 

Please consider supporting the Education Research Reading Room Podcast

If you're keen to help the ERRR podcast to stay independent and sustainable, please consider making a one-off or monthly donation through https://www.patreon.com/errr. Audio engineering, room hire, and audio-file web hosting cost Ollie upward of $70 per episode, and any donation (even as much as a cup of coffee), to help cover these costs is greatly appreciated : )

Listen to all past episodes of the ERRR podcast here.