The ERRR podcast can also be listened to on Spotify, apple podcasts, and all other podcasting apps. 

Since its invention by Gene Glass back in the 70's, the approach of the meta-analysis for aggregating research findings has taken the education world by storm. From the influential work of John Hattie to the newly released Teaching and Learning Toolkit, the meta-analysis has been lauded (in Australia and the U.K at least) as the ultimate methodology for identifying high impact teaching strategies and using that knowledge to direct teaching and learning from the classroom level to that of public policy.

But Adrian Simpson warns that we should be careful.

In his 2017 paper, The misdirection of public policy: Comparing and combining standardised effect sizes (download here) Adrian discusses the assumptions on which meta-analyses are based, and alludes to the repercussions if those assumptions are violated. This article forms the foundation for this Education Research Reading Room episode.

In this episode of the ERRR we hope to gain a more nuanced understanding of the process of meta-analysis as well as its limitations.

Note: If you'd like to listen to this podcast on your iPhone you can access it via the ‘Podcasts' app or check out the iTunes direct link here. For Android, the ‘Stitcher' app is free and easy to use, direct link to ERRR on stitcher here.

Our guest this episode

Adrian Simpson is Professor of Mathematics Education at Durham University in the UK and is The Principal of Josephine Butler College.

Adrian worked as a secondary mathematics teacher before becoming a lecturer in pure mathematics. A chance encounter with an education colleague trying to interpret a primary school pupil’s arithmetic led to a change of direction from researching the logical foundations of infinitesimals to mathematics education research.

While most of his research has been about teaching and learning mathematics at the undergraduate level, within mathematics education, he has been quite eclectic: writing research papers on conceptual growth, mathematical proof, understanding algebra, oral assessment and the use of videos in the development of teacher’s professional vision. With colleagues, he has had a number of grants including funding to study patterns of assessment in undergraduate mathematics; transferring Shanghai mathematics methods to the UK; and creating a nationwide programme for developing mathematics teachers’ skills in higher level school mathematics.

Adrian has used both qualitative and quantitative methods before another chance encounter, this time with a philosopher, led to a second change of direction to his most recent work looking at the evidential basis of those methods and particularly the use of quantitative methods in ‘evidence based policy’ within education.

Links mentioned during the interview

    • George Lilley's website looking in detail at various meta-analyses; here
    • David Freedman (Quoted as ‘David Evans' in the podcast): Freedman, D. A. (2009). Statistical models: theory and practice. Cambridge University Press
    • Pawson and Tilley: Pawson, R., & Tilley, N. (1997). Realistic evaluation. Sage.
    • Nancy Cartwright: Cartwright, N., & Hardie, J. (2012). Evidence-based policy: a practical guide to doing it better. Oxford University Press.
    • Tricia Greenhalgh: Greenhalgh, T., Howick, J., & Maskrey, N. (2014). Evidence based medicine: a movement in crisis?. Bmj, 348, g3725.
    • Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005). Realist review-a new method of systematic review designed for complex policy interventions. Journal of health services research & policy, 10(1), 21-34.
    • Andrew Gelman: this is a blog: andrewgelman.com
    • Richard Berk: Berk, R. (2007). Statistical inference and meta-analysis. Journal of Experimental Criminology, 3(3), 247-270.
    • Berk, R. A. (2004). Regression analysis: A constructive critique. Sage.
    • Berk, R. (2011). Evidence-based versus junk-based evaluation research: Some lessons from 35 years of the evaluation review. Evaluation Review, 35(3), 191-203.

Please consider supporting the Education Research Reading Room Podcast

If you're keen to help the ERRR podcast to stay independent and sustainable, please consider making a one-off or monthly donation through https://www.patreon.com/errr. Audio engineering, room hire, and audio-file web hosting cost Ollie upward of $70 per episode, and any donation (even as much as a cup of coffee), to help cover these costs is greatly appreciated : )

    • .

Related