Methods for meta-analysis with dependent effect size estimates led by Professor Larry Hedges
Committee Room No 2, Institute of Education, University of London, 20 Bedford Way, London, WC1H 0AL
Monday 7th June, 2010
10.00 a.m. to 3.00 p.m.
About Professor Larry Hedges:
The workshop was led by Professor Larry Hedges, Board of Trustees Professor of Statistics and Social Policy, Northwestern University (USA) and offered a rare opportunity to learn directly from an acknowledged expert in the field of educational statistics and evaluation.
Professor Hedges’ research straddles many fields—in particular those of sociology, psychology, and educational policy. He is best known for his work to develop statistical methods for meta-analysis (a statistical analysis of the results of multiple studies that combines their findings) in the social, medical, and biological sciences. It is a key component of evidence-based social research. Examples of some his recent studies include: understanding the costs of generating systematic reviews, differences between boys and girls in mental test scores, the black-white gap in achievement test scores, and frameworks for international comparative studies on education.
Widely published, he has authored or co-authored numerous journal articles and five books, including the seminal Statistical Methods for Meta-Analysis: A Practical Guide to Modern Methods of Meta-Analysis (with I. Olkin) and The Handbook of Research Synthesis (with H. Cooper and J. Valentine).
Who should attend:
The workshop was aimed at those conducting a systematic review and meta-analysis. A basic statistical understanding of effect sizes and meta-analysis was assumed.
The workshop was aimed at research students, research associates, academic researchers and those with a policy interest in quantitative synthesis.
About the workshop:
Conventional meta-analytic techniques rely of the assumption that effect size estimates from different studies are independent. The independence assumption is violated when studies produce several effect size estimates based on the same individuals. This workshop addressed methods of dealing with situations in which there are effect size estimates that are not independent.
The simplest class of methods for dealing with dependence involves reducing the problem back to one where there is a single (synthetic) effect size per study. This can be done by selecting one effect size, averaging effect sizes, or in some cases taking differences among effect sizes. The major shortcoming of this class of methods is that accurate variance estimation is difficult without knowledge of the dependence structure of the estimates.
The most complex strategy involves modeling the dependence structure among effect sizes within studies and using fully multivariate meta-analytic procedures. The major shortcoming of this strategy is the practical complexity of the computations required. In practice, both strategies suffer from the fact that studies rarely report the information needed to compute the correlation structure of effect size estimates computed from data on the same individuals. Professor Hedges discussed a new method (based on work by Hedges, Tipton, and Johnson) for dealing with correlated effect size estimates that provides valid statistical inference without the need for information about within-study correlation structure when the number of studies is not small. This method is easy to implement (much easier than multivariate methods) and is often suitable for meta-analyses involving 20 – 30 studies.
Please refer to the research article 'Robust variance estimation in meta-regression with dependent effect size estimates' by Larry V. Hedges, Elizabeth Tipton and Matthew C Johnson which was recently published online by Wiley Interscience in 'Research Synthesis Methods'.
Following the workshop, a public lecture entitled 'Context Effects, Experiments and Generalization in Educational Research' was led by Professor Larry Hedges. The lecture took place at the Elvin Hall, Institute of Eduation, University of London at 6.00 p.m. Further details can be found on the 'additional events' page.
Registration and costs:
There was no charge for the workshop. Refreshments and a buffet lunch were provided free of charge.
The number of places was limited to 25. Places were offered on a first come, first served basis.