Cookies on this website
We use cookies to ensure that we give you the best experience on our website. If you click 'Continue' we'll assume that you are happy to receive all cookies and you won't see this message again. Click 'Find out more' for information on how to change your cookie settings.

Objectives: To show the effects of chance on meta-analyses,and the potential dangers of being prompted to do a metaanalysisby one favourable trial. Design: In total, 100,000 trials were simulated and combinedinto 10,000 meta-analyses, using data from the controlgroup of a cancer trial. Each participant record wasrandomly coded to simulate allocation to 'treatment' or 'control'. Setting: Simulated study. Participants: De-identified records for 578 patients fromthe control group of a cancer trial, of whom 147 had died.Main outcome measure: Time to death from any cause. Results: Of the 100,000 trials, 4897 (4.9%) were statisticallysignificant at 2p < 0.05 and 123 (1.2%) of the 10,000meta-analyses were significant at 2p < 0.01. The mostextreme result was a 20% reduction (99% CI: 0.70-0.91;2p 1/4 0.00002) in the annual odds of dying in the 'treatment' group. If a meta-analysis contained at least one trial with astatistically significant result (at 2p < 0.05), the likelihood ofthe meta-analysis being significant (at 2p < 0.01) increasedstrikingly. For example, among the 473 meta-analyses inwhich the first trial in a batch of 10 was statistically significant(at 2p < 0.05), 18 (3.8%) favoured treatment at2p < 0.01.Conclusions: Chance can influence the results of metaanalysesregardless of how well they are conducted. Researchersshouldnotignorethiswhentheyplanameta-analysisandwhentheyreporttheirresults.Peoplereadingtheirreportsshouldalsobewary.Cautionisparticularlyimportantwhentheresultsofoneormoreincludedstudiesinfluencedthedecisiontodothemeta-analysis. © The Royal Society of Medicine.

Original publication




Journal article


Journal of the Royal Society of Medicine

Publication Date





116 - 119