10.4.3.1   Recommendations on testing for funnel plot asymmetry

For all types of outcome :

As a rule of thumb, tests for funnel plot asymmetry should be used only when there are at least 10 studies included in the meta-analysis, because when there are fewer studies the power of the tests is too low to distinguish chance from real asymmetry.

  Tests for funnel plot asymmetry should not be used if all studies are of similar sizes (similar standard errors of intervention effect estimates). However, we are not aware of evidence from simulation studies that provides specific guidance on when study sizes should be considered ‘too similar’.

Results of tests for funnel plot asymmetry should be interpreted in the light of visual inspection of the funnel plot. For example, do small studies tend to lead to more or less beneficial intervention effect estimates? Are there studies with markedly different intervention effect estimates (outliers), or studies that are highly influential in the meta-analysis? Is a small P value caused by one study alone? Examining a contour-enhanced funnel plot, as outlined in Section 10.4.1 , may further help interpretation of a test result.

When there is evidence of small-study effects, publication bias should be considered as only one of a number of possible explanations (see Table 10.4.a ). Although funnel plots, and tests for funnel plot asymmetry, may alert review authors to a problem which needs considering, they do not provide a solution to this problem.

Finally, review authors should remember that, because the tests typically have relatively low power, even when a test does not provide evidence of funnel plot asymmetry, bias (including publication bias) cannot be excluded.

For continuous outcomes with intervention effects measured as mean differences :

The test proposed by (Egger 1997a) may be used to test for funnel plot asymmetry. There is currently no reason to prefer any of the more recently proposed tests in this situation, although their relative advantages and disadvantages have not been formally examined. While we know of no research specifically on the power of the approach in the continuous case, general considerations suggest that the power will be greater than for dichotomous outcomes, but that use of the method with substantially fewer than 10 studies would be unwise.

For dichotomous outcomes with intervention effects measured as odds ratios:

The tests proposed by Harbord et al. (Harbord 2006) and Peters et al. (Peters 2006) avoid the mathematical association between the log odds ratio and its standard error (and hence false-positive test results) that occurs for the test proposed by Egger at al. when there is a substantial intervention effect, while retaining power compared with alternative tests. However, false-positive results may still occur in the presence of substantial between-study heterogeneity.

The test proposed by Rücker et al. (Rücker 2008) avoids false-positive results both when there is a substantial intervention effect and in the presence of substantial between-study heterogeneity. As a rule of thumb, when the estimated between-study heterogeneity variance of log odds ratios, tau-squared, is more than 0.1, only the version of the arcsine test including random-effects (referred to as ‘AS+RE’ by Rücker et al.) has been shown to work reasonably well. However it is slightly conservative in the absence of heterogeneity, and its interpretation is less familiar because it is based on an arcsine transformation. (Note that although this recommendation is based on the magnitude of tau-squared other factors, including the sizes of the different studies and their distribution, influence a test’s performance. We are not currently able to incorporate these other factors in our recommendations).

When the heterogeneity variance tau-squared is less than 0.1, one of the tests proposed by Harbord 2006, Peters 2006 or Rücker 2008 can be used. (Test performance generally deteriorates as tau-squared increases).

As far as possible, review authors should specify their testing strategy in advance (noting that test choice may be dependent on the degree of heterogeneity observed). They should apply only one test, appropriate to the context of the particular meta-analysis, from the above-recommended list and report only the result from their chosen test. Application of two or more tests is undesirable since the most extreme (largest or smallest) P value from a set of tests does not have a well-characterized interpretation.

For dichotomous outcomes with intervention effects measured as risk ratios or risk differences, and continuous outcomes with intervention effects measured as standardized mean differences:

Potential problems in funnel plots have been less extensively studied for these effect measures than for odds ratios, and firm guidance is not yet available.

Meta-analyses of risk differences are generally considered less appropriate than meta-analyses using a ratio measure of effect (see Chapter 9, Section 9.4.4.4 ). For similar reasons, funnel plots using risk differences should seldom be of interest. If the risk ratio (or odds ratio) is constant across studies, then a funnel plot using risk differences will be asymmetrical if smaller studies have higher (or lower) baseline risk.

Based on a survey of meta-analyses published in the Cochrane Database of Systematic Reviews , these criteria imply that tests for funnel plot asymmetry should be used in only a minority of meta-analyses (Ioannidis 2007b) .

Tests for which there is insufficient evidence to recommend use

The following comments apply to all intervention measures. The test proposed by Begg and Mazumdar (Begg 1994) has the same statistical problems but lower power than the test of Egger et al., and is therefore not recommended. The test proposed by Tang and Liu (Tang 2000) has not been evaluated in simulation studies, while the test proposed by Macaskill et al. (Macaskill 2001) has lower power than more recently proposed alternatives. The test proposed by Schwarzer et al. (Schwarzer 2007) avoids the mathematical association between the log odds ratio and its standard error, but has low power relative to the tests discussed above.

In the context of meta-analyses of intervention studies considered in this chapter, the test proposed by Deeks et al. (Deeks 2005) is likely to have lower power than more recently proposed alternatives. This test was not designed as a test for publication bias in systematic reviews of randomized trials: rather it is aimed at meta-analyses of diagnostic test accuracy studies, where very large odds ratios and very imbalanced studies cause problems for other tests.

  • Open access
  • Published: 16 April 2007

Assessment of funnel plot asymmetry and publication bias in reproductive health meta-analyses: an analytic survey

  • João P Souza 1 , 2 ,
  • Cynthia Pileggi 2 &
  • José G Cecatti 1  

Reproductive Health volume  4 , Article number:  3 ( 2007 ) Cite this article

14k Accesses

63 Citations

Metrics details

Despite efforts to assure high methodological standards, systematic reviews may be affected by publication bias. The objective of this study was to evaluate the occurrence of publication bias in a collection of high quality systematic reviews on reproductive health.

Systematic reviews included in the Reproductive Health Library (RHL), issue No 9, were assessed. Funnel plot was used to assess meta-analyses containing 10 or more trials reporting a binary outcome. A funnel plot, the estimated number of missing studies and the adjusted combined effect size were obtained using the "trim and fill method". Meta-analyses results that were not considered to be robust due to a possible publication bias were submitted to a more detailed assessment.

A total of 21 systematic reviews were assessed. The number of trials comprising each one ranged from 10 to 83 (median = 13), totaling 379 trials, whose results have been summarized. None of the reviews had reported any evaluation of publication bias or funnel plot asymmetry. Some degree of asymmetry in funnel plots was observed in 18 of the 21 meta-analyses evaluated (85.7%), with the estimated number of missing studies ranging from 1 to 18 (median = 3). Only for three meta-analyses, the conclusion could not be considered robust due to a possible publication bias.

Asymmetry is a frequent finding in funnel plots of meta-analyses in reproductive health, but according to the present evaluation, less than 15% of meta-analyses report conclusions that would not be considered robust. Publication bias and other sources of asymmetry in funnel plots should be systematically addressed by reproductive health meta-analysts. Next amendments in Cochrane systematic reviews should include this type of evaluation. Further studies regarding the evolution of effect size and publication bias over time in systematic reviews in reproductive health are needed.

Peer Review reports

Implementing best practices is a major goal in health services [ 1 ]. However, the identification of such practices depends on the evaluation and synthesis of a large amount of scientific information. This may be achieved by carrying out systematic reviews and meta-analyses, which have come to represent important sources of evidence-based knowledge for clinicians, policy makers and researchers [ 2 ].

A systematic review is an observational study of the scientific literature based on individual studies. It may contain meta-analyses, which are statistical procedures developed for summarizing effects across individual studies. The ideal meta-analysis should combine data appropriately to produce a more complete and meaningful estimate of the overall effect [ 3 ]. Nevertheless, despite efforts to assure high methodological standards, systematic reviews may be affected by publication bias, one of the major drawbacks of such studies and a threat to their validity. Publication bias occurs whenever the results of a set of published studies differ from the results of all the research performed on a specific topic [ 3 ]. A publication-biased meta-analysis may present an ineffective or unsafe intervention as being effective or safe, or not recommend an effective or safe intervention because the results of some studies already performed are not included. Furthermore, publication bias may be partially responsible for occasional discrepancies between the conclusions of previous meta-analyses and subsequent large multicenter trials [ 4 ].

The assessment of publication bias is a relatively new recommendation but reported in several relevant meta-analyses reporting guidelines [ 5 – 8 ]. However, it has been observed that only few meta-analyses have actually evaluated publication bias (3.2% – 6.5%) [ 9 ]. It is also unclear whether reproductive health meta-analyses are affected by publication bias, since we have been unable to identify any previous reports assessing publication bias and its effects on meta-analyses in reproductive health. The objective of this survey was, therefore, to evaluate the occurrence of publication bias in a collection of high quality systematic reviews on reproductive health.

This is an analytic survey carried out to evaluate the impact of publication bias on the results of meta-analyses of reproductive health interventions. Systematic reviews included in the World Health Organization (WHO) Reproductive Health Library (RHL), issue No 9, were assessed [ 10 ]. The RHL is a WHO instrument for documenting and disseminating best practices in the field. It reproduces the most relevant Cochrane systematic reviews related to reproductive health, adding some practical aspects and pertinent comments for developing country settings, as well as implications for research [ 10 ].

There are several methods of assessing the occurrence of publication bias. A common approach is based on scatter plots of the treatment effect estimated by individual studies versus a measure of study size or precision (the "funnel plot"). In this graphical representation, larger and more precise studies are plotted at the top, near the combined effect size, while smaller and less precise studies will show a wider distribution below. If there is no publication bias, the studies would be expected to be symmetrically distributed on both sides of the combined effect size line. In case of publication bias, the funnel plot may be asymmetrical, since the absence of studies would distort the distribution on the scatter plot [ 3 ].

The "trim and fill" method examines the existence of asymmetry in the funnel plot and is recommended as a tool for the assessment of the robustness of the results of meta-analyses (sensitivity analysis). The method consists of a rank-based data augmentation procedure that statistically estimates the number and location of missing studies. The main application of this method is to adjust for the possible effects of missing studies [ 11 ]. If the conclusion of the meta-analysis remains unchanged following adjustment for the publication bias, the results can be considered reasonably robust, excluding publication bias.

In the present study, funnel plot asymmetry was used to assess meta-analyses containing 10 or more trials reporting a binary outcome. In each review, the meta-analysis with the greatest number of trials was selected for evaluation. Therefore those meta-analyses may not necessarily represent the primary outcomes for the review. If two or more meta-analyses had a similar number of trials, the one listed first in the review was selected [ 12 ]. To achieve consistency across meta-analyses, endpoints were re-coded if necessary so that an effect size below 1 always indicated a beneficial effect of the intervention.

The following data were extracted from each meta-analysis: study name, subgroups within the study, data on effect size, year of publication of the most recent trial included and assessment of publication bias. After extracting the data and compiling a database, statistical analysis was performed using the Comprehensive Meta-Analysis ® software program (version 2.2.034, USA, 2006). The database was checked twice for the presence of inconsistencies. For each meta-analysis, a funnel plot, the estimated number of missing studies and the adjusted combined effect size were obtained using the "trim and fill method". This procedure was applied to both sides of each funnel plot. The model of effect (fixed-effects or random-effects) used was the same as that applied in the primary meta-analysis. Results of "trim and fill method" were validated by using Stata (Stata Corporation, College Station, TX, USA)

In Cochrane systematic reviews, the Mantel-Haenszel method is usually applied to estimate relative risk by using the Review Manager (RevMan) software program [ 8 ]. In order to comply with the requirements of the Comprehensive Meta-Analyses ® software program used in the present study, values obtained for the adjustment for publication bias were converted through the inverse variance method. In case of possible publication bias, the meta-analyses were submitted to a more detailed assessment.

The 9 th issue of the WHO Reproductive Health Library included 105 Cochrane Systematic Reviews. A total of 21 systematic reviews contained meta-analyses with 10 or more trials reporting a binary outcome. In this set of meta-analyses, the number of trials comprising each one ranged from 10 to 83 (median = 13), totaling 379 trials, whose results have been summarized. None of the reviews had reported any evaluation of publication bias or funnel plot asymmetry. During data extraction, the endpoints of one meta-analysis were re-coded to guarantee consistency across meta-analyses [ 13 ].

The main characteristics of the selected meta-analyses and the adjustments performed are shown in Table 1 . According to the "trim and fill method", some degree of asymmetry in funnel plots was observed in 18 of the 21 meta-analyses evaluated (85.7%). In meta-analyses in which asymmetric funnel plot was found, the estimated number of missing studies ranged from 1 to 18 (median = 3). All summary plots of meta-analyses, together with their respective "trimmed and filled" funnel plots, are shown in Appendix 1 [See Additional file 1 ].

In 18 of the 21 meta-analyses evaluated, the assessment procedure and the adjustments for publication bias had no effect on the conclusions (Table 1 ); however, in the remaining three (14.3%, 3:21), the conclusion cannot be considered robust due to a possible publication bias [ 14 – 16 ]. These results were obtained applying the same model of effect used in the primary meta-analysis (fixed-effects or random-effects), but were confirmed using the alternative method (data not shown).

Figures 1 and 2 show filled funnel plots with summary effect estimates before and after adjustment for the publication bias. Both meta-analyses included a similar number of trials and both presented asymmetric funnel plots of identified studies (open circles). After adjustment for the publication bias, the estimated number of missing studies entered into the funnel plot (filled circles) was moderate, 7 and 5 respectively. The summary of estimates obtained before (open diamond) and after the adjustment (filled diamond) indicates that, if really such a number of missing studies exists, the impact on the conclusion may not be negligible. Figure 3 also presents a filled funnel plot and, although the estimates suggest that only one study may be missing, the practical impact on the conclusion should be considered. Table 2 summarizes the practical impact of the adjustment for publication bias on conclusions in these three meta-analyses.

figure 1

A filled funnel plot of the antibiotics prophylaxis regimens for cesarean section data, with filled circles denoting the imputed missing studies. The bottom diamonds show summary effect estimates before (open) and after (filled) publication bias adjustment.

figure 2

A filled funnel plot of the continuous support for women during childbirth data, with filled circles denoting the imputed missing studies. The bottom diamonds show summary effect estimates before (open) and after (filled) publication bias adjustment.

figure 3

A filled funnel plot of the interventions for emergency contraception data, with filled circles denoting the imputed missing studies. The bottom diamonds show summary effect estimates before (open) and after (filled) publication bias adjustment.

The main results of this analytic survey suggest that some degree of asymmetry in funnel plots is a common finding in reproductive health meta-analyses. In about 14% of the selected meta-analyses, this type of asymmetry qualitatively impacted conclusions. However, the present study also has some limitations that have to be taken into consideration for the subject to be evaluated within context. If Cochrane meta-analyses differ in their methods from other meta-analyses or if meta-analyses with fewer than ten trials differ from those with more than ten, a selection bias may exist. Despite this possible selection bias, this source of meta-analyses was chosen because of the consistency in the methodology, associated with a widely recognized standard of quality. The number of trials was established as an inclusion criterion since this factor results in the best performance of the "trim and fill" method. Moreover, although other methods are available for the assessment of asymmetry in funnel plots, no consensus has been reached with respect to the superiority of any single method. Therefore, any method used for detecting asymmetry in funnel plots should be considered indirect and exploratory. In this study, we used the "trim and fill" method as an instrument for sensitivity analysis. Our principal concern was not the exact number of missing studies; we were, in fact, interested in how the effect size estimates would be qualitatively changed by the presence of an underlying publication bias.

Asymmetrical in funnel plots are linked to publication bias although there are other sources of asymmetry that have to be considered, including other dissemination biases, differences in the quality of smaller studies, the existence of true heterogeneity, and chance. Asymmetry in funnel plots may be an indicator that a more detailed investigation should be carried out on the presence of heterogeneity, such as sensitivity analysis.

Nevertheless, none of the meta-analyses evaluated in this study reported the use of sensitivity analysis. In the latest version of the software program generally used by Cochrane reviewers (RevMan, version 4.2.8, The Nordic Cochrane Centre, Rigshospitalet 2003), no formal test for the assessment of funnel plots is available. This software program permits the visual subjective interpretation of funnel plots, but such an approach has been shown to include a significant inter-observer variability [ 17 ]. These limitations may have restricted the use of this method in this selected sample of meta-analyses.

On the other hand, there is some concern regarding the evolution of effect size over time and the impact of including "old" trials in meta-analyses. In the past, it was possible that the determinants of data suppression and the intensity of publication bias were different when compared to those in current use. We observed that in one-third of meta-analyses, the most recent trials had been published prior to 1997 (8:21) and in more than half (11:21), the most recent trials had been published prior to 2000. In fact, it is unclear whether the date of publication would have any impact on meta-analyses in reproductive health, but caution should be taken when summarizing effects across older trials.

Rather than reviewing the conclusions of meta-analyses, the aim of this study was to provide evidence of publication bias and its consequences in selected reproductive health meta-analyses. Consequently, three examples of possible publication bias were identified. Following adjustment, a reduction in the discrepancy between the conclusions of larger trials and the conclusions of meta-analyses was seen (data not shown). In all three meta-analyses whose conclusions were not considered robust, their suggested post-adjustment conclusion was in agreement with those of the respective larger trials included. In the systematic review assessing interventions for emergency contraception [ 16 ], the meta-analysts adopted a different form of sensitivity analysis, by reanalyzing the data, and including only the trials that had adequate allocation concealment. Their findings were similar to the adjustment for funnel plot asymmetry or publication bias.

Implementing Best Practices in Reproductive Health. [ http://www.ibpinitiative.org/ ]

Villar J, Gülmezoglu AM, Khanna J, Carroli G, Hofmeyr GJ, Schulz K, Lumbiganon P: Evidence-based reproductive health in developing countries: The WHO Reproductive Health Library. 1999, Geneva, The World Health Organization, 2

Google Scholar  

Rothstein HR, Sutton AJ, Borenstein M: Publication Bias in Meta-analysis: prevention, assessment and adjustment. 2005, Chichester, UK:John Wiley & Sons, Ltd

Chapter   Google Scholar  

LeLorier J, Grégoire G, Benhaddad A, Lapierre J, Dederian F: Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med. 1997, 337: 536-42. 10.1056/NEJM199708213370806.

Article   CAS   PubMed   Google Scholar  

Oxman AD, Cook DJ, Guyatt GH: User's guide to the medical literature. JAMA. 1994, 272: 1367-71. 10.1001/jama.272.17.1367.

Maher D, Cook DJ, Eastwood S: Improving the quality of reports of meta-analysis of randomized controlled trials: the QUOROM statement. Lancet. 1999, 354: 1896-900. 10.1016/S0140-6736(99)04149-5.

Article   Google Scholar  

Stroup DF, Berlin JA, Morton SC: Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA. 2000, 283: 2008-12. 10.1001/jama.283.15.2008.

Higgins JPT, Green S, editors: Cochrane Handbook for Systematic Reviews of Interventions 4.2.5 [updated May 2005]. The Cochrane Library. 2005, Chichester, UK: John Wiley & Sons, Ltd, 3

Tallon D, Schneider M, Egger M: Quality of systematic reviews published in high impact general and specialist journals. Proceedings of 2nd symposium on systematic reviews: beyond the basics. 1999, Oxford

WHO: The Reproductive Health Library. 2006, Oxford: Update Software Ltd, [ http://www.rhlibrary.com ]

Duval S: The trim and fill Method. Publication Bias in Meta-analysis: prevention, assessment and adjustment. Edited by: Rothstein HR, Sutton AJ, Borenstein M. 2005, Chichester, UK: John Wiley & Sons, Ltd

Sutton AJ, Duval SJ, Tweedie RL, Abrams KR, Jones DR: Empirical assessment of effect of publication bias on meta-analysis. BMJ. 2000, 320: 1574-77. 10.1136/bmj.320.7249.1574.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Martin-Hirsch P, Jarvis G, Kitchener H, Lilford R: Collection devices for obtaining cervical cytology samples (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Hopkins L, Smaill F: Antibiotic prophylaxis regimens and drugs for cesarean section (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Hodnett ED, Gates S, Hofmeyr GJ, Sakala C: Continuous support for women during childbirth (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Cheng L, Gülmezoglu AM, Van Oel CJ, Piaggio G, Ezcurra E, Van Look PFA: Interventions for emergency contraception (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Villar J, Piaggio G, Carroli G, Donner A: Factors affecting the comparability of meta-analyses and largest trials results in perinatology. J Clin Epidemiol. 1997, 50: 997-1002. 10.1016/S0895-4356(97)00148-0.

Hofmeyr GJ: Amnioinfusion for meconium-stained liquor in labour (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Smaill F, Hofmeyr GJ: Antibiotic prophylaxis for cesarean section (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Smaill F: Antibiotics for asymptomatic bacteriuria in pregnancy (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Kenyon S, Boulvain M, Neilson J: Antibiotics for preterm rupture of membranes (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

McDonald H, Brocklehurst P, Parsons J: Antibiotics for treating bacterial vaginosis in pregnancy (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Abalos E, Duley L, Steyn DW, Henderson-Smart DJ: Antihypertensive drug therapy for mild to moderate hypertension during pregnancy (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Knight M, Duley L, Henderson-Smart DJ, King JF: Antiplatelet agents for preventing and treating pre-eclampsia (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

King JF, Flenady VJ, Papatsonis DNM, Dekker GA, Carbonne B: Calcium channel blockers for inhibiting preterm labour (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Atallah AN, Hofmeyr GJ, Duley L: Calcium supplementation during pregnancy for preventing hypertensive disorders and related problems (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Thacker SB, Stroup D, Chang M: Continuous electronic heart rate monitoring for fetal assessment during labor (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Crowley P: Interventions for preventing or improving the outcome of delivery at or beyond term (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Oates-Whitehead RM, Haas DM, Carrier JAK: Progestogen for preventing miscarriage (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Hodnett ED, Fredericks S: Support during pregnancy for women at increased risk of low birthweight babies (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Carroli G, Bergel E: Umbilical vein injection for management of retained placenta (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Johanson RB, Menon V: Vacuum extraction versus forceps for assisted vaginal delivery (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Hofmeyr GJ, Gülmezoglu AM: Vaginal misoprostol for cervical ripening and induction of labour (Cochrane Review). The Reproductive Health Library. 2006, Oxford: Update Software Ltd, (Reprinted from The Cochrane Library, Issue 1, 2006. Chichester, UK: John Wiley & Sons, Ltd.), [ http://www.rhlibrary.com ]9

Download references

Acknowledgements

The authors are pleased to thank Steven Tarlow for helping in data analysis.

Author information

Authors and affiliations.

Department of Obstetrics and Gynecology, School of Medical Sciences, University of Campinas, Brazil

João P Souza & José G Cecatti

Clinical Epidemiology Collaborative Group, Women's Integrated Health Care Center, University of Campinas, Brazil

João P Souza & Cynthia Pileggi

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to João P Souza .

Additional information

Competing interests.

The author(s) declare that they have no competing interests.

Authors' contributions

JPS participated in all the steps of the project, including the project development, data extraction, data analysis and writing the final report. CP participated in the project development and data extraction. JGC participated in the project development, data analysis and writing the final report. All authors provided suggestions for the manuscript, read it carefully, agreed on its content and approved the final version.

Electronic supplementary material

Additional file 1: appendix 1. filled funnel plots of 21 reproductive health meta-analyses. (doc 768 kb), authors’ original submitted files for images.

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2, authors’ original file for figure 3, rights and permissions.

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article.

Souza, J.P., Pileggi, C. & Cecatti, J.G. Assessment of funnel plot asymmetry and publication bias in reproductive health meta-analyses: an analytic survey. Reprod Health 4 , 3 (2007). https://doi.org/10.1186/1742-4755-4-3

Download citation

Received : 22 February 2007

Accepted : 16 April 2007

Published : 16 April 2007

DOI : https://doi.org/10.1186/1742-4755-4-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Publication Bias

Reproductive Health

  • Emergency Contraception
  • Cochrane Systematic Review
  • Analytic Survey

ISSN: 1742-4755

cochrane systematic review funnel plot

  • Open access
  • Published: 23 September 2013

The evolution of assessing bias in Cochrane systematic reviews of interventions: celebrating methodological contributions of the Cochrane Collaboration

  • Lucy Turner 1 ,
  • Isabelle Boutron 2 , 3 , 4 , 5 ,
  • Asbjørn Hróbjartsson 6 ,
  • Douglas G Altman 7 &
  • David Moher 1 , 8  

Systematic Reviews volume  2 , Article number:  79 ( 2013 ) Cite this article

16k Accesses

40 Citations

23 Altmetric

Metrics details

Peer Review reports

“…to manage large quantities of data objectively and effectively, standardized methods of appraising information should be included in review processes.

… By using these systematic methods of exploration, evaluation, and synthesis, the good reviewer can accomplish the task of advancing scientific knowledge”. Cindy Mulrow, 1986, BMJ.

The global evidence base for health care is extensive, and expanding; with nearly 2 million articles published annually. One estimate suggests 75 trials and 11 systematic reviews are published daily[ 1 ]. Research syntheses, in a variety of established and emerging forms, are well recognised as essential tools for summarising evidence with accuracy and reliability[ 2 ]. Systematic reviews provide health care practitioners, patients and policy makers with information to help make informed decisions. It is essential that those conducting systematic reviews are cognisant of the potential biases within primary studies and of how such biases could impact review results and subsequent conclusions.

Rigorous and systematic methodological approaches to conducting research synthesis emerged throughout the twentieth century with methods to identify and reduce biases evolving more recently[ 3 , 4 ]. The Cochrane Collaboration has made substantial contributions to the development of how biases are considered in systematic reviews and primary studies. Our objective within this paper is to review some of the landmark methodological contributions by members of the Cochrane Bias Methods Group (BMG) to the body of evidence which guides current bias assessment practices, and to outline the immediate and horizon objectives for future research initiatives.

Empirical works published prior to the establishment of the Cochrane Collaboration

In 1948, the British Medical Research Council published results of what many consider the first 'modern’ randomised trial[ 5 , 6 ]. Subsequently, the last 65 years has seen continual development of the methods used when conducting primary medical research aiming to reduce inaccuracy in estimates of treatment effects due to potential biases. A large body of literature has accumulated which supports how study characteristics, study reports and publication processes can potentially bias primary study and systematic review results. Much of the methodological research during the first 20 years of The Cochrane Collaboration has built upon that published before the Collaboration was founded. Reporting biases, or more specifically, publication bias and the influence of funding source(s) are not new concepts. Publication bias initially described as the file drawer problem as a bias concept in primary studies was early to emerge and has long been suspected in the social sciences[ 7 ]. In 1979 Rosenthal, a psychologist, described the issue in more detail[ 8 ] and throughout the 1980s and early 1990s an empirical evidence base began to appear in the medical literature[ 9 – 11 ]. Concurrent with the accumulation of early evidence, methods to detect and mitigate the presence of publication bias also emerged[ 12 – 15 ]. The 1980s also saw initial evidence of the presence of what is now referred to as selective outcome reporting[ 16 ] and research investigating the influence of source of funding on study results[ 10 , 11 , 17 , 18 ].

The importance of rigorous aspects of trial design (e.g. randomisation, blinding, attrition, treatment compliance) were known in the early 1980s[ 19 ] and informed the development by Thomas Chalmers and colleagues of a quality assessment scale to evaluate the design, implementation, and analysis of randomized control trials[ 20 ]. The pre-Cochrane era saw the early stages of assessing quality of included studies, with consideration of the most appropriate ways to assess bias. Yet, no standardised means for assessing risk of bias, or “quality” as it was referred to at the time, were implemented when The Cochrane Collaboration was established. The use of scales for assessing quality or risk of bias is currently explicitly discouraged in Cochrane reviews based on more recent evidence[ 21 , 22 ].

Methodological contributions of the Cochrane Collaboration: 1993 – 2013

In 1996, Moher and colleagues suggested that bias assessment was a new, emerging and important concept and that more evidence was required to identify trial characteristics directly related to bias[ 23 ]. Methodological literature pertaining to bias in primary studies published in the last 20 years has contributed to the evolution of bias assessment in Cochrane reviews. How bias is currently assessed has been founded on published studies that provide empirical evidence of the influence of certain study design characteristics on estimates of effect, predominately considering randomised controlled trials.

The publication of Ken Schulz’s work on allocation concealment, sequence generation, and blinding[ 24 , 25 ] the mid-1990s saw a change in the way the Collaboration assessed bias of included studies, and it was recommended that included studies were assessed in relation to how well the generated random sequence was concealed during the trial.

In 2001, the Cochrane Reporting Bias Methods Group now known as the Cochrane Bias Methods Group, was established to investigate how reporting and other biases influence the results of primary studies. The most substantial development in bias assessment practice within the Collaboration was the introduction of the Cochrane Risk of Bias (RoB) Tool in 2008. The tool was developed based on the methodological contributions of meta-epidemiological studies[ 26 , 27 ] and has since been evaluated and updated[ 28 ], and integrated into Grading of Recommendations Assessment, Development and Evaluation (GRADE)[ 29 ].

Throughout this paper we define bias as a systematic error or deviation in results or inferences from the truth[ 30 ] and should not be confused with “quality”, or how well a trial was conducted. The distinction between internal and external validity is important to review. When we describe bias we are referring to internal validity as opposed to the external validity or generalizability which is subject to demographic or other characteristics[ 31 ]. Here, we highlight landmark methodological publications which contribute to understanding how bias influences estimates of effects in Cochrane reviews (Figure  1 ).

figure 1

Timeline of landmark methods research [ 8 , 13 , 16 , 17 , 20 , 22 , 26 , 31 – 45 ] .

Sequence generation and allocation concealment

Early meta-epidemiological studies assessed the impact of inadequate allocation concealment and sequence generation on estimates of effect[ 24 , 25 ]. Evidence suggests that adequate or inadequate allocation concealment modifies estimates of effect in trials[ 31 ]. More recently, several other methodological studies have examined whether concealment of allocation is associated with magnitude of effect estimates in controlled clinical trials while avoiding confounding by disease or intervention[ 42 , 46 ].

More recent methodological studies have assessed the importance of proper generation of a random sequence in randomised clinical trials. It is now mandatory, in accordance with the Methodological Expectations for Cochrane Interventions Reviews (MECIR) conduct standards, for all Cochrane systematic reviews to assess potential selection bias (sequence generation and allocation concealment) within included primary studies.

Blinding of participants, personnel and outcome assessment

The concept of the placebo effect has been considered since the mid-1950s[ 47 ] and the importance of blinding trial interventions to participants has been well known, with the first empirical evidence published in the early 1980s[ 48 ]. The body of empirical evidence on the influence on blinding has grown since the mid-1990s, especially in the last decade, with some evidence highlighting that blinding is important for several reasons[ 49 ]. Currently, the Cochrane risk of bias tool suggests blinding of participants and personnel, and blinding of outcome assessment be assessed septely. Moreover consideration should be given to the type of outcome (i.e. objective or subjective outcome) when assessing bias, as evidence suggests that subjective outcomes are more prone to bias due to lack of blinding[ 42 , 44 ] As yet there is no empirical evidence of bias due to lack of blinding of participants and study personnel. However, there is evidence for studies described as 'blind’ or 'double-blind’, which usually includes blinding of one or both of these groups of people. In empirical studies, lack of blinding in randomized trials has been shown to be associated with more exaggerated estimated intervention effects[ 42 , 46 , 50 ].

Different people can be blinded in a clinical trial[ 51 , 52 ]. Study reports often describe blinding in broad terms, such as 'double blind’. This term makes it impossible to know who was blinded[ 53 ]. Such terms are also used very inconsistently[ 52 , 54 , 55 ] and the frequency of explicit reporting of the blinding status of study participants and personnel remains low even in trials published in top journals[ 56 ], despite explicit recommendations. Blinding of the outcome assessor is particularly important, both because the mechanism of bias is simple and foreseeable, and because evidence for bias is unusually clear[ 57 ]. A review of methods used for blinding highlights the variety of methods used in practice[ 58 ]. More research is ongoing within the Collaboration to consider the best way to consider the influence of lack of blinding within primary studies. Similar to selection bias, performance and detection bias are both mandatory components of risk of bias assessment in accordance with the MECIR standards.

Reporting biases

Reporting biases have long been identified as potentially influencing the results of systematic reviews. Bias arises when the dissemination of research findings is influenced by the nature and direction of results, there is still debate over explicit criteria for what constitutes a 'reporting bias’. More recently, biases arising from non-process related issues (i.e. source of funding, publication bias) have been referred to as meta-biases[ 59 ]. Here we discuss the literature which has emerged in the last twenty years with regards to two well established reporting biases, non-publication of whole studies (often simply called publication bias) and selective outcome reporting.

Publication bias

The last two decades have seen a large body of evidence of the presence of publication bias[ 60 – 63 ] and why authors fail to publish[ 64 , 65 ]. Given that it has long been recognized that investigators frequently fail to report their research findings[ 66 ], many more recent papers have been geared towards methods of detecting and estimating the effect of publication bias. An array of methods to test for publication bias and additional recommendations are now available[ 38 , 43 , 67 – 76 ], many of which have been evaluated[ 77 – 80 ]. Automatic generation of funnel plots have been incorporated when producing a Cochrane review and software (RevMan) and are encouraged for outcomes with more than ten studies[ 43 ].A thorough overview of methods is included in Chapter 10 of the Cochrane Handbook for Systematic Reviews of Interventions[ 81 ].

Selective outcome reporting

While the concept of publication bias has been well established, studies reporting evidence of the existence of selective reporting of outcomes in trial reports have appeared more recently[ 39 , 41 , 82 – 87 ]. In addition, some studies have investigated why some outcomes are omitted from published reports[ 41 , 88 – 90 ] as well as the impact of omission of outcomes on the findings of meta-analyses[ 91 ]. More recently, methods for evaluating selective reporting, namely, the ORBIT (Outcome Reporting Bias in Trials) classification system have been developed. One attempt to mitigate selective reporting is to develop field specific core outcome measures[ 92 ] the work of COMET (Core Outcome Measures in Effectiveness Trials) initiative[ 93 ] is supported by many members within the Cochrane Collaboration. More research is being conducted with regards to selective reporting of outcomes and selective reporting of trial analyses, within this concept there is much overlap with the movement to improve primary study reports, protocol development and trial registration.

Evidence on how to conduct risk of bias assessments

Often overlooked are the processes behind how systematic evaluations or assessments are conducted. In addition to empirical evidence of specific sources of bias, other methodological studies have led to changes in the processes used to assess risk of bias. One influential study published in 1999 highlighted the hazards of scoring 'quality’ of clinical trials when conducting meta-analysis and is one of reasons why each bias is assessed septely as 'high’, 'low’ or 'unclear’ risk rather than using a combined score[ 22 , 94 ]. Prior work investigated blinding of readers, data analysts and manuscript writers[ 51 , 95 ]. More recently, work has been completed to assess blinding of authorship and institutions in primary studies when conducting risk of bias assessments, suggesting that there is discordance in results between blind and unblinded RoB assessments. However uncertainty over best practice remains due to time and resources needed to implement blinding[ 96 ].

Complementary contributions

Quality of reporting and reporting guidelines.

Assessing primary studies for potential biases is a challenge[ 97 ]. During the early 1990s, poor reporting in randomized trials and consequent impediments to systematic review conduct, especially when conducting what is now referred to as 'risk of bias assessment’, were observed. In 1996, an international group of epidemiologists, statisticians, clinical trialists, and medical editors, some of whom were involved with establishing the Cochrane Collaboration, published the CONSORT Statement[ 32 ], a checklist of items to be addressed in a report of the findings of an RCT. CONSORT has twice been revised and updated[ 35 , 36 ] and over time, the impact of CONSORT has been noted, for example, CONSORT was considered one of the major milestones in health research methods over the last century by the Patient-Centered Outcomes Research Institute (PCORI)[ 98 ].

Issues of poor reporting extend far beyond randomized trials, and many groups have developed guidance to aid reporting of other study types. The EQUATOR Network’s library for health research reporting includes more than 200 reporting guidelines[ 99 ]. Despite evidence that the quality of reporting has improved over time, systemic issues with the clarity and transparency of reporting remain[ 100 , 101 ]. Such inadequacies in primary study reporting result in systematic review authors’ inability to assess the presence and extent of bias in primary studies and the possible impact on review results, continued improvements in trial reporting are needed to lead to more informed risk of bias assessments in systematic reviews.

Trial registration

During the 1980s and 1990s there were several calls to mitigate publication bias and selective reporting via trial registration[ 102 – 104 ]. After some resistance, in 2004, the BMJ and The Lancet reported that they would only publish registered clinical trials[ 105 ] with the International Committee of Medical Journal Editors making a statement to the same effect[ 40 ]. Despite the substantial impact of trial registration[ 106 ] uptake is still not optimal and it is not mandatory for all trials. A recent report indicated that only 22% of trials mandated by the FDA were reporting trial results on clinicaltrials.gov[ 107 ]. One study suggested that despite trial registration being strongly encouraged and even mandated in some jurisdictions only 45.5% of a sample of 323 trials were adequately registered[ 108 ].

Looking forward

Currently, there are three major ongoing initiatives which will contribute to how The Collaboration assesses bias. First, there has been some criticism of the Cochrane risk of bias tool[ 109 ] concerning its ease of use and reliability[ 110 , 111 ] and the tool is currently being revised. As a result, a working group is established to improve the format of the tool, with version 2.0 due to be released in 2014. Second, issues of study design arise when assessing risk of bias when including non-randomised studies in systematic reviews[ 112 – 114 ]. Even 10 years ago there were 114 published tools for assessing risk of bias in non-randomised studies[ 115 ]. An ongoing Cochrane Methods Innovation Fund project will lead to the release a tool for assessing non-randomized studies as well as tools for cluster and cross-over trials[ 116 ]. Third, selective reporting in primary studies is systemic[ 117 ] yet further investigation and adoption of sophisticated means of assessment remain somewhat unexplored by the Collaboration. A current initiative is ongoing to explore optimal ways to assess selective reporting within trials. Findings of this initiative will be considered in conjunction with the release of revised RoB tool and its extension for non-randomized studies.

More immediate issues

Given the increase in meta-epidemiological research, an explicit definition of evidence needed to identify study characteristics which may lead to bias (es) needs to be defined. One long debated issue is the influence of funders as a potential source of bias. In one empirical study, more than half of the protocols for industry-initiated trials stated that the sponsor either owns the data or needs to approve the manuscript, or both; none of these constraints were stated in any of the trial publications[ 118 ]. It is important that information about vested interests is collected and presented when relevant[ 119 ].

There is an on-going debate related to the risk of bias of trials stopping early because of benefit. A systematic review and a meta-epidemiologic study showed that such truncated RCTs were associated with greater effect sizes than RCTs not stopped early, particularly for trials with small sample size[ 120 , 121 ]. These results were widely debated and discussed[ 122 ] and recommendations related to this item are being considered.

In addition, recent meta-epidemiological studies of binary and continuous outcomes showed that treatment effect estimates in single-centre RCTs were significantly larger than in multicenter RCTs even after controlling for sample size[ 123 , 124 ]. The Bias in Randomized and Observational Studies (BRANDO) project combining data from all available meta-epidemiologic studies[ 44 ] found consistent results for subjective outcomes when comparing results from single centre and multi-centre trials. Several reasons may explain these differences between study results: small study effect, reporting bias, higher risk of bias in single centre studies, or factors related to the selection of the participants, treatment administration and care providers’ expertise. Further studies are needed to explore the role and effect of these different mechanisms.

Longer term issues

The scope of methodological research and subsequent contributions and evolution in bias assessment over the last 20 years has been substantial. However, there remains much work to be done, particularly in line with innovations in systematic review methodology itself. There is no standardised methodological approach to the conduct of systematic reviews. Subject to a given clinical question, it may be most appropriate to conduct a network meta-analysis, scoping review, a rapid review, or update any of these reviews. Along with the development of these differing types of review, there is the need for bias assessment methods to develop concurrently.

The way in which research synthesis is conducted may change further with technological advances[ 125 ]. Globally, there are numerous initiatives to establish integrated administrative databases which may open up new research avenues and methodological questions about assessing bias when primary study results are housed within such databases.

Despite the increase in meta-epidemiological research identifying study characteristics which could contribute to bias in studies, further investigation is needed. For example, as yet there has been little research on integration of risk of bias results into review findings. This is done infrequently and guidance on how to do it could be improved[ 126 ]. Concurrently, although some work has been done, little is known about how magnitude and direction in estimates of effect for a given bias and across biases for a particular trial and in turn, set of trials[ 127 ].

To summarise, there has been much research conducted to develop understanding of bias in trials and how these biases could influence the results of systematic reviews. Much of this work has been conducted since the Cochrane Collaboration was established either as a direct initiative of the Collaboration or thanks to the work of many affiliated individuals. There has been clear advancement in mandatory processes for assessing bias in Cochrane reviews. These processes, based on a growing body of empirical evidence have aimed to improve the overall quality of the systematic review literature, however, many areas of bias remain unexplored and as the evidence evolves, the processes used to assess and interpret biases and review results will also need to adapt.

Bastian H, Glasziou P, Chalmers I: Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?. PLoS Med. 2010, 7: e1000326-10.1371/journal.pmed.1000326.

Article   PubMed   PubMed Central   Google Scholar  

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009, 62: e1-e34. 10.1016/j.jclinepi.2009.06.006.

Article   PubMed   Google Scholar  

Hedges LV: Commentary. Statist Med. 1987, 6: 381-385. 10.1002/sim.4780060333.

Article   Google Scholar  

Chalmers I, Hedges LV, Cooper H: A brief history of research synthesis. Eval Health Prof. 2002, 25: 12-37. 10.1177/0163278702025001003.

Medical Research Council: STREPTOMYCIN treatment of pulmonary tuberculosis. Br Med J. 1948, 2: 769-782.

Hill AB: Suspended judgment. Memories of the British streptomycin trial in tuberculosis. The first randomized clinical trial. Control Clin Trials. 1990, 11: 77-79. 10.1016/0197-2456(90)90001-I.

Article   CAS   PubMed   Google Scholar  

Sterling TD: Publication decisions and their possible effects on inferences drawn from tests of significance - or vice versa. J Am Stat Assoc. 1959, 54: 30-34.

Google Scholar  

Rosenthal R: The file drawer problem and tolerance for null results. Psycholog Bull. 1979, 86: 638-641.

Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H: Publication bias and clinical trials. Control Clin Trials. 1987, 8: 343-353. 10.1016/0197-2456(87)90155-3.

Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR: Publication bias in clinical research. Lancet. 1991, 337: 867-872. 10.1016/0140-6736(91)90201-Y.

Dickersin K, Min YI, Meinert CL: Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. JAMA. 1992, 267: 374-378. 10.1001/jama.1992.03480030052036.

Light KE: Analyzing nonlinear scatchard plots. Science. 1984, 223: 76-78. 10.1126/science.6546323.

Begg CB, Berlin JA: Publication bias: a problem in interpreting medical data. J Royal Stat Soc Series A (Stat Soc). 1988, 151: 419-463. 10.2307/2982993.

Dear KBG, Begg CB: An approach for assessing publication bias prior to performing a meta-analysis. Stat Sci. 1992, 7: 237-245. 10.1214/ss/1177011363.

Hedges LV: Modeling publication selection effects in meta-analysis. Stat Sci. 1992, 7: 246-255. 10.1214/ss/1177011364.

Pocock SJ, Hughes MD, Lee RJ: Statistical problems in the reporting of clinical trials. A survey of three medical journals. N Engl J Med. 1987, 317: 426-432. 10.1056/NEJM198708133170706.

Hemminki E: Study of information submitted by drug companies to licensing authorities. Br Med J. 1980, 280: 833-836. 10.1136/bmj.280.6217.833.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Gotzsche PC: Multiple publication of reports of drug trials. Eur J Clin Pharmacol. 1989, 36: 429-432. 10.1007/BF00558064.

Kramer MS, Shapiro SH: Scientific challenges in the application of randomized trials. JAMA. 1984, 252: 2739-2745. 10.1001/jama.1984.03350190041017.

Chalmers TC, Smith H, Blackburn B, Silverman B, Schroeder B, Reitman D: A method for assessing the quality of a randomized control trial. Control Clin Trials. 1981, 2: 31-49. 10.1016/0197-2456(81)90056-8.

Emerson JD, Burdick E, Hoaglin DC, Mosteller F, Chalmers TC: An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. Control Clin Trials. 1990, 11: 339-352. 10.1016/0197-2456(90)90175-2.

Juni P, Witschi A, Bloch R, Egger M: The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999, 282: 1054-1060. 10.1001/jama.282.11.1054.

Moher D, Jadad AR, Tugwell P: Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care. 1996, 12: 195-208. 10.1017/S0266462300009570.

Schulz KF: Subverting randomization in controlled trials. JAMA. 1995, 274: 1456-1458. 10.1001/jama.1995.03530180050029.

Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995, 273: 408-412. 10.1001/jama.1995.03520290060030.

Naylor CD: Meta-analysis and the meta-epidemiology of clinical research. BMJ. 1997, 315: 617-619. 10.1136/bmj.315.7109.617.

Sterne JA, Juni P, Schulz KF, Altman DG, Bartlett C, Egger M: Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological’ research. Stat Med. 2002, 21: 1513-1524. 10.1002/sim.1184.

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD: The cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011, 343: d5928-10.1136/bmj.d5928.

Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P: GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008, 336: 924-926. 10.1136/bmj.39489.470347.AD.

Green S, Higgins J: Glossary. Cochrane handbook for systematic reviews of interventions 4.2. 5 [Updated May 2005]. 2009

Juni P, Altman DG, Egger M: Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ. 2001, 323: 42-46. 10.1136/bmj.323.7303.42.

Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I: Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA. 1996, 276: 637-639. 10.1001/jama.1996.03540080059030.

Cochrane AL: Effectiveness and efficiency: random reflections on health services. 1973

A proposal for structured reporting of randomized controlled trials. The standards of reporting trials group. JAMA. 1994, 272 (24): 1926-1931. 10.1001/jama.1994.03520240054041.

Moher D, Schulz KF, Altman DG: The CONSORT statement: revised recommendations for improving the quality of reports of pllel-group randomised trials. Lancet. 2001, 357: 1191-1194. 10.1016/S0140-6736(00)04337-3.

Schulz KF, Altman DG, Moher D: CONSORT 2010 statement: updated guidelines for reporting pllel group randomised trials. BMJ. 2010, 340: c332-10.1136/bmj.c332.

Schulz KF, Chalmers I, Altman DG, Grimes DA, Dore CJ: The methodologic quality of randomization as assessed from reports of trials in specialist and general medical journals. Online J Curr Clin Trials. 1995, 197: 81-

Egger M, Davey SG, Schneider M, Minder C: Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997, 315: 629-634. 10.1136/bmj.315.7109.629.

Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG: Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004, 291: 2457-2465. 10.1001/jama.291.20.2457.

De AC, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R: Clinical trial registration: a statement from the International Committee of Medical Journal Editors. N Engl J Med. 2004, 351: 1250-1251. 10.1056/NEJMe048225.

Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E: Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008, 3: e3081-10.1371/journal.pone.0003081.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008, 336: 601-605. 10.1136/bmj.39465.451748.AD.

Sterne JA, Sutton AJ, Ioannidis JP, Terrin N, Jones DR, Lau J: Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011, 343: d4002-10.1136/bmj.d4002.

Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J: Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med. 2012, 157: 429-438.

Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R: The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010, 340: c365-10.1136/bmj.c365.

Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC: Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007, 36: 847-857. 10.1093/ije/dym087.

Beecher HK: The powerful placebo. J Am Med Assoc. 1955, 159: 1602-1606. 10.1001/jama.1955.02960340022006.

Chalmers TC, Celano P, Sacks HS, Smith H: Bias in treatment assignment in controlled clinical trials. N Engl J Med. 1983, 309: 1358-1361. 10.1056/NEJM198312013092204.

Hróbjartsson A, Gøtzsche PC: Placebo interventions for all clinical conditions. Cochrane Database Syst Rev. 2010, 1:

Hrobjartsson A, Thomsen AS, Emanuelsson F, Tendal B, Hilden J, Boutron I: Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. BMJ. 2012, 344: e1119-10.1136/bmj.e1119.

Gotzsche PC: Blinding during data analysis and writing of manuscripts. Control Clin Trials. 1996, 17: 285-290. 10.1016/0197-2456(95)00263-4.

Haahr MT, Hrobjartsson A: Who is blinded in randomized clinical trials? A study of 200 trials and a survey of authors. Clin Trials. 2006, 3: 360-365.

PubMed   Google Scholar  

Schulz KF, Chalmers I, Altman DG: The landscape and lexicon of blinding in randomized trials. Ann Intern Med. 2002, 136: 254-259. 10.7326/0003-4819-136-3-200202050-00022.

Devereaux PJ, Manns BJ, Ghali WA, Quan H, Lacchetti C, Montori VM: Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials. JAMA. 2001, 285: 2000-2003. 10.1001/jama.285.15.2000.

Boutron I, Estellat C, Ravaud P: A review of blinding in randomized controlled trials found results inconsistent and questionable. J Clin Epidemiol. 2005, 58: 1220-1226. 10.1016/j.jclinepi.2005.04.006.

Montori VM, Bhandari M, Devereaux PJ, Manns BJ, Ghali WA, Guyatt GH: In the dark: the reporting of blinding status in randomized controlled trials. J Clin Epidemiol. 2002, 55: 787-790. 10.1016/S0895-4356(02)00446-8.

Hróbjartsson A, Thomsen ASS, Emanuelsson F, Tendal B, Hilden J, Boutron I: Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors. Can Med Assoc J. 2013, 185: E201-E211. 10.1503/cmaj.120744.

Boutron I, Estellat C, Guittet L, Dechartres A, Sackett DL, Hrobjartsson A: Methods of blinding in reports of randomized controlled trials assessing pharmacologic treatments: a systematic review. PLoS Med. 2006, 3: e425-10.1371/journal.pmed.0030425.

Goodman S, Dickersin K: Metabias: a challenge for comptive effectiveness research. Ann Intern Med. 2011, 155: 61-62. 10.7326/0003-4819-155-1-201107050-00010.

Sterling TD, Rosenbaum WL, Weinkam JJ: Publication decisions revisited: the effect of the outcome of statistical tests on the decision to publish and vice versa. Am Stat. 1995, 49: 108-112.

Moscati R, Jehle D, Ellis D, Fiorello A, Landi M: Positive-outcome bias: comparison of emergency medicine and general medicine literatures. Acad Emerg Med. 1994, 1: 267-271.

Liebeskind DS, Kidwell CS, Sayre JW, Saver JL: Evidence of publication bias in reporting acute stroke clinical trials. Neurology. 2006, 67: 973-979. 10.1212/01.wnl.0000237331.16541.ac.

Carter AO, Griffin GH, Carter TP: A survey identified publication bias in the secondary literature. J Clin Epidemiol. 2006, 59: 241-245. 10.1016/j.jclinepi.2005.08.011.

Weber EJ, Callaham ML, Wears RL, Barton C, Young G: Unpublished research from a medical specialty meeting: why investigators fail to publish. JAMA. 1998, 280: 257-259. 10.1001/jama.280.3.257.

Dickersin K, Min YI: NIH clinical trials and publication bias. Online J Curr Clin Trials. 1993, 50: 4967-

Dickersin K: How important is publication bias? A synthesis of available data. AIDS Educ Prev. 1997, 9: 15-21.

CAS   PubMed   Google Scholar  

Vevea JL, Hedges LV: A general linear model for estimating effect size in the presence of publication bias. Psychometrika. 1995, 60: 419-435. 10.1007/BF02294384.

Taylor SJ, Tweedie RL: Practical estimates of the effect of publication bias in meta-analysis. Aust Epidemiol. 1998, 5: 14-17.

CAS   Google Scholar  

Givens GH, Smith DD, Tweedie RL: Publication bias in meta-analysis: a Bayesian data-augmentation approach to account for issues exemplified in the passive smoking debate. Stat Sci. 1997, 221-240.

Duval S, Tweedie R: Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. 2000, 56: 455-463. 10.1111/j.0006-341X.2000.00455.x.

Sterne JA, Egger M: Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. J Clin Epidemiol. 2001, 54: 1046-1055. 10.1016/S0895-4356(01)00377-8.

Terrin N, Schmid CH, Lau J, Olkin I: Adjusting for publication bias in the presence of heterogeneity. Stat Med. 2003, 22: 2113-2126. 10.1002/sim.1461.

Schwarzer G, Antes G, Schumacher M: A test for publication bias in meta-analysis with sparse binary data. Stat Med. 2007, 26: 721-733. 10.1002/sim.2588.

Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L: Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J Clin Epidemiol. 2008, 61: 991-996. 10.1016/j.jclinepi.2007.11.010.

Moreno SG, Sutton AJ, Ades AE, Stanley TD, Abrams KR, Peters JL: Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Med Res Methodol. 2009, 9: 2-10.1186/1471-2288-9-2.

Moreno SG, Sutton AJ, Turner EH, Abrams KR, Cooper NJ, Palmer TM: Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications. BMJ. 2009, 339: b2981-10.1136/bmj.b2981.

Terrin N, Schmid CH, Lau J: In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol. 2005, 58: 894-901. 10.1016/j.jclinepi.2005.01.006.

Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L: Comparison of two methods to detect publication bias in meta-analysis. JAMA. 2006, 295: 676-680. 10.1001/jama.295.6.676.

Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L: Performance of the trim and fill method in the presence of publication bias and between-study heterogeneity. Stat Med. 2007, 26: 4544-4562. 10.1002/sim.2889.

Ioannidis JP, Trikalinos TA: The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey. CMAJ. 2007, 176: 1091-1096. 10.1503/cmaj.060410.

Sterne JAC, Egger M, Moher D: Chapter 10: addressing reporting biases. Cochrane handbook for systematic reviews of intervention. Version 5.1.0 (Updated march 2011) edition. Edited by: Higgins JPT, Green S. 2011, The Cochrane Collaboration

Hutton JL, Williamson PR: Bias in metaGÇÉanalysis due to outcome variable selection within studies. J Royal Stat Soc : Series C (Appl Stat). 2002, 49: 359-370.

Hahn S, Williamson PR, Hutton JL: Investigation of within-study selective reporting in clinical research: follow-up of applications submitted to a local research ethics committee. J Eval Clin Pract. 2002, 8: 353-359. 10.1046/j.1365-2753.2002.00314.x.

Chan AW, Krleza-Jeric K, Schmid I, Altman DG: Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ. 2004, 171: 735-740. 10.1503/cmaj.1041086.

von Elm E, Rollin A, Blumle A, Senessie C, Low N, Egger M: Selective reporting of outcomes of drug trials. Comparison of study protocols and published articles. 2006

Furukawa TA, Watanabe N, Omori IM, Montori VM, Guyatt GH: Association between unreported outcomes and effect size estimates in Cochrane meta-analyses. JAMA. 2007, 297: 468-470.

Page MJ, McKenzie JE, Forbes A: Many scenarios exist for selective inclusion and reporting of results in randomized trials and systematic reviews. J Clin Epidemiol. 2013

Chan AW, Altman DG: Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ. 2005, 330: 753-10.1136/bmj.38356.424606.8F.

Smyth RM, Kirkham JJ, Jacoby A, Altman DG, Gamble C, Williamson PR: Frequency and reasons for outcome reporting bias in clinical trials: interviews with trialists. BMJ. 2011, 342: c7153-10.1136/bmj.c7153.

Dwan K, Gamble C, Williamson PR, Kirkham JJ: Systematic review of the empirical evidence of study publication bias and outcome reporting BiasGÇöAn updated review. PloS one. 2013, 8: e66844-10.1371/journal.pone.0066844.

Williamson PR, Gamble C, Altman DG, Hutton JL: Outcome selection bias in meta-analysis. Stat Methods Med Res. 2005, 14: 515-524. 10.1191/0962280205sm415oa.

Williamson P, Altman D, Blazeby J, Clarke M, Gargon E: Driving up the quality and relevance of research through the use of agreed core outcomes. J Health Serv Res Policy. 2012, 17: 1-2.

The COMET Initiative: http://www.comet-initiative.org/ , Last accessed 19th September 2013

Greenland S, O’Rourke K: On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions. Biostat. 2001, 2: 463-471. 10.1093/biostatistics/2.4.463.

Article   CAS   Google Scholar  

Berlin JA: Does blinding of readers affect the results of meta-analyses? University of Pennsylvania Meta-analysis Blinding Study Group. Lancet. 1997, 350: 185-186.

Morissette K, Tricco AC, Horsley T, Chen MH, Moher D: Blinded versus unblinded assessments of risk of bias in studies included in a systematic review. Cochrane Database Syst Rev. 2011, MR000025-

Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M: Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?. Lancet. 1998, 352: 609-613. 10.1016/S0140-6736(98)01085-X.

Gabriel SE, Normand SL: Getting the methods right–the foundation of patient-centered outcomes research. N Engl J Med. 2012, 367: 787-790. 10.1056/NEJMp1207437.

Simera I, Moher D, Hoey J, Schulz KF, Altman DG: A catalogue of reporting guidelines for health research. Eur J Clin Invest. 2010, 40: 35-53. 10.1111/j.1365-2362.2009.02234.x.

Hopewell S, Dutton S, Yu LM, Chan AW, Altman DG: The quality of reports of randomised trials in 2000 and 2006: comptive study of articles indexed in PubMed. BMJ. 2010, 340: c723-10.1136/bmj.c723.

Turner L, Shamseer L, Altman DG, Weeks L, Peters J, Kober T: Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev. 2012, 11: MR000030

Simes RJ: Publication bias: the case for an international registry of clinical trials. J Clin Oncol. 1986, 4: 1529-1541.

Moher D: Clinical-trial registration: a call for its implementation in Canada. CMAJ. 1993, 149: 1657-1658.

CAS   PubMed   PubMed Central   Google Scholar  

Horton R, Smith R: Time to register randomised trials. The case is now unanswerable. BMJ. 1999, 319: 865-866. 10.1136/bmj.319.7214.865.

Abbasi K: Compulsory registration of clinical trials. BMJ. 2004, 329: 637-638. 10.1136/bmj.329.7467.637.

Moja LP, Moschetti I, Nurbhai M, Compagnoni A, Liberati A, Grimshaw JM: Compliance of clinical trial registries with the World Health Organization minimum data set: a survey. Trials. 2009, 10: 56-10.1186/1745-6215-10-56.

Prayle AP, Hurley MN, Smyth AR: Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: cross sectional study. BMJ. 2012, 344: d7373-10.1136/bmj.d7373.

Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P: Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 2009, 302: 977-984. 10.1001/jama.2009.1242.

Higgins JPT, Altman DG, Sterne JAC: Chapter 8: assessing risk of bias in included studies. Cochrane handbook for systematic reviews of interventions. Version 5.1.0 (Updated march 2011) edition. Edited by: Higgins JPT, Green S. 2011, The Cochrane Collaboration

Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Krebs SJ: Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009, 339: b4012-10.1136/bmj.b4012.

Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M: Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol. 2012

Higgins JP, Ramsay C, Reeves B, Deeks JJ, Shea B, Valentine J: Issues relating to study design and risk of bias when including non-randomized studies in systematic reviews on the effects of interventions. Res Syn Methods. 2012, 10.1002/jrsm.1056

Norris SL, Moher D, Reeves B, Shea B, Loke Y, Garner S: Issues relating to selective reporting when including non-randomized studies in systematic reviews on the effects of healthcare interventions. Research Synthesis Methods. 2012, 10.1002/jrsm.1062

Valentine J, Thompson SG: Issues relating to confounding and meta-analysis when including non-randomized studies in systematic reviews on the effects of interventions. Res Syn Methods. 2012, 10.1002/jrsm.1064

Deeks JJ, Dinnes J, D’Amico R, Sowden AJ, Sakarovitch C, Song F: Evaluating non-randomised intervention studies. Health Technol Assess. 2003, 7: iii-173.

Chandler J, Clarke M, Higgins J: Cochrane methods. Cochrane Database Syst Rev. 2012, 1-56. Suppl 1

Dwan K, Altman DG, Cresswell L, Blundell M, Gamble CL, Williamson PR: Comparison of protocols and registry entries to published reports for randomised controlled trials. Cochrane Database Syst Rev. 2011, MR000031-

Gotzsche PC, Hrobjartsson A, Johansen HK, Haahr MT, Altman DG, Chan AW: Constraints on publication rights in industry-initiated clinical trials. JAMA. 2006, 295: 1645-1646.

Roseman M, Turner EH, Lexchin J, Coyne JC, Bero LA, Thombs BD: Reporting of conflicts of interest from drug trials in Cochrane reviews: cross sectional study. BMJ. 2012, 345: e5155-10.1136/bmj.e5155.

Bassler D, Briel M, Montori VM, Lane M, Glasziou P, Zhou Q: Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA. 2010, 303: 1180-1187. 10.1001/jama.2010.310.

Montori VM, Devereaux PJ, Adhikari NK, Burns KE, Eggert CH, Briel M: Randomized trials stopped early for benefit: a systematic review. JAMA. 2005, 294: 2203-2209. 10.1001/jama.294.17.2203.

Goodman S, Berry D, Wittes J: Bias and trials stopped early for benefit. JAMA. 2010, 304: 157-159.

Dechartres A, Boutron I, Trinquart L, Charles P, Ravaud P: Single-center trials show larger treatment effects than multicenter trials: evidence from a meta-epidemiologic study. Ann Intern Med. 2011, 155: 39-51. 10.7326/0003-4819-155-1-201107050-00006.

Bafeta A, Dechartres A, Trinquart L, Yavchitz A, Boutron I, Ravaud P: Impact of single centre status on estimates of intervention effects in trials with continuous outcomes: meta-epidemiological study. BMJ. 2012, 344: e813-10.1136/bmj.e813.

Ip S, Hadar N, Keefe S, Parkin C, Iovin R, Balk EM: A Web-based archive of systematic review data. Syst Rev. 2012, 1: 15-10.1186/2046-4053-1-15.

Hopewell S, Boutron I, Altman DG, Ravaud P: Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomized trials: a cross-sectional review. Journal TBD. in press

Turner RM, Spiegelhalter DJ, Smith GC, Thompson SG: Bias modelling in evidence synthesis. J R Stat Soc Ser A Stat Soc. 2009, 172: 21-47. 10.1111/j.1467-985X.2008.00547.x.

Download references

Acknowledgements

We would like to thank the Canadian Institutes of Health Research for their long-term financial support (2005–2015)-CIHR Funding Reference Number—CON-105529 this financing has enabled much of the progress of the group in the last decade. We would also like to thank Jodi Peters for help preparing the manuscript, Jackie Chandler for coordinating this paper as one of the series. We would also like to thank the Bias Methods Group membership for their interest and involvement with the group. We would especially like to thank Matthias Egger and Jonathan Sterne, previous BMG convenors, and Julian Higgins, Jennifer Tetzlaff and Laura Weeks for their extensive contributions to the group.

Cochrane awards received by BMG members

The Cochrane BMG currently has a membership of over 200 statisticians, clinicians, epidemiologists and researchers interested in issues of bias in systematic reviews. For a full list of BMG members, please login to Archie or visit http://www.bmg.cochrane.org for contact information.

Bill Silverman prize recipients

2009 - Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine 2007 4(3): e78. doi: http://dx.doi.org/10.1371/journal.pmed.0040078 [full-text PDF].

Thomas C. Chalmers award recipients

2001 (tie) - Henry D, Moxey A, O’Connell D. Agreement between randomised and non-randomised studies - the effects of bias and confounding [abstract]. Proceedings of the Ninth Cochrane Colloquium, 2001.

2001 (runner-up) - Full publication: Sterne JAC, Jüni P, Schulz KF, Altman DG, Bartlett C, and Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in “meta-epidemiological” research. Stat Med 2002;21:1513–1524.

2010 - Kirkham JJ, Riley R, Williamson P. Is multivariate meta-analysis a solution for reducing the impact of outcome reporting bias in systematic reviews? [abstract] Proceedings of the Eighteenth Cochrane Colloquium, 2010.

2012 - Page MJ, McKenzie JE, Green SE, Forbes A. Types of selective inclusion and reporting bias in randomised trials and systematic reviews of randomised trials [presentation]. Proceedings of the Twentieth Cochrane Colloquium, 2012.

Author information

Authors and affiliations.

Ottawa Hospital Research Institute, Ottawa, Canada

Lucy Turner & David Moher

INSERM, U738, Paris, France

Isabelle Boutron

Hôpital Hôtel Dieu, Centre, d’Epidémiologie Clinique, Paris, France

Faculté de Médecine, University of Paris Descartes, Sorbonne, Paris Cité, Paris, France

French Cochrane Centre, Paris, France

Nordic Cochrane Centre, Rigshospitalet, Copenhagen, Denmark

Asbjørn Hróbjartsson

Centre for Statistics in Medicine, University of Oxford, Oxford, UK

Douglas G Altman

Department of Epidemiology and Community Medicine, University of Ottawa, Ottawa, Canada

David Moher

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Lucy Turner .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors’ contributions

This paper was invited for submission by the Cochrane Bias Methods Group. AH and IB conceived of the initial outline of the manuscript, LT collected information and references and drafted the manuscript, DM reviewed the manuscript and provided guidance on structure, DM, DGA, IB and AH all provided feedback on the manuscript and suggestions of additional literature. All authors read and approved the final manuscript.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Rights and permissions.

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article.

Turner, L., Boutron, I., Hróbjartsson, A. et al. The evolution of assessing bias in Cochrane systematic reviews of interventions: celebrating methodological contributions of the Cochrane Collaboration. Syst Rev 2 , 79 (2013). https://doi.org/10.1186/2046-4053-2-79

Download citation

Received : 01 May 2013

Accepted : 11 September 2013

Published : 23 September 2013

DOI : https://doi.org/10.1186/2046-4053-2-79

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Publication Bias
  • Primary Study
  • Cochrane Collaboration
  • Bias Assessment
  • Selective Reporting

Systematic Reviews

ISSN: 2046-4053

  • Submission enquiries: Access here and click Contact Us
  • General enquiries: [email protected]

cochrane systematic review funnel plot

Bias in meta-analysis detected by a simple, graphical test

Affiliation.

  • 1 Department of Social Medicine, University of Bristol. [email protected]
  • PMID: 9310563
  • PMCID: PMC2127453
  • DOI: 10.1136/bmj.315.7109.629

Objective: Funnel plots (plots of effect estimates against sample size) may be useful to detect bias in meta-analyses that were later contradicted by large trials. We examined whether a simple test of asymmetry of funnel plots predicts discordance of results when meta-analyses are compared to large trials, and we assessed the prevalence of bias in published meta-analyses.

Design: Medline search to identify pairs consisting of a meta-analysis and a single large trial (concordance of results was assumed if effects were in the same direction and the meta-analytic estimate was within 30% of the trial); analysis of funnel plots from 37 meta-analyses identified from a hand search of four leading general medicine journals 1993-6 and 38 meta-analyses from the second 1996 issue of the Cochrane Database of Systematic Reviews.

Main outcome measure: Degree of funnel plot asymmetry as measured by the intercept from regression of standard normal deviates against precision.

Results: In the eight pairs of meta-analysis and large trial that were identified (five from cardiovascular medicine, one from diabetic medicine, one from geriatric medicine, one from perinatal medicine) there were four concordant and four discordant pairs. In all cases discordance was due to meta-analyses showing larger effects. Funnel plot asymmetry was present in three out of four discordant pairs but in none of concordant pairs. In 14 (38%) journal meta-analyses and 5 (13%) Cochrane reviews, funnel plot asymmetry indicated that there was bias.

Conclusions: A simple analysis of funnel plots provides a useful test for the likely presence of bias in meta-analyses, but as the capacity to detect bias will be limited when meta-analyses are based on a limited number of small trials the results from such analyses should be treated with considerable caution.

Publication types

  • Research Support, U.S. Gov't, Non-P.H.S.
  • Meta-Analysis as Topic*
  • Randomized Controlled Trials as Topic
  • Regression Analysis
  • Statistics as Topic
  • Treatment Outcome

Banner

Systematic Reviews: How to Analyze a Systematic Review

  • How to Analyze a Systematic Review
  • Locating Sources
  • Other Literature Sources
  • Citations and Organization
  • Partnering with a Librarian

Evaluating a Systematic Review and/or Meta-analysis

Like every other type of article, systematic reviews and meta-analysis need to be evaluated and critiqued to checked for bias. You never use other systematic reviews in your own systematic review, but it is important to check for previous systematic reviews and meta-analysis on your topic. This not only gives you some ideas on criteria (such as limiting dates to only include more recent studies), but can also give you an idea of what a systematic review needs in order to be considered a quality systematic review. 

In order to evaluate a systematic review or meta-analysis, you need to understand what to look for. Below are some common graphs and charts that are typically included in a systematic review and/or meta-analysis.

The CASP Checklist is a checklist of 10 questions to help you make sense of a Systematic Review. This checklist is designed to help you think systematically and critically about systematic reviews and meta-analyses.

  • CASP Checklist

Forest Plots

Forest plots are visual summaries of essential information found in a meta-analysis. Each study included in the MA is plotted with its own line and includes the size of the intervention group and control group, the confidence interval, and the weight. These are all plotted against the Line of No Effect (either 0 or 1). 

undefined

Higgins JPT, Thomas J, Chandler J, et al., eds.  Cochrane Handbook for Systematic Reviews of Interventions . 2nd ed. Hoboken, NJ: Wiley-Blackwell|; 2019.

Above is an example of a forest plot from the Cochrane Handbook for Systematic Reviews of Interventions . In order to read this, you'll need to understand how forest plots are built.

  • Each study in the meta-analysis is represented as its own line on the plot, usually denoted by the author and year of the study.
  • The tick mark represents the confidence interval of the study, so the longer the horizontal line, the larger the CI.
  • The box or dot on that line represents the sample size; big box is a higher sample size, small box is a smaller sample size.
  • The diamond at the bottom is the representation of the combined CIs and sample size (this is calculated during the meta-analysis process).
  • The vertical line is called the Line of No Effect. To the right of the line is the intervention side; the left is the no intervention/placebo side. Studies that fall on the intervention side of the plot show that the intervention was effective. Studies on the placebo side did not have findings supporting the efficacy of the intervention. The Line of No Effect can either start at 0 or 1. This is determined during the statistical calculations of the meta-analysis. 

An important part of understanding a forest plot is understanding what the combined results show. If the diamond representing the combined results falls on the right of the Line of No Effect, the meta-analysis finds the interventions to be effective. If it touches the Line of No Effect or is completely on the left side (no intervention), the meta-analysis finds that the intervention was not effective, or is not statistically significant enough to make that determination. 

Another thing that a forest plot can show is whether or not the studies are homogenous. In order to do a quality meta-analysis, the studies must be homogenous. If they are not, it will appear on a forest plot. In order to check this, see if you can draw a line through all the tick marks (CIs) of all the studies. If you can, the meta-analysis is using homogenous studies. 

PRISMA Flow Chart

undefined

Yang C, Deng S. Laparoscopic versus open mesh repair for the treatment of recurrent inguinal hernia: a systematic review and meta-analysis. Annals of Palliative Medicine   2020 ;9(3):1164-1173. doi:10.21037/apm-20-968

Funnel Plots

Funnel plots are the graphical representation of the size of trials plotted against the effect size they report (Lee & Hatopf, 2012). They are scatter plots of the intervention effect estimates from a study against the study's size or precision (Cochrane Handbook, 2019). Chapter 13 of the Cochrane Handbook for Systematic Reviews of Interventions explains funnel plots in more detail. 

undefined

Higgins JPT, Thomas J, Chandler J, et al., eds. Cochrane Handbook for Systematic Reviews of Interventions . 2nd ed. Hoboken, NJ: Wiley-Blackwell|; 2019.

Risk of Bias Graphs

Risk of Bias tables summarize the judgment of bias for the studies used in the systematic review. Each study's judgments for types of bias are represented with colored dots and symbols. Green + dots denote low risk bias in an area, red - dots denote high risk of bias, and yellow ? dots denote unclear of unknown risk of bias. The types of bias included in these tables are determined by the researchers and what areas they believe to be most important within the context of the review. 

undefined

  • << Previous: Home
  • Next: Resources >>
  • Last Updated: Jun 20, 2023 1:56 PM
  • URL: https://library.kansascity.edu/systematicreviews

IMAGES

  1. cochrane systematic review funnel plot

    cochrane systematic review funnel plot

  2. cochrane systematic review funnel plot

    cochrane systematic review funnel plot

  3. Funnel Plot (example from Cochrane Handbook 1 )

    cochrane systematic review funnel plot

  4. cochrane systematic review funnel plot

    cochrane systematic review funnel plot

  5. cochrane systematic review funnel plot

    cochrane systematic review funnel plot

  6. cochrane systematic review funnel plot

    cochrane systematic review funnel plot

VIDEO

  1. Line Plot Sampling dan Uniform Systematic Distribution Sampling Peta Manual

  2. Abaft The Funnel by Rudyard Kipling

  3. Forest Plot vs Funnel plot #research #metaanalysis #systematicreview #publicationbias #analysis

  4. How to use our funnel plot tool

  5. Part 2: Search, study selection, data extraction, risk of bias assessment, analysis

  6. Cochrane Review mentioned on ER

COMMENTS

  1. 10.4.1 Funnel plots

    10.4.1 Funnel plots · A funnel plot is a simple scatter plot of the intervention effect estimates from individual studies against some measure of each study's

  2. 10.4.3.1 Recommendations on testing for funnel plot asymmetry

    Based on a survey of meta-analyses published in the Cochrane Database of Systematic Reviews, these criteria imply that tests for funnel plot asymmetry

  3. Funnel plots for detecting bias in meta-analysis

    E Nuesch, S Trelle, S Reichenbach, AW Rutjes, B Tschannen, DG Altman, M Egger, P Juni. BMJ (Clinical research ed.), 2010, 341, c3515 | added to CENTRAL: 30

  4. Assessment of funnel plot asymmetry and publication bias in

    Next amendments in Cochrane systematic reviews should include this type of evaluation. Further studies regarding the evolution of effect size

  5. The evolution of assessing bias in Cochrane systematic reviews of

    Automatic generation of funnel plots have been incorporated when producing a Cochrane review and software (RevMan) and are encouraged for

  6. Bias in meta-analysis detected by a simple, graphical test

    reviews from the Cochrane Database of Systematic Reviews x Critical

  7. Bias in meta-analysis detected by a simple, graphical test

    ... Cochrane Database of Systematic Reviews. Main outcome measure: Degree of funnel plot asymmetry as measured by the intercept from regression of standard

  8. How to Analyze a Systematic Review

    Chapter 13 of the Cochrane Handbook for Systematic Reviews of Interventions explains funnel plots in more detail. undefined. Higgins JPT, Thomas

  9. Typical funnel plot generated by RevMan showing small study bias

    Methods A literature review was performed using the databases MEDLINE, the Cochrane Database, Embase, Chinese National Knowledge Infrastructure (CNKI) and

  10. (PDF) Recommendations for examining and interpreting funnel plot

    ... Cochrane Handbook for Systematic Reviews. of Interventions.7. What is a funnel plot? A funnel plot is a scatter plot of the effect estimates from. individual

  11. Contour-enhanced meta-analysis funnel plots help distinguish

    [12] Cochrane Database of Systematic Reviews. In: The Cochrane Collab