Quality of evidence for bias from attributes of study design

Abstract text
Background Several meta-epidemiological studies have been published that examine the relationship between attributes of study design and bias. The findings from these studies show some variation, but generally favor common assumptions of smaller effect sizes for rigorous design.
Objectives The goal of this systematic review of meta-epidemiological studies was to evaluate the quality of the literature describing the impact of study design on the validity of study results and to assess the agreement between stated conclusions and the data provided.
Methods A search of Medline, Embase, Web of Science, Cochrane methodology register, and reference lists for meta-epidemiological studies was conducted. References were selected by predefined criteria and appraised with an adjusted Amstar tool. We examined the following study characteristics or quality measures: randomization, allocation concealment, blinding, study size, and sponsoring. The directions of conclusions were categorized as larger effect, smaller effect, or no effect from the presence of the quality measure. Effect sizes and significance or confidence intervals were also extracted. All steps were performed in duplicate. The agreement between available data and the direction of conclusions were analyzed with Chi2 test.
Results We found 43 meta-epidemiological studies that together stated 63 conclusions about the effect of one or more of the defined quality measures. From 22 of these studies we were able to extract 59 effect estimates of the quality measures. Eleven of these were statistically significant. Only 28/63 or 44% of conclusions were supported by the data. A conclusion of no effect was more often supported by data than a conclusion for a smaller effect from the presence of a quality measure (p<0.01).
Conclusions Most of the conclusions in meta-epidemiological studies are not properly supported by data, especially in cases where conclusions supported the common paradigm of smaller effects in studies that apply generally accepted quality measures.
Jacobs W1, Sandhu L2, Kruyt M3, Moojen W1, Oner C3
1 Department of Neurosurgery, Leiden University Medical Center (LUMC), The Netherlands
2 Division of General Surgery, Department of Surgery, University of Toronto, Toronto, Canada
3 Department of Orthopedic Surgery, University Medical Center Utrecht (UMCU), Utrecht, Netherlands
Presenting author and contact person
Presenting author: 
Wilco Jacobs
Contact person Affiliation Country
Wilco Jacobs (Contact this person) Leiden University Medical Center / University Medical Center Utrecht Netherlands
Date and Location
Oral session A4O1
Monday 1 October 2012 - 11:00 - 11:20