The Grading of Recommendations Assessment, Development and Evaluation Reliability Study (the GRADERS)

Abstract text
The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach has been widely adopted for summarizing, grading and presenting evidence by systematic reviewers and guideline developers.
1. Evaluate the inter-rater reliability of assessing quality of evidence (QoE) using the GRADE approach.
2. Evaluate the effect of assessing QoE in duplicate on the reliability of this approach.
All participants completed two training exercises prior to the study. Participants worked independently as single raters initially to assess the QoE of 4 outcomes from 4 systematic reviews. After recording their initial impression of QoE using a global rating on a visual analog scale, raters graded the QoE following the GRADE approach. Subsequently, we randomly paired raters and asked them to submit a consensus QoE rating. Investigators, data abstractors and data analysts were unaware of raters’ identification. We used generalizability theory to calculate a reliability coefficient.
Fifteen volunteers from the GRADE working group and 10 from the Health Research Methodology graduate program at McMaster University participated in the study. Members of the GRADE working group had more experience with GRADE approach at baseline.
The inter-rater reliability of the GRADE approach in assessing the QoE when single raters evaluated the body of evidence was 0.66 among the HRM students and 0.72 among members of the GRADE working group. The inter-rater reliability of a global rating of QoE using VAS (without using the GRADE approach) was lower – 0.31 and 0.27 respectively. The inter-rater reliability did not improve when QoE was assessed by pairs of raters reaching consensus on the final rating.
Our findings support the presumption that using GRADE approach by trained individuals is more reliable than intuitive judgments about the QoE. The results also suggest that single raters are sufficient to reliably assess the QoE using GRADE system.
Mustafa R1, Santesso N1, Brozek J1, Akl E2, Schunemann H1
1 McMaster University, Hamilton, Canada.
2 SUNY at Buffalo, NY, USA. McMaster University, Hamilton, Canada.
Presenting author and contact person
Presenting author: 
Reem Mustafa
Contact person Affiliation Country
Reem Mustafa (Contact this person) McMaster University Canada
Date and Location
Oral session C12O4
Wednesday 3 October 2012 - 12:00 - 12:20