Original Article
The development of a quality appraisal tool for studies of diagnostic reliability (QAREL)

https://doi.org/10.1016/j.jclinepi.2009.10.002Get rights and content

Abstract

Background and Objective

In systematic reviews of the reliability of diagnostic tests, no quality assessment tool has been used consistently. The aim of this study was to develop a specific quality appraisal tool for studies of diagnostic reliability.

Methods

Key principles for the quality of studies of diagnostic reliability were identified with reference to epidemiologic principles, existing quality appraisal checklists, and the Standards for Reporting of Diagnostic Accuracy (STARD) and Quality Assessment of Diagnostic Accuracy Studies (QUADAS) resources. Specific items that encompassed each of the principles were developed. Experts in diagnostic research provided feedback on the items that were to form the appraisal tool. This process was iterative and continued until consensus among experts was reached.

Results

The Quality Appraisal of Reliability Studies (QAREL) checklist includes 11 items that explore seven principles. Items cover the spectrum of subjects, spectrum of examiners, examiner blinding, order effects of examination, suitability of the time interval among repeated measurements, appropriate test application and interpretation, and appropriate statistical analysis.

Conclusions

QAREL has been developed as a specific quality appraisal tool for studies of diagnostic reliability. The reliability of this tool in different contexts needs to be evaluated.

Introduction

What is new?

  • (1)

    A new checklist has been developed for the quality appraisal of studies of diagnostic reliability.

  • (2)

    This provides a specific tool for use in systematic reviews of diagnostic reliability.

Quality appraisal tools are used to assess the quality of primary research studies and are used in the preparation of systematic reviews. The use of quality appraisal tools for systematic reviews of treatment interventions is well established [1], [2], [3], [4]. For diagnostic procedures, however, methods for systematic reviews have only more recently emerged [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16]. The two main characteristics of diagnostic procedures investigated in primary studies are accuracy and reliability.

Diagnostic reliability refers to the agreement between two or more observations of the same entity. Interrater reliability refers to the agreement between two or more raters whom observe the same entity, whereas intrarater reliability refers to the agreement between two or more observations of the same entity made by a single rater. In contrast, diagnostic accuracy is defined as the amount of agreement between an index test and a reference standard [17].

Because poor reliability adversely affects the accuracy of diagnostic tests, it has been recommended that reliability and accuracy be reported in diagnostic research [18]. The estimation of diagnostic accuracy, however, is difficult and complex in circumstances where there is no accepted reference standard. In this case, reliability is an important indicator of the potential of the test to be accurate.

Studies of diagnostic reliability contain unique design features that are not represented on tools that assess the quality of studies investigating diagnostic accuracy such as QUADAS (Quality Assessment of Diagnostic Accuracy Studies) [15]. Conversely, these tools contain items that are not relevant for studies of diagnostic reliability. Although other quality appraisal tools have been published for reliability studies [19], [20], [21], [22], [23], [24], [25], none have achieved widespread use. Comparison of these tools shows that although there is some overlap in the items included, there are also items that are unique to each tool. This diversity across checklists provided the rationale to develop a unifying tool informed by existing tools. In this article, we report the development of a quality appraisal tool for studies of diagnostic reliability (Quality Appraisal of Reliability Studies [QAREL]) for use across different content areas.

Section snippets

Methods

The development of QAREL was based on methodological procedures used in the design of existing quality appraisal tools [2], [15]. Quality appraisal tools often use numeric scoring systems to rank individual studies with an overall quality score [1], [2], [3], [4]; however, opinions differ over the weighting of the scores for individual items on the appraisal tool [13], [26], [27]. It has been recommended that each item on a quality appraisal tool be considered separately for its impact on the

Results

An initial list of 42 items was reduced to 11 items that form the QAREL checklist (Table 1). The wording of each item was revised numerous times for the purpose of improving the reliability and usability of the checklist.

Items 1 and 2 consider sampling bias and the representativeness of subjects and raters. Items 3–7 relate to rater blinding. Item 8 relates to the order in which raters or subjects were examined. Item 9 relates to the suitability of the time interval among repeated measurements.

Conclusions

QAREL is presented as a new tool for the assessment of the key aspects of the quality and applicability of reliability studies. This tool will be of particular relevance to studies of clinical reliability and diagnostic imaging. Instructions for its use are presented and a data extraction form for the collection of relevant information from primary studies is included. The reliability of QAREL in different contexts needs to be evaluated.

Acknowledgments

The authors thank Robert Moran (Unitec, New Zealand) and Luke Rickards (Adelaide, Australia) for their comments and feedback regarding their use of the QAREL checklist. The project was funded in part by National Health and Medical Research Council (NHMRC) Program grant 402764.

References (29)

  • J.J. Deeks

    Systematic reviews in health care: systematic reviews of evaluations of diagnostic and screening tests

    BMJ

    (2001)
  • W.L. Deville et al.

    Conducting systematic reviews of diagnostic studies: didactic guidelines

    BMC Med Res Methodol

    (2002)
  • J. Dinnes et al.

    A methodological review of how heterogeneity has been examined in systematic reviews of diagnostic test accuracy

    Health Technol Assess

    (2005)
  • A.R. Feinstein

    Misguided efforts and future challenges for research on “diagnostic tests”

    J Epidemiol Community Health

    (2002)
  • Cited by (267)

    View all citing articles on Scopus

    The authors contributed equally to this work. N.P.L conceived of the study and designed the initial protocol. P.M., L.I., and N.B. provided advice on the design of the study and participated as members of the reference group. N.P.L drafted the initial manuscript, and all authors contributed to and approved the final manuscript.

    View full text