What is new?
- (1)
A new checklist has been developed for the quality appraisal of studies of diagnostic reliability.
- (2)
This provides a specific tool for use in systematic reviews of diagnostic reliability.
What is new? A new checklist has been developed for the quality appraisal of studies of diagnostic reliability. This provides a specific tool for use in systematic reviews of diagnostic reliability.
Quality appraisal tools are used to assess the quality of primary research studies and are used in the preparation of systematic reviews. The use of quality appraisal tools for systematic reviews of treatment interventions is well established [1], [2], [3], [4]. For diagnostic procedures, however, methods for systematic reviews have only more recently emerged [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16]. The two main characteristics of diagnostic procedures investigated in primary studies are accuracy and reliability.
Diagnostic reliability refers to the agreement between two or more observations of the same entity. Interrater reliability refers to the agreement between two or more raters whom observe the same entity, whereas intrarater reliability refers to the agreement between two or more observations of the same entity made by a single rater. In contrast, diagnostic accuracy is defined as the amount of agreement between an index test and a reference standard [17].
Because poor reliability adversely affects the accuracy of diagnostic tests, it has been recommended that reliability and accuracy be reported in diagnostic research [18]. The estimation of diagnostic accuracy, however, is difficult and complex in circumstances where there is no accepted reference standard. In this case, reliability is an important indicator of the potential of the test to be accurate.
Studies of diagnostic reliability contain unique design features that are not represented on tools that assess the quality of studies investigating diagnostic accuracy such as QUADAS (Quality Assessment of Diagnostic Accuracy Studies) [15]. Conversely, these tools contain items that are not relevant for studies of diagnostic reliability. Although other quality appraisal tools have been published for reliability studies [19], [20], [21], [22], [23], [24], [25], none have achieved widespread use. Comparison of these tools shows that although there is some overlap in the items included, there are also items that are unique to each tool. This diversity across checklists provided the rationale to develop a unifying tool informed by existing tools. In this article, we report the development of a quality appraisal tool for studies of diagnostic reliability (Quality Appraisal of Reliability Studies [QAREL]) for use across different content areas.
The development of QAREL was based on methodological procedures used in the design of existing quality appraisal tools [2], [15]. Quality appraisal tools often use numeric scoring systems to rank individual studies with an overall quality score [1], [2], [3], [4]; however, opinions differ over the weighting of the scores for individual items on the appraisal tool [13], [26], [27]. It has been recommended that each item on a quality appraisal tool be considered separately for its impact on the
An initial list of 42 items was reduced to 11 items that form the QAREL checklist (Table 1). The wording of each item was revised numerous times for the purpose of improving the reliability and usability of the checklist.
Items 1 and 2 consider sampling bias and the representativeness of subjects and raters. Items 3–7 relate to rater blinding. Item 8 relates to the order in which raters or subjects were examined. Item 9 relates to the suitability of the time interval among repeated measurements.
QAREL is presented as a new tool for the assessment of the key aspects of the quality and applicability of reliability studies. This tool will be of particular relevance to studies of clinical reliability and diagnostic imaging. Instructions for its use are presented and a data extraction form for the collection of relevant information from primary studies is included. The reliability of QAREL in different contexts needs to be evaluated.
The authors thank Robert Moran (Unitec, New Zealand) and Luke Rickards (Adelaide, Australia) for their comments and feedback regarding their use of the QAREL checklist. The project was funded in part by National Health and Medical Research Council (NHMRC) Program grant 402764.
The authors contributed equally to this work. N.P.L conceived of the study and designed the initial protocol. P.M., L.I., and N.B. provided advice on the design of the study and participated as members of the reference group. N.P.L drafted the initial manuscript, and all authors contributed to and approved the final manuscript.