Materials for the study included the proposed Relationship Problem and Maltreatment (RPM) codes and features for ICD-11, RPM codes (i. e., Z-codes) for ICD-10, and proposed Mental and Behavioral Disorder (MBD) diagnostic definitions for ICD-11 (i. e., Mood Disorders and Anxiety and Fear-Related Disorders).

Vignettes were developed and tested according to standard procedures for ICD-11 field trials (e. g., Evans et al., 2015; Keeley, Reed, Roberts, Evans, Robles et al., 2016). Twelve case vignettes were generated for the study (see Table 2). Drs. Heyman, Slep, and Foran drafted the vignettes — based on actual clinical cases — to contain both key characteristics (i. e., MBD symptoms and RPM codes) and typical clinical presentations; “gold standard answers” for the vignettes were subsequently validated by 12 international RPM experts to ensure consensus agreement on the correct codes and diagnoses, and global applicability of the cases. Vignettes with less than 90% agreement across raters were revised by Drs. Reed and Kogan to address identified areas of disagreement. Vignettes described male and female adults across a range of ages who were currently in heterosexual relationships but omitted specific detail associated with any cultural group or religious practice. The vignettes reflected the four study conditions: (a) features consistent with both a RPM and a MBD, (b) features consistent with only a RPM (c) features consistent with only a MBD and (d) features consistent with neither a RPM nor a MBD (see Table 2). Specifically, each vignette described an individual or couple experiencing either the presence or absence of one of three ICD-11 RPMs (i. e., Relationship Distress with Spouse or Intimate Partner; Spouse or Partner Violence, Physical; or Spouse or Partner Abuse, Psychological) and the presence or absence of one of two common MBDs (i. e., Single Episode Depressive Disorder or Generalized Anxiety Disorder). Vignettes in which a RPM is absent described subthreshold features for the RPM, and vignettes in which a MBD is absent described subthreshold mental health symptoms. Due to concerns about the length of the study (i. e., time to complete), two lower prevalence ICD-11 RPM categories — (1) Spouse or Partner Violence, Sexual and (2) Spouse or Partner Neglect — were not included.

Because ICD-10 Z-codes contain only an omnibus “Problems in relationship with spouse or partner,” comparisons with ICD-11 necessarily required collapsing ICD-11 respondents’ classifications into “yes/no” for all four RPM types tested. However, subsequent analyses for individual ICD-11 RPM types were used to investigate the accuracy of clinician decisions using the proposed guidelines.

Performance by condition/vignette

Table 4 shows clinicians’ performance for any RPM (presence/absence) x MH (presence/absence) combination. Chi-square analyses were conducted for each ICD-10 and ICD-11 comparison (i. e., the frequency of correct responses for the ICD-10 versus the ICD-11 for RPM or for MH). For clinicians using ICD-11, compared with those using ICD-10, RPM guidelines were significantly more likely to correctly classify RPM presence/absence when a MH condition was present (see results for Conditions I and III in Table 4). When a MH condition was absent, there were no significant difference between clinicians using proposed ICD-11 and ICD-10 guidelines (although the direction of non-significant difference always favored ICD-11). A mixed measures ANOVA was run to test this finding across conditions. There was a significant interaction between condition (MH present I and III versus MH absent II and IV) and ICD version [(F (1, 736) = 4.502, p = .034; eta-squared=.006]. As shown in Figure 1, ICD-11 outperformed ICD-10, with more pronounced differences when MH conditions were absent.

Table 4.

Accuracy of clinicians’ classifications using ICD-11 (Proposed) and ICD-10 Criteria.

	Condition I (MH+ RPM+)		Condition II (MH- RPM+)		Condition III (MH+RPM-)		Condition IV (MH- RPM-)
ICD Criteria	MH	RPM	MH	RPM	MH	RPM	MH	RPM
ICD-11 (proposed)	72%	84%	80%	89%	78%	73%	78%	64%
ICD-10	67%	74%	78%	84%	78%	62%	73%	62%
χ2(1)	1.60	10.83*	0.43	4.51	0.001	10.34*	2.24	0.39
Cramer's V	0.047	0.121	0.024	0.078	0.001	0.118	0.055	0.023

Note. Classification accuracy could range between 0% (no agreement with the correct vignette classification) to 100% (complete agreement with the correct vignette classification). *p <.05 with Benjamini-Hochberg False Discovery Rate correction. Note: ICD = International Classification of Diseases; MH = Mental Health Problems; RPM = Relationship Problems and Maltreatment;+= Present; - = Absent; N = 738.

Figure 1.

Estimated marginal means of classification accuracy for ICD-11 vs. ICD-10 by condition.

(0.11MB).

This is the first study of the classification accuracy of ICD RPM codes. By experimentally manipulating case presentations, we were able to test clinicians’ diagnostic accuracy using either proposed ICD-11 descriptors for three adult intimate partner RPMs (relationship distress; physical violence, psychological abuse) or the existing ICD-10 single category (problems in relationship with spouse/partner) when combined with the presence or absence of individual mental health disorders. Prior clinic-based field testing of the guidelines (Heyman & Slep, 2006, 2009) indicated that the greater specificity of the guidelines for RPMs led to substantial increases in diagnostic accuracy. Thus, we expected the more thoroughly delineated ICD-11 guidelines to outperform those of the ICD-10.

Overall, these hypotheses were supported. The more detailed proposed ICD-11 guidelines performed better than the ICD-10 listing, with significant superiority when mental health problems were present. The framing of the individuals’ problems made a difference in whether the RPMs were salient to the rating clinician. Furthermore, correct classification of RPMs was equivalent to that of mental health problems, implying that general mental health clinicians are as adept at recognizing contextual problems as individual ones. It may be that both types of problems are present, clinicians turn to the guidelines to aid in differential diagnosis and that the increased specificity of the proposed ICD-11 guidelines boosts accuracy. WHO's training materials may want to highlight this inclination to improve accuracy in both multi-problem and family-problem only presentations.

We also found that the application of RPM diagnostic guidelines appears to be related to clinical presentations and not to clinician factors. Classification for the four clinical conditions (i.e., MH presence/absence by RPM presence/absence) was tested for eight demographic and professional factors and none of the 32 tests affected ICD-11 (proposed) or ICD-10 classifications. When ICD-11 and ICD-10 were collapsed, two of the 32 were significant. Differences were related to misclassification of a MH problem rather than to the misclassification of an RPM. Both younger and older clinicians, compared with middle-aged clinicians, were less likely to correctly diagnose the presence of a mental health problem when an RPM was absent. Clinicians from South-East Asia or Africa, compared with clinicians from Europe, were likely to misclassify a MH problem as being absent when only a RPM problem was present. Because this regional effect was for this particular condition, not for RPMs across the board, it could be spurious or it could reveal a regional inclination to attribute problems to individual, rather than relational, problems where plausible. Although mentioned in the brief training materials clinicians reviewed before rating, WHO may want to underscore this issue, especially with short vignette examples that highlight to clinicians situations in which classification guidelines for both types of problems are met.

Finally, as evidenced both by the patterns of incorrect type of RPM selected and by themes in the clinician feedback on their decisions, clinicians appeared to superimpose a hierarchy on the RPM codes, with relationship distress being less severe than psychological abuse, which is less severe than physical abuse. For example, when behaviors in fact met definitional requirements for psychological abuse but were not perceived as severe enough (as inferred from the open-ended responses justifying their decision-making), clinicians often would assign the relationship distress code, making it a catch-all category for psychological abuse as well as unhappiness.

Should the proposed abuse diagnoses be included in ICD-11, two training implications were brought to the fore by these results. First, the abuse diagnostic guidelines do not require severe harm, a pattern of abuse, an intent to exercise power and control, or any other inference beyond that contained in the field-tested, validated (Heyman, Slep, & Foran, 2015) criteria. Clinicians assess whether a qualifying act (or, in the case of neglect, omission) occurred and whether a qualifying impact was related to or exacerbated by the act/omission. This structure parallels the symptom-with-associated-harm structure of the ICD and DSM (Heyman & Slep, in press). Training should highlight both what the guidelines require and what they do not. Second and relatedly, many participants appeared to employ heuristics (e. g., representativeness, availability, Tversky & Kahneman, 1974) placing relationship problems on a severity continuum and used these heuristics, rather than the guidelines as written, to make their decisions. Training should approach this problem head-on, alerting clinicians to the error and suggesting heightened attention to the act and impact guidelines that operationalize RPMs.

The modest agreement between clinical diagnoses and the experimenter-fixed “correct” decision in this study for physical and psychological abuse is wholly consistent with similar, non-analogue comparisons in the real world. A meta-analysis (Rettew, Lynch, Achenbach, Dumenci, & Ivanova, 2009) aggregating 38 studies found poor agreement between clinical diagnoses — made with the ICD or with the DSM (American Psychiatric Association, 1980, 1987, 1994) — and those made by independent raters using gold-standard structured clinical interviews. This disparity could be due to clinicians using guidelines differently than intended (as noted in the open-ended responses) or to differences in the decision-making process itself, with clinicians making overall “yes/no” classifications and “gold standard” raters making criterion-by-criterion decisions and then applying the system's algorithmic rules for overall classification. In the earlier field trials that developed the RPM guidelines (Heyman & Slep, 2006) adapted by the ICD working group, field classifiers, who assessed for and classified only maltreatment, improved from 75% agreement with master reviewers to over 90% agreement when classification switched from an overall decision to criterion-by-criterion decisions guided by a computerized decision support system, that gathered the classifiers’ votes and used the guidelines’ logic to make an overall classification. Not only does criterion-by-criterion voting force the rater to attend to each guideline, but also it may change the process from a gestalt decision about whether the case matches implicit archetypes for that type of problem to a feature-based decision about the sub-elements of the clinical presentation. Although criterion-by-criterion decision-making is impractical in many field settings, perhaps it could be incorporated in training to highlight the usefulness of adhering to the written guidelines.

This study used a large, international panel of clinicians to test agreement among clinicians using proposed descriptors for ICD-11 adult Relational Problems and Maltreatment categories. ICD-11 proposed guidelines performed better than those from ICD-10, particularly in the presence of co-morbid mental health problems. Correctly identifying the presence of interpersonal violence is particularly important to improving a public health because of significant associated health risks. That ICD-11 guidelines performed better than ICD-10 in their detection in the context of mental disorders is important because it reflects how these issues typically present in clinical practice. This lack of hypothesized superiority may be due to (a) training issues related to some clinicians’ decisions being shaded by their archetypes for relationship distress and physical and psychological abuse (e. g., a severity continuum) and to (b) clinical classifications traditionally being made for presence or absence of problems/disorders overall compared with “gold-standard” determinations being made on a criterion-by-criterion basis. These data will be used to revise the RPM training recommendations prior to the release of ICD-11 in 2018.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.