Predictive validity of the Short-Term Assessment of Risk and Treatability (START) for multiple adverse outcomes: The effect of diagnosis

Highlights

• Inpatients present multiple risks: assessments should be valid across diverse diagnostic groups.

• We compared predictive accuracy of the START across multiple diagnostic groups.

• The tool was supported to an extent but cannot be recommended for use unreservedly.

• Clinicians should consider evidence for their patient group when assessing risk.

Abstract

The Short-Term Assessment of Risk and Treatability (START) assists risk assessment for seven risk outcomes based on scoring of risk and protective factors and assignment of clinically-informed risk levels. Its predictive validity for violence and self-harm has been established in males with schizophrenia, but accuracy across pathologically diverse samples is unknown. Routine START assessments and 3-month risk outcome data of N = 527 adult, inpatients in a UK secure mental health facility were collected. The sample was divided into diagnostic groups; predictive validity was established using receiver operating characteristics regression (rocreg) analysis in which potential covariates were controlled. In most single-diagnosis groups START risk factors ('vulnerabilities'), protective factors ('strengths'), and clinically-informed estimates predicted multiple risk outcomes with effect sizes similar to previous research. Self-harm was not predicted among patients with an organic diagnosis. The START risk estimates predicted physical aggression in all diagnostic groups, and verbal aggression, self-harm and self-neglect in most diagnostic groups. The START can assist assessment of aggressive, self-harm, and self-neglect across a range of diagnostic groups. Further research with larger sample sizes of those with multiple diagnoses is required.

Introduction

Structured risk assessment for violence is common and accepted practice in mental health and criminal justice settings. Actuarial tools comprise schedules of empirically-derived risk factors whose presence raters are required to determine, and subsequently subject to an algorithmic scoring system to determine the probability of an individual engaging in future violence (Hart et al., 2007). Structured professional judgement (SPJ) tools have extended this approach by combining the requirement to consider empirically-derived risk factors with a degree of latitude for clinical judgement about individual cases (Guy et al., 2012).

The growth of use of these tools has led to a number of developments, particularly in the case of SPJ instruments. First, the focus of most violence risk assessment tools has been on factors that increase risk. However, some authorities contend that protective factors, “variables that reduce the effect of risk factors or influence the outcome independently” (Braithwaite et al., 2010, p. 272), have been insufficiently addressed (O’Shea and Dickens, 2016a, Rogers, 2000, Stouthamer-Loeber et al., 2002, Webster et al., 2006). From this perspective, a focus on so-called risk factors constitutes an institutionalised focus on the patient's weaknesses or limitations which could lead to an over-estimation of risk and to subsequent overly-restrictive risk management interventions. Working with a patient to identify and bolster their protective factors, or strengths, may help to develop the therapeutic alliance and facilitate the implementation of more effective risk management strategies (de Ruiter and Nicholls, 2011, van den Brink et al., 2015, Wilson et al., 2010).

Second, structured risk assessment tools have focused on relatively static, historical risk factors such as history of violence. While offering important indicators of future risk behaviour, they are inherently insensitive to change and thus have limited utility in the identification of treatment targets. In contrast, dynamic factors, for example the severity and nature of active symptoms of major mental illness, can change with time, are associated with changes in risk behaviour (Hanson and Harris, 2000) and can add significant incremental validity to risk assessment (Doyle and Dolan, 2006). Therefore, they can usefully contribute to violence risk assessment (Chu et al., 2013, Grevatt et al., 2004, McNiel et al., 2003, Wilson et al., 2013), and management strategies (Whittington et al., 2014).

Additionally, most risk assessment tools used in mental health settings have traditionally been concerned with violence or suicide, and do not aim to inform clinicians about other important risk outcomes such as self-neglect (Gunstone, 2003), or absconding (Muir-Cochrane and Mosel, 2008). Some therefore contend that risk assessment should address a wider range of adverse outcomes to which patients might be at risk (Webster et al., 2009). Finally, there has been increased interest in the prediction of risk behaviour over shorter time-periods than achieved by established instruments like the HCR-20 (6 months; Webster et al., 1997). In inpatient settings, particularly, it may be advantageous to conduct more regular risk assessments and make management adjustments accordingly.

The authors of the START (Webster et al., 2004, Webster et al., 2009) attempted to address all of the above issues. The START requires raters to consider 20 items both as risk factors (“Vulnerabilities”) and protective factors (“Strengths”). Items were selected for their dynamic nature and thus suitability for identifying treatment targets. Raters are required to consider these factors to inform and augment their clinical judgement to make a Specific Risk Estimate (SRE) in seven risk domains: violence to others, self-harm, suicide, substance misuse, victimisation, unauthorized leave and self-neglect. Finally, the START aims to assist assessment for the three-month period ahead, half as long as that recommended for iteration of the widely used HCR-20 (Webster et al., 1997).

A systematic review and meta-analysis (O’Shea and Dickens, 2014) revealed that the START was internally consistent and has convergent reliability with other risk measures. Predictive validity of the tool for aggression in studies rated as low risk of bias (Braithwaite et al., 2010, Desmarais et al., 2012, Wilson et al., 2013) ranges from 0.65 to 0.84 (Strength scale), 0.66 to 0.82 (Vulnerabilities scale), and 0.52 to 0.89 (Violence Risk Estimate). Studies of other risk outcomes are rarer; pooled effect sizes from all available studies have produced small effect sizes for rediction of self harm, self neglect and victimisation (O’Shea and Dickens, 2014). The START is valued by mental health workers who find it easy to use and it has acceptable inter-rater reliability and good predictive validity for violence and self-harm (Doyle et al., 2008). Subsequent research has demonstrated that respective SREs are predictive of self-harm/suicidality and victimisation (O’Shea et al., 2016), and there is recent evidence that the START may, to an extent, be predictive of substance misuse and unauthorized leave (O’Shea and Dickens, 2015a). Despite this, most of the START literature has been conducted in relatively small samples of young Caucasian males with schizophrenia. Two of the current authors have recently shown that the START has better predictive validity for women than men for aggression and self-harm outcomes (O’Shea and Dickens, 2015b).

One remaining gap concerns the START's performance across different diagnostic groups. From a theoretical perspective, the predictive efficacy of risk assessment tools is expected to be maximal in populations similar to validation samples (Buchanan, 2013). Empirically, there is evidence that the predictive efficacy of the HCR-20 performs across diagnoses in a manner broadly consistent with this (Gray et al., 2011, O'Shea et al., 2014). Therefore, it would be expected that the START would predict risk behaviour best among samples with schizophrenia or personality disorder diagnoses (Nicholls et al., 2006). We have therefore conducted a study to test the predictive validity of the START Strength and Vulnerability scores and SREs as a function of diagnosis, whilst controlling for potential covariates such as gender, age and ethnicity.