A call to improve the validity of criterion-based content analysis (CBCA): Results from a field-based study including 60 children's statements of sexual abuse

https://doi.org/10.1016/j.jflm.2016.08.001Get rights and content

Highlights

  • Truthfulness of 60 verbatim statements of sexually abused children was analyzed.

  • The accuracy rate in the 40 confirmed allegations reached 90%.

  • The diagnostic accuracy of the CBCA needs to be further improved.

Abstract

The growing awareness of sexually-abused children has led to a major shift: previously considered untrustworthy, children are now regarded as competent in providing medico-legal evidence. Professionals undertaking the challenging task of assessing the child's credibility need to rely upon approved evaluation methods. The Criteria-Based Content Analysis is a tool developed to assess the truthfulness of a child's verbal statement. This field-based study explores its validity and its limitations.

Three independent experts rated the verbatim statements of 60 real-life alleged victims of sexual abuse. The CBCA scoring and final assessment of credibility were linked to the outcomes: confirmed or unconfirmed allegation of sexual abuse.

Inter-rater reliability coefficient was 0.74. The average overall accuracy rate corresponding to confirmed and unconfirmed cases was 75%. Among the confirmed allegations, the accuracy rate reached 90%, whereas the probability of discriminating the true negative cases within the unconfirmed cases was lower than chance level. Of all the 19 criteria, items 6 “Reproduction of conversation” and 12 “Accounts of subjective mental state” were the strongest predictors of genuine accounts. A significant association between age and CBCA scores was noted, the effect of age on CBCA scores was strongest in the unconfirmed cases.

Although some may argue that the validity of the CBCA is reasonably acceptable, results from this field study are less convincing. Increasing the diagnostic accuracy of the CBCA by adding new criteria, so as to raise the percentage of correct classifications in the confirmed accounts as well as in the unconfirmed accounts, would represent a major improvement.

Introduction

Children's statements in cases of suspected sexual abuse involve high medico-legal stakes. The scarcity of evidence at hand, sometimes solely limited to the victims' and the suspects' statements, underscores the crucial issue of establishing truthfulness in children's testimonies.

In this respect, Statement Validity Assessment (SVA) is probably the most popular tool worldwide for assessing the credibility of child statements in sexual alleged abuses.1 This method follows the Undeutsch hypothesis, stating that “truthful, reality-based accounts differ significantly in quantity and quality from unfounded, falsified, or distorted stories”.2 The central component of the SVA is the Criteria-Based Content Analysis (CBCA). This key tool assists in deciding the increased or decreased probability that the child's testimony is credible, based on the presence or absence of 19 specific criteria that are referred to as content criteria.3 As shown in Table 1, the 19 criteria are grouped into four major categories, from broad to specific aspects of the statement.

Substantial research has been generated on the CBCA. However, results emanating from laboratory-based studies have been the object of relative skepticism, based on the lack of underlying ecological validity.4 As stated by Trankell and Undeutsch,5 “laboratory studies cannot experimentally manufacture statements such that they imitate real truths and lies”.

To date, a limited number of field studies on the CBCA have been carried out. As far as we know, only six articles regarding the validity of the CBCA have been published.6, 7, 8, 9, 10, 11 Two of these studies included adult victims9, 12 and are therefore not fully transposable to children, especially given the fact that the CBCA was intentionally developed to evaluate statements from children who are witnesses or alleged victims in sexual abuse cases.13, 14 The earliest study seeking to ascertain the validity of the CBCA with respect to children's statements in a field setting7 was presented at the NATO conference in 1988. Its results received heavy criticism, notably because decisions on the veracity of the statements were not established through independent case facts,3 but also because one single evaluator was responsible for scoring the transcripts. Furthermore, the age ranges of the sample were large (3.5–17 years).15 These criticisms were consequently taken into consideration by the authors, and the study was later published as a book section.16 It described 20 confirmed cases of sexual abuse and 20 non confirmed testimonies. The CBCA scale with the 19 criteria was applied, using a two point Likert scale (0 = no, 1 = probably, 2 = absolutely). A significant difference between the two statement groups among all items was found in terms of total score, with no overlap between the distributions of scores assigned to each group.

Another survey, including ninety-eight statements of children aged 4–12 years,8 aimed to improve the ground truth evaluation by using a method named « Independent Case Fact Scales ». The CBCA version was limited to 14 items, each codified 0 or 1. The results disclosed a significantly larger CBCA score in the confirmed cases compared to the cases with no corroborating evidence.

A similar study led by Craig6 was conducted on a sample of 48 children's allegations aged 3–16 years. The CBCA version and the scaling method were identical to the one used by Lamb et al.8 However, in this case, the veracity of the statements was evaluated based on the confessions of the suspected perpetrator, the retraction of the statement by the presumed victim, or the polygraph results of the two protagonists. Again, results showed a mean CBCA significantly higher in the confirmed cases compared to the unconfirmed.

The latest study11 includes 109 statements of sexual abuse among children aged 4–14 years. The ground truth was particularly emphasized and based only on independent information. Two experts evaluated the children's statements using the 14 item CBCA, and by marking each item as 0 or 1. The results showed scores differing significantly between confirmed and unconfirmed cases, however these varied significantly depending on the children's age. Although the age of the child has very early on been considered important when applying the CBCA, no emphasis has been made to attempt to address the question as to what age group is the most suitable, on a scientific basis. To our knowledge, only one field study has focused on showing a strong effect of age on CBCA scores.11

In the works of Esplin and Raskin,16 the accuracy rate of the credibility assessments was 100%, which made one consider the results « too good to be true »,17 mainly caused by the fact that the decision about what was the ground truth relied partially on the verdict of the credibility assessment. In the two studies by Craig and by Roma6, 11 the methodology was more precise but the accuracy rate of the assessments was not stated.

The fact that the methodology differs considerably in each of the previously cited studies, especially with regards to establishment of ground truth, the number of CBCA criteria investigated and the type of score rating that was chosen, comparison between their respective findings does not seem appropriate.

The main problem identified has been the establishment of ground truth. Indeed, it can be viewed as an unavoidable weakness of field studies, however the robustness of this particular type of study can be increased, by using independent case facts when categorizing between truthful and not truthful. In this respect, the development of international guidelines and recommendations for establishing ground truth in order to increase the homogeneity of coming field studies, would represent a major improvement.

The intent of the present study is to explore the validity of the CBCA methodology in a forensic context, reaching as close as possible to ground truth. With this in mind, two main improvements have been added in our study in comparison to the previously cited studies. In order to increase the level of impartiality, three experienced raters independently scored the level of credibility of each case. Furthermore, the final proof of sexual abuse was solely based on medico-legal findings, such as medical findings and confession evidence by suspects, thereby excluding judicial decisions. We aimed to test the assumption that higher scores are observed in the confirmed abuse cases compared to the unconfirmed ones. Furthermore, we analyzed the relationship between the CBCA criteria and the final assessments given by the three expert opinions on whether the child's statement was credible or not. Another question of research was whether certain CBCA criteria were better than others in predicting the truthfulness of the allegation. Lastly, the effect of age on CBCA scores and on independent criteria was analyzed.

Section snippets

Sample

The initial sample consisted of 225 child testimonies carried out by the Geneva Police Department, Juvenile Division, between 2008 and 2011. The interview techniques were identical throughout the population sample, and followed the guidelines recommended by Steller.18 The interviews were audio taped and transcribed into verbatim transcripts, and each of these was attributed an anonymous identification number, rendering all 225 cases unidentifiable by the experts.

After excluding the auditions

Inter-rater reliability of CBCA total score and individual items

Individual ICC for the total score reached a coefficient of 0.755 (95% CI: 0.651–0.837; P < 0.001), whereas the degree that coders provided consistency in their final score across subjects was 0.902 (95% CI: 0.848–0.939; P < 0.001). The ICC coefficients of reliability for the individual CBCA criteria are presented in Table 1.

Criterion 6 “Reproduction of conversation” exhibited the highest alpha Krippendorf coefficient (α = 0.74), followed by 8 criteria with alpha levels between 0.50 and 0.60 (1

Reliability of total score and individual criteria

According to Fleiss,24 reliability scores between 0.60 and 0.75 should be considered as “good”, and scores over 0.75 as excellent. Hence, a coefficient of 0.76 for the total CBCA score using the ICC and an alpha Krippendorf coefficient of 0.74, suggests that the three raters had an acceptable degree of agreement, and suggests that the total CBCA scores reached a similar level across coders. Comparative results from previous studies are difficult to provide, given the various statistical methods

Conclusion

The findings for reliability with three raters suggest that CBCA is a « moderately » reliable instrument to assess the truthfulness of accounts provided by the alleged victims. Our findings also confirm those of previous studies, showing that CBCA criteria occur more frequently in truthful reports than in fabricated reports. However, the fact that a non-negligible number of individuals' “true” testimonies are judged non-credible is of major concern. As stated by Lamb et al.,8 the actual CBCA

Conflict of interests

The authors declare that they have no conflict of interests.

References (47)

  • R.A. Craig et al.

    Interviewer questions and content analysis of Children's statements of sexual abuse

    Appl Dev Sci

    (1999)
  • Esplin PW, Boychuck TD and Raskin DC. A field validity study of criteria based content analysis of children's...
  • A.D. Parker et al.

    Detection of Deception : statement Validity Analysis as a means of determining thruthfulness or falsity of rape allegations

    Leg Crim Psychol

    (2000)
  • E. Rassin

    Criteria based content analysis: the less scientific road to truth

    Expert Evid

    (1999)
  • E. Rassin et al.

    Characteristics of true versus false allegations of sexual offences

    Psychol Rep

    (2005)
  • C.R. Honts

    Assessing children's credibility: scientific and legal issues in 1994

    N. D Law Rev

    (1994)
  • S.W. Horowitz et al.

    Reliability of criteria based content analysis of child witness statements

    Leg Crim Psychol

    (1997)
  • G. Wells et al.

    Commentary: is this child fabricating? Reactions to a new assessment technique

  • D.C. Raskin et al.

    Assessment of children's statements of sexual abuse

  • F. Lamers-Winkleman et al.

    Children's testimony in The Netherlands: a study of statements validity analysis

  • M. Steller

    Recent developments in statement analysis

  • M. Steller et al.

    Criteria-based statement analysis. Credibility assessment of children's statements in sexual abuse cases

  • G. Niveau et al.

    Implementation of the SVA credibility assessing protocol in the French speaking forensic

    Swiss Arch Neurol Psych

    (2013)
  • Cited by (0)

    View full text