To: The Marshall Project
I read with interest the Marshall Project's piece on risk assessment ("Machine Bias") from this last Monday (May 23), and have two suggestions for how you might wish to follow this piece up.
The still controlling Supreme Court case for the use of clinical or actuarial assessments in the judicial system for assessing "dangerousness" (and particularly for Texas death penalty cases) is Barefoot v. Estelle (1983). I might suggest that any contemporary discussion of risk assessment should probably have this particular case and its conclusions as a starting point: no matter how we do it, clinically or actuarially, our ability to predict "dangerous" behavior is dismal, irrespective of how many risk assessment instruments are developed.
I've attached three items that might be of value in understanding the continuing importance of Barefoot v. Estelle: (1) redactions of the majority opinion and an elegant dissent by Justice Blackmun, (2) a brief from the American Psychiatric Association on how badly we do in predicting dangerous behavior, and (3) an unpublished paper by myself and a former student, Ehsan Bokhari: "The Prediction of 'Dangerous' Behavior: A Continuing Legacy for Barefoot v. Estelle (1983)."  By the way, the Marshall Project is welcome to the latter paper if it wants it or a version thereof.
The second suggestion has to do with the analysis of COMPAS data, and two specific things that should be included in the evaluation of any risk assessment instrument (RAI). The first is that the RAI needs to be compared against just using base-rates for prediction.  This is the Meehl-Rosen concept of clinical efficiency --- an instrument is clinically efficient if one can do better with it than merely "betting the base rates" (e.g., predicting everyone to be "not dangerous" because the base-rate for being "not dangerous" is higher than for being "dangerous").  I attach a Bokhari-Hubert paper published last year in Psychological Assessment, "A New Condition for Assessing the Clinical Efficiency of a Diagnostic Test" (bokhari_hubert.pdf).  It uses data from the MacArthur Violence Risk Assessment Study and its RAI called the Classification of Violence Risk (COVR).  Bottom line: the COVR is not clinically efficient.
The second item to include in assessing a RAI is a cross-validation study --- that is, how well does the RAI do on a "new" sample apart from the one it was constructed with.  This is necessary irrespective of whether using logistic regression (as in "Machine Bias"), or for any of the various tree construction methods, as in COVR.  I attach another Bokhari-Hubert paper that will appear in the Journal of Classification, entitled "The Lack of Cross-Validation Can Lead to
Inflated Results and Spurious Conclusions: A Re-Analysis of the MacArthur Violence Risk Assessment Study"

Larry Hubert
University of Illinois