The 2/3 Accuracy Ceiling & Why We Should Rethink Letting AI Predict Recidivism: A Replication of Dressel & Farid (2018)
In the modern criminal justice system, machine learning algorithms are commonly used to predict a defendant’s likelihood of recidivating, or repeating an offense. The risk assessment software COMPAS is one of the most widely used tools to do this, but research by Dressel & Farid finds that COMPAS’ predictions plateau at an accuracy rate of approximately 67%, a rate identical to that of the judgment of people with no special knowledge of the criminal justice system and of models much less complex than COMPAS. We formulate two logistic regression models with the features specified by Dressel & Farid and also find that these models are accurate only two-thirds of the time. We then build upon their results by calculating precision, recall, and F-1 Score to delve deeper into the reliability, or lack thereof, of employing machine learning algorithms for predicting recidivism. Our analysis supports Dressel & Farid’s notion that these algorithms have limited predictive power and may be too haphazard to be relied on in the criminal justice field.