In April, a classmate and I competed in the student competition at the 2021 UPSTAT Conference, a local research conference on applied statistical methods. This year’s theme was on applied social science for social justice in policing. In addition to holding presentations by researchers from Cornell, University of Texas in San Antonio, and the University of Rochester, the conference held a student competition that utilized publicly available datasets on policing outcomes in Austin, TX, one of the fastest growing cities in the US, and prompted students to develop visualizations and metrics that measure the fairness in traffic law enforcement. The competition was socially relevant given the recent social unrest in 2020, the well-reported link between gentrification and over-policing, and the central role that traffic enforcement has in police violence. I was exciting to participate.
My classmate and I submitted a paper on our analysis and got to present the results at the conference which won us first place along with another team (we tied) out of roughly 35 competing teams. You can see our full analysis here. On a personal note, I’ve spent most of the last two years feeling busy catching up in my stats knowledge, but winning this award was a huge validation that I was actually able to learn these disciplines on my own, outside of school.
Our analysis looked for racial bias in three areas of policing outcomes: traffic citations, searches, and use-of-force by the police. Counter to the competition’s instructions to formalize a “fairness” metric, we found it much more tractable to evaluate these outcomes based on disparity. In our paper, we point out that the term “fairness” in the academic literature defies easy definition and arbitrarily choosing one popular metric of fairness could ultimately harm the very people the metric was intended to help. By taking the alternative route and pointing to disparities in these three outcomes, we could potentially shine light on instances of unfair treatment, like identifying the fire through the smoke.
I was excited to learn about so much interesting social science research that goes into policing, and we comment the Austin City Council for being as transparent with their policing data as they are, but I still had a few criticisms. The statistical limitations of observational data of this kind pose serious limitations to identifying true instances of unfair treatment. In our paper, we warn against using simple metrics (of the kind implied by the competition organizers) in order to conjure the appearance of statistical fairness at the expense of critical examination of the underlying operations of law enforcement. No metric derived from observational data (the kind of data that this competition used) without the adequate controls can be definitive in assessing the fairness of any program in law enforcement.
Policing gets a lot of (understandable) political attention, but there’s a ton of social science research out there into better policing strategies (such as better oversight) which point to proven ways to reduce crime and improve public satisfaction. It’s my opinion that there’s a ton of low-hanging fruit that could improve the social services that policing is meant to provide. In comparison to the academic research, the public debates are far too reactionary to be fruitful, and I was surprised that this public discourse seems to affect the operations of government more than the academic debate. One surprising example of this was Austin’s Office of Police Oversight (OPO) annual report on racial profiling in traffic law enforcement. This competition was meant to mimic the mandate of this annual report, but reading these reports in preparation for this competition revealed to me just how low the bar was to be considered a “government report”. These reports are poorly reasoned and fails to demonstrate even the simplest of statistical understanding. I honestly hope they had an intern write them, because that would be less embarrassing… This was one of my biggest lessons from the experience of this conference: there is clearly a need to inject real academic rigor into these departments of government. The ideas discussed at the conference and in the academic literature could have real consequence, if only those in government would read them…
Stephen Gass 