SST, Inc., the company that sells ShotSpotter gunfire-detection systems to regional governments around the world, recently published a marketing pamphlet called the 2014 National Gunfire Index. The Seattle City Council might avoid purchasing another police boondoggle if it examines the phony data and confused arguments that pad out this piece of fake science.
Seattle Privacy Coalition ran across this document while compiling press clippings on ShotSpotter. Beginning in October, glowing press releases and media reports about the success of ShotSpotter began popping up around the country. For example, in Camden, NJ:
[County Commission] Director Louis Cappelli, Mayor Dana Redd and Chief Scott Thomson announced that ShotSpotter Flex[,] the global leader in gunfire detection and analysis, today announced that its National Gunfire Index revealed that gunfire incidents in Camden City for the first half of 2014 are down by 48.5 percent, compared to first half of 2013, where ShotSpotter was deployed during both periods.[1]
And in Kansas City, MO:
Newly released data from the makers of the ShotSpotter gunshot detection system indicates that gunfire has decreased significantly in Kansas City’s urban core over the last year.
The ShotSpotter system covers 3.55 square miles in Kansas City near the Troost MAX bus line. In comparing the first half of 2013 to the first half of 2014, gunfire incidents in those areas fell by 25.9 percent. That’s 55 fewer incidents. That keeps with the trend in 31 other cities across the United States and Caribbean that ShotSpotter serves: those cities averaged 25.9 percent fewer gunfire incidents, as well.[2]
Wow, those are amazing year-over-year crime reduction numbers! And the clear implication is that SpotShotter made this happen.
Let’s have a look at that data
An examination of ShotSpotter’s data and research methodology dispels any hope that it has a basis in legitimate science. There are several separate and false claims to be debunked.
False Claim 1 — Gunfire declined in SpotShotter cities from 2013 to 2014
Bizarrely, this most basic claim, the one politicians and media picked up on, is proved false (or unintelligible) by ShotSpotter’s own figures. These are summarized in a graphic[3]:
The researchers analyze their raw data (which consists of detected gunshots) in two ways, as total rounds fired, and as “incidents.” Gunfire “incidents” are never actually defined, nor are we told why this is a useful measure. As the graphic shows, incidents declined sharply in the 31 communities studied, while absolute rounds fired increased, and this 20.6% decline is what the the report touts on 7 of its 9 pages of content. The rounds-fired figure is mentioned on 3 pages.
The facts become murkier when we come to this baffling statement:
Rounds (bullets) fired per gunfire incident were up by 36%. On average, 3.2 rounds were fired per incident during the first half of 2014, up 10% from first half 2013 average of 2.9 rounds per incident.[4]
Now there are three different figures for rounds-fired:
- 14%
- 36%
- 10%
The supporting graphic (Web version[5]) both hides the high number and adds to the confusion with a whole new measure called “Total Number of Rounds Fired Per Incident” (emphasis ours). To see this, you have to mouse-over the bullet images, as shown in the following before-and-after versions:
Added confusion stems from the complete meaninglessness of “total rounds per incident” and how this relates to “average rounds.”
ShotSpotter does provide some actual raw numbers[6] (more on that below) that supposedly back up its generalizations.
Total Incidents | Total Rounds Fired | |
2013 | 14,703 | 42,830 |
2014 | 11,675 | 58,087 |
Year-over-year change | -20% | +36% |
At least that clarifies which of the calculated numbers are real, such as the high “total-rounds-per” number that the report tried to hide and fails to explain, and which turns out to be the real figure for rounds-per-incident. The scale of the problem becomes clear in the following graph — created by Seattle Privacy Coalition, not ShotSpotter — which illustrates the public safety significance of ShotSpotter’s two measures (as we understand them):
This makes it pretty clear that the “incidents” measure is here to obscure the fact that gunfire increased by 36% in the 31 ShotSpotter communities during the the period of the study. People dodging bullets don’t care how many people are firing at them. Yet these are the statistics that embolden SST President and CEO, Ralph A. Clark, to tell the Camden, NJ, newspaper:
“The gunfire index data is extremely encouraging and suggests what cities and their law enforcement agencies can accomplish with a comprehensive gun violence reduction effort focused on enhanced response and community engagement.”[7]
Or, as the Index itself puts it,
Gunfire incidents are down in almost every ShotSpotter Flex city. In the 31 communities that we were able to analyze both for 1H2013 and 1H2014, gunfire incidents were down in 28 of the 31 communities, or 90% of them.[8]
False Claim 2 — SpotShotter is responsible for reducing gun violence
Things look bad for ShotSpotter. Far from reducing gun violence, Its figures suggest it has aggravated gun violence in the communities where it is deployed. The only thing that saves it from that humiliating finding is the iron rule of statistics: correlation is not causation. ShotSpotter cannot actually be blamed for an increase in gun violence without controlled studies that rule out other factors that may be causing the increase. Furthermore, the sloppiness of the arithmetic and reasoning in the 2014 National Gunfire Index make us wary of actually trusting the figures presented. Without more data, there is no way to know how much damage ShotSpotter is or is not causing.
On the other hand, reliable independent crime statistics tell a story that is unhelpful to ShotSpotter’s case regardless of the soundness of the gunfire report data. The FBI’s uniform crime statistics document[9] a steady decline of all violent crime nationwide over the past 20 years:
This trend suggests that a decline of about 3% in the overall violent crime rate probably occurred between 2013 and 2014. Any claim that ShotSpotter reduces crime would have to take into account this background decline. It is troubling but not surprising that the study ignores this, since it is much more enjoyable to claim credit for whatever good is happening on your watch. Of course, that’s not science.
The most plausible inference to make in the face of the FBI’s figures is that ShotSpotter’s figures, showing a 36% increase in gunfire over the last year, are simply too aberrant to be trusted without confirmation by qualified researchers.
False Claim 3 — ShotSpotter bases its claims on real-world data
ShotSpotter spends a lot of time in this report stressing its careful comparison of “apples-to-apples” data. Unfortunately, it appears that the researchers only got halfway through that research methods course. Even knowing the shoddiness of the National Gunfire Index‘s methodology and analysis, it comes as a surprise that ShotSpotter actually made up data to fill out gaps in its observed gunfire tracking. The note on methodology at the end of the Index explains how this worked in considerable detail (emphasis ours)[10]
So, in other words, up to 45% of any particular community’s data over a six month period was “imputed” by means of this process of “proration.”
That explains a lot.
Notes
[1] “ShotSpotter Index Measures a Large Decrease in Gun Violence.” www.camdencounty.com, October 8, 2014. http://www.camdencounty.com/county-news/shotspotter-index-measures-large-decrease-gun-violence. Accessed 2014/11/-21.
[2] “ShotSpotter Success: Gunfire down by 26 percent in Kansas City areas by ShotSpotter following transit-police partnership.” www.kcata.orgOct 13, 2014. http://www.kcata.org/news/spotshotter_success. Accessed 2014/11/21.
[3] “2014 National Gunfire Index.” [Web version.] http://www.shotspotter.com/1H2014NGI
[4] 2014 National Gunfire Index, p. 7. [PDF, 2014.]http://www.shotspotter.com/download-2014ebook. Also archived at http://192.168.42.120/wp-content/uploads/2014/11/ShotSpotter_2014NGI-eBook.pdf
[5] “2014 National Gunfire Index.” [Web version.] http://www.shotspotter.com/1H2014NGI
[6] 2014 National Gunfire Index, pp. 5, 7. [PDF, 2014.]http://www.shotspotter.com/download-2014ebook. Also archived at http://192.168.42.120/wp-content/uploads/2014/11/ShotSpotter_2014NGI-eBook.pdf
[7] “ShotSpotter Index Measures a Large Decrease in Gun Violence.” www.camdencounty.com, October 8, 2014. http://www.camdencounty.com/county-news/shotspotter-index-measures-large-decrease-gun-violence. Accessed 2014/11/-21.
[8] 2014 National Gunfire Index, p. 6. [PDF, 2014.]http://www.shotspotter.com/download-2014ebook. Also archived at http://192.168.42.120/wp-content/uploads/2014/11/ShotSpotter_2014NGI-eBook.pdf
[9] “FBI Uniform Crime Reports: Crime in the United States 2013: Table 1.” http://www.fbi.gov/about-us/cjis/ucr/crime-in-the-u.s/2013/crime-in-the-u.s.-2013/tables/1tabledatadecoverviewpdf/table_1_crime_in_the_united_states_by_volume_and_rate_per_100000_inhabitants_1994-2013.xls. Downloadable as a spreadsheet at http://www.fbi.gov/about-us/cjis/ucr/crime-in-the-u.s/2013/crime-in-the-u.s.-2013/tables/1tabledatadecoverviewpdf/table_1_crime_in_the_united_states_by_volume_and_rate_per_100000_inhabitants_1994-2013.xls/output.xls.
[10] 2014 National Gunfire Index, p. 10. [PDF, 2014.]http://www.shotspotter.com/download-2014ebook. Also archived at http://192.168.42.120/wp-content/uploads/2014/11/ShotSpotter_2014NGI-eBook.pdf