ShotSpotter makes up its gunfire data, but it STILL doesn’t make any sense

SST, Inc.,  the company that sells ShotSpotter gunfire-detection systems to regional governments around the world, recently published a marketing pamphlet called the 2014 National Gunfire Index. The Seattle City Council might avoid purchasing another police boondoggle if it examines the phony data and confused arguments that pad out this piece of fake science.

Seattle Privacy Coalition ran across this document while compiling press clippings on ShotSpotter. Beginning in October, glowing press releases and media reports about the success of ShotSpotter began popping up around the country. For example, in Camden, NJ:

[County Commission] Director Louis Cappelli, Mayor Dana Redd and Chief Scott Thomson announced that ShotSpotter Flex[,] the global leader in gunfire detection and analysis, today announced that its National Gunfire Index revealed that gunfire incidents in Camden City for the first half of 2014 are down by 48.5 percent, compared to first half of 2013, where ShotSpotter was deployed during both periods.[1]

And in Kansas City, MO:

Newly released data from the makers of the ShotSpotter gunshot detection system indicates that gunfire has decreased significantly in Kansas City’s urban core over the last year.

 

 

The ShotSpotter system covers 3.55 square miles in Kansas City near the Troost MAX bus line. In comparing the first half of 2013 to the first half of 2014, gunfire incidents in those areas fell by 25.9 percent. That’s 55 fewer incidents. That keeps with the trend in 31 other cities across the United States and Caribbean that ShotSpotter serves: those cities averaged 25.9 percent fewer gunfire incidents, as well.[2]

Wow, those are amazing year-over-year crime reduction numbers! And the clear implication is that SpotShotter made this happen.

Let’s have a look at that data

An examination of ShotSpotter’s data and research methodology dispels any hope that it has a basis in legitimate science. There are several separate and false claims to be debunked.

False Claim 1 — Gunfire declined in SpotShotter cities from 2013 to 2014

Bizarrely, this most basic claim, the one politicians and media picked up on, is proved false (or unintelligible)  by  ShotSpotter’s own figures. These are summarized in a graphic[3]:

shot-2014-11-21_23-54-58

The researchers analyze their raw data (which consists of detected gunshots) in two ways, as total rounds fired, and as “incidents.” Gunfire “incidents” are never actually defined, nor are we told why this is a useful measure. As the graphic shows, incidents declined sharply in the 31 communities studied, while absolute rounds fired increased, and this 20.6% decline is what the the report touts on 7 of its 9 pages of content. The rounds-fired figure is mentioned on 3 pages.

The facts become murkier when we come to this baffling statement:

Rounds (bullets) fired per gunfire incident were up by 36%. On average, 3.2 rounds were fired per incident during the first half of 2014, up 10% from first half 2013 average of 2.9 rounds per incident.[4]

Now there are three different figures for rounds-fired:

  • 14%
  • 36%
  • 10%

The supporting graphic (Web version[5]) both hides the high number and adds to the confusion with a whole new measure called “Total Number of Rounds Fired Per Incident” (emphasis ours). To see this, you have to mouse-over the bullet images, as shown in the following before-and-after versions:

two_views

Added confusion stems from the complete meaninglessness of “total rounds per incident” and how this relates to “average rounds.” 

ShotSpotter does provide some actual raw numbers[6] (more on that below) that supposedly back up its generalizations.

Total Incidents Total Rounds Fired
2013 14,703 42,830
2014 11,675 58,087
Year-over-year change  -20%  +36%

At least that clarifies which of the calculated numbers are real, such as the high “total-rounds-per” number that the report tried to hide and fails to explain, and which turns out to be the real figure for rounds-per-incident. The scale of the problem becomes clear in the following graph — created by Seattle Privacy Coalition, not ShotSpotter — which illustrates the public safety significance of ShotSpotter’s two measures (as we understand them):

NGI

This makes it pretty clear that the “incidents” measure is here to obscure the fact that gunfire increased by 36% in the 31 ShotSpotter communities during the the period of the study.  People dodging bullets don’t care how many people are firing at them. Yet these are the statistics that embolden SST President and CEO, Ralph A. Clark, to tell the Camden, NJ, newspaper:

“The gunfire index data is extremely encouraging and suggests what cities and their law enforcement agencies can accomplish with a comprehensive gun violence reduction effort focused on enhanced response and community engagement.”[7]

Or, as the Index itself puts it,

Gunfire incidents are down in almost every ShotSpotter Flex city. In the 31 communities that we were able to analyze both for 1H2013 and 1H2014, gunfire incidents were down in 28 of the 31 communities, or 90% of them.[8]

False Claim 2 — SpotShotter is responsible for reducing gun violence

Things look bad for ShotSpotter. Far from reducing gun violence, Its figures suggest it has aggravated gun violence in the communities where it is deployed. The only thing that saves it from that humiliating finding is the iron rule of statistics: correlation is not causation. ShotSpotter cannot actually be blamed for an increase in gun violence without controlled studies that rule out other factors that may be causing the increase. Furthermore, the sloppiness of the arithmetic and reasoning in the 2014 National Gunfire Index make us wary of actually trusting the figures presented. Without more data, there is no way to know how much damage ShotSpotter is or is not causing.

On the other hand, reliable independent crime statistics tell a story that is unhelpful to ShotSpotter’s case regardless of the soundness of the gunfire report data. The FBI’s uniform crime statistics document[9] a steady decline of all violent crime nationwide over the past 20 years:

FBI

This trend suggests that a decline of about 3% in the overall violent crime rate probably occurred between 2013 and 2014. Any claim that ShotSpotter reduces crime would have to take into account this background decline. It is troubling but not surprising that the study ignores this, since it is much more enjoyable to claim credit for whatever good is happening on your watch. Of course, that’s not science.

The most plausible inference to make in the face of the FBI’s figures is that ShotSpotter’s figures, showing a 36% increase in gunfire over the last year, are simply too aberrant to be trusted without confirmation by qualified researchers.

False Claim 3 —  ShotSpotter bases its claims on real-world data

ShotSpotter spends a lot of time in this report stressing its careful comparison of “apples-to-apples” data. Unfortunately, it appears that the researchers only got halfway through that research methods course. Even knowing the shoddiness of the National Gunfire Index‘s methodology and analysis, it comes as a surprise that ShotSpotter actually made up data to fill out gaps in its observed gunfire tracking. The note on methodology at the end of the Index explains how this worked in considerable detail (emphasis ours)[10]

shot-2014-11-21_23-26-56

So, in other words, up to 45% of any particular community’s data over a six month period was “imputed” by means of this process of “proration.”

That explains a lot.

 

Notes

[1] “ShotSpotter Index Measures a Large Decrease in Gun Violence.” www.camdencounty.com, October 8, 2014. http://www.camdencounty.com/county-news/shotspotter-index-measures-large-decrease-gun-violence. Accessed 2014/11/-21.

[2] “ShotSpotter Success: Gunfire down by 26 percent in Kansas City areas by ShotSpotter following transit-police partnership.” www.kcata.orgOct 13, 2014. http://www.kcata.org/news/spotshotter_success. Accessed 2014/11/21.

[3] “2014 National Gunfire Index.” [Web version.] http://www.shotspotter.com/1H2014NGI

[4] 2014 National Gunfire Index, p. 7. [PDF, 2014.]http://www.shotspotter.com/download-2014ebook. Also archived at http://192.168.42.120/wp-content/uploads/2014/11/ShotSpotter_2014NGI-eBook.pdf

[5] “2014 National Gunfire Index.” [Web version.] http://www.shotspotter.com/1H2014NGI

[6] 2014 National Gunfire Index, pp. 5, 7. [PDF, 2014.]http://www.shotspotter.com/download-2014ebook. Also archived at http://192.168.42.120/wp-content/uploads/2014/11/ShotSpotter_2014NGI-eBook.pdf

[7] “ShotSpotter Index Measures a Large Decrease in Gun Violence.” www.camdencounty.com, October 8, 2014. http://www.camdencounty.com/county-news/shotspotter-index-measures-large-decrease-gun-violence. Accessed 2014/11/-21.

[8] 2014 National Gunfire Index, p. 6. [PDF, 2014.]http://www.shotspotter.com/download-2014ebook. Also archived at http://192.168.42.120/wp-content/uploads/2014/11/ShotSpotter_2014NGI-eBook.pdf

[9] “FBI Uniform Crime Reports: Crime in the United States 2013: Table 1.” http://www.fbi.gov/about-us/cjis/ucr/crime-in-the-u.s/2013/crime-in-the-u.s.-2013/tables/1tabledatadecoverviewpdf/table_1_crime_in_the_united_states_by_volume_and_rate_per_100000_inhabitants_1994-2013.xls. Downloadable as a spreadsheet at http://www.fbi.gov/about-us/cjis/ucr/crime-in-the-u.s/2013/crime-in-the-u.s.-2013/tables/1tabledatadecoverviewpdf/table_1_crime_in_the_united_states_by_volume_and_rate_per_100000_inhabitants_1994-2013.xls/output.xls.

[10] 2014 National Gunfire Index, p. 10. [PDF, 2014.]http://www.shotspotter.com/download-2014ebook. Also archived at http://192.168.42.120/wp-content/uploads/2014/11/ShotSpotter_2014NGI-eBook.pdf

One Reply to “ShotSpotter makes up its gunfire data, but it STILL doesn’t make any sense”

  1. To: The Seattle Privacy Coalition
    Date: November 26, 2014

    We appreciate the chance to address and correct a few points made in the article, “ShotSpotter makes up its gunfire data, but it STILL doesn’t make any sense”. The article puts forth three supposedly false claims, which we will speak to in the order they appeared:
    False Claim 1 — Gunfire declined in ShotSpotter cities from 2013 to 2014
    “Gunfire ‘incidents’ are never actually defined, nor are we told why this is a useful measure.”
    “Added confusion stems from the complete meaninglessness of ‘total rounds per incident’ and how this relates to ‘average rounds.’”

    We did not define the term ‘incident’ in our First Half 2014 Gunfire Index. However, the definition of ‘incident’ it is made perfectly clear in our 2013 Full Year Index. An ‘incident’ is one or more gunshots (multiple rounds) that have been detected by ShotSpotter and, based on time and location, can reasonably be grouped into a single incident. To see the Full Year 2013 index, go here.

    As the most complete source of gunfire data in cities where ShotSpotter is deployed, we can state for certain that incidents of gunfire definitely *did* decrease, significantly, by 20.6% from 14,703 in 31 communities monitored by ShotSpotter in 1H2013 compared to 11,675 in the same 31 communities in 1H2014.

    Furthermore, the ‘total rounds per incident’ is the total number of gunshots (bullets fired) detected by ShotSpotter in the coverage area we’re analyzing. The ‘average rounds’ is the average of gunshots per incident. The average adjusts for the change in the number of incidents. Both numbers are used because they measure different things –
     The ‘number of incidents’ is a reflection of the number of gun-related crimes.
     The ‘number of rounds per incident’ provides and indication of the overall intensity of multiple round shooting event and to some degree the use of higher capacity firearms.

    “This makes it pretty clear that the “incidents” measure is here to obscure the fact that gunfire increased by 36% in the 31 ShotSpotter communities during the period of the study. … Far from reducing gun violence, its figures suggest it has aggravated gun violence in the communities where it is deployed.”

    With the understanding above of differentiating incidents from rounds fired, you see the data showed that, while the average ShotSpotter community saw a 20.6% decrease, some police agencies and their respective cities saw much greater decreases of 40%, 50% and more. The maximum reduction we saw year-on-year was in Springfield, Massachusetts, where gunfire incidents decreased by 60% from 1H2013 to 1H2014.

    False Claim 2 — SpotShotter is responsible for reducing gun violence
    Correlation, causation and the “clear implication that ShotSpotter made this happen.”
    “… correlation is not causation.”

    ShotSpotter does not claim that we are exclusively and directly responsible for any decrease in gunfire incidents. But this cause-and-effect problem occurs for all crime-reduction efforts. There is no conclusive proof that can measure the direct result of any one specific, isolated, crime-fighting program or technology. Most of the agencies we work with combine several approaches into a comprehensive and proactive program to bring about a reduction in crime. We believe the most powerful attribute in reducing gun crime is community collaboration and cooperation with public safety agencies that have earned the trust in those communities. That trust is earned by providing higher levels of targeted responsiveness and service around gun crime that we know happens much more frequently than gets reported to police. We do believe very much that we help enable that process when our technology is combined with agency best practices and other gun crime reduction efforts.

    False Claim 3 — ShotSpotter bases its claims on real-world data
    The 14,703 gunfire incidents in 1H2013, and the 11,675 gunfire incidents in 1H2014 are an aggregation of our own database of actual shots fired in our ShotSpotter-covered areas. We have collected, analyzed and confirmed each of those incidents with both an audio and a visual review, before publishing out to the local law enforcement agency. That’s what we would refer to as ‘real world data.’

    “The supporting graphic (Web version[5]) both hides the high number …”
    “Bizarrely, this most basic claim, the one politicians and media picked up on, is proved false (or unintelligible) by ShotSpotter’s own figures.”

    Nothing is hidden. All of the numbers are presented the same way in the graphics and in the text. The Index highlighted the bad news of the increase in the number of rounds per incident the same way it highlighted the good news of the drop in the number of incidents.

    “… it comes as a surprise that ShotSpotter actually made up data to fill out gaps in its observed gunfire tracking.”
    “So, in other words, up to 45% of any particular community’s data over a six month period was ‘imputed’ by means of this process of ‘proration.’”

    Your writer shows a lack of understanding of basic statistical imputation.
     Missing data arise in almost all serious statistical analyses , and imputation is a standard and accepted technique for filling in missing data in a statistical analysis. It is similar to a seasonal adjustment. It was used for those communities that did not have full coverage over our comparison period. If, for example, a community started using ShotSpotter starting on February 1, 2013, we will be comparing 5 months of 2013 to 6 months of 2014. We must deal with the missing data from January 2013.
     If we simply take the averages, we are making the hidden assumption that January 2013 is the same as the average of the rest of 2013. But we know that the missing month does not necessarily look like the other 5 months due to seasonal variation. Imputation takes other information from all the rest of our data into account to make a more intelligent estimate for January. We did not “make up data”, and MOST importantly, we were very clear and transparent in footnoting the fact that we did use imputation.

    Those are the facts. We are a company that strives for perfection and are passionate about helping make a real difference in the public safety posture of many underserved communities while being commercially viable. We are open to having dialogue about the facts of what we do but are concerned when we see people or organizations purposely miss state and/or miss lead those facts. Our hope is that your blog posting was done hastily and led to conclusions based on misunderstandings versus intentional misdirection.

    Thank you in advance for publishing our rebuttal.

    SST, Inc.

Leave a Reply

Your email address will not be published.