ShotSpotter makes up its gunfire data, but it STILL doesn’t make any sense

SST, Inc., the company that sells ShotSpotter gunfire-detection systems to regional governments around the world, recently published a marketing pamphlet called the 2014 National Gunfire Index. The Seattle City Council might avoid purchasing another police boondoggle if it examines the phony data and confused arguments that pad out this piece of fake science.

Seattle Privacy Coalition ran across this document while compiling press clippings on ShotSpotter. Beginning in October, glowing press releases and media reports about the success of ShotSpotter began popping up around the country. For example, in Camden, NJ:

[County Commission] Director Louis Cappelli, Mayor Dana Redd and Chief Scott Thomson announced that ShotSpotter Flex[,] the global leader in gunfire detection and analysis, today announced that its National Gunfire Index revealed that gunfire incidents in Camden City for the first half of 2014 are down by 48.5 percent, compared to first half of 2013, where ShotSpotter was deployed during both periods.[1]

And in Kansas City, MO:

Newly released data from the makers of the ShotSpotter gunshot detection system indicates that gunfire has decreased significantly in Kansas City’s urban core over the last year.

The ShotSpotter system covers 3.55 square miles in Kansas City near the Troost MAX bus line. In comparing the first half of 2013 to the first half of 2014, gunfire incidents in those areas fell by 25.9 percent. That’s 55 fewer incidents. That keeps with the trend in 31 other cities across the United States and Caribbean that ShotSpotter serves: those cities averaged 25.9 percent fewer gunfire incidents, as well.[2]

Wow, those are amazing year-over-year crime reduction numbers! And the clear implication is that SpotShotter made this happen.

Let’s have a look at that data

An examination of ShotSpotter’s data and research methodology dispels any hope that it has a basis in legitimate science. There are several separate and false claims to be debunked.

False Claim 1 — Gunfire declined in SpotShotter cities from 2013 to 2014

Bizarrely, this most basic claim, the one politicians and media picked up on, is proved false (or unintelligible) by ShotSpotter’s own figures. These are summarized in a graphic[3]:

The researchers analyze their raw data (which consists of detected gunshots) in two ways, as total rounds fired, and as “incidents.” Gunfire “incidents” are never actually defined, nor are we told why this is a useful measure. As the graphic shows, incidents declined sharply in the 31 communities studied, while absolute rounds fired increased, and this 20.6% decline is what the the report touts on 7 of its 9 pages of content. The rounds-fired figure is mentioned on 3 pages.

The facts become murkier when we come to this baffling statement:

Rounds (bullets) fired per gunfire incident were up by 36%. On average, 3.2 rounds were fired per incident during the first half of 2014, up 10% from first half 2013 average of 2.9 rounds per incident.[4]

Now there are three different figures for rounds-fired:

The supporting graphic (Web version[5]) both hides the high number and adds to the confusion with a whole new measure called “Total Number of Rounds Fired Per Incident” (emphasis ours). To see this, you have to mouse-over the bullet images, as shown in the following before-and-after versions:

Added confusion stems from the complete meaninglessness of “total rounds per incident” and how this relates to “average rounds.”

ShotSpotter does provide some actual raw numbers[6] (more on that below) that supposedly back up its generalizations.

	Total Incidents	Total Rounds Fired
2013	14,703	42,830
2014	11,675	58,087
Year-over-year change	-20%	+36%

At least that clarifies which of the calculated numbers are real, such as the high “total-rounds-per” number that the report tried to hide and fails to explain, and which turns out to be the real figure for rounds-per-incident. The scale of the problem becomes clear in the following graph — created by Seattle Privacy Coalition, not ShotSpotter — which illustrates the public safety significance of ShotSpotter’s two measures (as we understand them):

This makes it pretty clear that the “incidents” measure is here to obscure the fact that gunfire increased by 36% in the 31 ShotSpotter communities during the the period of the study. People dodging bullets don’t care how many people are firing at them. Yet these are the statistics that embolden SST President and CEO, Ralph A. Clark, to tell the Camden, NJ, newspaper:

“The gunfire index data is extremely encouraging and suggests what cities and their law enforcement agencies can accomplish with a comprehensive gun violence reduction effort focused on enhanced response and community engagement.”[7]

Or, as the Index itself puts it,

Gunfire incidents are down in almost every ShotSpotter Flex city. In the 31 communities that we were able to analyze both for 1H2013 and 1H2014, gunfire incidents were down in 28 of the 31 communities, or 90% of them.[8]

False Claim 2 — SpotShotter is responsible for reducing gun violence

Things look bad for ShotSpotter. Far from reducing gun violence, Its figures suggest it has aggravated gun violence in the communities where it is deployed. The only thing that saves it from that humiliating finding is the iron rule of statistics: correlation is not causation. ShotSpotter cannot actually be blamed for an increase in gun violence without controlled studies that rule out other factors that may be causing the increase. Furthermore, the sloppiness of the arithmetic and reasoning in the 2014 National Gunfire Index make us wary of actually trusting the figures presented. Without more data, there is no way to know how much damage ShotSpotter is or is not causing.

On the other hand, reliable independent crime statistics tell a story that is unhelpful to ShotSpotter’s case regardless of the soundness of the gunfire report data. The FBI’s uniform crime statistics document[9] a steady decline of all violent crime nationwide over the past 20 years:

This trend suggests that a decline of about 3% in the overall violent crime rate probably occurred between 2013 and 2014. Any claim that ShotSpotter reduces crime would have to take into account this background decline. It is troubling but not surprising that the study ignores this, since it is much more enjoyable to claim credit for whatever good is happening on your watch. Of course, that’s not science.

The most plausible inference to make in the face of the FBI’s figures is that ShotSpotter’s figures, showing a 36% increase in gunfire over the last year, are simply too aberrant to be trusted without confirmation by qualified researchers.

False Claim 3 — ShotSpotter bases its claims on real-world data

ShotSpotter spends a lot of time in this report stressing its careful comparison of “apples-to-apples” data. Unfortunately, it appears that the researchers only got halfway through that research methods course. Even knowing the shoddiness of the National Gunfire Index‘s methodology and analysis, it comes as a surprise that ShotSpotter actually made up data to fill out gaps in its observed gunfire tracking. The note on methodology at the end of the Index explains how this worked in considerable detail (emphasis ours)[10]

So, in other words, up to 45% of any particular community’s data over a six month period was “imputed” by means of this process of “proration.”

That explains a lot.

Notes

[1] “ShotSpotter Index Measures a Large Decrease in Gun Violence.” www.camdencounty.com, October 8, 2014. http://www.camdencounty.com/county-news/shotspotter-index-measures-large-decrease-gun-violence. Accessed 2014/11/-21.

[2] “ShotSpotter Success: Gunfire down by 26 percent in Kansas City areas by ShotSpotter following transit-police partnership.” www.kcata.orgOct 13, 2014. http://www.kcata.org/news/spotshotter_success. Accessed 2014/11/21.

[3] “2014 National Gunfire Index.” [Web version.] http://www.shotspotter.com/1H2014NGI

[4] 2014 National Gunfire Index, p. 7. [PDF, 2014.]http://www.shotspotter.com/download-2014ebook. Also archived at http://192.168.42.120/wp-content/uploads/2014/11/ShotSpotter_2014NGI-eBook.pdf

[5] “2014 National Gunfire Index.” [Web version.] http://www.shotspotter.com/1H2014NGI

[6] 2014 National Gunfire Index, pp. 5, 7. [PDF, 2014.]http://www.shotspotter.com/download-2014ebook. Also archived at http://192.168.42.120/wp-content/uploads/2014/11/ShotSpotter_2014NGI-eBook.pdf

[7] “ShotSpotter Index Measures a Large Decrease in Gun Violence.” www.camdencounty.com, October 8, 2014. http://www.camdencounty.com/county-news/shotspotter-index-measures-large-decrease-gun-violence. Accessed 2014/11/-21.

[8] 2014 National Gunfire Index, p. 6. [PDF, 2014.]http://www.shotspotter.com/download-2014ebook. Also archived at http://192.168.42.120/wp-content/uploads/2014/11/ShotSpotter_2014NGI-eBook.pdf

[9] “FBI Uniform Crime Reports: Crime in the United States 2013: Table 1.” http://www.fbi.gov/about-us/cjis/ucr/crime-in-the-u.s/2013/crime-in-the-u.s.-2013/tables/1tabledatadecoverviewpdf/table_1_crime_in_the_united_states_by_volume_and_rate_per_100000_inhabitants_1994-2013.xls. Downloadable as a spreadsheet at http://www.fbi.gov/about-us/cjis/ucr/crime-in-the-u.s/2013/crime-in-the-u.s.-2013/tables/1tabledatadecoverviewpdf/table_1_crime_in_the_united_states_by_volume_and_rate_per_100000_inhabitants_1994-2013.xls/output.xls.

[10] 2014 National Gunfire Index, p. 10. [PDF, 2014.]http://www.shotspotter.com/download-2014ebook. Also archived at http://192.168.42.120/wp-content/uploads/2014/11/ShotSpotter_2014NGI-eBook.pdf

November 21, 2014November 23, 2014

Revisiting ShotSpotter with Seattle Council

At or around 10:00 am on Monday, November 24, as part of their budget deliberations, the Seattle City Council will once again take up the question of whether or not to fund acquisition of an “acoustic gunshot locator system,” more familiarly known to readers of this blog and law-enforcement watchers as ShotSpotter. (Agenda).

To prep for talking to Council during the public-hearing/performance-art section of Monday’s meeting, we’ve refreshed our ShotSpotter Fact Sheet to help inform council members and the public about the problems with this technology.

ShotSpotter (SST, Inc.) Fact Sheet

In addition to marveling at the evidence that the system manages to be expensive, intrusive, AND ineffective at the same time, we’re asking ourselves, What’s the rush? Why no public discussion?

In fact, this proposal would be a good one to run by the privacy advisory committee that Council Members Harrell and O’Brien, and Mayor Murray are still putting together. Attempting to ram it through now, before the committee’s first meeting (December 2) might lead a cynic to wonder if someone is trying to beat the committee’s recommendations.

If Council and the Mayor are serious about this privacy initiative, they should put the ShotSpotter proposal on the back burner.

We don’t really expect to get more information about the internal discussion at the city about ShotSpotter in time for Monday’s discussion, but for those who’d like to follow along, here is Phil’s PRA request for any guidance provided by Central Staff to City Council on this topic:
<https://www.muckrock.com/foi/seattle-69/acoustic-gunshot-locator-system-guidance-from-city-council-central-staff-14295/>

For your reference, here’s a PDF of the “green sheet” that puts a proviso (i.e., restriction) on the funds allocated for ShotSpotter. Council will discuss this issue at the Budget Committee meeting at 10:00 am or after the Full Council meeting.

Hope to see you Monday!

November 19, 2014December 5, 2014

In search of privacy advocacy from underrepresented communities

In an attempt to bring together various privacy stakeholders in Seattle, particularly from the Muslim community, I attended my first Muslim, Sikh and Arab Advisory Council meeting looking for specific privacy cases to learn about. I later wrote to the Seattle Police Department with various questions and concerns. In light of the City of Seattle establishing a welcomed Privacy Initiative, it seems prudent to involve privacy stakeholders from various local and underrepresented communities.

Below is an email sent to a Seattle Police Department program manager regarding the the Muslim, Sikh and Arab Advisory Council to the Seattle Police Department on November 19th, 2014. The program manager, Maggie Olsen, promptly replied stating that my email had been forwarded to the Commander of the Community Outreach Section, Captain John Hayes. Jan and I have not yet received a reply.

Hello Maggie,

My name is Christopher Sheats, a concerned citizen of Seattle and a volunteer for the Seattle Privacy Coalition (SPC) (seattleprivacy.org).
My colleague Jan is CC'd, she is the SPC director. I'm writing with regard to the Muslim, Sikh and Arab (MSA) Advisory Council to the Seattle Police Department.

This email may be better directed to Detective Yanal Vwich, or possibly Chief Kathleen O'Toole. I attended my first MSA meeting on October 2nd, 2014. I advertised that the Citizens Technology and Telecommunications Advisory Board (CTTAB) was planning a privacy symposium to focus on the privacy impact to vulnerable populations in Seattle. The privacy symposium is supposed to happen next year, but I am not yet sure about details.

I also want Detective Yanal Vwich and the MSA to be aware that the City of Seattle is establishing a special, and likely permanent, privacy advisory board for the city. For more information about the new privacy
board:

CITY OF SEATTLE LAUNCHES DIGITAL PRIVACY INITIATIVE http://murray.seattle.gov/city-of-seattle-launches-digital-privacy-initiative/

Seattle Takes the Lead in Nationwide Surveillance vs. Privacy Debate http://192.168.42.120/press-release-response-to-seattle-privacy-initiative/

Composition of City’s Privacy Advisory Board (written by Jan) http://192.168.42.120/composition-of-citys-privacy-advisory-board/

I have several questions, please help where possible.

Q1- It appears that MSA and the East African Advisory Council's have been combined and that their meeting schedule has been severely reduced.
Why is that?

Q2- Given the privacy and trust implications of these vulnerable populations, I was surprised to learn that these meetings were being held at a government facility. I know that this has a negative impact on attendance. Why can't it be changed to a community center, with more access to bus routes?

NSA Surveillance Chilling Effects: HRW and ACLU Gather More Evidence https://www.eff.org/deeplinks/2014/07/nsa-surveillance-chilling-effects

Q3- Would't it be prudent to have a member of the Muslim community to be directly involved with the Seattle Privacy Advisory Board?

Latest Snowden Leaks: FBI Targeted Muslim-American Lawyers http://www.wired.com/2014/07/snowden-leaks/

One of the attendees bluntly accused that the FBI was spying on people in his community. The FBI attendee blatantly lied in response--given the Snowden revelations that go into specific proof that says otherwise. The Seattle City Council, the Seattle Police Department, local FBI offices, and the City's IT Dept all share the responsibility in gaining trust though cooperation and privacy enhancements.

Having gone to just one MSA meeting, I was almost overwhelmed with the amount of distrust between attending members of vulnerable populations and the various US Gov attendees. As an empathetic white male, it was uncomfortable.

On behalf of the Seattle Privacy Coalition, I would like to assist wherever possible to help bridge this divide, regarding trust through privacy. Please share the following information to the respective Advisory Councils where appropriate. It would also be great to get members of MSA involved with our citizen group (Seattle Privacy Coalition) for greater diversity:

Send an empty email to the following address to be put on our announce
list:
seattleprivacy-announce-unsubscribe@lists.riseup.net

Seattle Privacy Coalition can be contacted directly:
contact@192.168.42.120

I can be contacted directly, and I have a public PGP key available:
yawnbox@riseup.net
7DFF 4EE5 1C63 9060 C9C9
A48A BEAF 1420 523A EB46

If any of you would like help setting up your own PGP key-pair so that people can securely contact you, or an alternative secure chat application, please let me know.

Lastly, with your permission, I'd like to publish any or all responses to our blog, SeattlePrivacy.org, so that the public can be further educated about these ongoing issues.

Cheers

Christopher Sheats
citizen

November 14, 2014November 14, 2014

“If not for Seattle, this history would be different”

Laura Poitras’s Citizenfour reminds us that courage is local

A few days before the Seattle City Council announced its precedent-setting privacy initiative, the year’s most anticipated documentary, Citizenfour, opened at the Uptown SIFF Cinema. Laura Poitras’s third film about the post-9/11 American security state tells the story of Edward Snowden, the NSA whistle-blower who made “dragnet surveillance” a household term.

Seattle’s step toward privacy and accountability was well-covered in the local press and also made the leap to a couple of governance trade journals. Seattle Privacy made sure that Laura Poitras herself knew what had happened here at the same time that her film was drawing capacity crowds. She sent us congratulations:

It is fitting that Seattle is first to respond – it is the home of NSA
PRISM partners such as Microsoft, as well a strong community of people
building alternatives to dragnet surveillance. These alternatives, as
well as informing and engaging with the people of Seattle, are a step
toward regaining meaningful democratic oversight relating to security
and privacy in our country.

If not for Seattle, this history would be different.

When the Seattle Privacy Coalition came together in early 2013, the city’s political establishment issued us the tin-foil hats reserved for people who worry about government surveillance. The disgraced, federally supervised Seattle Police Department was so used to getting its way in technology matters that it shrugged off negative public reaction to the “port security” camera network. In talks with city officials, we provoked eye-rolls and knowing smirks by suggesting that the city should pass up federal grant money that paid for boondoggles such as police drones. [Note: See the update at the end of this post. It ain’t over.]

After Snowden, the complacency was gone. Little has changed at the national or state levels — the security agencies still run Congress and the White House, Boeing still dictates to Olympia. But locally, there is movement. DHS-funded spying and cops in tanks have become issues with names: Oakland, Ferguson. The city establishment’s dread of controversy now works in favor of privacy advocates. The security lobby will have a hard time influencing every petty municipality the way it influences the federal government.

An evolving model for political action emerges from Citizenfour. In a world where democracy and the press have ceased to function at the highest levels, we watch lone individuals making fateful choices grounded in their private experience. These precise moments of integrity contrast with farcically mediated global contexts: archival footage of NSA Director Keith Alexander and National Security Director James Clapper telling extravagant lies to Congress; a frantic scrum of boom-bearing reporters around Glenn Greenwald and his partner (and taking care to edit themselves out of the film they will broadcast); or the recurring apparition of Wolf Blitzer playing Wolf Blitzer. Always there is a strong implicit case for what it real and what is not, and where personal agency lies.

“There’ll be the breaking of the ancient western code / Your private life will suddenly explode.” — Leonard Cohen

Poitras, not Snowden, is the first example of this in Citizenfour. Out of the blue, Snowden sends her an encrypted email message, an event recreated on-screen as white text unspooling in the black void of a Linux computer terminal. Disembodied in this weirdly intimate environment, an as yet anonymous Snowden tells her he is a spy, that he has classified disclosures to make, that there is great danger, and that their joint government adversary can attempt one trillion password cracks per second. Her private decision to accept this mysterious challenge leads to the events of the movie. When she later asks “Citizenfour” why he had chosen her, He tells her, “You chose yourself.”

Poitras next tells the story of NSA veteran William Binney. After the end of the Cold War, he developed systems to automate the collection and analysis of telecommunications metadata. Originally, the targets were foreign, but shortly after 9/11, NSA turned Binney’s work into the basis of its new program of blanket domestic surveillance. His internal protests against NSA’s lawless, ineffective, and wasteful policies went nowhere, and he soon left the agency. After being raided at gun-point in 2007 during an FBI leak investigation (in which he was later cleared), Binney gained prominence as one of the most outspoken NSA whistle-blowers prior to Snowden.

The misguided raid on Binney was provoked in part by the revelations of Mark Klein, who is not actually in the movie, though we do see a hearing from one the lawsuits that resulted. Klein was a technician for AT&T who discovered that Room 641a at 611 Folsom Street in San Francisco was an NSA diversion site for all of AT&T’s Internet and telephone traffic. Appalled by what amounted to a tap on the entire Internet, Klein took his story to the Los Angeles Times, which refused (under government pressure) to print it. He next took it to the New York Times, which also bowed to government pressure for a year before finally publishing it in 2005.

Seattle Privacy’s co-founder Jacob Appelbaum turns up twice in the film, once before and once after his NSA reporting forced him into Berlin exile. In one segment, he presses an Occupy Wall Street audience to consider whether they have been personally under surveillance, and lists ways it could have happened — not just by means of telephones, email, and the Web, but also credit cards, travel passes, etc. He calls them canaries in a coal mine who are experiencing what everybody will experience in the near future. (As Jacob likes to say, “My present is your future,” though he now thinks the future has pretty much arrived for everyone.) The personal experience entails the universal problem, and is the key to fighting it.

We also meet Ladar Levison, the [former] proprietor of the secure email service Lavabit. Its most famous customer: Edward Snowden. Levison built an encrypted mail service that collected no information on its users, and thus had nothing to give law enforcement even when subpoenaed. Unable to identify Snowden’s correspondents in the usual way by seizing metadata, the FBI told Levison to give up Lavabit’s master SSL encryption keys, which would allow them to uncloak the entire Lavabit customer base secretly in real time. Levison instead shut down his business rather than betray his customers’ privacy. Try to imagine that in a corporatized setting where profit is paramount and ethical concerns are actionable in civil court.

In bare outline, Snowden’s own story is that he gave up his prior life and risked life imprisonment (or worse) to expose the actions of NSA and its partners. Most will remember his principled if fatalistic rationale from the original June 2013 interview. In Citizenfour, Snowden’s anxiety and regret become palpable. He masters his fear and steps through the hotel room door into what may be the waiting arms of a hostile government. Though Snowden repeatedly downplays his role in leaking the documents — “I’m not the story” — his choice is the story.

At Seattle Privacy, we hope to change how citizens are treated by their local government and by the police. The recent good news notwithstanding, we will continue to push the City Council to follow through on its stated intentions. We don’t want the promised oversight structure to end up a dead letter like Ordinance 124142, another privacy “first” that was passed 18 months ago and never enforced. At stake is a role for Seattle as a national model of awakened democratic government. It took bold individuals to expose the corrupt surveillance state, and it will take a bold community to prove Laura Poitras right: “If not for Seattle, this history would be different.”

Update:

Even as I wrote and published this, the City Council threatened to reverted business-as-usual by planning a budget hearing for a ShotSpotter-type system. For information about the city’s past flirtation with outdoor audio surveillance (and some sleazy video of Seattle politics at its worst) see our ShotSpotter fact sheet. Rest assured we will communicate to our leaders what we think of their renewed interest in ShotSpotter.

November 7, 2014November 7, 2014

Composition of City’s Privacy Advisory Board

I was curious about how the city was going to populate its newly formed Privacy Advisory Board (which, hurray!) so I asked Ginger in the Department of Information Technology. This was a huge question we had at Seattle Privacy Coalition when we first started trying to figure out how something like this would look.

Here’s what she wrote to me in email.

“As we considered this group of advisers, we wanted to reach out to a broad range of experts in the field of privacy as we work on our privacy initiative. This includes thought-leaders with expertise from academia, including the areas of law, technology, ethics, philosophy and political science; private and federal law; government and corporate privacy policy and compliance; open data; cyber security, civil rights, citizen advocacy and community members with a strong personal investment in our local policies and practices. We want this group to review our work, provide meaningful feedback and challenge us to reach beyond the usual bureaucratic processes and decision-making.

“We have been gathering names over the past several months from internal and external privacy professionals and are now speaking and meeting with potential members to invite their participation. We are hoping for 7-9 members with this vast body of experience and the good news is that we are making progress.”

I know the Privacy Advisory Board meets for the first time on December 2, so I’m sure they’re scrambling to get people to commit.

It’s too early to be anything but grateful that this is happening at all, but I do wish there was explicit mention of a hardware engineer or a cryptographer. I guess probably “cyber security” refers to system administrators, although just having the word “cyber” in there at all gives me shivers of discomfort.

Here’s the list we had originally envisioned, which came from 8-10 people sitting in Office Nomads, tossing out ideas:

University of Washington security and privacy research center
Representative from the police accountability and review board
Computer hardware developer
Security software developer
Sociologist
Two attorneys with expertise in privacy law
Near Future Security Researcher
ACLU technology expert
Community member(s) with stake in free expression (artist? blogger?)
Small business owner

Even at the time, however, it didn’t feel complete. Not enough “general public” representation. Not enough attention to vulnerable populations, who are on the front lines of privacy violations. Still, there’s enough of an overlap that it feels like a good start.

As with other committees that this Mayor has formed, the process of identifying people to serve has been quiet and behind the scenes. I look forward to seeing who the people are who are willing and able to serve on this probably fairly challenging committee. Naturally, SPC will be doing a gap analysis on the board’s composition, after we hear more about it.