Randomness: The Guarantee of Success

5 October 2016

Wednesday 21 September 2016 was the day of “Project Edward”, created by Tispol, the European Traffic Police Network, as the “European Day Without A Road Death”.

You read that right. A day without a road death anywhere in Europe—which sees on average around 70 deaths per day—just somehow happening.

Maybe they meant it was a project which took a step towards such a day? No: they were very specific. They meant exactly what they said on the tin:

“TISPOL’s target is that no one should die on the roads of Europe on Wednesday 21 September.”

So that was their target: that they would select a day in the near future and the 70-ish deaths that happen day in, day out would simply not happen.

Bold targets need bold action, so how would they achieve this?

Well, there was an online pledge form. They managed to get a little over 100,000 people to fill it in. “Impressive results” according to EU transport commissioner Violeta Bulc.

edward-bulc

In addition there were apparently “75 road safety actions”—a mighty 2.5 actions per Tispol member country. It’s not clear what the actions were, but from looking at Twitter it seemed that traffic police were basically doing the normal things that traffic police do, with a hashtag.

Some went out of their way, though. In Northamptonshire they took a break from pulling people over for offences to set up a trestle table and take some photos of people holding up cards pledging not to do things like commit the criminal offence of using their phone while driving. (Unfortunately the original tweet was deleted following a farcical argument about driving offences not being criminal offences.) Maybe they could use this approach for other criminal offences.

northants-rpu-not-a-crime

I’m forced to wonder whether a trestle table manned by police officers is the sort of thing that is going to attract the dangerous drivers of Europe, let alone cause a behaviour shift in their behaviour. I wonder the same about an online form which people find out about mainly by having previously subscribed to social media posts from police and road safety organisations and which they fill in because they want to spend two minutes of their day expressing a desire for decent road safety. These measures seem likely to reach only the people who don’t need to be reached.

Anyway, with all that hard-hitting action being undertaken to really make sure Wednesday was death-free, for simplicity’s sake let’s assume for the moment that it had no effect whatsoever and return to the matter of how statistically likely it was that Project Edward would meet its target.

Inevitable failure

Let’s look at just one country because, unsurprisingly, it’s the only one for which I’ve got detailed data to hand: the UK (well—England, Wales and Scotland anyway). What are the chances of even a UK-only Project Edward succeeding?

In order to estimate that, here’s what we’ll do. We’ll check the last three years and see how many of those days saw no fatal incidents. It’s not a perfect approach, but it’s good enough for the purposes of this argument.

So: three years, or 1095 days. And how many of those 1095 days were fatality-free? 14. So the probability of picking a day at random and seeing no fatalities in the UK is less than 1 in 78 (and those odds certainly won’t improve if we also take Northern Ireland into account). You’ve got a significantly better chance of pulling the ace of spades from a well-shuffled deck of cards.

In other words, there was just under a 99% chance that the UK mainland would single-handedly cause Project Edward to fail.

Now, imagine the probability of zero fatalities when you’re looking at the whole of the EU, with a total fatality rate fifteen times that of the UK. It’s vanishingly small. (You have a slightly better chance of buying one UK lottery ticket each week for four weeks and winning the jackpot every time.)

Failure was all but inevitable, yet Tispol decided to set this as an apparently genuine target.

But, as we’ll see, when it comes to the politics of “road safety”, the decision to set that target may not necessarily be quite as stupid as it first appears.

Why? Because the nature of randomly distributed events is that things look very different if you’re only looking at part of the picture.

Inevitable success

The crucial point is that the near-certainty of at least one fatality somewhere in the UK doesn’t mean near-certainty of at least one fatality everywhere in the UK, which can be broken down in a number of ways. There are, for example, 42 police forces in the area covered by STATS19—and if you pick a day at random there’s actually a very strong chance that a force will see zero fatalities.

For example: in Nottinghamshire, our 1095 days of data predict a 94% chance of no fatalities occurring on any randomly-chosen day. Which makes Nottinghamshire police’s tweet look like a bit of a fuss about nothing.

edward-notts

In Northamptonshire there’s also a 94% chance of no fatalities and, amazingly, their horse romped home as well. Maybe the trestle table was an effective lucky charm after all.

edward-northants

So far the tweets have only hinted at the relationship between Project Edward and the statistically extremely likely non-occurrence of local fatalities, but some went a little further.

Sussex and Surrey (89% and 93% respectively) reported that “Project Edward concluded successfully”. It’s an interesting thing to say: that this absence of death was a matter of Project Edward supposedly achieving its objective rather than one of overwhelming statistical likelihood. What they’re saying is that this highly probable non-event of no deaths locally implies the highly improbable success of the overall objective of no deaths across a vast area.

edward-sussex

edward-surrey

Lincolnshire is part of a wider road policing unit, which was lucky enough to be able to crow about zero fatalities across the whole of its area, an occurrence which—thanks to this enlarged area—had a mere 75% chance of occurring “during Project Edward”.

edward-lincs

North Wales is yet another area with a 94% chance of seeing zero deaths on any given day. But, for them, the occurrence of this very likely result means that it was a “successful Project Edward”.

edward-north-wales

Either these forces don’t understand the fact that the vast majority of days in their individual areas are fatality free, or they are determined to pretend they don’t in order to promote inherently hopeless schemes such as Project Edward. North Wales seemed to show their hand somewhat: minutes after I pointed out that their claim of success was bogus, they blocked me. Is that a sign of wanting to discuss randomness, or… not?

All of this celebration is, whilst mathematically illiterate, politically astute. Lots of people see regional police forces’ messages, and it was regional forces who were able to declare the day a success with a veneer of legitimacy: No fatalities here—it worked!

And all this hot air was wind in Tispol’s sails, who abandoned all pretence of their stated aim and proudly declared that they were “overwhelmed by the success” of Edward.

The politics of randomness

The rounds of back-slapping as Edward sailed off into the distance were, however, not a new thing. It’s far from uncommon for authorities and politicians to use the distribution characteristics of broadly random events as a mark of success for some intervention, when in fact they’re almost certainly just statistical noise.

When Queensland introduced its still-contentious metre passing law in 2014, it did so on the back of gradually increasing cycling fatalities: from 2008 to 2013 there had been respectively 7, 8, 7, 9, 10 and 13 fatalities. To most people, that looks like a rising trend, but with such low numbers it’s perfectly possible that it’s just random distribution.

However, 2014 saw 9 fatalities and 2015 just 4, so in early 2016 there was much crowing about the effectiveness of the legislation, because that looks like a downward trend. The timing of that crowing was prudent: in early 2016, when those press releases came out, statistics was already giving a harsh lesson in random distribution. In just the first four months of the year, at least 7 cyclists died on Queensland’s roads. All you have to do is shake a couple of 2013 fatalities into 2014 and a couple of 2016 fatalities into 2015 and most people wouldn’t really perceive a trend at all, but this is all a perfectly plausible result of randomness in distribution.

Of course, the official fatality figures won’t be released until 2017: only people actively looking at the reports from the front line would have known of the swing of the pendulum of randomness. By the time the public at large are made aware of the rise in fatalities, the passing law will be old news: the rise will be attributed to something other than natural randomness, something which will fit whatever action the relevant authorities feel politically empowered to take. Personally I’ll wager it will be attributed to allegedly deteriorating cyclist behaviour—watch this space.

We must also remember, of course, that correlation is not causation: I have yet to see a single discussion of the law which actually looked at the nature of fatal collisions before and after the legislation to assess whether any may have been related to passing distance rather than just statistical noise (though I at least started). And nor will I see one, because it is in the interest of no organisation or authority to undermine public belief in the supposed efficacy of this law. Not even the academic assessment of the legislation looked into this. However, it did at least show an understanding of statistics: whilst being cautiously optimistic about an apparent improvement in the average distance left by drivers in non-collision cases (which is largely irrelevant to casualty figures) it remarked that “While there was a 35% reduction in the rate of fatalities for cyclists, this reduction was not statistically significant due to the small numbers involved…It is premature to draw conclusions regarding the road safety benefits of the road rule at this stage.”

Most people would think a 35% reduction is a big thing. But with these data it’s not, because the numbers are small and the randomness of distribution is strong. Calendar years are just buckets that we use to quantify things: the first day of January is not special when it comes to changes in road danger or vehicle use, it is just a completely arbitrary line that we draw, and shaking just a couple of events either side of some of those lines gives people an unrealistic impression of reality.

When people see the other manifestation of the same phenomenon—a clustering of events rather than a sparseness of them—they also react strongly. When London saw six cycling fatalities in a fortnight in November 2015, the Metropolitan Police deployed officers in a reprise of Operation Safeway. Widely criticised by cycling campaigners, Safeway was a highly visible roadside operation that focused on some easily-eyeballed things rather than actually analysing whether there was any reason—any at all, never mind the things it actually tackled—other than simple randomness for the closeness of the events. It was merely about being seen to respond to something that the public and the media had started to discuss.

In fact, London saw fewer cycling fatalities in 2015 than in any of the ten years that preceded it. It just happened to be the case that they were randomly distributed in a way that most people wouldn’t perceive as random. Yet this clustering of distribution isn’t even a statistical anomaly: it’s statistical normality.

Of course, such an intervention—timely but facile—is politically astute, since it is inevitable that before long the random clustering of events ceases. When it does, the rate of fatalities will return to—or, more likely, fall below—whatever passes for normal and, since most people confuse correlation and causation, that politically uncontroversial intervention will appear to have been a success. Jump on that wave of randomness at the right point and you can surf the success all the way to the beach.

All of this arises because we’re dealing with a largely randomised distribution of infrequent events, and the public at large tend not to fully comprehend that. Waiting for a cluster and then launching an intervention is a virtually guaranteed winning technique: if it correlates and it’s at least superficially plausible, people will put faith in it. It’s homeopathy as a political device.

Ignorance or insidiousness?

In truth, there were at least five fatalities in the UK on the day of Project Edward. The daily average in 2015 was 4.7. So by the “logic” of using a single day’s fatalities in an arbitrary geographical area as an indicator of performance, we could say that Project Edward made things a little worse. Of course, it didn’t: anyone genuinely believing it had any effect at all on the day’s fatality figures for any area is hopelessly mistaken.

And while several forces are hanging out the bunting simply because the laws of probability behaved entirely normally, you won’t find any self-congratulatory tweets from Essex, Lancashire, Thames Valley, Norfolk or Cumbria. They’re keeping quiet, because they were the randomly chosen recipients of the entirely normal number of fatal incidents on that day. Of course, Tispol made no reference to them either.

When Project Edward was announced, it was obvious that its stated goal was delusional: the idea of a 70-a-day death rate simply stopping overnight is absurd. It would be absurd even with a plan of serious action, but when the interventions over and above normal police activities essentially appear to be an online form and a hashtag it’s hard to have faith in any organisation that does anything other than ridicule it. Those that vocally support it and claim some sort of success should be viewed with a critical eye: anyone claiming anything on the basis of the occurrence or non-occurrence of a sparsely and randomly distributed event should be challenged. When the police pull up the shutters and block public engagement on this issue, we should be especially concerned. (Point of note: an interesting thing to have in mind is Nolnah’s Razor, a reversal of Hanlon’s Razor.)

Ignorance can be relatively easily fixed. Insidiousness less so. Any organisation that doesn’t want to be tarnished with either of these labels shouldn’t be supporting initiatives that can only reasonably be classified as one or the other. So let’s see who continues to support Project Edward next year, because it simply encourages statistical randomness to be misused to defend seemingly minimal and/or questionable changes in action.

To be honest, I’m not even sure “Edward” even transposes well across European languages. Maybe an Iberian audience, for instance, would prefer “Project Eduardo”. And why not? “European Day of Unimaginably Anomalous Results? Dream On!” just about covers it, I think.

Epilogue

A little under a month after Project Edward, Tispol published the “results”. Unsurprisingly it was billed as a great success:

On Wednesday 21 September 2016 (Project EDWARD day) there were 43 road deaths across 31 countries. This compares with 70 fatalities on the same day in 2015 and represents a like-for-like reduction of almost 39 per cent.

Sounds good, doesn’t it? Well, obviously the first caveat has been covered above: fatality figures on any given day vary wildly, and the fact that two days a year apart saw a different number of fatalities means absolutely nothing.

But it actually gets worse. Let’s keep that “reduction of almost 39%” in mind as we look at the data sheet that accompanies the press release.

Take a look at the row at the bottom, which lists the fatalities in the UK. How many were recorded? Three.

But, as noted above, there were at least five. Which means Tispol’s UK figure is under-reporting by at least 40%. Now, I have neither the time nor the linguistic skills to go trawling the web for similar information for each of the countries in question, so I won’t speculate as to the level of under-reporting elsewhere. But when at least some of the under-reporting exceeds the cited fall in fatalities, it makes the self-congratulatory claims look fairly ridiculous.

Violeta Bulc, whilst inevitably giving a glowing review of the whole affair, was curiously on the money with this neatly-worded statement (the emphasis is mine):

This year, we recorded a significant improvement compared to 2015.

Comments

  1. dan 5 October 2016 11:43am #

    The Poisson distribution, if I remember my statistics course of many years ago.

    Something that is statistically extremely unlikely at a single point of time in a single location is statistically likely over a wide area and long time. There will be no fatal accidents outside your front door this afternoon; there will be a fatal accident in the UK this week

    It is ludicrous that our police forces should be spending time doing something which won’t change behaviours or outcomes.

    • Meltdblog 6 October 2016 8:38am #

      Thinking carefully, the Poisson distribution only holds for independent events. We could improve the model by taking distributions for each integer of correlated deaths in multiple fatality accidents.

  2. Notak 5 October 2016 2:44pm #

    My main conclusion from this would be that every police force needs a statistical service to help it decide what’s worth doing, and a media interpreter of statistics to explain to the public (and to governments, action groups, etc) why they won’t be supporting certain actions. The latter probably harder to do than the former.

  3. Dr Robert Davis 5 October 2016 6:32pm #

    Excellent post. John Adams has been saying this sort of thing for years in “Risk”, “Risk and Freedom” etc. See http://www.john-adams.co.uk/

  4. Dr Robert Davis 5 October 2016 6:35pm #

    And of course you can keep fatalities down by having fewer “vulnerable road users” outside cars out there in the first place….

    • Meltdblog 5 October 2016 10:00pm #

      You certainly can, moving all vulnerable transport users (pedestrians and cyclists) into cars would reduce the road toll, but it would be even lower if we instead moved all the people from cars onto public transport and non-motorised transport. Both extreme situations have lower road tolls that the current mix, but neither are achievable.

      Requiring people to use cars for their safety is good news for the automotive industry and bad news for the environment/energy supply, only one of these has strong advertising and a lobby group. We need to be moving to a situation where the car is less desirable than the alternatives, not superior.

  5. paulc 6 October 2016 4:22am #

    “from 2008 to 2013 there had been respectively 7, 8, 7, 9, 10 and 13 fatalities. To most people, that looks like a rising trend, but with such low numbers it’s perfectly possible that it’s just random distribution.”

    a better guide to a real trend would be the KSI figures… not the fatalities…

  6. jennifer mooney 16 August 2017 9:59pm #

    My husband died when he crashed his bike into a tree last year.
    How does anyone expect to stop things like that happening.
    #cyclingsucks

Leave a Reply to Meltdblog Cancel reply

Your email address will not be published. Required fields are marked *