How bad science and its misuse is damaging society and you

In 1999 a solicitor Sally Clark was put on trial for murdering her two babies and she was sentenced to life imprisonment. In her trial, Professor Sir Roy Meadow, an expert in ‘parents who harm their children’, gave expert evidence. He said that the chance of two children in the same family from dying of Sudden Death Syndrome (SIDS) was ‘one in 73 million’ implying she must have murdered them as it was so unlikely that they had died from SIDS.

The judge of the case, Sally’s defence team, the appeal court judges, and almost all the journalists and legal commentators reporting in the case seemed to accept this (as none of them challenged it). Yet a relatively simple understanding of statistics shows he was wrong for two reasons…

Firstly, Professor Meadow used a simple statistical method as the chance of a child dying from SIDs in UK was about one in 8544, so Professor Meadow multiplied it again by 8544 to get the chance of two children dying from SIDs in the same family, which assumes that the deaths are independent. Just as if the chance of tossing a coin and getting a heads is ‘one in two’, then the chance of getting two heads in a row is in ‘two times two’ or one in four. But although the cause of SIDs is not known, it is quite likely that there is some cause and that this underpinned the two deaths. In that case multiplying the numbers leads to the wrong probability.

Not letting ‘probability’ overawe you

But even if the first concern does not apply there is a second more serious error. Let’s assume there is a ‘one in 73 million’ chance. So it is highly unlikely that Sally’s two children died from SIDs taken on their own. But there were about 20 million families in the UK and a significant number will have two or more children. The odds that one of these families will suffer the loss of two children to SIDs is quite high. In the same way that it is highly unlikely that you can pick the next lottery winner but the chances are pretty high that someone will win the lottery. So Sally won a terrible ‘lottery’ out of these  families.

Looked at another way - there were only two possibilities put forward by the prosecution: either both children died from SIDs or both were murdered by their mother, Sally. Both possibilities are highly unlikely but comparatively they are likely to be the same order of magnitude (double homicides by mothers in the UK is very rare too). The odds are quite possibly similar which means rather than there being a ‘one in 73 million’ chance of the two children dying from SIDs – it may be about 50/50 that they either died from SIDs or were murdered. This would not add to the evidence to take it 'beyond reasonable doubt'.

Asking the right question first

It was a tragic case and Sally was released after serving 4 years in Jail only to die of alcohol poisoning 4 years later in 2007. This type of error is called the prosecutor’s fallacy and is behind the misuse of DNA testing when convicting defendants.

Let’s take the case of a defendant whose DNA matches that found on the murder weapon and let’s assume only 1 in 5 million people in the United Kingdom have that DNA. This seems like good evidence for convicting a criminal? But it is the wrong answer because we are asking the wrong question. The table below shows why:

The question commonly asked to support the prosecution’s case is: ‘what is the probability that someone’s fingerprints will be there if he or she is innocent’? The answer is 11/60 million – in other words it is close to zero that an innocent person’s DNA will match.

But the situation was being assessed after the fact that this DNA was there, it is not about the probability of it being there. It is a bit like working off the chance that a roulette wheel will get 10 reds in a row. It is very small and when I was a student I went to a casino and waited until there had been 9 reds in a row then bet black. I lost all the money on me because once you have 9 reds in a row the chance of getting a 10th red is approximately 50/50 once again (on a technical note; they are in fact 18 in 37, or 48.6/51.4, on a single zero roulette wheel!).

The key point is that it is irrelevant what the chances of getting into that situation were. The question is, ‘what is the probability once you are in it’. So there is a person whose matching DNA has been found at the scene of the crime. The correct question is, ‘what is the chance that he or she is innocent?’ Now you can see the answer is 11 out of 12 or a 92% chance of being innocent! In the vast majority of cases, DNA matching only helps prove innocence not guilt.

Seeing through the ‘it’s scientific fact’ argument

But bad science can be used by some organisations across different industries in a way that directly affects you. For instance a financial organisation may proudly declare its system correctly classifies 99% of legitimate transactions as legitimate while correctly classifying 99% of fraudulent transactions as fraudulent.

This seems pretty accurate but that assessment does not account for the base rate fallacy. Let’s assume 1 in 1000 credit card transactions are fraudulent. So of 1000 transactions – nine will be incorrectly flagged as fraudulent and one correctly classified. So despite the system processing 99% accurately, it will correctly class a transaction as fraudulent only 10% of the time because of the high ‘base rate’.

Keeping to drugs that improve your health

The effectiveness of drugs is also prone to many statistical errors and can mean you are harming yourself by taking ineffective ones. A drug trial is generally considered to indicate if a drug is effective if there is no more than a 1 in 20 chance of that drug getting the positive results it did randomly. So it would be expected that for every 1000 trials there will be 50 false positives (as 1000/20 is the number of random times this will happen).

It is a requirement of medicinal regulatory authorities that multiple trials must be conducted to weed out the false positives, but this is where the publication bias takes effect. Papers describing the results of trials have to be written up and submitted to journals for publication. The papers which appear to show an effect are more likely to be written up, submitted and selected. A selection process is happening whereby the papers that showed an effect purely by chance of a particular drug are disproportionately submitted and accepted for publication.

The other papers showing no effect are disproportionately omitted. So the drug looks good and you then get the placebo affect kicking in to help make it look like it is effective (the placebo affect is something working because we think it will work and the magnitude of this affect can be surprisingly large). So you may end up using a ‘scientifically proven’ drug which provides illusory benefits but real side effects!

Another classic error is to decide what the hypothesis is for what the drug does after carrying out the experiment and collecting the data. If you look closely enough you can usually find a correlation in data to something but ‘correlation is not causation’.

So what has all this got to do with leadership?

This is a 'heavy' lead article and if you have reached this point then our congratulations on the self-discipline to follow the arguments through. We believe at MarkTwo that you can only lead others to the extent that you lead yourself. An important part of leading yourself includes making good decisions. We each make up to ten thousand decisions a day and the quality of our life is a direct reflection of the quality of our decisions. Using science properly, and avoiding bad science, is one way to help us make better decisions in many areas of our life. While it takes self-leadership to make the effort to do this, the benefits in our life, and those around us, can be very significant. The more significant the decision the more important it is that we look at the methodology behind the ‘scientific facts’. Also the healthier you keep yourself then the more energy you will have to lead others.

For more insights here I recommend watching the TED video called Battling Bad Science (see below) as well as reading HOW NOT TO BE WRONG THE HIDDEN MATHS OF EVERYDAY LIFE by Jordan Ellenberg. 

Battling with your teenager? Problems with your partner? In the wrong job? Can't find your purpose? 

MarkTwo Personality Training can help you deal with these issues and more! Book now to save $660!

This 2-day worskhop will provide you with a practical understanding of why people behave the way they do, whether it is your partner, child, colleague or boss. This could potentially help you improve your relationships.

You will also gain a deeper understanding of your own personality style, key strengths and roadblocks preventing you to reach your potential. Seats are limited.  

Register Now to get the early bird price and save $660!

This course trains people in the same Jungian model as the Myers-Briggs Type Indicator® (MBTI) but adds a third dimension which takes it much further.

When: Fri 20 - Sat 21 March, 2015
Where: Tudor Box Hill Hotel, Box Hill, Melbourne, Victoria, Australia
Price: $990 (Earlybird extended to 27th February, 2015)
Note: This extraordinary course is only run once per year for 25 participants and sells out.

Get more details or book the earlybird and save $660 inc GST

Click here to see full list of events.

In one word I would call the workshop: powerful...also say enlightening, energising, thought provoking, fun and challenging. Really enjoyed it - Chris Balazs, Senior Manager Quality Control, CSL Behring

TED Talk

Every day there are news reports of new health advice, but how can you know if they're right? Doctor and epidemiologist Ben Goldacre shows us, at high speed, the ways evidence can be distorted, from the blindingly obvious nutrition claims to the very subtle tricks of the pharmaceutical industry.

Landing in a sea of colour

I am heading to India in early March for some meetings with key business leaders in Hyderabad, Mumbai, Pune, Thiruvanthapuram and Bengaluru.

Looking forward to experience my first taste of "holi" - the Indian festival of colours! The festival of Holi is a celebration of the Colors of Unity & Brotherhood - an opportunity to forget all differences and indulge in unadulterated fun.

Holi has traditionally been celebrated in high spirit without any distinction of cast, creed, color, race, status or gender. It is one occasion when sprinkling colored powder ('gulal') or colored water on each other breaks all barriers of discrimination so that everyone looks the same and universal brotherhood is reaffirmed.

Upcoming Events

We are running a range of public programs due to the intense demand arising from the impact of our corporate programs.

Our 3-dimensional model on human nature and motivation called the Universal Hierarchy of Motivation is such a model and it is both holistic and holographic. To find out more about the benefits of doing our courses and to save hundreds $$$$ by getting the earlybird prices, click here

Quotes of the Month

These days people seek knowledge, not wisdom. Knowledge is of the past, wisdom is of the future - Vernon Cooper

Knowledge comes by taking things apart: analysis. But wisdom comes by putting things together - John A. Morrison