31-05-2013, 10:50 PM
It seems the comparison with DNA use in court throws up valuable lessons. It’s very easy to blur the distinction between “the odds” and the claims made for those odds - as they are two different entities.
Anyone speaking of odds of billions to one is very likely to be overlooking one or two practical realities along the way. Contrary to those claims that the number is too big to doubt, it seems DNA matching is quite a complex, subjective process with lots of choices made before that impressive, ‘unchallengeable’ statistic is unveiled by the guy in the white gloves to wow the court.
First let’s examine the goods from a commonsense viewpoint. If the odds against a DNA match being wrong are, as claimed, billions to one, then in a population of 7 billion - the vast majority of whom have never been convicted on DNA evidence - you would be unlucky if there were even ONE conviction based on DNA evidence where the DNA evidence was later found to be wrong. However a quick search turned up half a dozen (I won’t post links as they are to crimes). There seem to be plenty more. The main problem being contamination en route to the lab or before. In others it was (to be charitable) over zealousness on the part of officials talking up weak evidence.
This study puts the theoretical claims into perspective:
False result fear over DNA tests
Nick Paton Walsh
The Observer, Sunday 27 January 2002
“One in every hundred forensic tests performed on the DNA of suspected criminals may give a false result, according to the first research of its kind into laboratory error rates.”
…The findings will shock British DNA laboratories, which deny that errors exist.
…British forensic experts expressed alarm at the errors and stressed they were more numerous than expected.”
<!-- m --><a class="postlink" href="http://www.guardian.co.uk/uk/2002/jan/27/ukcrime.research">http://www.guardian.co.uk/uk/2002/jan/2 ... e.research</a><!-- m -->
DNA can be planted; although perhaps not very likely, the possibility means those billion to one odds go out the window. It also transpires that DNA can be easily fabricated in the lab.
Useful lawyery essay here featuring the issues <!-- m --><a class="postlink" href="http://www.lawlink.nsw.gov.au/lawlink/pdo/ll_pdo.nsf/vwPrint1/PDO_dnadealingwithincourt">http://www.lawlink.nsw.gov.au/lawlink/p ... ithincourt</a><!-- m -->
As Phil notes, the assumption of statistical independence (needed because they multiply the probabilities for each DNA fragment together, which leads to the quoted small probability of a mismatch) is not guaranteed, and has meant that for example in some populations a much higher rate of pairs of people matching than would be predicted by the ‘statistics’ in use. See para 28 here, where 122 pairs of profiles matched in the Arizona offender database of 65,493 profiles. <!-- m --><a class="postlink" href="http://www.state.il.us/court/opinions/appellatecourt/2012/2nddistrict/2091328.pdf">http://www.state.il.us/court/opinions/a ... 091328.pdf</a><!-- m -->. Also 206 matching pairs were found from 33,858 profiles in a survey of DNA databases by the National Institute of Forensic Science, <!-- m --><a class="postlink" href="http://www.lawlink.nsw.gov.au/lawlink/pdo/ll_pdo.nsf/vwPrint1/PDO_dnadealingwithincourt">http://www.lawlink.nsw.gov.au/lawlink/p ... ithincourt</a><!-- m -->
So, no billion to one chance there. These are partly due to using partial DNA matches rather than full ones – that’s what you get when you widen the goalposts, as people do.
DNA testing IS useful, just as probability calculations for chess engine matches IS useful. It’s not the science that’s at fault, but the human (over) interpretation. Blind faith in numbers isn't standing up.
Anyone speaking of odds of billions to one is very likely to be overlooking one or two practical realities along the way. Contrary to those claims that the number is too big to doubt, it seems DNA matching is quite a complex, subjective process with lots of choices made before that impressive, ‘unchallengeable’ statistic is unveiled by the guy in the white gloves to wow the court.
First let’s examine the goods from a commonsense viewpoint. If the odds against a DNA match being wrong are, as claimed, billions to one, then in a population of 7 billion - the vast majority of whom have never been convicted on DNA evidence - you would be unlucky if there were even ONE conviction based on DNA evidence where the DNA evidence was later found to be wrong. However a quick search turned up half a dozen (I won’t post links as they are to crimes). There seem to be plenty more. The main problem being contamination en route to the lab or before. In others it was (to be charitable) over zealousness on the part of officials talking up weak evidence.
This study puts the theoretical claims into perspective:
False result fear over DNA tests
Nick Paton Walsh
The Observer, Sunday 27 January 2002
“One in every hundred forensic tests performed on the DNA of suspected criminals may give a false result, according to the first research of its kind into laboratory error rates.”
…The findings will shock British DNA laboratories, which deny that errors exist.
…British forensic experts expressed alarm at the errors and stressed they were more numerous than expected.”
<!-- m --><a class="postlink" href="http://www.guardian.co.uk/uk/2002/jan/27/ukcrime.research">http://www.guardian.co.uk/uk/2002/jan/2 ... e.research</a><!-- m -->
DNA can be planted; although perhaps not very likely, the possibility means those billion to one odds go out the window. It also transpires that DNA can be easily fabricated in the lab.
Useful lawyery essay here featuring the issues <!-- m --><a class="postlink" href="http://www.lawlink.nsw.gov.au/lawlink/pdo/ll_pdo.nsf/vwPrint1/PDO_dnadealingwithincourt">http://www.lawlink.nsw.gov.au/lawlink/p ... ithincourt</a><!-- m -->
As Phil notes, the assumption of statistical independence (needed because they multiply the probabilities for each DNA fragment together, which leads to the quoted small probability of a mismatch) is not guaranteed, and has meant that for example in some populations a much higher rate of pairs of people matching than would be predicted by the ‘statistics’ in use. See para 28 here, where 122 pairs of profiles matched in the Arizona offender database of 65,493 profiles. <!-- m --><a class="postlink" href="http://www.state.il.us/court/opinions/appellatecourt/2012/2nddistrict/2091328.pdf">http://www.state.il.us/court/opinions/a ... 091328.pdf</a><!-- m -->. Also 206 matching pairs were found from 33,858 profiles in a survey of DNA databases by the National Institute of Forensic Science, <!-- m --><a class="postlink" href="http://www.lawlink.nsw.gov.au/lawlink/pdo/ll_pdo.nsf/vwPrint1/PDO_dnadealingwithincourt">http://www.lawlink.nsw.gov.au/lawlink/p ... ithincourt</a><!-- m -->
So, no billion to one chance there. These are partly due to using partial DNA matches rather than full ones – that’s what you get when you widen the goalposts, as people do.
DNA testing IS useful, just as probability calculations for chess engine matches IS useful. It’s not the science that’s at fault, but the human (over) interpretation. Blind faith in numbers isn't standing up.