31-05-2013, 01:04 AM
Thanks Graham, It wasn’t the choice of 25 ply I was questioning, it was afterwards varying the ply level to include some fourth or fifth choices that I thought was questionable. Sorry, Sally Clark was brought up on the other thread on this topic.
Hi Andrew. My point was, that as it stands it wouldn’t get to court, because a court would have certain requirements of the accusations, requirements that haven’t been met. I’ve already mentioned some of them. Anyway, before you can talk about ‘incredibly small’ odds you have to make sure the calculation that produces those odds is valid. At the moment the calculation is not even on view for Ivanov (or a lawyer or statisticians perhaps) to question.
I think I know what you mean though – the engine match ‘seems’ so strong that there ‘shouldn’t’ be any problem putting together a valid case, and even if the case has to be weakened slightly to remain valid, it shouldn’t matter? That would be putting the cart before the horse though. The case (and any calculation) has to be stated before any judgement could be made!
Let’s have a go anyway. Going by what has been said, it seems to me that the accusation against Ivanov, if formalized, might look something like this:
“The probability of a player rated X, playing normally, attaining a match of Y% against the top move of Houdini 3 engines in Zadar games 1-9, is Z%. Z is very small. Therefore he was not playing normally.”
It’s actually a bit of a step from the last statement to “therefore he was cheating”, but let’s ignore that for now.
To complete the accusation, several choices have to be made:
Is it really the top move, or the top 3?
Which rating to count, the pre-tournament or post tournament one, and should his improved performance not be allowed for?
Is it Houdini 3 or Rybka? Or perhaps Houdini 2, as claimed by the protesting players at Zadar?
Do you need to exclude certain games - which ones, and why? This also brings in questions relating to the method of cheating.
You might think well, you just look at the data, and pick the best choices for the accusation - but there is a statistical snag with this; every choice of parameter that you estimate from the data from which the result is to be obtained weakens the statistical strength of any positive result. I don’t know enough to quantify this, but if there are several choices made, I think it could mean dividing the final odds by several numbers relating to those choices.
There is another factor that reduces the statistical strength of the conclusion; the fact that the suspicion arose from the same data as is being used to prove the suspicion increases the chance of a false result. I’m not saying it’s a complete no-no, but I’m not sure how you account for it. An expert like Andy Muir might be able to advise on these questions. Had there been a stated prior suspicion, or some other objective reason to examine the data, it would be stronger.
I’m not just raising procedural objections in an open and shut case. Here is an extreme example to show you how these big astronomical odds can fall to earth if there is a problem with the data.
People eg Lilov are bit blasé talking about top matches or top 3 matches interchangeably, even allowing fourth or fifth match to be counted to make a point. So - let’s say someone allowed the first four choices over Ivanov’s 200 (say) Zadar games, and then someone else did a calculation based on them, but (wrongly) counting as if it were only the top three. What difference would that make to the odds? It might surprise you that in the probability calculation this could cause an error factor of (almost) a million million million million. This number comes from 3/4 to the power 200. As I say it’s an extreme example and I’m not saying anyone has done a calculation in this way, but it’s an example of what can happen if you hand-wave the numbers too much.
So while I 'think' he did cheat, I won’t be convinced the case is in any way ‘proven’ until I see a definite, precise accusation made by a reasonably competent body (who takes responsibility for making it) and it’s scrutinized by the wider community. I don’t mind if the competent body are a little bit biased (difficult to avoid), as long as any judgements made by them within the data are visible for all to see.
As I said before, a comparison with other players would help. In your opening gambit you seem to suggest these are around – if you could post any links I’d be interested, thanks.
Cheers
Hi Andrew. My point was, that as it stands it wouldn’t get to court, because a court would have certain requirements of the accusations, requirements that haven’t been met. I’ve already mentioned some of them. Anyway, before you can talk about ‘incredibly small’ odds you have to make sure the calculation that produces those odds is valid. At the moment the calculation is not even on view for Ivanov (or a lawyer or statisticians perhaps) to question.
I think I know what you mean though – the engine match ‘seems’ so strong that there ‘shouldn’t’ be any problem putting together a valid case, and even if the case has to be weakened slightly to remain valid, it shouldn’t matter? That would be putting the cart before the horse though. The case (and any calculation) has to be stated before any judgement could be made!
Let’s have a go anyway. Going by what has been said, it seems to me that the accusation against Ivanov, if formalized, might look something like this:
“The probability of a player rated X, playing normally, attaining a match of Y% against the top move of Houdini 3 engines in Zadar games 1-9, is Z%. Z is very small. Therefore he was not playing normally.”
It’s actually a bit of a step from the last statement to “therefore he was cheating”, but let’s ignore that for now.
To complete the accusation, several choices have to be made:
Is it really the top move, or the top 3?
Which rating to count, the pre-tournament or post tournament one, and should his improved performance not be allowed for?
Is it Houdini 3 or Rybka? Or perhaps Houdini 2, as claimed by the protesting players at Zadar?
Do you need to exclude certain games - which ones, and why? This also brings in questions relating to the method of cheating.
You might think well, you just look at the data, and pick the best choices for the accusation - but there is a statistical snag with this; every choice of parameter that you estimate from the data from which the result is to be obtained weakens the statistical strength of any positive result. I don’t know enough to quantify this, but if there are several choices made, I think it could mean dividing the final odds by several numbers relating to those choices.
There is another factor that reduces the statistical strength of the conclusion; the fact that the suspicion arose from the same data as is being used to prove the suspicion increases the chance of a false result. I’m not saying it’s a complete no-no, but I’m not sure how you account for it. An expert like Andy Muir might be able to advise on these questions. Had there been a stated prior suspicion, or some other objective reason to examine the data, it would be stronger.
I’m not just raising procedural objections in an open and shut case. Here is an extreme example to show you how these big astronomical odds can fall to earth if there is a problem with the data.
People eg Lilov are bit blasé talking about top matches or top 3 matches interchangeably, even allowing fourth or fifth match to be counted to make a point. So - let’s say someone allowed the first four choices over Ivanov’s 200 (say) Zadar games, and then someone else did a calculation based on them, but (wrongly) counting as if it were only the top three. What difference would that make to the odds? It might surprise you that in the probability calculation this could cause an error factor of (almost) a million million million million. This number comes from 3/4 to the power 200. As I say it’s an extreme example and I’m not saying anyone has done a calculation in this way, but it’s an example of what can happen if you hand-wave the numbers too much.
So while I 'think' he did cheat, I won’t be convinced the case is in any way ‘proven’ until I see a definite, precise accusation made by a reasonably competent body (who takes responsibility for making it) and it’s scrutinized by the wider community. I don’t mind if the competent body are a little bit biased (difficult to avoid), as long as any judgements made by them within the data are visible for all to see.
As I said before, a comparison with other players would help. In your opening gambit you seem to suggest these are around – if you could post any links I’d be interested, thanks.
Cheers