Thursday, October 12, 2006

A Brief Note on Statistics

Yesterday, regarding the Lancet survey of casualties in Iraq, I noted that rejecting this survey out of hand - a properly randomized survey with a large sample size - effectively requires you to reject every single public opinion poll ever done. Nevertheless, the talking point among all innumerate rightists seems to be that the methodology is "flawed" somehow. Nevermind that no specific methodological criticisms have been levelled that pass the laugh test.

Broadly speaking, the right is rejecting this survey based on the fact that it is a survey, not a body-by-body counting. But nobody with any expertise - anywhere - believes that a body-count would seriously produce a more accurate number. Let's illustrate with a historical example: The United States Census Bureau has been counting humans - free and otherwise - longer than there's been an Iraq, or a Canada for that matter. The US Constitution requires that the Census be taken every 10 years and that Americans be counted individually. However, leading up to the 2000 census, there was a minor controversy over whether the Census bureau would be allowed to use statistical sampling to estimate where the discrepancies lay between the official tally (required by the Constitution to apportion seats in Congress) and reality.

Nobody in either party denied that statistical methods would provide a better, more accurate picture of where Americans lived and what kind of people they are. But head counts systematically undercount minorities and the poor, while overcounting the wealthy whites. Guess which party wins and loses in this state of affairs? Guess which party controls Congress? Guess which party claims the majority of the Supreme Court vote? The Supreme Court ruled - correctly, I hasten to say - that the letter of the Constitution is very clear, and statistical sampling could not be used to apportion seats, but could be used for other purposes.

The point is this: The Census Bureau - with more continuous experience doing head counts than any other institution on Earth - believes that statistical sampling is a more accurate way to collect data about a large population than a simple head count.

Statistical surveys are, in a very real way, the foundation of the modern technocratic state. GDP, unemployment, inflation, wages, all the numbers that make up the headlines to our lives are based on statistics - collected in much the same way as this Iraqi casualty survey was taken. The authors of the Lancet study have been open and honest in all of their methods, their data, their assumptions, and the potential faults to their conclusions. Methodologically, this survey is as sound as any other - or else the Lancet wouldn't have risked it's considerable reputation to publish it, especially after the 2004 survey.

If the pro-war crowd can't stomach the results of their splendid little war, maybe - just maybe - they should have thought a bit more about it beforehand.

Just remember - 92% of the Iraqis were able, when asked, to provide valid physical documentation of the deaths they claimed. Every single objection to this survey is resting its hopes on the remaining 8%.


Olaf said...


The Supreme Court ruled - correctly, I hasten to say - that the letter of the Constitution is very clear, and statistical sampling could not be used to apportion seats, but could be used for other purposes.

Who the hell knows stuff like this???

Secondly, great post.

I think that the stat thing about it that bothers me, is that it implies that every death was the fault of Americans (or at least that's what I take from it, and the way many are framing it)... and while I suppose you could call the US the initial source, I'd be interested in knowing how many Iraqi's have been killed by the quasi-civil war going on, or as casualties in Jihadist attacks on US troops.

I don't want to look at any document, and was hoping that you could tell me whether the two sources of direct death were separately accounted for. I mean if you know the Supreme Courts decision on statistical sampling, you must know this.

Also, for your interest, if you didn't know, apparently a high ranking British general is saying he wants British troops out of Iraq immediately, or soonish. Apparently, I saw it on the news as I typed.

And finally, where have you been on my MAD to the Max series... I'm counting on the reasonable people out there to suggest why I'm wrong, because I'm starting to believe it myself.

john said...

"Who the hell knows stuff like this???"

An unemployed person with access to Wikipedia is a dangerous thing.

The Lancet survey broke down the violent deaths between "caused by the coalition", "other", and "unknown". The US/UK were responsible for 31% of the 600K violent deaths (of 650K total.)