Polling for dummies

McCain supporters are probably jumping up and down on word that the Investor's Business Daily tracking poll has the Republican nominee within one point of Obama. But look closely at one of the breakouts:

AGE Obama McCain Not Sure
18-24* 22% 74% 4%
25-44 47% 45% 8%
45-64 49% 36% 15%
65+ 40% 46% 14%

See something strange? As in John McCain getting 74 percent support among 18-24 year olds? The asterisk indicates this particular age group has a small sample size that’s known to fluctuate. No kidding. Sometimes in life a dose of common sense can come in handy, even among pollsters. Nate Silver at FiveThirtyEight dissects the numbers:

Suppose that the true distribution of the 18-24 year old vote is a 15-point edge for Obama. This is a very conservative estimate; most pollsters show a gap of anywhere from 20-35 points among this age range. About 9.3 percent of the electorate was between age 18-24 in 2004. Let's assume that the percentage is also 9.3 percent this year. Again, this is a highly conservative estimate. The IBD/TIPP poll has a sample size of 1,060 likely voters, which would imply that about 98 of those voters are in the 18-24 age range. What are the odds, given the parameters above, that a random sampling of 98 voters aged 18-24would distribute themselves 74% to McCain and 22% to Obama?

Using a binomial distribution, the odds are 54,604,929,633-to-1 against. That is, about 55 billion to one. So, there is an 0.000000002% chance that IBD/TIPP just got really unlucky. Conversely, there is a 99.999999998% chance that one of the following things is true: (i) They're massively undersampling the youth vote. If you only have, say, 30 young voters when you should have 100 or so in your sample, than the odds of a freak occurrence like this are significantly more likely; or- (ii) Something is dramatically wrong with their sampling or weighting procedures, or their likely voter model.

Before this monster miscue, Silver provided an interesting primer on all the tracking polls. Here's what he said about IBD:

Track Record: IBD/TIPP touts itself as the most accurate pollster based on its strong result in 2004, when they nailed the Bush-Kerry numbers within a couple of tenths of a point. One good result does not a pollster make, however, and in 2000, their performance was only average, missing the Bush-Gore margin by 2.5 points. Results were slightly erratic and counterintuitive earlier this year, when for instance they showed an 11-point Obama lead in mid-May when most other pollsters showed him struggling at that time.