Zogby = Garbage

As was said of this over at the Corner, Zogby jumps the shark from bad pollster to hack. The inside scoop from Mystery Pollster blogger, Mark Blumenthal:


The Post-Standard newspaper in Syracuse and WSYR-TV had asked Zogby to conduct a second poll of the race after the pollster acknowledged that his firm had improperly weighted the results of a survey last week. In that case, Zogby polled the 25th district but then weighted the data using voter registration information from the more-Republican 24th district.

Zogby promised the two media outlets that he would do a new poll from scratch, but when the results of that survey came in both declined to run them. Jim Tortora, the news director of WSYR-TV, wrote on the station's Web site that after consulting with outside polling experts, he was concerned that Zogby had conducted the second poll using the same larger sample of 5,000 likely voters as he had on the first survey.

"With respect to Mr. Zogby, we felt the questions raised ... left us with only one choice: We had to pull the poll," Tortora wrote.

Used the same sample? Here is the explanation from WSYR's Tortora about their analysis of the second poll:

This time, the Post Standard arranged an independent expert from the University of Connecticut's Department of Public Policy to review the findings of the second Zogby poll. Late Tuesday, we discovered that some of the same people who were called for the first poll, were called again. Zogby confirmed they did indeed use the same larger sample of 5000 likely voters, to come up with this "new" poll sample of 502 likely voters. Our independent expert felt this raised a red flag...an unknown variable. The concern? How would you react if you were called twice in about a week, to answer the same questions? Would you answer differently? The same? Would you even take the call?

And as the Syracuse Post-Standard reported, "27 people responded to both polls." If that were not enough, even after all of this came to light, Tortora repots:

Mr. Zogby firmly stands by his findings. He insists his methodology is sound, and was prepared to join us live at 5:30pm to explain his findings and back-up his results. He points out pollsters often disagree about each other's methods.

Red flag indeed. I've a modest background in research. By no means am I an expert and I, like anyone with a basic stats class under their belt, can understand exactly why you don't do this. Blumenthal finishes by highlighting research professional's on why this is a problem.

The best part? They called Zogby a rookie before even knowing who'd conducted the poll:


"I think it's sort of a rookie mistake if you're including people a second time from a database," said Cliff Zukin, past president of the American Association for Public Opinion Research, an industry group based in Lenexa, Kan.

A bad practice

Zukin, a professor of public policy and political science at Rutgers University, spoke before he was told who conducted the poll.

He said it's considered a bad practice to call the same people twice for "random" polls.
"The problem is the first interview activates them," Zukin said. "They follow the news differently. So the people become different from a random citizen.

"If you didn't purge those people from the database," he said, "then that is a significant methodological problem. It gives you a problem to make any inference from these data."

Yes, we often disagree, but there are limits to what can be waived off as a mere difference of opinion. If Mr. Zogby or any other pollsters want to explain and defend the practice of reusing sample, our Guest Pollster's Corner is wide open.

Sampling is all about bias. In fact, every possible methodology available to a research professional is about minimizing response bias.

For me, I've never been much of a Zogby fan. True, he came closest to all of calling 2000 correctly. However, he backed into it. For example, Scott Rasmussen had Bush winning in 2000 by a comfortable 3-5%, something that potentially equated to a 100-150+ electoral vote blowout.

It didn't materialize because of the DUI story which suppressed Republican turnout. The result synched nicely with Zogby's prediction, but only because of the well-timed October surprise.

As for 2002 and 2004, it's well documented just how off he was. This pretty much cements the reputation as hack far as I'm concerned.