A few days ago, kos
posted a thread on "Rating the polls", giving the data for many major polls for the recent primaries. I gave some extra
data to help measure the performance of the polls. Yesterday, I posted an
abortive diary with more data on Zogby but screwed up the html.
This entry give the Zogby data again as well as the data for ARG and SUSA. A short analysis is: all these pollsters made significant enough errors that their results should not be trusted. For the long analysis...
We look more closely at the data.
In the last line of each of the tables below, we have the difference between the errors made in the poll for that state and the maximum allowable error given by the MOE calculation. Three lines up from that you see counts of how many times the result was at or outside the MOE.
Analysis: Zogby
Zogby was very accurate for NH, OK and AZ although, in those states, Zogby still had 1 result at the MOE (Edwards in OK). In SC and MO, however, Zogby was over the maximum allowable error (although not by much), but had three results outside the MOE. This is unacceptable for any pollster and places Zogby's methodology in doubt. Moreover, as Zogby eliminated "undecideds" his numbers generally benefited from higher accuracy.
Analysis: ARG
ARG was only accuracte for NH. In every other state, ARG was significantly outside the maximum allowable error. Moreover, ARG had a whopping 15 instances where a candidate's percentage was outside or at MOE. This is ridiculous. Anyone out there who wants some polling sone should ject call me. I'll ask all my friends and report back - the result will be just as accurate. I hope someone compares these numbers to the numbers in the predictions threads. I suspect people's guesses will, by and large, be more accurate than ARG.
Analysis: SUSA
SUSA did very well in all states except SC and MO, where they were significantly off. Although some of this error is undoubtably from the remaining "undecideds" in their polls, the error is high enough to cast doubt on the methodology. Moreover, SUSA had 10 instances of a candidate's percentage was outside or at MOE. Again, this points to a failure of methodology.
General Conclusion: While Zogby is clearly the best of the three, he still leaves quite a bit to be desired. If I had polling to do, I wouldn't hire any of these firms. Caveat emptor.
DATA:
| | NH | SC | OK | MO | AZ | DE |
Zogby | | | | | | | |
Clark | | 9 | 8 | 31 | 6 | 28 | |
| Actual | 12 | 7 | 30 | 4 | 27 | |
| Difference | -3 | 1 | 1 | 2 | 1 | |
Dean | | 24 | 8 | 6 | 9 | 15 | |
| Actual | 26 | 5 | 4 | 9 | 14 | |
| Difference | -2 | 3 | 2 | 0 | 1 | |
Edwards | | 12 | 36 | 26 | 17 | 7 | |
| Actual | 12 | 45 | 30 | 25 | 7 | |
| Difference | 0 | -9 | -4 | -8 | 0 | |
Kerry | | 37 | 32 | 29 | 56 | 42 | |
| Actual | 38 | 30 | 27 | 51 | 43 | |
| Difference | -1 | 2 | 2 | 5 | -1 | |
Lieberman | | 9 | 5 | 6 | 3 | 7 | |
| Actual | 9 | 2 | 6 | 4 | 7 | |
| Difference | 0 | 3 | 0 | -1 | 0 | |
Sharpton | | | 8 | | | | |
| Actual | | 10 | | | | |
| Difference | 0 | -2 | 0 | 0 | 0 | |
| | | | | | | |
MOE | | 4.1 | 4.1 | 4.1 | 4.1 | 4.1 | |
# outside MOE | | 0 | 1 | 0 | 2 | 0 | |
# at MOE | | 0 | 0 | 1 | 0 | 0 | |
Error (RMS) | | 3.74 | 10.39 | 5.00 | 9.70 | 1.73 | |
Max error | | 9.17 | 10.04 | 9.17 | 9.17 | 9.17 | |
Difference | | -5.43 | 0.35 | -4.17 | 0.53 | -7.44 | |
| | | | | | | |
| | NH | SC | OK | MO | AZ | DE |
ARG | | | | | | | |
Clark | | 13 | 11 | 28 | 6 | 21 | 8 |
| Actual | 12 | 7 | 30 | 4 | 27 | 10 |
| Difference | 1 | 4 | -2 | 2 | -6 | -2 |
Dean | | 25 | 9 | 8 | 7 | 10 | 14 |
| Actual | 26 | 5 | 4 | 9 | 14 | 10 |
| Difference | -1 | 4 | 4 | -2 | -4 | 4 |
Edwards | | 15 | 31 | 21 | 15 | 11 | 9 |
| Actual | 12 | 45 | 30 | 25 | 7 | 11 |
| Difference | 3 | -14 | -9 | -10 | 4 | -2 |
Kerry | | 35 | 24 | 25 | 46 | 32 | 27 |
| Actual | 38 | 30 | 27 | 51 | 43 | 50 |
| Difference | -3 | -6 | -2 | -5 | -11 | -23 |
Lieberman | | 6 | 4 | 7 | 3 | 9 | 16 |
| Actual | 9 | 2 | 6 | 4 | 7 | 11 |
| Difference | -3 | 2 | 1 | -1 | 2 | 5 |
Sharpton | | | 10 | | | | |
| Actual | | 10 | | | | |
| Difference | 0 | 0 | 0 | 0 | 0 | |
| | | | | | | |
MOE | | 4 | 4 | 4 | 4 | 4 | 4 |
# outside MOE | | 0 | 2 | 1 | 2 | 2 | 2 |
# at MOE | | 0 | 2 | 1 | 0 | 2 | 1 |
Error (RMS) | | 5.39 | 16.37 | 10.30 | 11.58 | 13.89 | 24.04 |
Max error | | 8.94 | 9.80 | 8.94 | 8.94 | 8.94 | 8.94 |
Difference | | -3.56 | 6.57 | 1.35 | 2.63 | 4.95 | 15.10 |
| | | | | | | |
| | NH | SC | OK | MO | AZ | DE |
SUSA | | | | | | | |
Clark | | 12 | 16 | 29 | 6 | 28 | 9 |
| Actual | 12 | 7 | 30 | 4 | 27 | 10 |
| Difference | 0 | 9 | -1 | 2 | 1 | -1 |
Dean | | 28 | 9 | 7 | 15 | 18 | 12 |
| Actual | 26 | 5 | 4 | 9 | 14 | 10 |
| Difference | 2 | 4 | 3 | 6 | 4 | 2 |
Edwards | | 14 | 34 | 27 | 20 | 7 | 11 |
| Actual | 12 | 45 | 30 | 25 | 7 | 11 |
| Difference | 2 | -11 | -3 | -5 | 0 | 0 |
Kerry | | 33 | 17 | 26 | 44 | 34 | 42 |
| Actual | 38 | 30 | 27 | 51 | 43 | 50 |
| Difference | -5 | -13 | -1 | -7 | -9 | -8 |
Lieberman | | 7 | 4 | | | | 10 |
| Actual | 9 | 2 | | | | 11 |
| Difference | -2 | 2 | 0 | 0 | 0 | -1 |
Sharpton | | | 12 | | | | |
| Actual | | 10 | | | | |
| Difference | 0 | 2 | 0 | 0 | 0 | |
| | | | | | | |
MOE | | 4 | 4 | 5.3 | 3.5 | 5 | 4.8 |
# outside MOE | | 1 | 3 | 0 | 3 | 1 | 1 |
# at MOE | | 0 | 1 | 0 | 0 | 0 | 0 |
Error (RMS) | | 6.08 | 19.87 | 4.47 | 10.68 | 9.90 | 8.37 |
Max error | | 8.94 | 9.80 | 10.60 | 7.00 | 10.00 | 10.73 |
Difference | | -2.86 | 10.08 | -6.13 | 3.68 | -0.10 | -2.37 |