Why Exit Polls Go Wrong?

It is 2022 and the election season is around in India. In the next few months, we are going to have State Legislative Elections in five states. This includes elections in politically the most important state in India: Uttar Pradesh, and the heart of the Farmers’ protest: Punjab. But, before the results will be announced, we will see a lot of news channels conducting exit polls in these states to show you the probable winners.

If we look back at the last year’s state election of West Bengal, the performance of exit polls was very poor. Almost every big pollster predicted that the Bhartiya Janta Party (BJP) would give a tough fight to the incumbent All India Trinamool Congress (AITMC). Some even predicted a win for the BJP. But, as it turned out, AITMC won by a more-than-comfortable margin of 138 seats. Opinion polls inflated BJPs seats by 42% – 148%, which is a huge blunder. So, why did it happen? And how are exit polls, more often than not, wrong?

An opinion poll is conducted by surveying a randomly chosen set of people which is representative of the entire population. The survey is done by various techniques which can alter depending upon the surveyor company. For instance, the ABP-CVoter opinion poll for Uttar Pradesh was conducted using Computer Assisted Telephone Interviewing (CATI) technique which means interviewing people on a call and recording their answers on a computer.

Opinion polls are generally a subject of sampling error. It means the risk that the chosen set of people is, purely by chance, not a true representative of the entire population. This defect is commonly set right by giving a range to the prediction, and by surveying a maximum number of people. The larger the sample, the sampling error would be that small.

Contrary to this, the 1936 US election poll conducted by Literary Digest surveyed humongous 2.4 million people and still showed Landon winning against Roosevelt by a comfortable margin (the actual result was that Roosevelt crushed Landon by a huge margin). Whereas, the opinion poll conducted by George Gallup was much closer to the result by interviewing only three thousand people.

Tim Harford writes in How to Make the World Add Up: Ten Rules for Thinking Differently About Numbers,

“But if three thousand interviews were good, why weren’t 2.4 million much better? The answer is that sampling error has a far more dangerous friend: sampling bias… [It] is when the sample isn’t randomly chosen at all. George Gallup took the pain to find an unbiased sample because he knew that was far more important than finding a big one.”

Literary Digest mailed people a form from their contacts in automobile and telephone directories. And at that time, people with a car and a telephone were fairly prosperous. Hence, they were only surveying the rich population. Also, something known as survivor bias was at work here. To explain this, let me take you to World War II.

During the war, British Royal Air Force wanted to protect its airplanes from German anti-aircraft guns. So, they decided to put heavy plating on the plane. But, as the plating was heavy, it could not be used to cover the entire plane, otherwise, it would not be able to fly. Analysis showed that the fuselage had more holes than the engine, hence it was decided to cover the fuselage, until statistician Abraham Wald arrives to the scene. His insight was quite exceptional: the reason these airplanes have survived is that they have got hit on places other than the engine. The engine was never hit which is why they were able to come back. Those who got hit on the engine never came back from the war. Hence, plating should be done where there are no or fewer holes, and not where there are maximum.

The British analyzed the surviving airplanes which were in front of them and forgot about the missing ones. Similarly, Literary Digest took the people who responded to their mail into consideration and never cared about the missing people who never replied. It turned out that more Landon supporters replied to the mail. Survivor bias also adds to the sampling bias.

Even the census goes through such problems. The 2011 census in India also omitted 2.3% of the population, according to its Post Enumeration Report. The percentage looks small, but a small fraction of a large number is still a large number. As Harford writes,

“The census takers know that certain kinds of people are less likely to respond when the official-looking census form lands with a thud on their doormat: people who live in multiple-occupancy houses such as a shared student house; men in their twenties… As a result, 5 per cent may look very different from the 95 per cent (response rate of British census) who do.”

To conclude, that is the long and the short of opinion polls. And I hope it might change your views the next time you look at one of them.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s