IpsosMORI – we need to talk about your tiny sample size.

“The web survey was launched on 26 January 2011, with extensive publicity across BBC television and radio, and newspaper coverage. Responses were enthusiastic and, by July 2011, 161,400 complete surveys had been submitted. However, examination of the data revealed that the GBCS web survey suffered from a strong selection bias, with respondents being predominantly drawn from the well-educated social groups. As one BBC journalist told us when we reported this problem, ‘yes, you seem to have got a typical BBC news audience there’. To address this problem, the BBC therefore agreed to conduct a separate, nationally representative face-to-face survey using identical questions. This survey, with 1026 respondents, was conducted using quota sampling methods by the well-known survey firm GfK in April 2011. Tests from its field division and by ourselves indicate that its demographics are nationally representative. In this article we refer to the nationally representative survey as GfK and the web survey as GBCS.”

Table 1. Reproduced from Savage et al.

Putting this table into graph form gives Figure 1, where the descriptive categories have been converted into numbers by Excel:

Figure 1.

Here, the biases mentioned by Savage et al. can be seen clearly, where the professional classes are heavily over-represented and the lower classes are under-represented. If the GBCS had been exactly representative of social structure in Britain as a whole, the graph would have been  a horizontal straight line of value 1. Anything above 1 means that that class is over-represented; anything below means that it is under-represented in the survey. In other words, this graph shows the biases of the GBCS sample.

IpsosMORI classify the respondents to their surveys into four demographic categories: AB, C1, C2 and DE. These are compressed categories derived from the following standard demographic categories:

Table 2: Standard demographic classifications.

By combining the nine categories in Table 1 into the six standard demographic classifications and then further compressing them into the four categories used by IpsosMORI, the likelihood of responding to the GBCS survey becomes:

Figure 2: Likelihood of responding to GBCS (compressed social classes).

Note that the R² value in Figure 1 is 0.696 indicating a good fit for the polynomial curve; whilst the R² value in Figure 2 is 0.984 indicating a near perfect fit. This suggests that the compression of the social classes has been conducted correctly.

In this curve, social classes C2 and DE are under-represented in the GBCS survey, C1 is about right because it is close to a value of 1; and AB are over-represented by 1.8 times.

This method of deriving ratios from the comparison of known proportions in the real population with proportions in a survey such as the GBCS, suggests that a similar method could be used to compare real or known data with poll data to determine demographic sampling biases by the polling companies.

When this is done for the IpsosMORI poll of 18th May 2016, by social class, the following curve is the result:

Figure 3

“Furthermore, comparisons with the Culture, Class, Distinction (Bennett et al., 2008) study showed that it was not possible to simply weight the GBCS to deal with these skews because the GBCS respondents from routine classes turn out to be highly unrepresentative of their peers in these classes. They possess relatively more cultural and social capital than their peers, and indeed the very act of participating in the GBCS was a ‘performative’ way of claiming cultural stakes [my emphasis] (as discussed by Bourdieu, 1984; Skeggs, 2004). Let us be clear, therefore, that GBCS alone cannot be used to derive a representative model of class.”

The parallels between the respondents to the GBCS and polling panels seem to be clear. From all social classes, polling panelists will be confident of their own beliefs (after all, they are regularly consulted about what they think of a brand or a political party). This means that they are less diffident about their lives and the society in which they live. All this points towards a largely liberal perspective on life.

There may even be two further problems with self selected polling panels, and which are not discussed or even acknowledged by the polling companies. The first is that hard left wingers are taught to ‘stack’ an audience – a problem known to occur in the BBC Question Time audience. This means that even though they may know each other, they spread out in the audience and make as much noise as possible. This gives the impression of the whole audience being much more united in an opinion than they really are. This technique is used for public meetings and will mean that an otherwise minority view will often prevail over the majority. The Left are extremely good at making it sound as if they are more numerous than they really are.

The second problem is the possibility that some people will like doing polls so much that they will join more than one polling panel. It is completely unknown how big this problem is, but if it is widespread (and I suspect it is because of the ‘stacking’ phenomenon) then online polls could potentially be all polling essentially the same panel. That is, not just people with similar views, but the same individuals. This is one possible reason for the similarity between polling results by the different companies. To my knowledge, online polling companies do not test or correct for this problem.

So this leaves the phone polling which, if conducted properly, is theoretically the only way of getting a truly random and representative sample. Except that it isn’t. Apart from the difficulties of getting a representative sample during weekdays and working hours, when many people are at work (Some polls may correct for this by focussing their phoning during the evenings) there is still the issue of a different type of self selection, which is similar to the kind of self selection that happens in the online polls. This takes the form of a willingness to answer a lot of highly intrusive questions about one’s life and finances to a total stranger over the phone. With increasing wariness over phone based selling and downright scams, many people are immediately suspicious of plausible sounding patter. Most will refuse to answer questions and put the phone down. Against this background of the politically reticent refusing to answer questions, the phone poll becomes a sample of those who are politically motivated. This is probably the root cause of many of the biases detected in the IpsosMORI poll analysed above.

Add to this the normal reticence that many people have about sharing their political views with strangers (or even relatives, friends or colleagues) and it becomes apparent that there is a big problem with formal, properly scientific polling methods.

Despite all of these problems, the polling companies think that they can get within two or three percent in a General Election. However, the forthcoming referendum is not a General Election. In British politics, referenda are very rare events. Furthermore, the subject goes right to the heart of how we see ourselves as a sovereign country and independent people. The issues of economic, demographic and democratic concern cut right across party politics, but tend to do so unevenly. For example, most members of the Conservative Party are known to be highly Eurosceptic. Meanwhile, the intellectual Left are more inclined to be Europhiles. The working classes, normally thought of as ‘belonging’ to the Left, are more likely to be patriotic and suspicious of foreigners. A referendum on Brexit is therefore much more difficult to call.

Add to this the fact that nearly all of the casual online polls in the newspapers and other places are almost without exception, heavily in favour of Leave with sample sizes in the tens of thousands. It will be argued by the polling companies, that these are all ‘unscientific’. Furthermore, they all suffer from the criticism above, in that they are all self selected. Some of them can be criticised because they do not prevent people from making multiple votes by logging in and out of the poll. Some of them do actively weed out multiple voting because they identify e-mail addressed and prevent multiple voting, but even so, they are still self selecting. Nevertheless, the huge sample sizes of these online polls should cancel out any demographic or educational bias. But what is absolutely clear is that they are polling a very different sector of society; and that that sector is in favour of Leave by a margin of approximately 40%.

The principal difference between the casual online polls and the formal polls is that people do not have to give away lots of information about themselves before they get a chance to vote. Furthermore, they get an instantaneous result.

People I have spoken to are adamant that most of their friends and colleagues are for Leave. I have even spoken to one Remain campaigner who thinks Leave will win, because of the greater passion of the ‘outers’. And therein lies a clue as to what seems to be going on below the radar. There seems to be a seething discontent with the status quo in British politics today. Almost everywhere you look, there is contempt for many politicians who are seen to be careerist and self-serving. The more arrogant of the media élite will dismiss this phenomenon as ‘populism’. They will continue to regard the rest of us as vulgar, unwashed, uncircumcised, knuckle dragging proles whose views are rarely sought and needed even less. But I feel that things are changing.

This referendum is talked about in pubs and discussed at work (places where people would never normally discuss politics). There are debates all over the country. Twitter is alive with those on both sides commenting about it. I have never before seen this level of political engagement amongst ordinary people. Something is happening that is not being picked up by the pollsters and certainly not David Cameron and his team. The reticent are beginning to speak. And I think that, in the quietest and most unassuming way characteristic of the British way of doing things, they are angry – and they are going to vote Leave in very large numbers.

And the only place where this is being picked up is in the casual online newspaper polls with huge sample sizes.