Saturday, April 14, 2007
The art of polling
Debate on the NOP/ITV Wales poll has been kicked off on the internet once more by a post from BBC Welsh Affairs Editor, Vaughan Roderick, on his Welsh language blog. A translation is available here.
Vaughan refers to a strange apology that appeared in the Western Mail on Monday 9th April in which they sought to clarify the outcome of the poll: "In our analysis of an NOP opinion poll for ITV Wales on voting intention at the National Assembly election, we incorrectly suggested that a predicted 3.5% swing from Labour to the Conservatives would result in a Conservative gain from Labour at Carmarthen West and Pembrokeshire South. In fact, the seat would be won by Plaid Cymru"
Vaughan argues that the raw statistics actually show Plaid Cymru marginally ahead of the Tories on the constituency vote (182 preferences to 180) and that the position is reversed only when the figures are adjusted for propensity to vote. He believes that there are two interpretations that can be placed on the poll and that Plaid Cymru have been harshly treated.
Although, I have concerns with this poll, these centre more on how it is being applied rather than the published outcome itself. I believe for example that it is common practise to weight polls so as to give more precedence to the views of those likely to vote. Accordingly, I am much more inclined to accept the weighted outcome than the raw figures as an accurate indication of people's preferences. I also believe that the larger sample helps in dealing with the particular circumstances of Wales, an issue that has led to larger margins of error on more standard data sets.
It is my also view that the attempt to turn the polling figures into seats is fatally flawed. Although the data set is large enough to provide a snapshot of opinion across Wales, it cannot hope to do the same for an individual constituency, especially one as large and as diverse as Carmarthen West and South Pembrokeshire. Applying standard swings is all very well, but these sort of standard swings do not tend to happen anymore, and in very marginal seats, especially three-way marginals, there are many other factors that need to be considered that can only be accounted for in an individual poll aimed at that seat. If there is one lesson that we can pick up from the 2005 General Election it is that voting has become more regionalised and localised, with some trends being confined to certain constituencies or groups of constituencies.
Secondly, the application of the poll to the regional list seems to have gone awry in the north. I find it difficult to believe for example that on the stated level of support in the poll, that the Tories can win two constituencies AND two lists seats in the North Wales region. That outcome is both unlikely and too convenient in supporting the notion that the Tories will be the second largest party for my liking.
The fact remains though that whichever set of figures one looks at this poll is a disappointment for Plaid Cymru. It puts them at a level of support well below their own claims. It leaves them struggling to maintain their status as second party and it makes some of their current seats more vulnerable, most notably Ceredigion. Arguing around the margins in interpreting the figures does not change that.
But, the real problem with this poll lies in its isolation. It is a snapshot in time and we have already moved on. There are no other polls to compare it with so as to give us perspective. There is no way of measuring trends. The danger is that we will set it in stone and leave it in a field as a monument to political punditry when actually we need a lot more information both to awaken people's interest in this election and to generate an edge that will get some real debate going.
I would argue that this is the media's responsibility. They are the only body of people who are sufficently independent to produce credible polls. They should also recognise that regular polling will assist them in their duty to inform and interpret the news. In Scotland there is a real buzz because regular polls have set challenges to all the parties and caused them to raise their game.
Without the same here we will muddle along but we will not set the world on fire. We will debate the issues in a worthy and dutiful manner and 40% or so of the electors will come out and vote. The challenge will be to secure differential turnout, not to change the course of a nation. Politicians have a duty here as well, but our job is made so much easier if there is regular polling and an air of expectation and excitement as to the outcome.
Instead of quibbling over figures journalists and broadcasters need to start taking the temperature of the nation more frequently and in a way that informs and interests ordinary people. If this election is quiet or dull it is because they have not been doing that. In these circumstances a low turnout becomes a self-fulfilling prophecy and that cannot be good for democracy.
Vaughan refers to a strange apology that appeared in the Western Mail on Monday 9th April in which they sought to clarify the outcome of the poll: "In our analysis of an NOP opinion poll for ITV Wales on voting intention at the National Assembly election, we incorrectly suggested that a predicted 3.5% swing from Labour to the Conservatives would result in a Conservative gain from Labour at Carmarthen West and Pembrokeshire South. In fact, the seat would be won by Plaid Cymru"
Vaughan argues that the raw statistics actually show Plaid Cymru marginally ahead of the Tories on the constituency vote (182 preferences to 180) and that the position is reversed only when the figures are adjusted for propensity to vote. He believes that there are two interpretations that can be placed on the poll and that Plaid Cymru have been harshly treated.
Although, I have concerns with this poll, these centre more on how it is being applied rather than the published outcome itself. I believe for example that it is common practise to weight polls so as to give more precedence to the views of those likely to vote. Accordingly, I am much more inclined to accept the weighted outcome than the raw figures as an accurate indication of people's preferences. I also believe that the larger sample helps in dealing with the particular circumstances of Wales, an issue that has led to larger margins of error on more standard data sets.
It is my also view that the attempt to turn the polling figures into seats is fatally flawed. Although the data set is large enough to provide a snapshot of opinion across Wales, it cannot hope to do the same for an individual constituency, especially one as large and as diverse as Carmarthen West and South Pembrokeshire. Applying standard swings is all very well, but these sort of standard swings do not tend to happen anymore, and in very marginal seats, especially three-way marginals, there are many other factors that need to be considered that can only be accounted for in an individual poll aimed at that seat. If there is one lesson that we can pick up from the 2005 General Election it is that voting has become more regionalised and localised, with some trends being confined to certain constituencies or groups of constituencies.
Secondly, the application of the poll to the regional list seems to have gone awry in the north. I find it difficult to believe for example that on the stated level of support in the poll, that the Tories can win two constituencies AND two lists seats in the North Wales region. That outcome is both unlikely and too convenient in supporting the notion that the Tories will be the second largest party for my liking.
The fact remains though that whichever set of figures one looks at this poll is a disappointment for Plaid Cymru. It puts them at a level of support well below their own claims. It leaves them struggling to maintain their status as second party and it makes some of their current seats more vulnerable, most notably Ceredigion. Arguing around the margins in interpreting the figures does not change that.
But, the real problem with this poll lies in its isolation. It is a snapshot in time and we have already moved on. There are no other polls to compare it with so as to give us perspective. There is no way of measuring trends. The danger is that we will set it in stone and leave it in a field as a monument to political punditry when actually we need a lot more information both to awaken people's interest in this election and to generate an edge that will get some real debate going.
I would argue that this is the media's responsibility. They are the only body of people who are sufficently independent to produce credible polls. They should also recognise that regular polling will assist them in their duty to inform and interpret the news. In Scotland there is a real buzz because regular polls have set challenges to all the parties and caused them to raise their game.
Without the same here we will muddle along but we will not set the world on fire. We will debate the issues in a worthy and dutiful manner and 40% or so of the electors will come out and vote. The challenge will be to secure differential turnout, not to change the course of a nation. Politicians have a duty here as well, but our job is made so much easier if there is regular polling and an air of expectation and excitement as to the outcome.
Instead of quibbling over figures journalists and broadcasters need to start taking the temperature of the nation more frequently and in a way that informs and interests ordinary people. If this election is quiet or dull it is because they have not been doing that. In these circumstances a low turnout becomes a self-fulfilling prophecy and that cannot be good for democracy.
Comments:
<< Home
The basic problem is that in each of the 40 first past the post seats there are different circumstances. Glyn Davies for example makes an interesting point that the end of dual candidates seem to have affected voters reactions to him as a candidate. Some even thought that he had retired! The result on May 3rd will depend on turnout and how the vote is divided.Labour could survive in some areas because the anti Labour vote is divided. On the other hand if there is a surge behind one candidate to turf out the sitting Labour member there could be some shocks. With 3 weeks to go all the parties need to up their efforts.I travel around Wales and quite frankly so far the reaction has been 'what election?' It is a real disappointment and you sense the boredom already in the media with an election that does not seem to inspire anyone except the political junkies. It isn't good for Welsh democracy or the future status of the Assembly. There is too much consensus and too few personalities. Russell Deacon is quite right to argue that it is stifling Welsh politics.
# posted by 
: 9:59 AM
: 9:59 AM
"Instead of quibbling over figures journalists and broadcasters need to start taking the temperature of the nation more frequently and in a way that informs and interests ordinary people. If this election is quiet or dull it is because they have not been doing that."
So it's obviously nothing to do with either the politics of the past four years or the politics promised for the next four years:)
So it's obviously nothing to do with either the politics of the past four years or the politics promised for the next four years:)
Ciaran, that depends on whether you want to take one sentence out of context as you have done, or not. I said that politicians have a responsibility as well, but it seems to me that one reason why there is no sense of expectation out is there is because people do not see a genuine race. Polls can help to attract interest and give some shape to an election that is otherwise being fought in the press by spin doctors.
A good post Peter. People like a 'competition' and opinion polls create continuing interest. I've even been known to watch the scoring in the Eurovision Song Contest - not having watcghed the contest itself !
I agree with Alwyn's view that polls should be specially designed for Wales, instead of using UK psephology..
# posted by : 9:53 PM
: 9:53 PM
It is difficult to disagree with any of that. Betsan Powys noted her frustration in her blog at the BBC's continuing refusal to publish 'state of the parties' polling, and at about £10,000 a pop there aren't other media outlets who can afford to commission a big poll. Though I hear the Western Mail plan a slightly cheaper one later in the campaign.
Peter Black is right to say the pundits are on shaky ground extrapolating seat predictions from a Wales-wide sample. But in fairness to Denis Balsom with ITV's being the only poll in the field how else can he try and make sense of the findings other than try and explain the percentages by predicting what they'd mean in terms of seats?
Journalists and politicians alike say they think the campaign is boring (as if they don't have a part in it!). It would help if the campaign wasn't spread over five weeks. It would be easier to generate a sense of occassion if the official campaign period was shorter, perhaps 2-3 weeks.
Peter Black is right to say the pundits are on shaky ground extrapolating seat predictions from a Wales-wide sample. But in fairness to Denis Balsom with ITV's being the only poll in the field how else can he try and make sense of the findings other than try and explain the percentages by predicting what they'd mean in terms of seats?
Journalists and politicians alike say they think the campaign is boring (as if they don't have a part in it!). It would help if the campaign wasn't spread over five weeks. It would be easier to generate a sense of occassion if the official campaign period was shorter, perhaps 2-3 weeks.
Very interesting discussion arising from a provocative post. I've blogged enough about my own concerns about the poll, but at least it is one and I think you're right Peter that the data becomes less meaningful when mapped on to individual seats.
"It would be easier to generate a sense of occassion if the official campaign period was shorter, perhaps 2-3 weeks."
This seems to chime with my recollections of the 2003 election, where the campaign only really seemed to come alive in this period. If the same happens in this election then today could well mark the gear change. Hooray!
Post a Comment
"It would be easier to generate a sense of occassion if the official campaign period was shorter, perhaps 2-3 weeks."
This seems to chime with my recollections of the 2003 election, where the campaign only really seemed to come alive in this period. If the same happens in this election then today could well mark the gear change. Hooray!
# posted by : 8:23 AM
: 8:23 AM<< Home
