Computers, Coders, and Voters: Comparing Automated Methods for Estimating Party Positions

TY - JOUR

T1 - Computers, Coders, and Voters: Comparing Automated Methods for Estimating Party Positions

AU - Hjorth, Frederik Georg

AU - Klemmensen, Robert Tranekær

AU - Hobolt, Sara Binzer

AU - Hansen, Martin Ejnar

AU - Kurrild-Klitgaard, Peter

PY - 2015/4/1

Y1 - 2015/4/1

N2 - Assigning political actors positions in ideological space is a task of key importance to political scientists. In this paper we compare estimates obtained using the automated Wordscores and Wordfish techniques, along with estimates from voters and the Comparative Manifesto Project (CMP), against expert placements. We estimate the positions of 254 manifestos across 33 elections in Germany and Denmark, two cases with very different textual data available. We find that Wordscores approximately replicates the CMP, voter, and expert assessments of party positions in both cases, whereas Wordfish replicates the positions in the German manifestos only. The results demonstrate that automated methods can produce valid estimates of party positions, but also that the appropriateness of each method hinges on the quality of the textual data. Additional analyses suggest that Wordfish requires both longer texts and a more ideologically charged vocabulary in order to produce estimates comparable to Wordscores. The paper contributes to the literature on automated content analysis by providing a comprehensive test of convergent validation, in terms of both number of cases analyzed and number of validation measures.

AB - Assigning political actors positions in ideological space is a task of key importance to political scientists. In this paper we compare estimates obtained using the automated Wordscores and Wordfish techniques, along with estimates from voters and the Comparative Manifesto Project (CMP), against expert placements. We estimate the positions of 254 manifestos across 33 elections in Germany and Denmark, two cases with very different textual data available. We find that Wordscores approximately replicates the CMP, voter, and expert assessments of party positions in both cases, whereas Wordfish replicates the positions in the German manifestos only. The results demonstrate that automated methods can produce valid estimates of party positions, but also that the appropriateness of each method hinges on the quality of the textual data. Additional analyses suggest that Wordfish requires both longer texts and a more ideologically charged vocabulary in order to produce estimates comparable to Wordscores. The paper contributes to the literature on automated content analysis by providing a comprehensive test of convergent validation, in terms of both number of cases analyzed and number of validation measures.

KW - Faculty of Social Sciences

KW - Text as data

KW - automated content analysis

KW - party positions

KW - Wordscores

KW - Wordfish

KW - CMP

U2 - 10.1177/2053168015580476

DO - 10.1177/2053168015580476

M3 - Journal article

SN - 2053-1680

VL - 2

SP - 1

EP - 9

JO - Research and Politics

JF - Research and Politics

IS - 2

ER -