EXPERT SURVEYS and PARTY MANIFESTOS

1. EXPERT SURVEYS and PARTY MANIFESTOS Limits and potentials

2. STRUCTURE • Expert survey by Benoit and Laver on party policy • Some reviews: Curini’s analysis Steenbergen and Marks • Party Manifestos by CMP • Curini’s analysis

3. DESIGNING THE EXPERT SURVEY Benoit and Laver conducted a set of systematic surveys in order to collect the information required to locate all politically relevant parties on a wide range of policy dimensions The surveys were conducted from 2002-03, covered 47 countries, resulted in 1,491 valid expert responses, locating 387 different political parties on scales relating to a total of 37 unique policy dimensions.

4. 1.1 IDENTIFYING EXPERTS The typical expert is an academic specializing in political parties and electoral politics of his or her country. Method divided in steps: • Contact the national political science association of the country, if it exists, with a request for its membership lists.

5. 1.1 IDENTIFYING EXPERTS • If it does not exist, “snowball” strategy is used: short list of well-known experts from the country, and ask each to name as many additional experts as possible • Then contact each of those experts, asking them in turn to name as many additional experts as possible Additional lists from universities and NGO, and exclude journalists and political actors

6. 1.2 IDENTIFYING PARTIES “Politically relevant” parties where chosen, when they met any one of three criteria: • Every existing national party that won seats in the national legislature at the country’s most recent election • Every existing national party that had won at least 1 percent of the vote nationally at the country’s most recent election

7. 1.2 IDENTIFYING PARTIES • Any other parties indicated by local experts as politically relevant despite not meeting the other two criteria Even small parties are included since they’re crucial for the analysis of political competition (ex. in forming gov) also to avoid to exclude parties that would play significant roles in party competition over a longer period.

8. 1.3 POLICY DIMENSIONS Core set of four substantive policy dimensions in order to allow for direct comparison between countries: • ECONOMIC POLICY • SOCIAL POLICY • ENVIRONMENTAL POLICY • DECENTRALIZATION OF DECISION MAKING For each dimension a scale from 1 to 20, with the lower numbers indicating a ‘left-wing’ position and the higher a ‘right-wing’ position.

11. LIMITS of EXPERT SURVEYS 2.1 HOW TO EVALUATE BIAS BL are aware that there could be a systematic relationship between the experts’ own ideologies and their judgments about party policy positions SYMPATHY SCALE: Experts are asked to place all parties on a scale indicating their own closeness to each party’s policies.

12. To assess bias BL used the following regression-based technique: • For each country-party section, they regress experts’ placements on their sympathy scores for the party in question • They compute the mean and the variance of the predicted left–right score for an expert with an ‘indifferent’ sympathy score of 10.5 (the midpoint of the 1-to-20 scale), taking this to be the ‘corrected’ placement

13. they compare the ‘corrected’ expert placements with the ‘actual’ placements (i.e. the mean score of parties resulting from the expert survey) to see whether or not these differences are statistically distinguishable From this analysis only seven parties appeared to involve sistematic bias that cuold be statistically distinguished from zero.

15. 2.2 SAMPLING/IDEOLOGICAL BIAS • Ideological bias known as rationalisation or projection: respondents tend to place the parties they like closer to where they perceive themselves to be (assimilation effect) and to place those parties they dislike farther away (contrast effect). • Sampling bias: respondents are systematically unrepresentative of all possible experts (more left-wing or liberal)

16. 2.3 SOLVING IDEOLOGICAL BIAS Focus on the sympathy question alone: risk of under/overstimate IB Curini: • Use an unfolding model in order to estimate the location of the ideal point using respondent’s preferences • “corrected score”: focus on the relationship between respondents’ ideal points and parties’ ideological stance

17. Scoreij= aj+ βj[s(Policyi– Scorei)] + εi Policyi : respondents’ position (unfolding method) on the left-right scale Scorei : mean party position on the same scale Bj : indicator of ideological bias if it is statistically distinguishable from zero (as it was the case for 53 parties out of 158) a direct way of “correcting” biased score, neutralising the ideological bias

19. 2.4 EVALUATING EXPERT SURVEYS Budge four type of problems: • what ‘party’ is being judged by the expert? • what criteria do experts bring to bear when they judge party positions? • do experts judge the intentions of parties or their behavior? • what is the time frame for the judgments that we ask experts to make?

20. HOW DO EXPERTS INTERPRET THE QUESTIONS IN EXPERT SURVEYS AND HOW DO THEY LINK SUBSTANTIVE KNOWLEDGE ABOUT PARTIES TO THOSE QUESTIONS? Steenbergen and Marks: these concerns can be alleviated in the expert survey design: • Avoid ambiguous terms • Give them a more circumscribed meaning limit interpretation but there may still be an interpretative space for experts that could distort their judgments

21. The 1999 expert survey of national party positions on European integration Ray: “What was the overall orientation of the party leadership towards European integration in 1999” • 7 response options • Specification of the object that was to be evaluated (party leadership) • Specification of the time frame (1999) provide to the experts a common frame of mind

22. Assess variance in the experts’ judgments: • Compute the standard deviation of their placements of parties • Perform a variance components analysis: limited statistically significant variation across the experts rather similar judgments about EU stances of the political parties • Attributes of a party or party system: measure party differentiation within a party system as the standard deviation of party positions in that country variation is a function of party differentiation, salience and dissent

23. Steenbergen and Marks’ model • The “congeneric test model”: the correlation between the expert judgments should be greater, perhaps much greater, when they are judging the same trait than when they are judging different traits • similarity coefficients: compare the correlational patterns across experts, we should expect to see a high similarity coefficient between two experts if those experts indeed evaluate the same trait

24. THE ITALIAN EXCEPTION Reliability of expert judgments on Italian parties is worse than that for other countries. This is primarily due to one expert (E4).

25. The correlations between the judgments of this expert (E4) and the remaining five experts are low, suggesting that he or she may have used different criteria for placing Italian parties. good grounds for doubting the validity of this expert responses although it is worth emphasizing that even after including this expert the final results for Italy look quite good

26. ALTERNATIVE MEASURES • Manifesto Research Group: extract party positions from party manifestos. It has coded both favorable and unfavorable mentions of European integration • Voter perceptions of party positions. The EES project asked respondents where parties stood on the issue of European integration. Subject to projection effects voters may project their own stance on European integration onto the party • Use members of parliament (MPs) and members of the European Parliament (MEPs) to gauge party support for European integration

27. ANALYSING PARTY MANIFESTOS In 1979 the Manifesto Research Group (MRG) was formed to analyze documents in 90 democracies comparatively and within a common framework. Manifesto: written as a whole and as overall balance of subjects carefully considered IT IS OVERALL PRIORITIES THAT MATTER RATHER THAN PARTICULAR STANCES TAKEN IN ISOLATION

28. The idea behind the MRG is that parties argued with each other by emphasizing different policy priorities rather than confronting each other on the same issues Since 1989 CMP (the Comparative Manifesto Project) has coded and standardized the statements of party election programs of 25 democracies since 1945 into 56 policy categories, such that each quasi-sentence of every election program is coded into one, and only one, specific category. SENTENCE WHICH IS THE VERBAL EXPRESSION OF ONE POLITICAL IDEA OR ISSUE (GENERAL CODING UNIT)

29. Basis of the policy estimates: textual data, examination of parties’ and governments’ own statements of policy, in the shape of election programmes (manifestos) and declarations in the Parliamentary debate before a vote of confidence or investiture • estimates on what actors themselves have said rather then on people’s judgements • Major source of evidence: texts, autoritative documents issued by parties and governments • Collect and organize it in statistical form

30. 13 categories as indicators of policy positions emphasised by the “left” and another 13 as indicators of policy positions emphasised by the “right”. The left-right position is constructed for each party in each election by subtracting the sum of its left statements from the sum of its right statements. As a result the range of this scale is between –100 (i.e., the score of a party devoting its entire programme to Left-wing issues) and +100 (i.e., the score of a party devoting its entire programme to Right-wing issues).

32. ADVANTAGES OF PARTY MANIFESTOS They allow three types of comparison: • Changes in policy position over time within specific parties • Differences in policy positions across parties even in different countries • Differences across countries

33. RELIABILITY and VALIDITY Two measurement concerns: Reliability: ARE THE PROCEDURES DERIVED FROM THE THEORY RELIABLE, GIVING THE SAME RESULTS EVERY TIME THEY ARE APPLIED? (computers will always apply some procedures to the same text in the same way: this is not always true of the human coders who have produced the Manifesto data) Validity: ARE PROCEDURES VALID IN THE SENSE OF ACTUALLY TELLING US WHAT WE THINK THEY TELL US ABOUT CONTENT (OF MANIFESTOS) AND SENDERS (PARTIES)? (is the underling theory itself correct? Are the codings correlated with other measures of the same conceptual construct?)

34. THE ITALIAN CASE CURINI using Manifesto scale, a trade-off can emerge: if on one hand this methodology guarantees a more theoretically sound comparison between countries and time, on the other hand it does not allow for specific characteristics of individual party systems (that could therefore be buried under a uniform left-right dimension).

35. The Italian party system was characterized for a long period by: • THE PRESENCE OF TWO ANTI-SYSTEM PARTIES, THE COMMUNIST PARTY AND THE ITALIAN SOCIAL MOVEMENT ON THE RIGHT • THE ABSENCE OF ANY CREDIBLE ALTERNATIVE TO A GOVERNMENT FORMULA BUILT AROUND THE CHRISTIAN DEMOCRATIC PARTY • A HIGH NUMBER OF RELEVANT PARTIES POLARIZED PLURALISM

37. PLACEMENT OF THE 4 MAIN ITALIAN PARTIES DURING THE “FIRST REPUBLIC” (1948-92), USING THE MANIFESTO SCORE Some results seem quite odd: in two occasion the DC is located at the right of the MSI, along the 70s the PSI is at the left of the PCI, and the PCI in the 1953 election is the furthest right party.

38. CURINI’S METHOD For scholars who want to apply models that rely on the policy preferences of the Italian political parties and that wish to use PMD, Curini proposes a method that consists of the following steps: • The first change needed is the application of a reliability analysis to the original Manifesto scale, in order to assess the scale’s internal consistency: the Cronbach’s alpha coefficient is a famous indicator of consistency. Ideally, it should be above .7. The scalability of the original Manifesto scale for the country investigated should be assessed

39. If the Cronbach’s alpha is satisfactory, parties’ score should be smoothed by recognizing how the score of a party at time t does not translate directly in its position (If however the reliability analysis shows some inconsistency, the possibility to drop some categories included in the original left-right scale should be considered before smoothing the parties’ scores)

40. CRONBACH’S ALPHA RESULTS POSITIVE (.54) ALTHOUGH LOW BY THE STANDARDS FOR LIKERT SCALE. TO TEST WHETHER ALL THE INDICATORS BELONG TO THE SAME SCALE, VALUES FOR THE CRONBACH ALPHA ARE COMPUTED ALSO WHEN SPECIFIC ITEMS ARE REMOVED FROM THE SCALE. AS A RESULT THE CATEGORY “DEMOCRACY” SEEMS NOT TO TAP THE SAME CONCEPTUAL DIMENSION AS THE OTHER INDICATORS. REMOVING THIS CATEGORY HELPS A BIT THE CRONBACH ALPHA RESULT (.58). HOWEVER, THE STILL UNSATISFACTORY VALUE OF THE SCALE SUGGESTS THAT THE EQUAL WEIGHTING ASSUMPTION OF ALL THE ITEMS WHICH CHARACTERISES THE ADDITIVE MANIFESTO SCALE BE DROPPED.

41. Second modification: the Manifesto score for a given party need not be identical to its actual positional value THE GENERAL PUBLIC IS EXPECTED TO WEIGHT INFORMATION FROM THE PAST AS WELL AS PROMISES AND DECLARATIONS OF THE LATEST MANIFESTOS TO IDENTIFY THE ACTUAL POSITION OF A GIVEN PARTY Then the position of a party at time t should reflect not only the score obtained in its current electoral program but also the party’s general image emerging from the cumulation of all its previous manifestos

42. To produce measures that capture this dynamics, the exponentially weighted moving average of current and past manifesto positions is computed: POSITIONt = a MANIFESTOt +(1.0 – a)POSITIONt-1 Curini analyzes the 1953 general election and, by adding the scores obtained from the Manifesto scale without the category “Democracy”, and his final scores, he obtains more plausible results: the PCI moves from being the most rightist party to the most leftist, the PSI moves to the left of the DC, MSI becomes the most rightist one.

44. CURINI’S CONCLUSIONS Considering the 2001 Italian general election, an advantage of his method stands out: IN THAT ELECTION MOST PARTIES (WITH THE EXCEPTION OF COMMUNIST REFOUNDATION PARTY) DID NOT ISSUE THEIR OWN ELECTORAL MANIFESTOS, BUT FORMED PRE-ELECTORAL COALITIONS AND SUBSCRIBED TO A JOINT PLATFORM. THIS IMPLIED THAT THE CMP WAS ABLE TO CODE JUST ONE MANIFESTO FOR EACH COALITION. NONETHELESS, ESTIMATION OF EACH PARTY’S INDIPENDENT POLICY POSITIONS IS ALSO REQUIRED AS PARTIES ARE STILL THE PRE-EMINENT POLITICAL ACTORS WITHIN ITALY

45. CURINI’S CONCLUSIONS By contrast, the smoothing procedure included in his method allows him to estimate different scores for different parties even if at time t parties agree on the same manifesto, precisely because different parties (usually) have different histories.

46. Policy estimates of Manifesto data are ideal also for operationalizing and testing coalition theories. As an additional check for the validity of his scores, Curini uses them as dependent variables to test a widely accepted theory: the central role of the median party in policy bargaining.

47. CONCLUSIONS McDonald/Mendes: expert surveys are deficent and the manifesto measures are useful on two concerns: • The experts place the parties in such stable locations that there is little hope of using expert surveys to investigate party policy dynamics. The CMP is the only viable data source for observing such dynamics and analysing the party movements

48. Expert surveys may produce suspect results about where the parties stand on different dimensions of politics and policy. There is doubt that expert respondents actually can and do make clear distinctions between policy dimensions. Steenbergen & Marks: there are good reasons to trust expert survey results on party positions: • Consistency of experts’ responses • Expert placements of political parties converge with other measures

49. As we have seen both methods have confident support. BL admit and analyse some divergent estimates between expert survey and party manifestos. They claim that these differences are due to the understimation of some issues in the CMP analysis, and obviously that their method is the “correct” one…

50. OUR CONCLUSIONS Expert surveys rely on a certain leeway accorded to respondents and therefore are subject to the risk of bias But when using Manifestos, still the analysis is based on certain categories that can be ambiguously assigned both to the left and the right side, or on a certain statements that sometimes could be more linked to strategies (due to the elections) then to the real nature of that party. How can one really find the “hidden” real party position on certain issues?