1 / 24

Epistemology and Methods Content Analysis and Survey Research May 13 2008

Epistemology and Methods Content Analysis and Survey Research May 13 2008. Overview. Content Analysis Survey Research Group Exercise. Content Analysis. Mannheim and Rich 1995, chapter 10

edan
Download Presentation

Epistemology and Methods Content Analysis and Survey Research May 13 2008

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Epistemology and MethodsContent Analysis and Survey ResearchMay 13 2008

  2. Overview • Content Analysis • Survey Research • Group Exercise

  3. Content Analysis • Mannheim and Rich 1995, chapter 10 • Content Analysis: the systematic counting, assessing, and interpreting of the form and substance of communication • Precondition: physical record of communication • Type of communication: books, magazines, newspapers, documentaries, films, recordings, photographs, meeting minutes, government documents, diplomatic communiqués, cartoons, political advertisement, speeches, letters and diaries, blog,…. • Relatively low cost research method

  4. Mannheim/Rich • Steps in using this method: • Define the population of communications we want to study (this is determined by the research question!) • Once population is defined, one possibility is sampling (still a large n, sufficient level in generalizing from results) • Focus on • Type of communication • Time period of communication • The size of communication (e.g. op-ed articles) • The frequency of communication (e.g. monthly memos) • The distribution of communication (e.g. home delivered newspapers) • The location of communication • The parties to the communication (sender, receiver)

  5. Mannheim/Rich • Define the unit of analysis • Word (e.g. peace in speeches of presidents) • Pitfalls: • Nonstandardized measures can lead to biased results (e.g. word references by 1000 words) • Different meanings of the word (e.g. we seek peace vs. we never allow peace): counting the words in context • We can use judges or coders • Or we move to a second unit of analysis: Theme

  6. Mannheim/Rich • Theme: context+ / complexity+ • Example: Development (trade openness increases economic development; the developmental concerns of LDCs, …this contributes to sustainable development) • Spend time on coding rules…

  7. Mannheim/Rich • Item: the communication itself taken as a whole • Example: e.g. how many times GB senior referred to Saddam Hussein as Hitler-like prior to 1991 Persian Gulf War • Full-text search (e.g. NYT) for words “George Bush”, “Saddam Hussein” and “Hitler” • These words appear within 10 words in either direction

  8. Mannheim/Rich • Create a “dictionary”: define each and every observation we might make and allocate a particular coding • What constitutes a salient reference: e.g. Cuban school books (American, US, Imperialists, Outlaw Regime in Washington, etc…) • Evaluation of salient references (good or bad, pro or anti, ranking, intensity) • E.g. ranking statements to measure “intensity” of newspaper endorsement of candidates (Figure 10.1) • Assistance from a group of judges (about the meaning or intensity of a term) • The role of pre-testing!

  9. Mannheim/Rich • Various procedures: • E.g. define distribution scale (1-x), tell judges to distribute a certain quota for each category (1-x) • The mean is calculated for each statement • E.g. pair-comparison scaling • Pitfalls: number and selection of judges • Pair-comparison for 100 to 200 items!

  10. Mannheim/Rich • Special problems: • Communications are issues for a purpose “All Chinese people believe that the new agricultural policy is a major step….” • If we want to assess the impact of communication, we need to know whom it reaches… • What is the degree of our own access to communication (free choice over material we analyze?) • Is the research sample representative • Intercoder reliability (human judgments): degree of consensus among coders

  11. Laver/Garry • Estimating Policy Positions from Political Texts • The Manifesto Research Group (MRG) / Comparative Manifestos Project (CMP): biggest show on the road (started in the early 1980s) • Party manifestos are strategic documents written by politically sophisticated party elite with many different objectives in mind • Usually issues during elections campaigns, reflect party positions • Provide evidence party position changes over time

  12. Laver/Garry • MRG: measuring relative emphasis of an issue (salience, not party position!), position # emphasis (attention paid to) • MRG categories unipolar (e.g. law and order) • MRG categories bipolar (e.g. social service expansion) • In order to retrieve information on position, additional coding of MRG raw data (e.g. Harmel-Janda Party Change Project (PCP)) • Definition of 19 issues of interests, coding -5 to +5, few experts engage in coding) • Caveat: Prior knowledge (subjective placement of parties)…

  13. Laver/Garry • Estimating policy positions / expert coding: • Data reduction (coding scheme) • in MRG sentence count for 54 coding categories, in PCP transforming text into scores on 19 policy scales • New approach for systematic coding: an example of hierarchical decision-making process (Table 1): • 1 ECONOMY (Role of state in economy) • 11 Economy/+State+ (Increase role) • 111 Economy/+State+/Budget (Budget) • 1111 Economy/+State+/Budget/Spending (Public Spending) • 11111 Health • 11112 Education and Training • 11113 Housing • 11114 Transport • …. • 1112 Economy/+State+/Budget/Taxes (Increase Taxes) • 1113 Economy/+State+/Budget/Deficit (Increase Budget Deficit)

  14. Laver/Garry • Estimating policy positions / computer coding • Quantitative content analysis • Dictionary – mechanical criteria • But need to focus on less ambiguous words or phrases (unipolar) • Example on “taxes”…it’s possible to lower taxes and to raise taxes, but in manifestos in favor of lowering taxes (question is how often is one wrong in order to correct?) • The validity costs are offset by gains in reliability (Computer coding is reliable) and fewer costs • Computer codes without knowledge of context (no pre-information…), computer could potentially more easily detect policy changes… • Laver/Garry on policy positions in UK and Ireland • Cross-validation of techniques (Computer, revised expert, MRG, expert surveys), high correlation on economic policy, less so on social policy (e.g. MRG technique of saliency…)

  15. Content Analysis • Questions?

  16. Survey Research • Mannheim and Rich 1995, chapter 7 • Survey research is a method of data collection in which information is obtained directly from individual persons who are selected so at to provide a basis for making inferences about some larger population • Methods: direct questioning (face-to-face or telephone interviews) or questionnaires… • Types of information: facts, perceptions (what respondents know (or think they know)), opinions, attitudes (more stable preferences), behavioral reports

  17. Survey Research • Stages of Survey Research Preparation • Conceptualization (objective, theory, specific set of questions) • Survey Design (explorative, descriptive, explanatory) • linking objective with data-collecting method; • cross-sectional or longitudinal design (trend and cohort studies, panel studies) • Panel studies – watch out for: costly, keep track of sample, moving towards a biased sample? (respondent changes due to interviews, drop-outs)

  18. Survey Research • Instrumentation (operationalization): define content, form, format, wording and order of questions • Content: Limit the number of questions and time, keep number of hypotheses to be tested small (degree of data!) • Form: Open-ended vs. closed-ended questions (choice of options influences responses) • Format: Techniques how questions are presented (e.g. visual aids…)

  19. Survey Research • Wording and order of questions • compare with prior research • are respondents competent to answer?, use contingency questions to find out (to assess the level of knowledge) • statements are more useful than questions (e.g. to measure intensity of opinion, respondents use the same frame of reference) • but tendency of a response set (tendency to agree with statements), items should be mixed

  20. Survey Research • Generally 4 steps: explanation, warm-up questions, substantive questions, demographic questions • Explanation: should not reveal study information that would bias responses (eliminating fear of study, importance to warrant time and attention) • Warm-up: establish a good relationship with the respondents • Substantive questions: check question ordering, pre-test will give some guidance (experiment with different orderings) • Demographic items: at the end, personal information, prevent respondents’ being ill at ease… • Do not crowd items

  21. Survey Research • Techniques: • Face-to-face surveys:… • Mail surveys: • Telephone surveys: …. • Discussion on pro and cons? • Sampling, Validity and Reliability

  22. Brady: Contributions to Political Science • No other method for understanding politics is used more • Sample surveys analyzed with the help of statistical techniques • New survey methods involve “experiments embedded in surveys”: modify questions wordings to determine whether counter-arguments, subtle cues, rhetorical, emotional or cognitive factors can change opinion or behaviors (how stable are opinions and how do respondents threat information) • Examples: The Use of Competing Frames or Priming • Experiments have made surveys even stronger methods for testing theories, they potentially score high on conceptual richness and policy relevance • Surveys have the great virtues to ask the questions you want to ask, when and where you want!...

  23. Designing an experiment-survey on • Measuring • a) the attitude of citizen vis-à-vis free trade • Questions / Statements • Stability of attitudes (Framing, Priming) • b) the attitude of citizen vis-à-vis international law (e.g. WTO law) • Questions / Statements • Stability of attitudes (Framing, Priming)

  24. Good luck for the moot and assignment 2!

More Related