Fitting Oval Pegs into Round Holes: Tensions in Evaluating & Publishing Qualitative Research in Top-Tier North Ameri

1. Fitting Oval Pegs into Round Holes:Tensions in Evaluating & Publishing Qualitative Research in Top-Tier North American Journals* Michael G. Pratt University of Illinois at Urbana-Champaign * Forthcoming in Organizational Research Methods (July, 2008)

2. In my experience qualitative and quantitative papers are evaluated in precisely the same way -- both need to follow cannons of good empiricism and logic.[Qualitative researcher on being asked about his/her criteria for evaluating qualitative research.] The problem is when it is not evaluated differently -- when one standard is used for all research. [Qualitative researcher on being asked what the difference is between how qualitative and quantitative research is evaluated].

3. Goals • Setting expectations • Background: Crux of the debate & pragmatic concerns • Study & results: examining criteria-in-use  • Author • Evaluator (reviewer or editor) • Explicit comparison: qualitative vs. quantitative • Three fundamental tensions in publishing qualitative research • Publishing tips • Managing publishing tensions • Increasing your chances for a revise-and-resubmit

4. Setting Expectations This talk will be unlike many other CARMA talks: • No equations • Not about data analysis, but about publishing research (see Cummings & Frost, 1985; Gephardt, 2004; Golden-Biddle & Locke, 1993, 2007; Lee, 1999; Suddaby, 2006)

5. Crux of the Debate • Should qualitative research be held to similar criteria as quantitative research? • If different, is it even possible to come up with a common set of criteria for evaluating all qualitative research?

6. Crux of the Debate Similar Criteria Different Criteria Paradigms are too different – “such a resolution appears highly unlikely and would probably be less than useful” (Lincoln & Guba, 2000: 185) Qualitative methods are too disparate for a common set • Research is research • One can approximate “quantitative” criteria (e.g., generalizability) or use analogous criteria (e.g., credibility vs. internal validity – Lincoln & Guba, 1985)

7. So What?: Pragmatic Concerns • Lack of consensus means it is difficult to publish • If you do not publish… • Publishing in top-tier North American journals are often critical for career success (note: criteria may differ by journal, discipline and by geography)

8. Study • Sample 1: Polled authors of published articles that were at least partially qualitative, that appeared between 1995-2005 in the following journals: • Academy of Management Journal • Administrative Science Quarterly • Strategic Management Journal • Organization Science

9. Sample 1 • Published Authors: 336 authors identified – 107 usable responses • 64 could not be reached (e.g., technical, leaves, or loss of life) • Of the 272 that could be reached, I received responses from111 (40.8%) • Of these 4 did not feel qualified

10. Sample 2 • Put out a call to RMNet to all authors who submitted an article during 1995-2005 to AMJ, ASQ, OS, or SMJ • Received 22 additional responses who filled out the same survey

11. Survey • On-line (Survey Monkey) • Forced choice: topics • study design (clear purpose, methodological completeness & appropriateness) • framing and contribution (rq’s, relationship to theory and contribution) • stance as a researcher (voice, authenticity, data presentation) * Note: I also included space to comment on the criteria

12. Survey • Open-ended: • For those who have had their qualitative papers accepted at top-tier journals, briefly describe the 3-5 reasons for your acceptance. (AUTHOR) • If you have been an editor or reviewer of a qualitative paper submitted to a top-tier North American journal, please list 3-5 criteria that you feel are the most important for evaluating qualitative research. (EVALUATOR) • In what ways, if any, do you feel that qualitative research is evaluated differently than quantitative research? (QUAL. VS. QUANT). • Other issues?

13. Analysis • Quantitative – counts & percentages • Qualitative • Frequency counts of themes (e.g. novelty) OR • Iterative, grounded-theory approach (Locke, 2001; Strauss & Corbin, 1998)

14. Findings • (3) Perspectives • Author • Evaluator (reviewer or editor) • Qualitative vs. Quantitative

15. Author Perspective: What Respondents View as Important • Analyzed forced-choice and open-ended. • 10% cut-off • Table 1: Qualitative responses for top 3-5 criteria:

16. Author Perspective • Table 2: Quantitative

17. Author Perspective • Convergence around: • Contribution to theory • Well-articulated & complete methods • Some overlap in terms of writing and clarity (e.g., articulating the purpose of article) • Differences in: • Qualitative: role of reviewers & editors (as champions) • Quantitative: showing vs. telling (Golden-Biddle & Locke, 2007)

18. Evaluator Perspective • Analyzed open-ended question about 3-5 most important criteria as editor or reviewer • Used 10% cut-off in deciding whether to report a theme

19. Evaluator Perspective

20. Evaluator Perspective • Again, convergence on most critical criteria: contribution to theory, transparent & well-articulated methods • Novelty becomes a little less important –and primarily linked with theory; explaining the appropriateness of methodology becomes more important • Showing-telling tension more nuanced: [T]he manuscript must strike a balance between data and interpretation - with the balance being a moving target depending on the topic and the richness of the data. Data are typically preferred to interpretation, but careful guidance in processing data is always appreciated. • Showing vs. telling linked to “good writing”

21. Quantitative vs. Qualitative Perspective • Some informants talked explicitly about how qualitative research differs from quantitative research • This perspective on the data revealed new insights

22. s Problems with Editor or Reviewers Inappropriate Higher (Qualitative) No “Boilerplate” Standards Criteria Barriers to Publishing (e.g., length of manuscript) Themes in Comparing Qualitative to Quantitative Studies

23. Qual. vs. Quant. Perspective: Themes • Overly high standards If I held “quanto” papers up to my standards, ASQ and AMJ would go out of business, since I do not believe that a study of process (and most are) that does not directly measure or observe that process is acceptable science. I do not hold these papers up to that standard, so they get published! Bottom line, quant work has luxury of lower standards in some ways.

24. Qual. vs. Quant. Perspective: Themes • Overly high standards I have never had a quantitative study rejected by a journal because I am a fairly good researcher and writer (to A, A- and B journals). On the other hand of the six times I have submitted qualitative work to the same level journals, I have had only 1 acceptance (the one that put me on your contact list).

25. Reasons for Overly High Standards • Lack of common reviewing standards (no “boilerplate”) I think that qualitative research does not have the commonly understood methodological shorthand that quantitative research does, so researchers have to explain and justify their methods much more (compared to writing something like "we conducted a multiple regression analysis"). Consequently, evaluations are more skeptical and looking for “proof” of adequate rigor..

26. Reasons for Overly High Standards • Problems with editors & reviewers The biggest issue is that qualitative papers are often reviewed by reviewers who are either not adequately trained in qualitative methods, or who may not regard qualitative research as being as valid as quantitative research. As such, qualitative research must meet a higher standard. Even boring quantitative research gets published; careful, yet uneventful qualitative work will get ditched. Qualitative researchers are usually very stringent when reviewing, which is fine, but makes the bar much higher; oddly enough, constructs in quantitative can be daft and unrelated to any sense of reality and it's okay as long as the regressions have stars, clearly not so in qualitative!

27. Qual. vs. Quant Perspective: Themes • Inappropriate (quantitative) standards In general, the positivist perspective is applied by most reviewers in evaluating qualitative research -- for example, complaints that research is not "scientific" and interpretations are "subjective.”

28. Reason for Inappropriate Standards • Lack of “boilerplate” In evaluating qualitative research, there is insufficient recognition of its variety in methodology and epistemology. So, review comments often assume or adopt one way of how the research should look that usually corresponds with the deductive quantitative model of doing and presenting research. More focus on generating possible alternative hypotheses for phenomenon observed and presented.

29. Reasons for Inappropriate Standards • Lack of adequately trained evaluators I believe most quantitative reviewers - which make most of the reviewers in top tier American journals - do not know enough about qualitative epistemology and methodology to actually appreciate qualitative work in its own terms. Thus, they use evaluations that better fit a quantitative approach.

30. Outcome of Overly High & Inappropriate Standards • Barriers to publishing – especially length The burden is higher to "prove" the results are valid. With quantitative research, people follow the numbers and fairly well-accepted protocols for analysis. In qualitative research, I have to walk the reader through every step of data collection and analysis and justify why I made each choice. The editor kept asking for more and more quantitative analyses, although the qualitative analysis was good and based on well known methodological sources. Qualitative manuscript is longer because of the many quotes, but the editor did not accept longer paper than the standard.

31. s Problems with Editor or Reviewers Inappropriate Higher (Qualitative) No “Boilerplate” Standards Criteria Barriers to Publishing (e.g., length of manuscript) Themes in Comparing Qualitative to Quantitative Studies

32. Key Tensions in Publishing Qualitative Research • Break from existing theory, while also being embedded in it. • Write well: showing vs. telling (Golden-Biddle & Locke, 2007) • Be expansive (detailed & transparent), but also adhere to journal formats

33. Oval Pegs in Round Holes • At an abstract level, the criteria for qualitative and quantitative research are the same (e.g., contribution, good writing, & clear and appropriate methods) • However, how you fulfill these abstract goals differs and creates specific frustrations (e.g., showing vs. telling) – which are compounded by differing criteria in different journals, disciplines & geographic areas

34. Publishing Tips • … for qualitative papers (but many apply to quantitative, too).

35. Managing Publishing Tensions • Make “Oval” Pegs Seem “Rounder” • Mimicking quantitative format: intro, methods, results (findings), discussion (G-B & L, 2007) or do “closet” qualitative research (Sutton, 1997) • Mimicking quantitative criteria (Kirk & Miller, 1986; LeCompte & Goetz, 1982; Lee, 1999; Lincoln & Guba, 1985 as cited in Marshall & Rossman, 1999; Miles & Huberman, 1994) • Generalize to theory, not sample (Yin, 2003) • Naturalistic generalization (e.g., sparrow). • Use mixed-methods

36. Managing Publishing Tensions 2. Make Round Holes Larger (More Oval Friendly) • Better trained qualitative evaluators at journals • Change in doctoral student education

37. Managing Publishing Tensions • Focus on the Process of Fitting • Create open theoretical frames -- reviewing and critiquing the literature in such a way that you are able to delineate what has been written -- and at the same time, create space for representing where research has largely been silent (Pratt, 2008: 18) • Examples: Barley (1986); Pratt, Rockmann & Kaufmann (2006)

38. Managing Publishing Tensions “Filling in” these theoretical frames speaks to more general issue of presenting theory vs. data • interweaving throughout (e.g., Golden-Biddle & Rao, 1997) • “data sandwich” (e.g., Gephart, 1978) • “open-faced data sandwich” (e.g., Pratt & Rosa, 2003) • first- vs. second-order findings (Gioia & Chittipeddi, 1991)

39. Managing Publishing Tensions b. Creating a hands-on exhibit: Showing & telling within page limits -- invite reader to interact with your data in a structured way • Translation trade-offs: Balancing “contribution-to-length” and “doing violence to experience”

40. Managing Publishing Tensions c. Create checklists • “boilerplates” may be both unwanted and not possible • “Best papers” in AMJ & ASQ had some commonalities on what they mentioned

41. Managing Publishing Tensions: Proposed Checklist • Why this study? (Elsbach & Kramer, 2003) • Why are qualitative methods appropriate? • Am I building, elaborating, or testing theory? • Why study here? • What is the nature of the context I am examining? • What was my rationale for choosing this context? (e.g., extreme case – Dutton & Dukerich, 1991 vs. more prototypical cases – Uzzi, 1997). • What am I studying and why? • Am I sampling events (Isabella, 1990), cases (Zbaracki, 1998), people (Rafaeli & Sutton, 1988) etc.? • What is my sampling strategy? • How did I study these things? • How did I analyze the data? • How did I link data with theory? (see Figure 2, Corley & Gioa, 2004)

42. Improving your Chances for a R & R • Proviso: most first drafts will have misaligned theory and data • Have several people read it through first (including quantitative researchers) • Establish yourself as knowledgeable – remember you are the analytical instrument. • Know the literature(s) • Give hook + rationale for why it is important to fill it! • Thoughtful & transparent methods (and better to cite methods sources than empirical examples) • # of pages is not that impressive

43. Improving your Chances for a R & R • Review the methods checklist – especially purpose and sampling criteria • Show your data (use power & proof quotes) • Tell a story … that even non-academics can understand • Know your outlet & its mission/ focus • Examine editorial board • Look at previously published articles • Know about page limits vs. length-to-contribution ratio • Don’t spend too much time “gaming” it

44. Closing Thoughts • No paper – even a sole authored paper – is a solitary exercise. Use your academic ties, the editor, and the reviewer to make the paper the best it can be. • Publication is a matter of preparation, persistence, & luck!

45. Any Questions?

46. Relevant Literature • Alvermann, D., O’Brien, D., Dillon, D. (1996). On writing qualitative research. Reading Research Quarterly, 31(1): 114-120). • Amber, A., Adler, P.A., Adler, P., & Detzner, D. (1995). Understanding and evaluating qualitative research, Journal of Marriage and the Family, 57 (November): 879-893. • Crestwell, J. (1998). Data analysis and representation. Qualitative inquiry and research design: Choosing among 5 traditions. Thousand Oaks, CA: Sage. • Daft, R. (1985). Why I recommend that your manuscript be rejected and what you can do about it. In L. Cummings and P. Frost (Eds.) Publishing in the organizational sciences (pp. 193 - 209). Homewood Il: Richard D. Irwin, Inc. • Denzin, N. & Y. Lincoln (Eds.) (2000). Handbook of Qualitative Research, 2nd Edition. Thousand Oaks, CA: Sage. • Eisenhardt, K. (1989). Building theories from case research. Academy of Management Review, 14, 532-550. • Fine, G. & Elsbach, K. (2000). Ethnography and experiment in social psychological theory building: Tactics for integrating qualitative field data with quantitative lab data. Journal of Experimental Social Psychology, 35, 51-76.

47. Relevant Literature • Gephardt, R. (2004). From the editors: Qualitative research and the Academy of Management Journal, Academy of Management Journal, 47(4): 454-462. • Glaser , B. G. and A. L. Strauss (1967). The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago: Aldine. • Golden-Biddle, K, & Locke, K. (1993). Appealing work: An investigation of how ethnographic texts convince. Organization Science, 4(4): 595-616. • Golden-Biddle, K. & Locke, K. (1997). Composing qualitative research. Thousand Oaks, CA: Sage. • Howe, K. & Eisenhardt, M. (1990). Standards for qualitative (and quantitative) research: a Prolegomenon. Educational Researcher, 19(4): 2-9. • King, N. (2000). Commentary – Making ourselves heard: The challenges facing advocates of qualitative research in work and organizational psychology. European Journal of Work and Organizational Psychology, 9(4): 589-596. • Lee, T. (1999). Using qualitative methods in organizational research. Thousand Oaks, CA: Sage. • Lee, T., Mitchell, T. & Sablynski, C. (1999). Qualitative research in organizational and vocational behavior. Journal of Vocational Behavior, 55: 161-187.

48. Relevant Literature • Lincoln, Y. (1995). Emerging criteria for quality in qualitative and interpretive research. Qualitative Inquiry, 1(3): 275-289. • Locke, K. (2001). Grounded theory in management research. Thousand Oaks, CA: Sage. • Marshall, C & Rossman, G. (1989). Designing qualitative research. Newbury Park, CA: Sage. • McGrath, J. (1982). Dilemmatics: The study of research choices and dilemmas. In McGrath, J. E., Martin, J., & Kulka, R.A. (Eds.), Judgment calls in research. Beverly Hills, CA: Sage. • Miles, M. (1979). Qualitative data as an attractive nuisance: The problem of analysis. Administrative Science Quarterly, 24: 590-601. • Miles, M. & Huberman, A.M. (1994). Qualitative data analysis, 2nd Edition. Thousand Oaks, CA: Sage. • Perrow, C. (1985). Journalling careers. In L. Cummings and P. Frost (Eds.) Publishing in the organizational sciences (pp. 220 - 230). Homewood IL: Richard D. Irwin, Inc. • Prasad, P. (forthcoming). Crafting qualitative research: Working in the postpositivist traditions. ME Sharpe.

49. Relevant Literature • Pratt, M.G. (2000). Some thoughts on publishing qualitative research.Research Methods Forum.http://www.aom.pace.edu/rmd/pratt_files/pratt.htm • Smith, J. & Deemer, D. (2000). The problem of criteria in the age of relativism. In N. Denzin & Y. Lincoln (Eds.) Handbook of Qualitative Research, 2nd Edition (pp. 877-896). Thousand Oaks, CA: Sage. • Spradley, J. (1979). The ethnographic interview. New York: Holt, Rinehart & Winston, Inc. • Stake, R. (2000). Case studies. In N. Denzin & Y. Lincoln (Eds.) Handbook of Qualitative Research, 2nd Edition (pp. 435-454). Thousand Oaks, CA: Sage. • Strauss, A. & Corbin, J. (1998). Basics of qualitative research, 2nd Edition. Thousand Oaks: Sage. • Suddaby, R. 2006. From the editors: What grounded theory is not. Academy of Management Journal, 49(4): 633-642. • Sutton, R. 1997. The virtues of closet qualitative research. Organization Science, 8(1): 97 – 106. • Van Maanen, J (1979). The fact of fiction in organizational ethnography. Administrative Science Quarterly, 24: 539-550. • Yin, R. (2003). Case study research 3rd Ed. Beverly Hills, CA: Sage Publications.

50. Qualitative Methods • Grounded theory • Case study • Ethnography • Phenomenology • Biography / Life history • Action research • And more….