SURVEYS IN TS SOME PRINCIPLES AND A CASE STUDY

1. SURVEYS IN TSSOME PRINCIPLES AND A CASE STUDY daniel.gile@yahoo.com www.cirinandgile.com Gile SURVEYS

2. WHY SURVEYS? IN EMPIRICAL RESEARCH “OBJECTIVE” DESCRIPTION AND MEASUREMENT CANNOT COVER EFFICIENTLY ALL RELEVANT PHENOMENA IN PARTICULAR: THOUGHTS, MOTIVATIONS, OPINIONS, INTENTIONS CANNOT BE OBSERVED OR MEASURED DIRECTLY INDIRECT EXPERIMENTAL PROCEDURES ON INDICATORS ARE COMPLEX AND SOMETIMES TRICKY ASKING PEOPLE IS ANOTHER OPTION Gile SURVEYS

3. CAUTION PEOPLE’S UTTERANCES ARE NOT ALWAYS TRUTHFUL OR ACCURATE THEY ARE NOT NECESSARILY EASY TO INTERPRET BUT THEY ARE OFTEN A USEFUL COMPONENT OF RESEARCH FOR: • PRELIMINARY EXPLORATION • COMPLEMENTARY COLLECTION OF INFORMATION • VALIDATION OF INFORMATION OBTAINED THROUGH OTHER METHODS RELEVANCE TO RESEARCH INTO TRANSLATION IS OBVIOUS BECAUSE OF: COMPLEX PROCESSING IS PART OF TRANSLATION, ONLY PRODUCT IS VISIBLE, CORPORA ARE DIFFICULT TO OBTAIN, INTERPRETATION OF INDICATORS IS DIFFICULT Gile SURVEYS

4. THERE ARE MANY TYPES OF SURVEYS LARGE SCALE/SMALL SCALE SPOKEN (INTERVIEWS) VS. QUESTIONNAIRES COLLECT OBJECTIVE INFO VS. SUBJECTIVE INFO FOR QUALITATIVE PURPOSES / SIMPLE QUANTITATIVE PURPOSES / ADVANCED QUANTITATIVE PROCESSING INDEPENDENT / PART OF A LARGE PROJECT WITH OTHER COMPONENTS Gile SURVEYS

5. (SOME) OBSTACLES TO COLLECTION OF TRUTHFUL, ACCURATE INFO IN SURVEYS LACK OF WILLINGNESS OF PEOPLE TO INVEST EFFORTS IN SURVEYS → LOW RESPONSE RATES, INCOMPLETE ANSWERS - RESPONSES VARY AS A FUNCTION OF SELF-IMAGE AND IMAGE SUPPOSEDLY GIVEN TO INTERVIEWER (SO LESS THAN TRUTHFUL RESPONSE POSSIBLE) - RESPONSES VARY AS A FUNCTION OF ULTERIOR INTERESTS - QUESTIONS (AND ANSWERS) MAY BE INTERPRETED DIFFERENTLY BY AUTHORS AND READERS (RESPONDENTS AND ANALYSTS) Gile SURVEYS

6. IMPORTANT STRATEGIES • MINIMIZE EFFORTS FOR RESPONDENTS (CLEAR QUESTIONS, EASY TO ANSWER, LENGTH OF QUESTIONNAIRES) • PILOT THE QUESTIONNAIRES TO MAKE SURE PEOPLE UNDERSTAND PILOTING IS ALSO IMPORTANT TO DETECT MISSING ITEMS, AMBIGUITY AND OTHER WEAKNESSES • TRY TO ELIMINATE OR REDUCE EFFECT OF SELF-IMAGE (QUESTIONING STRATEGIES, TRAINING OF INTERVIEWERS) IF NOT POSSIBLE, USE RESULTS CONSERVATIVELY Gile SURVEYS

7. SAMPLING AND REPRESENTATIVITY SIZE IS NOT THE ONLY CONCERN REPRESENTATIVITY IS PARAMOUNT “POPULATION” “SUB-POPULATION” “BIAS” (SYSTEMATIC ERROR – versus SAMPLING ERROR) AVOIDING BIAS THROUGH RANDOM SAMPLING MAY WANT STRATIFIED RANDOM SAMPLING (RANDOM SAMPLING WITHIN SUB-POPULATIONS THAT YOU WANT TO INCLUDE) Gile SURVEYS

8. SAMPLING ISSUES SMALL SAMPLE SIZE DOES NOT IMPLY BIAS THROUGH IT MAY NOT HAVE FEATURES CLOSE ENOUGH TO THOSE OF THE POPULATION DUE TO SAMPLING ERROR SAMPLING ERROR DECREASES WITH SAMPLE SIZE (BUT PROPORTIONALLY TO SQUARE ROOT OF SAMPLE SIZE – MUTLIPLY SAMPLE SIZE BY 4 NAND YOU WILL ONLY REDUCE SAMPLING ERROR BY 2) SO INCREASING SAMPLE SIZE IS ONLY EFFICIENT TO A MODERATE EXTENT ESPECIALLY AS PATTERNS STABILIZE Gile SURVEYS

9. CASE STUDY: PUBLISHING IN TS EST SEMINAR IN GHENT IN 2005 EDITORS ASKED TO TALK ABOUT: - CRITERIA FOR PUBLICATION - DIFFICULTIES IN PUBLISHING ACTIVITY • REFEREEING DURING SEMINAR: JOURNAL EDITORS DID NOT SEEM AWARE OF THE FACT THAT THEY ARE IN COMPETITION DID NOT MENTION PROBLEMS WITH REFEREEING SO SURVEY TO CHECK BOTH Gile SURVEYS

10. PURPOSE OF THE SURVEY • IDENTIFY FACTORS WHICH MAY BE RELEVANT/IMPORTANT (NOT QUANTIFY THEM) • DO SOME EXPLORATION IN PREPARATION FOR FURTHER STUDIES IF INTERESTING ISSUES COME UP SURVEY DESIGN TAKES THESE OBJECTIVES ON BOARD (LESS STRINGENT REQUIREMENTS THAN IF HYPOTHESES WERE TO BE TESTED OR IF INFLUENCES WERE TO BE QUANTIFIED) Gile SURVEYS

11. QUESTIONNAIRE OR INTERVIEWS?QUESTIONNAIRES - INTERNET WHAT QUESTIONS? ONE SET ON CRITERIA FOR SELECTION ONE SET ON REFEREEING PROBLEMS (Not standard procedure, but for efficiency and higher response rate) • THINK OF CERTAIN CRITERIA FOR SELECTION • BUT LET RESPONDENTS ADD • ON REFEREEING, ASK ABOUT POSITIVE EXPERIENCE AND NEGATIVE EXPERIENCE • ASK ABOUT REFEREE’S EXPERIENCE • PLUS QUESTIONS ON RESPONDENTS TO ANALYZE BY EXPERIENCE, COUNTRY, LANGUAGES IF POSSIBLE (MINIMAX) Gile SURVEYS

12. CLASSIFY RESPONDENTS AS BEGINNERS / EXPERIENCED / IN THE MIDDLE BECAUSE THIS MAY BE RELEVANT TENTATIVE DUAL CRITERION WITH ARBITRARY THRESHOLDS (TURNED OUT TO BE ALL RIGHT – PERHAPS BECAUSE DESIGNER OF SURVEY “KNEW” THE FIELD WELL ENOUGH) - How long have you been in the field of TS? - Less than 5 years - 5 to 10 years - More than 10 years • How many papers have you published so far in the field, either in journals or in collective volumes? - Less than 5 - 5 to 10 - More than 10 Gile SURVEYS

13. SELECTION CRITERIA WHEN CHOOSING JOURNALS • Reputation/Rating - ATTENTION this is subjective • Speed of publication • Choosing the language of publication Why these criteria? What scale? - Not at all important - Not very important - Quite important - I don’t know Why not numerical? Why not more? Gile SURVEYS

14. REFEREEING - How often have you felt that the referee’s comments were useful to you? - ATT - How often have you felt that referees misunderstood your statements? - ATT - If you have acted as a referee yourself, how often have you felt that you had to perform this function because nobody else with the right qualification was available or willing to do the job, but were not fully qualified because of language problems or because you were not familiar enough with the field, topic or research methods concerned? – ATT • Often/Sometimes / Rarely / I don’t know NOTE: • BOTH POSITIVE AND NEGATIVE Why? • AUTHOR’S AND REFEREE’S VIEWPOINT Why? Gile SURVEYS

15. PILOTING DISTRIBUTION MODE : INTERNET ADVANTAGES AND DISADVANTAGES? SENT TO COLLEAGUES EXPERIENCED IN SURVEYS FOR INPUT (CHIARO AND NOCELLA) Why? SOME CHANGES ON BASIS OF THEIR COMMENTS AND SUGGESTIONS - PILOTING 1: SENT TO ABOUT 15 SCHOLARS (AFTERWARDS, THEY ARE NO LONGER AVAILABLE AS RESPONDENTS) - PILOTING 2: INITIAL PRODUCTION: FIRST BATCH – POSSIBILITY OF FINDING FURTHER PROBLEMS AND CORRECTING - THEN FINAL “PRODUCTION” Gile SURVEYS

16. PREPARATION OF DATA FOR ANALYSIS • EXCEL FOR SIMPLICITY, PORTABILITY (DATA BASE MORE POWERFUL, BUT MORE DIFFICULT TO EXPORT TO OTHER COLLEAGUES/SYSTEMS) • “NAMES”, MEANINGFUL (FOR EASIER READING AND INTERPRETATION THAN CODES) • W/ DIFFERENT INITIALS (FOR AUTOMATIC COPYING BY EXCEL – SAVES TIME) • COLUMN FOR FURTHER COMMENTS • IDENTIFIERS (NOT NAMES OF RESPONDENTS, UNLESS EXPLICIT PERMISSION) • INITIAL INFO ON RESPONDENTS TO HELP FIND OUT ABOUT INFLUENCE OF POTENTIALLY RELEVANT FACTORS (COUNTRY, EXPERIENCE) Gile SURVEYS

17. Gile SURVEYS

18. COUNTRIESn = 85 Gile SURVEYS

19. WHY RELEVANT AND WHAT CAN WE SAY ABOUT THIS SAMPLE? • FROM ALL OVER THE WORLD • BUT STRONG WEST EUROPEAN BIAS • POSSIBILITY OF CHECKING BY REGIONS • POSSIBILITY OF CHECKING BY COUNTRIES FOR SOME FACTORS (IF HYPOTHESES ON LANGUAGE, REFEREEING W/O FULL QUALIFICATIONS) Gile SURVEYS

20. Gile SURVEYS

21. INTERPRETING THE RESULTS CRITERIA FOR CHOOSING JOURNAL • REPUTATION: Q 88% CLEARLY IMPORTANT • SPEED: Q 60% - NV 37% OFTEN IMPORTANT – When? • LANGUAGE: Q 45% - NV 38% SOMETIMES NOT CLEAR BY COUNTRY IN SPITE OF HYPOTHESIS – FURTHER EXPLORATION POSSIBLE REFEREEING • USEFUL: O 50% - S 40% - R 5% POSITIVE ON THE WHOLE – OVERASSESSED BECAUSE POLITICALLY CORRECT? • MISUNDERSTANDINGS: O: 4% - S: 42% - R: 49% OK ON THE WHOLE • I NOT COMPETENT: O: 3% - S: 37% • NOT INSIGNIFICANT – UNDERASSESSED? – NOT CLEAR BY COUNTRY DESPITE SPECIFIC SITUATIONS – FURTHER EXPLORATION Gile SURVEYS

22. SOME LIMITATIONS AND SHORTCOMINGS OF THE STUDY • NOT RANDOM SAMPLE, DIFFICULT TO ASSESS BIAS - QUESTIONNAIRE SENT IN ENGLISH ONLY POSSIBLE LANGUAGE BIAS, PARTICULARLY IMPORTANT FOR QUESTION ON RELEVANCE ON LANGUAGE - IN LAST QUESTION, POSSIBLE AMBIGUITY BETWEEN NON APPLICABLE AND “NEVER”. SHOULD HAVE ADDED “NON APPLICABLE” CATEGORY Gile SURVEYS

23. WAS THE EXERCISE USEFUL? • QUANTITATIVE RESULTS CONFIRM INTUITIVE KNOWLEDGE, • GIVE GENERAL ORDER OF MAGNITUDE • NO BIG SURPRISES • PERHAPS USEFUL TO KNOW THAT MORE THAN A THIRD OF THE RESPONDENTS SAY THAT SOMETIMES THEY ARE NOT FULLY QUALIFIED • “QUALITATIVE” RESULT: MANY RESPONDENTS WANT TO INTERACT WITH REFEREES – PERHAPS USEFUL FOR EDITORS’ POLICY DECISIONS Gile SURVEYS