Proposal Review Board Member Training

1. Proposal Review BoardMember Training 2012 HGSE Student Research Conference

2. Welcome • Thank you for coming!!! • Grab some refreshments • Sign the attendance sheet that is going around • Look over the materials in your packet. You should have: • Proposal submission guidelines • 2 scored sample proposals for training (1 empirical & 1 non-empirical) • 2 proposals to score with blank rubrics (1 empirical, 1 non-empirical) • Sample comments for rubric

3. Roadmap First 20 minutes: • SRC History & Goals • PRB Responsibilities & Review Process • PRB Timeline • PRB Logistics Next hour: • SRC Submission Guidelines & Scoring Rubric • Scoring Examples

4. SRC History • Started 17 years ago by Susan Moore-Johnson and Kitty Bowles • Purpose to provide students with an opportunity to enter the research world in a non-threatening environment • Open to students from Harvard and beyond • Great experience for HGSE students in presenting and conference leadership

5. The Review Process • Each proposal will be blind-reviewed by two PRB members using the rubrics we will present today. • If the two reviews are discrepant (one recommends acceptance and the other recommends rejection), a third reviewer will be assigned to review the proposal to break the “tie.” • In late February, the PRB Chair will notify those who submitted proposals whether their proposal is accepted or rejected. • This year, we are not providing an opportunity for revision and re-submission • We may, however, make a recommendation regarding the presentation format (panel, roundtable, or poster)

6. PRB Member Responsibilities • Review proposals throughout the month of February. The exact number depends on the number of proposals received. • Review a few additional proposals in February if need arises (for instance to break a tie).

7. PRB Members & Submission • We encourage PRB members to also submit their own proposals to the SRC • We will not assign you to review your own proposal • Our system may be imperfect: • If you get a proposal and you can tell who wrote it for whatever reason (it’s your own or you helped someone revise their proposal), please let us know by email ASAP

8. Timeline • Now: Proposal submission open & ongoing • Jan. 27th: Deadline for applicants to submit proposals • Jan. 30th: Proposals go out to reviewers • Feb 13th: Reviews due back • Feb 16th – Feb 20th: Tiebreaker review • Feb 20th: Tiebreaker decisions due back • Feb. 24th: Decision emails sent to applicants • April 6th: 2012 HGSE Student Research Conference

9. PRB Logistics • STEP 1: You will receive an email with a list of the proposals you are responsible for reviewing, which will be designated by a four-digit code. • STEP 2: You will then download your allocated proposals from the isiteon a password protected tab called “Proposal Review Board Members.” • STEP 3: You will submit comments via an online poll similar to the one you filled out to RSVP to this training.

10. The Rubric • Complete one poll submission for each proposal you review • You don’t need to review them all in one sitting • You have a copy of the rubric in your packet

11. Submission Guidelines • Empirical v. non-empirical • Panel, roundtable & poster • Recommended outline

12. Empirical v. Non-empirical Empirical study • collect & analyze data • research question • qualitative or quantitative methods used to collect and analyze data • findings or hypothesized findings and conclusion based on analysis of data Non-empirical paper • analytic papers, theoretical arguments, policy briefs and literature reviews • research problem • an argument based more on logic and consideration of other perspectives and counter arguments • conclusion based on thoughtful analysis

13. Presentation Formats Most formal • Panel • 3-4 presenters with a common theme • 15 minute presentation by each panelist followed by a question and feedback period • HGSE faculty member or doctoral moderator • Audience • Paper complete or in progress (at least preliminary results must be ready to present by April 6th), especially QPs, dissertation chapters, term projects, etc. • Roundtable • Small group of students with common theme • 5-10 minute presentation by each participant followed by a question and feedback period • Especially suited to research projects in development • HGSE faculty member or doctoral moderator • Smaller audience (mostly participants) • Poster • Presenters prepare a tri-fold poster for display • All posters displayed simultaneously in a conference space – presenter stands by poster • Poster session open to all conference attendees - succinctly describe work with students, faculty, and visitors who may or may not be familiar with the presentation topic Least formal

14. Recommended Outline 1. RESEARCH PROBLEM: A.    Empirical study: State at least one research question B.    Non-empirical paper: State the research question, problem, or position.    2. CONCEPTUAL BACKGROUND: Explain the conceptual background for your research. Address why your research is important and include citations of published references.  3. METHODS/APPROACH: A.    Empirical study: Describe your research design and methods, including the plan for collecting and analyzing data to answer your research question B.    Non-empirical paper: Describe the argument, including how it will consider different perspectives and counter-arguments.  4. FINDINGS/CONCLUSIONS: State appropriate conclusions, tentative findings, or predictions.  5. IMPLICATIONS: Discuss the significance of your work for further educational research, theory, policy, or practice.

15. The Rubric • This training is meant to familiarize you with the rubric for proposal evaluation, and to involve you in the process of increasing our reliability in using it. • We’ll go through each question on the rubric • We’ll go through an example of each element • We’ll then score proposals together to check our inter-rater reliability.

16. Overall Scoring Guidelines • We want to be as inclusive as possible • We want the conference to be well-attended • This is not a professional conference, but rather a studentresearch conference • Rejection should be very rare – most likely you will not reject anyone (more on rejection later) • Goal is consistency across reviewers • Alignment w/ PRB guidelines • Focus is substance rather than form

17. Quantitative Scoring Criteria Evaluation Scale 3: Element meets criteria. It is fully articulated, convincing, and logical. 2: Element generally meets criteria. May have some small weaknesses. 1: Element may only be alluded to, or have serious weaknesses. 0: Element missing. Majority 1 or lower – can be considered for rejection. Anything higher, recommend acceptance. When in doubt…round up!

18. Qualitative Comments • Applicants will not see their quantitative scores (we use those internally), but they WILL receive these comments back, so please take care • Reviewer’s name will not be revealed to applicant • Goal = learning • If recommending acceptance: strengthening presentation • If recommending rejection: improving proposal for future submission • Suggested comments sheet in your packet – pull it out • Balance positive and constructive comments • Length of comments: • Length will vary based on quality of proposal • Up to you where to make your comments (for individual elements, overall, or both) • If you’re giving a 0 or 1 on a particular element, give a comment on that element • All comments should add up to at least a short paragraph

19. Rejection • Before you reject, think about: • Could comments help strengthen the presentation • Alternative formats: • Roundtable (For Panel Requests) • If you do recommend rejection: • Keep in mind the applicant will receive your comments • Please provide a clear rationale in your comments • Don’t be negative – please find something encouraging to say • Reach out to us if you would like help with phrasing etc.

20. Scoring: An Empirical Proposal

21. Empirical Proposal Ex. How does accountability impact school climate? A qualitative study of two urban middle schools -Let’s go element by element - although the proposals themselves will not explicitly separate the elements (500 Word Max Proposal Format)

22. Empirical study Conceptual backgroundDoes the proposal make connections to existing educational research, addressing why this research is important, and include citations? 3: According to the National Forum to Accelerate Middle-Grades Reform (2006), effective middle schools are academically rigorous, developmentally appropriate, and socially equitable. Comer's (2004) holistic view of child development supports the idea that schools need to balance academic rigor with a socially and emotionally nurturing environment. However, some education reformers claim that increased pressures to meet academic targets under No Child Left Behind, have caused many schools to focus on improving test scores at the expense of affective goals, negatively impacting students (Brigham et al, 2004; Lee, 2007). 2: According to the National Forum to Accelerate Middle-Grades Reform, effective middle schools are academically rigorous, developmentally appropriate, and socially equitable. Comer's holistic view of child development supports the idea that schools need to balance academic rigor with a socially and emotionally nurturing environment. However, some education reformers claim that increased pressures to meet the demands of educational accountability, such as those established by No Child Left Behind, have caused schools to focus too narrowly on teaching to the test. 1: According to the research literature that I have read, effective schools are academically rigorous, developmentally appropriate, and socially equitable. The holistic view of child development supports the idea that schools need to balance academic rigor with a socially and emotionally nurturing environment so that we can raise students who will be good citizens and there will be less bullying in schools. I do not think it is taking tests or having teachers teach to the test or pressuring young people to meet the demands of educational accountability which have been on the rise in America for many years.

23. Empirical study Research questionDoes the proposal contain a clear research question? 3: In this mixed-methods pilot study I investigate how, if at all, the climate of two Boston urban middle schools is changing as a result of actions their school administrators are taking with the express purpose of achieving Annual Yearly Progress (AYP). I define “climate” in a broad sense, taking it to refer to characteristics associated with effective schools (Chenowith, 2009), including teacher and student perceptions of their school, the quality of interactions taking place within it, and the commitment of students to their school and learning. 2: This study examines the effect of current educational accountability reform, --i.e., No Child Left Behind—on the climate of urban middle schools. This study seeks to determine how the educational climate of two urban middle schools with similar demographics and size may be changing as a result of educational accountability. It also seeks to understand the decisions their school administrators may be making about diverse aspects of school administration that may have an effect on the climate of these schools, and how school administrators and teachers reflect on school climate, accountability, and how these two factors may be related. 1: This study looks at what is happening in American schools today. My study is seeking to determine how the educational climate of urban middle schools may be changing. It also seeks to understand school administrators’ decision making and school climate.

24. Empirical study MethodsIs there a description of research methods that relate to the research question? 3: The sample consists of two Boston-area urban middle schools of similar size and student populations. One school meets AYP; the other has been labeled a “School In Need of Improvement.” I will evaluate changes in school climate by examining trends in the results of validated climate surveys (Shann, 1990; 1998) administered to approximately 90% of students at both schools over the past three consecutive years. I will conduct semi-structured interviews with teacher leaders (n=20), administrators (n=10), and school personnel (n=10) to assess their perceptions of how and why at the school climate has altered in recent years. I will also analyze school records related to attendance, behavior, budgets, professional development, and schedules to gain additional insights into the potential relationship between accountability and school climate at these schools and to triangulate my findings from the surveys and interviews (Wilkins, 2008). 2: The sample consists of two urban middle schools of similar demographics, but different accountability status. Several measures are used to document the climate of the schools. These measures include school climate surveys, interviews with some teachers, administrators at the two schools and some other school personnel, and analysis of school records. Surveys will be administered to measure student and teacher perceptions climate and factors they believe contribute to climate ir school, including the quality of relationships among school participants, administrative leadership, teacher autonomy, and student commitment to the school and learning. Interviews with teachers, administrators, and available school personnel will provide additional details of the climate and will also focus on how the climate may have changed. School records related to attendance, behavior, budgets, professional development, and schedules, as well as general observations of school activities, will also be analyzed to provide additional insight into accountability and school climate. 1: The sample consists of urban middle schools of similar demographics, but different accountability status. It is important that schools have similar demographics or our comparison will not be valid. Random sampling is also important. Several measures will be used to document the climate of the schools such as surveys, interviews, focus groups, and school records. We have already obtained permission slips from all participants. We will also be checking for threats to validity as well as internal consistency in our interviews.

25. Empirical study Findings Does the proposal indicate findings, predictions, or conclusions? 3: I hypothesize that a greater emphasis on accountability is associated with some deterioration in school climate, as indicated by increases in student absenteeism among under-performing students and students generally commenting unfavorably about the school curriculum in their climate surveys. However, I believe that in some ways increased accountability may be associated with improvements in school climate, such as students appreciating their teachers’ efforts to help them pass their tests, particularly at the school that has met AYP. 2: I anticipate finding that a greater emphasis on accountability is associated with mainly negative changes in school climate. For example, students will say that they find school more boring and there will be an increase in absenteeism. 1: I anticipate finding, based on our study of two schools, that schools that have a greater emphasis on accountability always have a bad school climate.

26. Empirical study Implications:Does the proposal include implications for emerging educational research, theory, policy and/or practice? 3: Although my results will be tentative and based on a small sample size, my study could shed light on the ways in which accountability measures affect the climate for better or worse within middle schools. Furthermore, my results could indicate important areas for further research. For example, is climate affected differently in schools deemed to be failing rather than succeeding? If students and/or teachers express dissatisfaction with specific actions taken by school authorities to improve test scores, can alternative interventions be designed that are less detrimental to school climate? 2: The results of this study have implications for immediate actions that school leaders can take to improve school climate, while bolstering student academic outcomes. The results may also have implications for further research to be conducted by these and other researchers. Furthermore, I expect the findings to make an important contribution to theory about how schools work and how changes should be based on influences of the larger socio-cultural context. Administrators need to know what effect they are having on young people’s lives and whether students are coping with the stresses and strains of having to take high-stakes tests that will affect the Adequate Yearly Progress of their schools. 1: The results of this study have many important implications for research, theory, and practice which will be explored. We will also present specific ideas for immediate changes in school policy which should be advocated based on our expected findings.

27. Non-empirical Ex. Language of Instruction Policy in Madagascar

28. Major Differences • Instead of a research question, non-empirical papers may have a research problem or a position statement. • Instead of methods, non-empirical papers may have an approach that includes a theoretical argument, with alternative perspectives or counter-arguments • Instead of findings or hypothesized findings non-empirical papers will have a conclusion.

29. Non-empirical study Conceptual Background:Does the proposal make connections to existing educational research, addressing why this research is important, and include citations? Research shows that children learn to read best when taught in the language they speak at home (International Reading Association, 2001; UNESCO, 2006). However, students in Malagasy schools who are taught in their mother tongues tend to score lower on the Progress in International Reading Literacy Study (PIRLS) exam than their counterparts in schools which use French and sometimes English, as languages of instruction (Boomer et al, 2007). What rating would you give and why? 3! • Existing research • Connected to RQ • Citations!

30. Non-empirical study Research Question:Does the proposal contain a clear research question, problem, or position statement? Education in the mother tongue should not be of lesser quality than education in so-called prestige languages, such as English. What causes this to be the case? How can we improve the reading of Malagasy students? It is necessary to look more deeply at language of instruction policies. It is necessary both to provide culturally relevant and high quality educations in order to achieve international educational targets. What rating would YOU give and why? How would you phrase your comment(s)? The author IS presenting a question, but could more clearly state what she means by “this” • Compelling problem is presented, but would be stronger if “we” were defined: • Teachers? • Academics? • Policymakers? 2!

31. Non-empirical study Methods/Approach:Is there a description of the argument, including how it will consider different perspectives and counter-arguments? One method is to look at language instruction policy from an international perspective. Here I will examine language policies in countries near to Madagascar. What rating would you give and why? How might you phrase your comment(s)? How? Which countries? 1 • No clear argument presented • Counterarguments could be inferred, but they are not explicitly confronted

32. Non-empirical study Findings/Conclusions:Does the proposal indicate findings, predictions, or conclusions? I then propose a reform of the Malagasy language policy as it is my belief that it is not the mother tongue itself, but poor teaching practice, that causes Madagascar’s low test scores. What rating would you give and why? How might you phrase your comment(s)? Author is stating a belief rather than a finding What kind of reform? 1

33. Non-empirical study Implications:Does the proposal include implications for emerging educational research, theory, policy and/or practice? • Vague • What kind of countries? • What kind of policy change? • What is the importance of language instruction policy? The results of this study have many important implications for research, theory, and practice in international realms. Adoption of a language of instruction policy change in Madagascar will serve as an example for other countries. What rating would you give and why? 1 Studies like this one are vitally important because a successful policy in one context may serve to inform decision-makers facing similar problems elsewhere. In fact, very few system-wide educational decisions carry the same importance or are as difficult as the choice of a language of instruction. Choosing one language over another to educate a society’s youth is a decision that carries substantial, political, social, and economic weight. This example could help to serve as a prototype for policies that protect the increasing number of economic migrants, who can be considered bereft of many of their rights since they fall between belonging in at least two nations, just like many of the marginalized ethnolinguistic groups of Madagascar. 2!

34. Putting it all together Now let’s score an empirical study: “Fact or Fiction?”: Children’s Ontological Beliefs in Historical Figures

35. Putting it all together Scoring a non-empirical paper

36. Questions/Concerns • SRC Email: src2012@gse.harvard.edu • Eve Ewing, SRC Training Chair: eve_ewing@mail.harvard.edu

37. EXAMPLE NON-EMPIRICAL PROPOSAL Format Submitted: Roundtable The purpose of this paper is to address the application of Hostetler’s framework of educational well-being to the globalization phenomenon occurring at a rapid pace in higher education. Globalization differs from internationalization in that it is marked by fluid borders, the concept of increased interconnections between nation states, and the idea that there is a global knowledge that is achievable through shared research and development. As the globalization trend has grown, a distinct lack of clarity of aims and intentions has presented itself. There is a lack of definition as to not only what makes an institution global, but also whether the institution is successful in its mission in the host country. While there is an imperative for globalization in higher education due to the framework of the global economy and mandate for human rights throughout the world, the outcomes of such efforts have proved themselves to be mixed for the hosting countries. Some African, South American, and developing Asian countries have been susceptible to heavy influence from Western interests (e.g., international institutions and corporations) that have the competitive advantage in setting the agenda for the institutions established. The paper discusses the effects of the transformation from the internationalization movement to one of globalization, and then goes on to address many outcomes of this development over the last two decades. Specific examples focused on are the UK’s movement into the Middle East, Australia’s institutes in the Pacific Rim, and American developments in India. While there have been attempts at regulation through standards provided by local governments (e.g., the UK’s QAA and the EU’s Lisbon Strategy) as well as ranking systems (The Times Higher Education Supplement Rankings of Universities and Shanghai Jiao Tong University), efforts at regulation by global organizations, such at The World Bank, UNESCO, and the Organization for Economic Co-operation and Development have proven themselves to be largely ineffective. The World Trade Organization’s framework for governance in the General Agreement on Trade in Services provides for economic equality but not for the well-being of the institution and its participants in the setting of the host country. Thus the authors of the paper propose to use Hostetler’s framework of educational well-being (2005) to examine the current state of globalization as well as propose a series of ethical guidelines to ensure that the interests of all nations and cultures involved are continuously considered as globalization transforms higher education.