BOBBY ALLEN, PAM LACH, STEPHANIE BARNWELL 2012 October 2nd

1. Valuing & Evaluating Digital Humanities Practice BOBBY ALLEN, PAM LACH, STEPHANIE BARNWELL 2012 October 2nd www.digitalhumanities.unc.edu

2. contexts of DIGITAL HUMANITIES practice The practice and the profession of history are undergoing changes quite unlike any we have experienced before.  I increasingly believe that the digital revolution is yielding transformations so profound that their nearest parallel is to Gutenberg’s invention of moveable type more than half a millennium ago. William Cronon | President, American Historical Association In “The Public Practice of History in and for a Digital Age” | January 2012 Digital scholarship is the inevitable future of the humanities and social sciences. [D]igital literacy is a matter of national competitiveness and a mission that needs to be embraced by universities, libraries, museums, and archives. . . .  A robust cyberinfrastructure should include centers that support collaborative work with specialized methods. . . .  Public funds should be at the forefront of support to such national centers of excellence in digital humanities and social science, as crucial seedbeds of further innovation. Our Cultural Commonwealth| 2006Report of the Commission on Cyberinfrastructure in the Humanities and Social Sciences American Council of Learned Societies

3. contexts of DIGITAL HUMANITIES practice I believe we must finally speak aloud the truth: that some arbitrary volume of published papers, on some narrowly defined points of debate, is not necessarily more worthy than other activities. . . . The central point is that the time is right, at this moment, to focus intently on the quality of your work and its impact on our students, our disciplines, and our communities. Quality and impact. Those must be the two central considerations in our reward system. And I am urging the provost and the deans to attend particularly to those two criteria in promotion and tenure decisions. Gordon Gee| President, Ohio State University In “Right Here, Right Now: A Crystallization of Purpose” | 7 October 2009

4. contexts of DIGITAL HUMANITIES practice Digital tools are transforming the practice of history, yet junior scholars and graduate students are facing obstacles and risks to their professional advancement in using methods unrecognized as rigorous scholarly work. Their peers and evaluators are often unable or unwilling to address the scholarship on its merits. Opportunities to publish digital work, or to even have it reviewed are limited. Finally, promotion and tenure processes are largely built around 19th-century notions of historical scholarship that do not recognize or appropriately value much of this work. The disconnect between traditional evaluation and training and new digital methods means young scholars take on greater risks when dividing their limited time and attention on new methods that ultimately may not ever face scholarly evaluation on par with traditional scholarly production. A Call to Redefine Historical Scholarship in the Digital TurnOpen Letter to AHA Research Division | 26 January 2012

5. DIGITAL Humanities

6. explore more resources: http://digitalhumanities.unc.edu/resources/

7. a few DH PROJECTS from UNC

8. contexts of DIGITAL HUMANITIES practice at UNC the Carolina Digital Humanities Initiative is… afive-year coordinated intervention in humanities research, graduate training, undergraduate teaching, and public engagement that will produce an adaptable and sustainable model of transformative academic practice in the humanities, change the role and reach of humanities at UNC, and serve as a model for innovation in academic practice in the humanities for other research universities

9. how will the CDHI work? cluster faculty hires Mellon Graduate Fellowships in Digital Humanities Mellon Postdoctoral Fellowships in Digital Humanities DIL/IAH Faculty Fellowships graduate certificate program in Digital Humanities cyberinfrastructure taskforce data studies curriculum

10. how will the CDHI work?

11. five-year deliverables of the CDHI 3 tenure-track digital humanists 12 Mellon Graduate Fellowships 4 Mellon Postdoctoral Fellowships 8 DIL/IAH Faculty Fellowships and projects graduate certificate program coordinated DH curriculum with Duke and NCSU DH skills/tools training program cyberinfrastructure best practices policies implemented and evaluated data studies curriculum 19 new DH courses developed, offered, and assessed

12. what could DH PRACTICE transform? how we ask research questions how and what we teach how students learn how we engage with the public how we train graduate students how humanists and scientists work together

13. what is distinctive about DH PRACTICE? social, collaborative interdisciplinary, trans-domain dynamic, iterative, open-ended use of sources, i.e. big data cuts across traditional distinctions: research, teaching & service can leverage engaged scholarship, effective use of digital technologies, and interdisciplinary collaborations

14. what is distinctive about DH PRACTICE? cost: from fellowship to grant funding technology & technical know-how external funding model not scalable requires access to technologies and the skills to use them experimental (risk of failure) Critical need to reduce barriers to entry and create sustainable business models!

15. VALUING digital humanities practice Our Cultural CommonwealthReport of the Commission on Cyberinfrastructure in the Humanities and Social Sciences American Council of Learned Societies (2006) The Commission believes that digital scholarship is the inevitable future of the humanities and social sciences, and that digital literacy is a matter of national competitiveness and a mission that needs to be embraced by universities, libraries, museums, and archives. In order to foster digital research, teaching, and publishing, we recommend specifically that there be: • policies for tenure and promotion that recognize and reward digital scholarship and scholarly communication; recognition should be given not only to scholarship that uses the humanities and social science cyberinfrastructure but also to scholarship that contributes to its design, construction, and growth.

16. VALUING digital humanities practice We might expect younger colleagues to use new technologies with greater fluency and ease, but with tenure at stake, they will also be more risk-averse. There is a widely shared perception that academic departments in the humanities and social sciences do not adequately reward innovative work in digital form. A handful of recent examples provide exceptions to the norm, but in the most elite universities, traditional scholarly work, in the form of a single-authored, printed book or article published by a university press or scholarly society, is the currency of tenure and promotion; work online or in new media—especially work involving collaboration—is not encouraged. Senior scholars now have both the opportunity and the responsibility to take certain risks, first among which is to condone risk taking in their junior colleagues and their graduate students, making sure that such endeavors are appropriately rewarded. Our Cultural CommonwealthReport of the Commission on Cyberinfrastructure in the Humanities and Social Sciences American Council of Learned Societies (2006)

17. SUPPORTING digital humanities practice • articulate clear expectations upon hire • reformulate and tweak these expectations, each year, to set benchmarks for achievement and for assessment • at mid-term review, collaboratively identify specific benchmarks, and the mode of evaluation that will be used to determine successful completion -- this is a portion of a contract for tenure, in effect • identify potential referees early on and revise as necessary • identify outside resources and potential collaborators at the same time (granting agencies, donors, legitimating agencies like NINES, potential collaborators) • identify audiences and expected impact of the research: what is it, who is it for and why is it significant? • document effort • use informal avenues to communicate with colleagues in a collegial manner so all faculty feel an interest in and responsibility for success Digital Humanities Scholarship: Recommendations for Chairs in Language and Literature Departments | Alison Booth, Pamela K. Gilbert, Steve Olsen, Brad Pasanek and the NINES/NEH Summer Institute Group, 2011

18. FORMS ofdigital humanities practice • websites • online peer-reviewed publication • scholarly electronic editions • research tools • engaged digital scholarship(partnerships with cultural heritage/community orgs) • blogs • standards and specifications • instructional technology • digital art/new media production • AND. . . .

19. EVALUATING digital humanities practice

20. EVALUATING digital humanities practice 2001

21. EVALUATING digital humanities practice 1. Fundamentals for Initial Review: The work must be evaluated in the medium in which it was produced and published 2. Crediting: Digital projects are often collaborative in nature, involving teams of scholars who work together in different venues over various periods of time. Authors of digital works should provide a clear articulation of the role or roles that they have played in the genesis, development, and execution of the digital project. 3. Intellectual Rigor: Digital projects vary tremendously and may not "look" like traditional academic scholarship; at the same time, scholarly rigor must be assessed by examining how the work contributes to and advances the state of knowledge of a given field or fields 4. Crossing Research, Teaching, and Service: Digital projects almost always have multiple applications and uses that enhance—at the same time—research, teaching, and service. 5. Peer Review: Digital projects should be peer reviewed by scholars in fields who are able to assess the project's contribution to knowledge and situate it within the relevant intellectual landscape 6. Impact: Digital projects can have an impact on numerous fields in the academy as well as across institutions and even the general public. They often cross the divide between research, teaching, and service in innovative ways that should be remarked Presner, et al, “How to Evaluate Digital Scholarship”

22. EVALUATING digital humanities practice 7. Approximating Equivalencies: Is a digital research project "equivalent" to a book published by a university press, an edited volume, a research article, or something else? These sorts of questions are often misguided since they are predicated on comparing fundamentally different knowledge artifacts and, perhaps more problematically, consider print publications as the norm and benchmark from which to measure all other work. Reviewers should be able to assess the significance of the digital work based on a number of factors: the quality and quantity of the research that contributed to the project; the length of time spent and the kind of intellectual investment of the creators and contributors; the range, depth, and forms of the content types and the ways in which this content is presented; and the nature of the authorship and publication process. 8. Development Cycles, Sustainability, and Ethics: It is important that review committees recognize the iterative nature of digital projects, which may entail multiple reviews over several review cycles, as projects grow, change, and mature. 9. Experimentation and Risk-Taking: Digital projects in the Humanities, Social Sciences, and Arts share with experimental practices in the Sciences a willingness to be open about iteration and negative results. As such, experimentation and trial-and-error are inherent parts of digital research and must be recognized to carry risk. Presner, et al, “How to Evaluate Digital Scholarship”

23. EVALUATING digital humanities practice Lee, Valerie and Cynthia L. Selfe, Cynthia L. "Our Capacious Caper: Exposing Print-Culture Bias in Departmental Tenure Documents." ADE Bulletin 145 (Spring 2008): 51-58

24. EVALUATING digital humanities practice

25. EVALUATING digital humanities practice Digital technologies do more than propose new ways of thinking, as did theory; they require new modes of being. To put this in less dramatic terms, the digital revolution requires us as a profession to make conscious the motivations and values inhering in material practices, from putting a manuscript in the mail to a publisher to requiring for tenure “a book between covers.” We must transfer the values informing these activities and practices onto new modes of activity, so that we understand, value, and evaluate theoretical decisions about database modeling, algorithms, and information flows to best support new research and reading practices. “Evaluating Digital Scholarship, Introduction” Susan Schreibman, Laura Mandell, and Stephen Olsen

26. EVALUATING digital humanities practice

27. EVALUATING (engaged) digital humanities practice • Communication/Dissemination • Does the scholar use a suitable styles and effective organization to present the work? • Does the scholar communicate/disseminate to appropriate academic and public audiences consistent with the mission of the institution? • Does the scholar use appropriate forums for communicating work to the intended audience? • Does the scholar present information with clarity and integrity? • Reflective Critique • Does the scholar critically evaluate the work? • What are the sources of evidence informing the critique? • Does the scholar bring an appropriate breadth of evidence to the critique? • In what way has the community perspective informed the critique? • Does the scholar use evaluation to learn from the work and to direct future work? • Is the scholar involved in a local, state and national dialogue related to the work? • Goals/Questions • Does the scholar state the basic purpose of the work and its value for public good? • Is there an “academic fit” with the scholar’s role, departmental and university mission? • Does the scholar define objectives that are realistic and achievable? • Does the scholar identify intellectual and significant questions in the discipline and in the community? • Context of theory, literature, “best practices” • Does the scholar show an understanding of relevant existing scholarship? • Does the scholar bring the necessary skills to the collaboration? • Does the scholar make significant contributions to the work? • Is the work intellectually compelling? • Methods • Does the scholar use methods appropriate to the goals, questions and context of the work? • Does the scholar describe rationale for election of methods in relation to context and issue? • Does the scholar apply effectively the methods selected? • Does the scholar modify procedures in response to changing circumstances? • Results • Does the scholar achieve the goals? • Does the scholar’s work add consequentially to the discipline and to the community? • Does the scholar’s work open additional areas for further exploration and collaboration? • Does the scholar’s work achieve impact or change? Are those outcomes evaluated and by whom? Evaluation Criteria for the Scholarship of Engagement Purdue University (2001)

28. EVALUATING (engaged)digital humanities practice

29. challenges to evaluating DH practice • collaboration/authorship • “curatorial practice” (scholarly editions, digital archives, discovery tools, databases) • tool-making (software, games, standards and specifications, search tools, visualization) • research vs. service • interdisciplinarity • lack of evaluative expertise at department level • assessing the intellectual value of digital work • time frame for evaluating open-ended work • documenting, contextualizing, explaining DH practice & process • evaluating digital work in the form in which it was meant to be experienced • experimental nature of DH practice • measuring impact

30. vehicles forevaluation • peer review • --18th Connect, NINES • reviews in journals (digital and analog) • --http://digitalhumanities.org/dhq/ • prizes • --http://www.historians.org/prizes/Rosenzweig_Fellowship.cfm • financial support (fellowships, internal and external grants, gifts) • --http://www.neh.gov/divisions/odh • conference presentations • --http://www.dh2012.uni-hamburg.de/ • external expert reviews • citations/scholarly references/blog posts • --http://www.dancohen.org/ • qualitative user feedback • -- http://docsouth.unc.edu/support/about/readers.pdf • quantitative use/impact measures • --downloads, visitor counts (http://www.google.com/analytics/)

31. vehicles forevaluation visualizations of 1-year Google Analytics data for Going to the Showhosted by CDLA at UNC-Chapel Hill

32. EXAMPLES of promotion and tenure practice University of Victoria“Report on Academic Computing Recognition” | 1998

33. EXAMPLES of promotion and tenure practice

34. EXAMPLES of promotion and tenure practice

35. EXAMPLES of promotion and tenure practice

36. STAY CONNECTED join the listserv | cdhi@unc.edu