CAR PURCHASE COMPARI

1. CAR PURCHASE COMPARI Trust and authority in scholarly communications David Nicholas and Anthony Watkinson, CIBER • text • text

2. Trust and authority under massive attack CAR PURCHASE COMPARI • Trust, authority and reliability are the very watchwords of scholarly communication. System built upon quality assurance • In today’s crowded, dynamic, diverse and dis-intermediated digital scholarly environment it is ever more difficult to establish and authority of information • More sources, channels, platforms, environments, players and consumers – all muddying the water. • Difficult to even know whose information it is anymore • More metrics and proxies; more gaming all brought on by increasing competition among academics • Certainly some VERY challenging behaviour coming along from the Google Generation [picture on right] and smartphone users

3. Background: the project CAR PURCHASE COMPARI • CIBER and UT with funding from Sloan Foundation researching how emerging digital behaviours/platforms are (possibly) challenging and changing concepts of trust and authority in scholarly world. In particular to discover: • How academic researchers assign and calibrate authority and trustworthiness to the scholarly sources and channels they choose to use, cite and disseminate; • Whether social media and open access are having an impact on conventional practices of establishing the authority and trustworthiness • Scope: 09/2012 – 11/2013; global with focus on UK & USA; science, social science • Working with publishers to identify, recruit and access researchers. T&F , Elsevier, Sage, BMC, Wiley, PloS and Thompson Reuters. • Methods: a) 14 focus groups to scope boundaries and identify important issues and questions (DONE); b) 80 critical incident interviews to supply the personal detail (DONE); c) international questionnaire to obtain quantitative data (PLANNING STAGE)

4. General: characteristics of a reliable and trustworthy source/channel CAR PURCHASE COMPARI Few researchers really dealt head-on with what was meant by reliability, quality, trustworthiness; discussions usually turned into a debate about relative merits of personal vs. proxy methods of establishing reliability and quality Trustworthiness was said be down to reputation of the author or journal in their research specialism Quality could be best determined by personal inspection and judgment, but because of a shortage of time, too much material to get through and the strictures of their managers and institutions (in respect to Impact Factors, for instance) this was not always feasible. While everyone thought it was an important issue nobody really thought it was a big or pressing issue. Was it just a question of what is useful?

5. General: personas CAR PURCHASE COMPARI Researchers function as authors, editors, reviewers, citers and users of the literature. Thus can bring different trust judgments to bear for each function Editors seemed to take a different stance; their thinking was dominated by their own picture of themselves as information providers, people offering trustworthy collections for others to trust. They appeared much more traditional Citation behaviour tends to be much stricter, more focussed and complex than use Researchers have considerably more freedom as to what they use - they can use blogs, OA etc. to their heart’s content Most researchers undertake a variety of scholarly roles. However, when they talk about trust and authority in scholarly communication they rarely preface what they say by stating that, for instance, as user or citer I do this. It might be so because the journal is so central to their views on quality

6. General: peer review CAR PURCHASE COMPARI Provides a degree of certainty about the quality of product. Shows that someone has put in an effort and that it had been validated by a community of scholars. Therefore, in theory, it is an important scholarly attribute which enables researchers to search, use, cite and disseminate with confidence On the one hand researchers want to be published in journals that have robust peer review mechanisms (despite the heartache involved) and on the other they want to feel secure in citing peer reviewed material. While there is a strong attachment to peer review most people preface their expression of trust with recognition that there are problems with the way it is undertaken. However, when you drill down to the mechanisms of peer review there is no consensus at all on how it might be improved. Biggest impact REF has had is in creating an institutional peer review in universities throughout the UK; a system by which researchers are constantly monitored.

7. General: metrics CAR PURCHASE COMPARI Evidence that the scholarly world being driven and governed by algorithms and the consequence was that creativity and new ideas were being driven out by a (high) metric-driven culture, which was standardising scholarly communication behaviour Some differences in attitude according to discipline of researcher. Scientists largely unquestioning about merits of metric system; social scientists slightly uneasy but felt there was no choice; the few humanities scholars clearly felt culturally uncomfortable and alienated, but uncomfortable or not were part of it Most extreme case of force-fed metrics was found in Business/Economics, where researchers told where to publish and what to cite by the academic journal quality guide produced by the Association of Business Schools Early career researchers in social sciences/humanities thought themselves as ‘slaves’ to a metric-based/journal focussed system; they have to adhere to rules to climb academic ladder but thought the ladder was broken. Journals a manifestation of all that was wrong with scholarly communication system. Yes, it’s all crazy, but I am not here to reshape academe.

8. General: social media CAR PURCHASE COMPARI Only a few – mostly young, thought social media nothing but social and transient; no question it should be trusted, and certainly not a substitute for the peer reviewed journal While expressing lack of interest in social media themselves many knew people that used it and felt guilty about lack of involvement Lack of interest explained by trust and validity problem but also other reasons: a) many were novices; b) they were antagonistic towards it; c) had no free time; d) were put off going down that route by the current HE climate; e) the (informal) language of social media not suitable for scholarly discourse. Recognition that social media most valuable for: a) getting new ideas and stimulation; b) self-promotion of publications, especially in regard to outreach. What most interested participants (when told) was the fact that social media could increase their citations. Early career researchers made extensive use of social media but scared to embrace it fully and camouflaged its use. Elsevier research suggests they are getting bolder. Many academic benefits for them: a) develop a personal network; b) facilitates collaboration; c) finding fellow researchers to work with (in real-time); d) staying in touch with what is going on; e) following authors (stalking!); f) easier to find someone with a particular point of view. Fast-track development.

9. General: open access CAR PURCHASE COMPARI Distrust of open access from an author and reader perspective evident. Distrust diminishes considerably (but not quite evaporates) when OA journals are published by an established publisher Few researchers aware of the pioneering efforts of PloS1, although they did like, in principle, what it has achieved (quick and popular). Researchers from teaching intensive universities more positive on openness grounds. Felt also that OA journals might help your career: Universities might rank IF journals more highly but Google ranks OA higher in hits list, so if you want to be listed first to impress head hunters then OA provides a good career route. Early career researchers liked principle of open access , but scared to embrace it because felt academe has not made up its mind about it. If published in an OA journal, or cited one, might have backed the wrong horse and found, for instance, that they had published in a second-rate journal. There might be a reputation threat.

10. General: better/worse than a decade ago? CAR PURCHASE COMPARI Researchers acknowledged that there was more bad stuff around, because it is just more accessible and there are more opportunities to publish, but accepted , overall, quality had risen over the years. The rise in quality meant they could live with the bad and boring stuff Rise in quality result of more people entering the field and the greater competition that comes with it. It is the niche/specialist journals that have taken full benefit of an abundance of run-of-the-mill material So not OA journals that are being blamed for rise in poor or mediocre work, but the big increase in subscription journals ‘There is a massive sea of mediocrity now because it is just easier to publish, but at the higher end the quality is better because of better training, greater competition and rewards for publishing’

11. Usage CAR PURCHASE COMPARI Low key.Played down usage and trust, because, if something was out there, their long-established networks would tell them about it; they did not need to go looking.Intuitive where you turned for information. But at the heart of it all was networks of people that you had developed over the years. Establishing worth.‘If I don’t recognise the author, then be careful; if additionally you don’t recognise the institutional affiliation be even more careful, and if you don’t recognise the journal as well, it is definitely not even worth looking at’. Impact Factor. Not that important determining what to read: a) too narrow a view of research literature; b) attached to a journal, not an article; c) lottery as to which journals get IFs; d) an IF meant journal bombarded by low grade authors trying to enhance careers; e) high IF journals tend to be stylized, lack innovative and fresh papers; f) not aware of what the IF is for a journal Bigger fields the more important IFs - do not know all authors. IFs useful in peripheral fields. High IF journals get best referees so papers of merit Peer review. If knew the journal or the editorial group researchers were fairly confident but cannot always guarantee quality.Some dubious peer reviewed stuff out there.

12. a. Usage CAR PURCHASE COMPARI Usage factor. Researchers not aware of possibilities on offer. Negatives: 1) too easily gamed; 2) highly used articles not in the best ones (by editorial opinion); 3) downloads do not represent readings; 4) media exposure raises an articles profile greatly. Altmetrics? Interesting/original sometimes trumps authority/ranking. Occasions when authority/ ranking are of secondary concern in determining usage, when researchers looking for new, fresh and creative content. Mainly social scientists. In these cases a quick and dirty search in Google/Google Scholar productive, especially in delivering inter-disciplinary material. Good writing a determinant. Reading/role of abstracts in trust /authority judgments. Too much published, too little time to read it and widespread adoption of skittering leads to a dependence on proxies (abstracts). Screening and cross-comparisons made at the abstract level. Few researchers read full-text – when writing/reviewing an article. In fact reading an article means looking 5-10% of it. Wanted them quality-controlled, properly reviewed. We need to be able to trust the abstract. Abstracts offered free to view adds to their value in this role. Other bibliographical trust points. Methodology (more so in science; social scientists thought to be too mechanistic a measure), conclusions, bibliography and theoretical stance.

13. Usage: types of sources CAR PURCHASE COMPARI Digital sources. Trust enhanced if something also available in print. Merits in paper and some associated with quality/reliability. Thus: a) paper was thought to provide more confidence because less ephemeral; b) because there were clear length limitations this focussed the mind and produced better copy. Early career researchers thought the opposite; could not believe anyone would think like that and thought digital was the real. Data. Some trust concerns because of a presumed absence of peer review; needed because could not decide for yourself given size and complexity of data. Idea of data being attached to an article, the article giving it its authority, thought to be a good idea. Also felt that name of the author was even more important in case of data. Conferences. Not REF material. Also: a) peer reviewing mechanisms poor, absent or done on the back of an abstract; b) increasing number of sharks out there that sully the name of conferences; c) despite the best indexing efforts of Web of Knowledge et al proceedings do not have the same visibility and standing. But they can attract large number of citations.

14. Usage: searching CAR PURCHASE COMPARI Search platforms. ‘Trusted’ big fat information pipe researchers connected to not the publisher platform or library big deal; it is in the Internet. Hence popularity of Google. Google Scholar thought to be surprisingly good. Librarians/JISC argue discovery terrain becoming increasingly difficult, with OA, but not according to researchers. Did not mention their publisher’s website nor for that matter university catalogue or federated search engine (JISC project!). Libraries mentioned in a negative and nostalgic fashion. Role of librarians bound up with the buildings in which they work. Libraries, once guardians of quality have no role at all to play today. Just did not see libraries as the point of entry to the information they are looking. Libraries seen as incomplete sources of information and researchers do not trust librarians to make the critical decisions on what is and what is not in the walled garden on their behalf. Trusted searches strategies: 1): follow-up citations from trusted sources and then you check the abstracts to establish worth and quality; 2): enter the name of a classic work in the field in Google Scholar and then look at the citations to it. Any barrier will put you off chasing something. Early career researchers did follow people and publishers, but the difference did not do this via a publisher database, but via social media

15. CAR PURCHASE COMPARI Usage: open access and social media • Open access. All the researchers we spoke to were beneficiaries of the big deal, which gave them easy access to journals. So OA no big deal. Nobody actually said they would rather not use OA material. However, even if they wished to, what is not clear as to how they could discriminate in the first place • Concerns of poor (or absent) peer reviewing of OA articles did not result in widespread checking of OA journals’ peer review polices. But the interesting thing was that some actually did say that they did check policies which we bet they do not do for subscription journals • Social media. Scientific tweets almost always point to a journal article; they are just to link. They all use Wikipedia, love it and happy to say so. Academe.edu mentioned positively for accessing difficult to access material. Young career workers talk about social media enabling them to have a conversation.

16. Citing CAR PURCHASE COMPARI • Not as liberal as use. Authors weigh up sources carefully (reject social media). There are political issues; need to cover their backs; people you have to cite to get accepted; only have real choice over a few of your citations. A lot of window dressing • Citation practices/gaming: • 1) cite your own work to raise your H index; • 2) cite papers in journal to which you submit; • 3) reviewers ask you to cite their own papers; • 4) cite very high impact articles because they set the agenda/benchmark and represent the very pinnacles of science (halo effect); • 5) cite post hoc – use citations to support your position, give your ideas more weight; this is particularly important if your ideas are novel; • 6) cite the first source on the topic and the most recent one (bookending); • 7) cite review articles as a bibliographical shorthand. We know that people are gaming because they say they do not do it but know people that do

17. Citing CAR PURCHASE COMPARI • Early career researchers. Pressure on them from supervisors to cite peer-reviewed articles. You see interesting things elsewhere (i.e. social media) but you cannot use/cited them. Cited social media sources as ‘personal communications’ • Academics from teaching intensive universities different:could not cite something they have not read and would cite anything, including trade publications. • Social media. twitter not used as an information source: it would be like citing a conversation in the bar. Blogs not to be sourced: many blogs ‘were just streams of consciousness stuff’.

18. a. Publishing/Dissemination CAR PURCHASE COMPARI • Pressures to publish and the impact on quality. Biggest and most important concern was not so much growth in the literature, but what was responsible for driving this growth, the increased pressure to publish. Trend is creating an ‘avalanche of bad/mediocre journal articles’. It is ‘publish or perish’ multiplied. And ‘there is a huge amount of junk floating around the [scholarly] system.’ This manifests itself in massive journal rejection rates ‘50% are rubbish’, coming from everywhere, but mostly developing countries • The massive influence of REF (in UK): • REF is all-pervading and persuasive. REF says citation scores not everything but no-one believes this and carry on attempting to publish in high impact journals. Who is going to question an article in Nature? Leads to a negative impact on creativity and a distortion in where articles really should be placed. Penalising inspirational and creative academics • REF is guilty of forcing people to publish more than they otherwise would, something which leads to higher levels of poor content being published. More and more universities are prescribing metrics as a proxy for quality, meaning lower ranked universities joining the game, when they really should not.

19. Publishing: peer review CAR PURCHASE COMPARI Few thought that the review process was broken. Early career researchers least fond Publishers. Organising peer review central role of publisher. Rejection rates. Badge of honour/quality. The practice. a) liked blind reviewing - reviewers freer to comment; b) mixed opinions about benefits of author-suggested referees (bad: suggest their friends; good: avoid referees who suspect of foul-play); c) light touch peer review not liked; heavy leads to better papers; d) open refereeing not popular as inhibits reviewer; e) not sure about post publication peer review; e) editors should be ultimate judges; be proactive and not always heed reviewers; f) editors should function as a release valve for peer-review process, when fails to allow for difference, freshness and innovation; g) referees improve an article even if they rejected it. It is worth submitting to Nature, even if you had no chance being accepted, just to get quality feedback. Dislikes: too slow. Need to obtain decision within two months; weakness predatory OA publishers take full advantage of (in their advertising, if not in reality). Concerns: Quality of reviewing thought to be variable and explained by the pressures on reviewers to get the job done quickly; as a consequence, quality is being sacrificed.

20. Publishing: Plagiarism and unethical practices CAR PURCHASE COMPARI Plagiarism. With the article avalanche comes dodgy content. Thought to be more rife and widespread than generally thought. While early career researchers agreed that plagiarism was a no-no, they were less antagonistic towards cut-and-paste behaviour, providing attributions were given. While academics from teaching intensive universities were not supportive either they were not that hard on it. Thought it was a fuzzy area and understood the reasons/pressures for doing it. Self-plagiarism they seemed to think was a less serious offence; maybe not an offence at all.   Publisher data shows plagiarism and duplication detected down from 30% to 10% in last 3 years. CrossCheck Making it up/fabrication. Everyone knew cases: senior academics included. Big pressure to do so in some parts of the world. It was alleged it could be as much as 10-20% in biological sciences. Difficult to establish.

21. Publishing: Open access CAR PURCHASE COMPARI • Few researchers admitted to OA publishing; deep seated dislike of OA for many. Being imposed (political reasons), rather than something actually needed. It’s open anyway. Few realised that traditional publishers produce OA articles and journals • Worries: • 1) OA material poor quality. ‘Crap’ was one term used to describe it; other terms used were ‘vanity publishing’, ‘self-deluded authors’.‘Why would you want to (pay to) publish in something in a start-up, which is easy to get into and has no reputation or pedigree. The journal would have to have a brand’; • 2) Business model liable to undermine rigorous review; concerns about possible two track peer review process, with OA articles being treated more leniently because of payment; • 3) Uneasy about the author pay model that underpins gold OA publishing. Passing of money seems to sully the transaction. Some concern that you could pay your way into publishing; • 4) Poorly run and unprofessional. Fails to recognise that there is a good deal of professionalism and standards behind publishing; • 5)Concerns that academics might have to publish in OA journals and that subscription journals might end up featuring different content and authors.

22. Publishing: Social Media CAR PURCHASE COMPARI Nobody saw it being an alternative to journal or book publishing. Raised the results of a previous CIBER focus group on social media which revealed that in one discipline at least younger researchers saw social media giving them a way of communicating ideas and information which they could not publish in journals in the hands of a cabal of old white males, fixated on impact factor scores. While there was some sympathy for this view, nobody came forth with similar examples