Library Philosophy and Practice 2009

ISSN 1522-0222

Effect of Peer Review on Citations in the Open Access Environment

Mohammad Hanief Bhat
Senior Librarian
Islamia College of Science & Commerce
Srinagar (India)

Introduction

Editorial peer review is widely used to select submissions to journals for publication and is presumed to improve their usefulness (Jefferson, Alderson, Wager & Davidoff, 2002). Publishing a peer reviewed article in a prestigious journal remains the highest validation for a work of scholarship. Peer review has served scholars well for centuries. The concept of reporting and validating research findings began in 1665, with the foundation of the Philosophical Transactions of the Royal Society of London. Peer review has enhanced the rigour and relevance of many scientific breakthroughs (Banks, 2006). Practically no historical accounts of the evolution of peer review exist. Contrary to common assumption, editorial peer review did not grow out of or interact with grant peer review. Editorial peer review procedures did not spread in an orderly way; they were not developed from editorial boards and passed on from journal to journal. Instead, casual referring out of articles on an individual basis may have occurred at any time, beginning in the early to mid-19th century (Burnham, 1990).

Peer review is not perfect, and when it is done sloppily, journals publish research that is flawed. Even when peer review is rigorous, flawed research sometimes gets into the literature. Journals have long relied on peer review, yet concerns about its limitations have often been expressed. Critics point out that some reviewers are unqualified and others, because of personal or professional rivalry, are biased. Editors may even select reviewers on the basis of the reviewers' biases. Furthermore, two or more reviewers may have widely discrepant opinions about a study. Critics also make the point that peer review not only fails to prevent the publication of flawed research but also permits the publication of research that is fraudulent. Some have described peer review as arbitrary, subjective, and secretive. In addition, many critics (including some of the popular press) maintain that it is simply unnecessary and slows the communication of information to the public ( Kassirer & Campion 1994). The present study makes an attempt to measure the effect of peer review process on research impact of publications in comparison to those which have not gone through the process via citation analysis.

Objectives

The following objectives are laid down for the study:

• To assess the impact of peer review on citations
• To compare the research impact of refereed and working papers

Scope

The scope of the present study is limited to research articles published in Information Research: An International Electronic Journal from 1998-2002.

Hypothesis

In view of the criticism of peer review process from various quarters, let us assume that it does not improve the quality of research output and consequently does not affect the research impact of publications and formulate the hypothesis “The research impact of refereed and working papers does not differ significantly” for the purpose of testing.

Methodology

Information Research: An International Electronic Journal is a high impact factor open access journal in the field of information science publishing working papers side by side with refereed articles. The publication output of five years (1998-2002) was selected for determining their research impact through citation analysis. The details of refereed and working papers were recorded separately across all the issues covering a time period of 1998-2002.

All the 101 articles (74 refereed and 26 working papers) were searched in Scopus database for citations. Two articles (one refereed and one working paper) are not indexed by Scopus are thus not included in the study. The number of citations, self citations and other details were recorded for all the 99 articles. Standard statistical techniques were used to estimate various statistical tests. The data is tabulated and analysed in a systematic manner to reveal findings in accordance with desired objectives.

Related literature

The institutionalization of the peer review process took place mostly in the 20 th century, either to handle new problems in the numbers of articles submitted or to meet the demands for expert authority and objectivity in an increasingly specialized world (Burnham, 1990). Although widely used it is largely untested and its effects are uncertain (Jefferson, Alderson, Wager & Davidoff, 2002).The process being expensive, slow, prone to bias, open to abuse, possibly anti-innovatory and unable to detect fraud (Smith, 1997). It will remain almost impossible to assess or improve its effectiveness, unless the objectives are properly defined (Jefferson, Wager & Davidoff, 2002). However, vigilance can improve fairness in peer review process and alleviate the effects of various pitfalls. The journal editors should conduct periodic internal and external evaluations of their journal's peer review process and outcomes with participation of reviewers, contributors, readers, and owners (Hojat, Gonnella & Caelleigh, 2003).

There is a need for the development of an international online programme for accredation of potential refrees (Benos, Bashari, Chaves, Gagger, Kapoor, LaFrance, et al., 2007). Davidoff (2004) found that quality of many manuscript reviews is excellent, but in many other it is unfortunately still far from optimal. The editors of journals might understandably look for ways to improve reviewer's performance. One such step is the anonymity of peer review which decreases both personal as well as geographical bias (Opthof, Coronel & Janse, 2002). It is found that manuscripts receive significantly higher priority ratings when reviewers and authors originate from the same country (Opthof, Coronel & Janse, 2002). A survey of 873 corresponding authors of manuscripts under consideration by the Annals of Emergency Medicine between May 1999 to October 2000 found that author satisfaction is associated with acceptance and not with review quality (Weber, Katz, Waeckerie & Callaham, 2002).

Despite various pitfalls the peer review process assists in improving the quality of the submitted manuscripts, whether the manuscripts are accepted or rejected (Opthof, Coronel & Janse, 2002). It is found that papers rejected by the Journal of Clinical Investigation were cited at lower frequency when published by other journals (Wilson, 1978), and similarly manuscripts rejected by Cardiovascular Research were cited at significantly lower frequency even if published by journals with a higher impact factor (Opthof, Furstner, Van Geer, Coronel, 2000).

Result and Discussion

Out of 101 research articles published between 1998-2002 in Information Research, only 99 are indexed in Scopus (73 refereed and 26 working papers). The total of 568 citations are received by 99 articles (Mean=5.73, S.D. = 8.85), out of which 67 (11.79%) are self citations. 21 articles received zero citations, 17 received one citation each and one article received 65 citations. The highest number of citations i.e., 208 are received by 24 research articles published in 2002 (Mean=8.06, S.D. =13.35) and the lowest number of citations i.e., 56 are received by 14 articles published in 1999 (Mean=4.0, S.D. =6.55). However the distribution of citations in 2002 is highly skewed owing to high Standard Deviation. This is due to the fact that one article has received 65 citations. Almost 50 percent of all self citations are received by the publications of 2002 alone (Table 1).

Table 1: Citation statistics of research publications

The 73 refereed articles have received 485 citations (Mean=6.64, S. D. =9.66), out of which 65 (13.4%) are self citations. Out of 73 articles 13 received zero citations, 8 received one citation each and one article received 65 citations. The highest number of citations are received for 22 research articles published in 2002 i.e., 194 (Mean=8.81, S.D. =13.81) and the least number of citations are received for 21 articles published in 2001 i.e., 98 (Mean=4.66, S. D. =5.57) (Table 2).

Table 2: Citation statistics of refereed papers

The 26 working papers have received 83 citations (Mean=3.19, S. D. =5.23), out of which only 2 (2.4%) are self citations. The 7 working papers received zero citations, 9 received one citation and one article received 22 citations. The highest number of citations are received for 2 working papers published in 2002 i.e., 14 (Mean=7.0, S. D. =8.48). The 3 working papers published in 2001 have received no citations (Table 3). All the working papers are cited in reputed peer review journals (Table 4).

Table 3: Citation statistics of working papers

Table 4: List of cited journals with no. of citations

Verification of hypothesis

The 73 refereed articles have received 485 citations (Mean=6.64, S. D. = 9.66) whereas 26 working papers have received 83 citations (Mean=3.19, S. D. =5.23). It implies that refereed articles are cited twice those of working papers. However the percentage of self citations amongst refereed papers is much higher than those of working papers. It is also revealed from the results that the distribution of citations in both the cases is some what skewed (S.D. being higher than mean). To test the hypothesis that “The research impact of refereed and working papers does not differ significantly” the Chi Square test is carried out

?² = S (O-E) 2 /E

E = Expected citations

O = Observed citations

The value of ?² is 39.8. At 1 df this is highly significant at .05 level, .01 level and even at .001 level. Thus the null hypothesis “The research impact of refereed and working papers does not differ significantly” is rejected and it is concluded that the research impact of refereed papers is higher than those of working papers. Even if we exclude self citations from the results the hypothesis is still rejected.

Conclusion

The present study reveals that despite criticism from various quarters and pitfalls the process of peer review has significant effect on the research impact of articles in terms of number of citations. Thus it is concluded that the process of peer review will remain an integral part of the scholarly communications system even in the emerging open access environment.

References

Banks, M. A.(2006). Towards a continuum of scholarship: The eventual collapse of the distinction between grey and non-grey literature. Publishing Research Quarterly 22 (1). Available: http://eprints.rclis.org/archive/00005803/01/GL7Paper_Final.pdf

Benos, D. J., Bashari, E., Chaves, J. M., Gagger, A., Kapoor, N., LaFrance, M., et al. (2007). The ups and downs of peer review. Advances in Physiology Education 31: 1043-4046. Available: http://advan.physiology.org/cgi/content/abstract/31/2/145

Burnham, J.C. (1990). The evolution of editorial peer review. JAMA 263 (10). Available: http://jama.ama-assn.org/cgi/content/abstract/263/10/1323

Hojat, M., Gonnella, J.S., & Caelleigh, A. S. (2003). Impartial judgment by the “gatekeepers” of science: Fallibility and accountability in the peer review process. Advances in Health Sciences Education 8 (1). 75-96.

Jefferson, T., Alderson, P., Wager, E., & Davidoff, F.(2002). effects of editorial peer review: A systematic review. JAMA 287 (21): 2784-2786. Available: http://jama.ama -assn.org/cgi/content/abstract/287/21/2784

Jefferson, T. Wager, E., & Davidoff, F. (2002). Measuring the quality of editorial peer review. JAMA 287 (21). 2786-2790. Available: http://jama.ama-assn.org/cgi/content/abstract/287/21/2786

Kassirer, J.P., & Campion, E. W.(1994). Peer review: Crude and understudied, but indispensable. JAMA 272: 96-97. Available: http://www.ama-assn.org/public/peer/7_13_94/pv3089x.htm

Opthof, T., Coronel, R., & Janse, M. J. (2002). The significance of the peer review process against the background of bias: priority ratings of reviewers and editors and the prediction of citation, the role of geographical bias. Cardiovascular Research 56 (3). 339-346. Availble: http://cardiovascres.oxfordjournals.org/cgi/content/extract/56/3/339

Opthof T., Furstner F., Van Geer M., & Coronel, R. (2000). Regrets or no regrets ? No regrets! The fate of rejected manuscripts. Cardiovascular Research 45: 255-258.

Smith, R. (1997). Peer review: Reform or revolution?: Time to open up the black box of peer review. BMJ 315. 759-760. Available: http://www.bmj.com/cgi/content/full/315/7111/759

Weber, E. J., Katz, P.P., Waeckerie, J. F., & Callaham, M. L. (2002). Author perception of peer review: Impact of review quality and acceptance on satisfaction. JAMA 287 (21). 2790-2793. Available: http://jama.ama-assn.org/cgi/content/abstract/287/21/2790

Wilson, J.D. (1978). Peer review and publication. Journal of Clinical Investigation 61: 1697-1701