D-Lib Magazine
November/December 2009

Volume 15 Number 11/12

ISSN 1082-9873

Measuring Citation Advantages of Open Accessibility

Samson C. Soong
The Hong Kong University of Science and Technology
<soong@ust.hk>

Introduction

Over the past decade, a good number of studies have attempted to demonstrate that open access journal articles have higher citation rates than traditionally published ones.¹ These open access articles are either published in open access journals or are made publicly available through open access repositories. More methodologically sound studies, however, are needed to pinpoint more exactly the impact, if any, of open accessibility on citation rates and to look at the issue of impact (or non-impact) more closely. For instance, of those journal articles that have been made openly accessible, about what percent of them benefit from open online availability? What percent of them do not?

This article describes a study, involving a set of articles published in scholarly journals by faculty members of the Hong Kong University of Science and Technology (HKUST) that have also been deposited in the HKUST Institutional Repository. The study was conducted to measure the actual effect of their open accessibility on citation rates. More importantly, the study suggests another quantitative method, and it is hoped that the study will inspire more effective methods of measuring the net impact of open access.

Level of research at HKUST

Established in 1991, HKUST was ranked 35^th among the top universities in the world by the Times Higher Education Supplement in 2009. The University, by a whole range of measures, is considered one of the leaders both in education and research. Although a relatively young university, it has already gained an international reputation for innovation and cutting-edge research in a number of fields, including nanoscience and nanotechnology, biological sciences, wireless and information technology, environment and sustainable development, and management science.

HKUST Institutional Repository

In 2003 the HKUST Library established a digital institutional repository (IR) to host, preserve and facilitate access to the scholarly output of the University community. This repository uses DSpace as its platform, which complies with the Open Archives Initiative (OAI), thus allowing archived documents to be easily discovered by web search engines and services, such as Google Scholar, OAIster, and Scirus.² There are currently over 5,600 documents in the Repository.

Documents deposited in the IR include conference papers, research and technical reports, book chapters, theses and dissertations, patents, and research datasets, as well as published journal articles. By submitting articles to the Repository after they are published by a journal, researchers at HKUST make their work more readily available to the global scholarly community. Depositing published articles in an open access repository usefully complements their publication by scholarly journals. Thus, the IR serves as a secure central repository of the scholarly output of the institution and offers greater visibility for the research carried out by the University community.

Methodology and data sources

Since methodological soundness is critical to assessing the results of any study, a full account of the methodology and data sources used to obtain the citation rates is given here.

A total of 50 archived journal articles that already have 10 or more citation counts in Scopus³ were randomly selected for inclusion in this study. This could be done quickly, as the system grabs in real time the current total citation count from Scopus and displays that piece of information as part of each record in the Repository, e.g., <http://repository.ust.hk/dspace/handle/1783.1/191>.

Each "full item record" in the Repository includes fields on "date.accessioned" and "date.available" which help to ascertain when a full-text journal article was actually added to the Repository and made openly accessible. Elsevier's Scopus database was then searched to get the citation counts of these 50 articles in each of the years since they were published, including the years after these articles were deposited in the HKUST IR.

For example, the Scopus record on the journal article "Computer supported augmentation and collaborative decision making: the HERMES system" by Karacapilidis and Papadias⁴ provides the following citation accounts: 2008(10 citations), 2007(8 citations), 2006(6 citations), 2005(5 citations), 2004(6 citations), 2003(2 citations), 2002(1 citation).

The absolute citation counts were used to derive two average-per-year citation counts, before and after the articles are made openly accessible through the IR. The Karacapilidis and Papadias article was published in 2001 and deposited in the IR in 2003, according to the "date.accessioned" field and the "date.available" field of the full item record for this archived journal article. Thus, the average-per-year citation counts before and after this article becomes openly accessible are 1.5 and 7 respectively. The number of citations in 2009 is not included in the calculation, as the year is not over yet. The number of citations during the year of publication (most of the time during part of the year) is also excluded in the computation.

The following table shows the results of this study.

Table of Study Results

Key Findings

Of the 50 articles included in the study, 29 (or 58%) have had a higher average-citation-rate after they have been deposited in the IR and made openly accessible than they had prior to being available in the IR. The rest of the open access articles, or 42 %, have not yet experienced a similar increase in the average-citation-rate. Although we cannot simply contrast these two percentages to derive an overall rate or a potential rate of increase, these results suggest that there is a citation advantage for a larger percent of articles that are made publicly accessible.

While there is an increase in the average citation rate for 58% of the articles, there is no obvious year-by-year incremental increase. In other words, we have not seen that the number of years an article becomes openly available has any particular effect. Much bigger increases for some articles suggest that the subject areas of these articles matter to a large extent. For instance, a number of articles in Chemical and Biomolecular Engineering in this study stand out in particular.

Conclusions

The citation advantage of open access needs to be studied more in depth to determine their impact in different subject areas or disciplines. An easy-to-follow framework for citation impact analysis of open accessibility is presented here. This framework allows for direct measurement and comparison of citation rates before and after journal articles are made openly available. This approach also allows for easy contrast between open access articles that have produced citation impact or advantage and those that have not. Any impact study would be undermined, however, if such a study does not take difference in disciplines into consideration. Although the value of time intervals to the study framework was not particularly highlighted, it is totally feasible to conduct the study longitudinally to see if the impact of open access changes over time.

The rapid growth of institutional repositories has helped the open access movement take hold with many more scholars and researchers in the world. Additional methodologically sound studies should be encouraged to explore more effective ways to measure and ascertain citation advantage of open accessibility of journal articles. The entire scholarly community will significantly benefit when open access to research output is compellingly advocated and maximized.

Notes and references

1 For instance, Steve Lawrence, "Online or invisible?" Nature 411 (6837):521, 2001. Available at: <http://www.nature.com/nature/debates/e-access/Articles/lawrence.html>; Stevan Harnad and Tim Brody, "Comparing the impact of open access (OA) vs. non-OA articles in the same journals", D-Lib Magazine, Vol 10, No. 6, June 2004. Available at: <doi:10.1045/june2004-harnad>.

2 "Scirus is the most comprehensive scientific research tool on the web. With over 450 million scientific items indexed at last count, it allows researchers to search for not only journal content but also scientists' homepages, courseware, pre-print server material, patents and institutional repository and website information." – Scirus website at <http://www.scirus.com/>.

3 "Scopus is the largest abstract and citation database of peer-reviewed literature and quality web sources with smart tools to track, analyze and visualize research. Scopus is designed and developed with over 500 users and librarians internationally. Updated daily, it includes the abstracts and references of 15,000 peer-reviewed journals from more than 4,000 international publishers, ensuring broad interdisciplinary coverage." – Scopus websit at <http://www.elsevier.com/wps/find/electronicproductdescription.cws_home/704746/description#description>.

4 Karacapilidis N., Papadias, D., "Computer supported argumentation and collaborative decision making: The HERMES system", 2001, Information Systems, 26 (4), pp.259-277. Professor Papadias is a faculty member of the Department of Computer Science and Engineering at HKUST.

Top | Contents
Search | Author Index | Title Index | Back Issues
Previous Article | Next Article
Home | E-mail the Editor

D-Lib Magazine Access Terms and Conditions

doi:10.1045/november2009-soong