|
eMJA Online Peer Review Trial - original proposal
This is the proposal for funding submitted to the Electronic Working Group of the
Australian Vice Chancellors' Committee for the first eMJA Internet peer review study. The section on budgets has been removed. The
funding sought represented only part of the estimated expenses of the trial.
Thursday 11 May 1995
University of Sydney Library - Medical Journal of Australia
Joint
submission to the AVCC Electronic Publishing Working Group
Proposal: examination of an open peer review process by electronic publication
Back to contents
Examination of an open peer review process by electronic publication The University
of Sydney Library and the Medical Journal of Australia have agreed to combine their expertise
and resources for this project. Objectives
1. To develop electronic publishing skills and expertise within the Library, in order to
position the Library as a key resource centre on campus for future academically-oriented
electronic publishing ventures. 2. Using electronic publishing on the Internet, to
develop a new model of peer review that allows:
- Open scrutiny of articles
and the peer review process itself, by electronically publishing articles with reviewers'
comments
- Faster publication of research by electronic delivery without
prejudice to print publication, which is regarded as the "gold standard" of academic
publication.
3. To compare the effectiveness of the new model with conventional peer
review. Outline of methods
An Internet World Wide Web site will be created for the MJA (eMJA). Here the MJA
will publish research articles that have been accepted for print publication, together with
comments provided by the peer reviewers. Readers on the Internet will be able to review the
articles and commentary and post their own comments to the eMJA. These comments will be
filtered editorially to remove irrelevant material, then passed on to the authors and peer
reviewers as feedback. Selected comments may be electronically published with the papers and
reviews as additional commentary; authors will be able to respond or revise their paper in
response to comments. The full text of all documents will be indexed and an online search
facility will be provided. After a period on the Internet, papers will be removed
from the eMJA review list and published in print in the MJA. Quantitative data will be
collected via the computer system and qualitative assessments will be sought from authors,
reviewers, editorial staff and an external Project Review Group. A conference of
involved parties will discuss the results with a view to making recommendations on open peer
review and electronic publication. Anticipated outcomes
A working protocol for open online peer review.
A development path for scholarly journals from printed publication to complementary
printed and electronic publications. Opening peer review to wider scrutiny should lift
the standard of peer review by encouraging higher performance from reviewers and providing
new feedback on their performance. Open peer review may improve the quality of
published articles. Authors should be satisfied with faster and wider publication to
the international research community and a potentially higher standard and broader range of
peer review. This may lead to a higher standard of contributions to the MJA.
Back to contents
1. To develop electronic publishing skills and expertise within the Library, in order to
position the Library as a key resource centre on campus for future academically oriented
electronic publishing ventures. 2. Using electronic publishing on the Internet, to
develop a new model of peer review that allows: Open scrutiny of journal
articles and the peer review process itself, by electronically publishing articles with reviewers'
comments. Faster publication of research by electronic delivery without
prejudice to print publication, which is regarded as the "gold standard" of academic
publication. 3. To compare the effectiveness of the new model with conventional peer
review. Meeting these objectives will contribute to achievement of a broader, longer-term
goal: 4. To develop protocols and models for an orderly extension of scholarly
publishing from print to electronic media.
Back to contents
Can authors make the cultural shift required to have articles published electronically on
the Internet before appearing in print? Can authors benefit from a wider ranging peer review
process? Will peer reviewers accept the electronic publication of their reviews? Will they
demand the cloak of anonymity or opt to be publicly identified? Is Internet "noise"
(i.e. excessive, trivial or even malign communications via the Internet) a barrier to scholarly
uses of the Internet? Will electronic publication reach a wider audience, in Australia
and overseas? Does the new model of peer review result in more effective review and
a higher quality of article?
Back to contents
University of Sydney Library
The largest university library in Australia, with the most diverse holdings.
Serves the University of Sydney Faculty of Medicine through three medical branch
libraries and also manages library branches supporting nursing studies and health sciences.
These collections are major national resources, and the Library has nominated 15 subjects as
of national importance in the Distributed National Collection Agreement in Medicine. Provides electronic resources to the Faculty of Medicine through local area networks
and is working with the Faculty on the introduction of information technologies in teaching
and research. Through its Information Services and Systems Departments, offers
proven skills and expertise in providing World Wide Web facilities, the creation and
maintenance of HTML (hypertext markup language) documents and the management of
contractual work in these fields. Library staff directly involved in this project
include:
- Medical Librarian: Pamela Leuzinger, BA(Hons), DipLib,
DipJuris.
- Systems Coordinator: Linda McDonald, BAppSc(CompSc)(Hons).
- Collection Management Librarian: Ross Coleman, BA(Hons), ALIA.
- Reference Services Librarian: Paula Garrett, BA, MS(LibraryInfoSc).
- Associate Librarian, Reader Services: Kate Sexton, BA, DipLib.
- Coordinator of Networked Information Resources: Steven Ryan, BA, DipInfoManagementLibrarianship.
The Medical Journal of Australia (MJA)
Research and education journal of the Australian Medical Association (AMA),
published since 1914 by the Australasian Medical Publishing Company on behalf of the AMA. Largest circulation medical journal in Australia (22,000 Australian doctors
twice a month). Fully indexed in Index Medicus.
Member of the Vancouver group of medical journals, with The Lancet, New
England Journal of Medicine, British Medical Journal, Canadian Medical Association Journal,
Journal of the American Medical Association, New Zealand Medical Journal, Index Medicus,
Tidsskrift for Den Norske Laegeforening and Annals of Internal Medicine. MJA is a peer reviewed journal, and peer review is supplemented by the use of a Content
Review Committee, of which the members are: - Dr Paul Glasziou , MB BS, PhD, Senior Lecturer in Clinical Epidemiology, University of Queensland, Deputy Director, NHMRC Clinical Trials Centre.
- Dr Paddy Phillips, MB BS, DPhil, FRACP, Senior Lecturer, Department of Medicine, University of Melbourne, Consultant Physician, Austin Hospital.
- Professor Adrian Mindel, MD, FRCP, FRACP, Academic Unit in Sexual Health Medicine, Sydney Hospital, Editor - Genitourinary Medicine.
MJA is published via desktop publishing technology
(Quark Xpress, Illustrator, Photoshop). Journal Staff (14):
- Editor: Dr Martin Van Der Weyden, MD, FRACP, FRCPA. Previously
Professor of Haematology, Monash University and Chief of Investigative Services, Alfred
Health Care Group, Alfred Hospital, Melbourne.
- Consulting Editor: Dr
Stephen Lock, CBE, MD, FRCP, FACP (Editor Emeritus, British Medical Journal).
- Senior Assistant Editor: Dr Bronwyn Gaut, MB BS, DCH, DA.
- Assistant Editor: Dr Bronwen Ross, MB BS.
- Publication Coordinator: Craig Bingham, BA(Hons), DipEd.
- Chief Copy Editor: Helen Randall, BSc, DipOT. Copy
- Editors: Elsina Meyer, BSc; Kerrie Lawson, BSc(Hons), PhD, MASM; John Strigas,
BSc(Hons).
- Librarian: Deirdre Ward, BSW, DipLib, AALIA.
- Supported by: Proof Reader, Production Manager, Production Assistant, Editorial
Administrator and an Editorial Assistant.
Journal staff directly involved in this
project include the Editor, Assistant Editors, Copy Editors and Publication Coordinator (Craig
Bingham, who will be Project Coordinator for the project). The Content Review Committee
will play a role in assessing the methods and results of the project.
Back to contents
Outline:
An Internet World Wide Web site will be created for the MJA (eMJA). Here the MJA
will publish research articles that have been accepted for print publication, together with
comments provided by the peer reviewers. Readers on the Internet will be able to review the
articles and commentary and post their own comments to the eMJA. These comments will be
filtered editorially at the MJA to remove irrelevant material, then passed on to the authors and
peer reviewers as feedback. Selected comments may be electronically published with the
papers and reviews as additional commentary; authors will be able to respond or revise their
paper in response to comments. The full text of all documents will be indexed and an online
search facility will be provided. After a period on the Internet, papers will be removed
from the eMJA review list and published in print in the MJA. Quantitative data will be
collected via the computer system and qualitative assessments will be sought from authors,
reviewers, editorial staff and an external Project Review Group. A conference of involved
parties will discuss the results with a view to making recommendations on open peer review
and electronic publication.
Measuring outcomes:
Access and use
- Number of visitors to the eMJA Web site and to individual articles; their
location (Australian/overseas).
- Number of comments received on articles.
Costs (time and resources)
- Time and resources spent creating and maintaining the eMJA Web site.
- Time and resources spent administering the open review process.
- Comparison of time to electronic publication with time to print publication (with and without
electronic publication).
Quality of open review process
- Proportion of comments received during open review that are found to be
relevant.
- Number of comments leading to changes in articles; number leading to
further comments from MJA reviewers or authors.
- Editorial assessment of reviews
and articles submitted to open peer review (by means of established checklists of performance
and qualitative assessment).
- Author's and peer reviewers responses to the process
(questionnaires and interviews) and opinions of its effectiveness.
- Content Review
Committee (an MJA standing committee of peers) assessment of the process and of the quality
of reviews and papers.
- Assessment by the Project Review Group (an expert
panel recruited by the university to provide "arms-length" overview of the method and
results).
- Reader's responses (collected via Internet and by focus group
interviews).
Technology:
The call for submissions asks for the innovative use of information and
communications technology. The project is based on use of the World Wide Web, the
hypertext publication environment that is now the leading medium of electronic publication, as
it supports text, graphics, hyperlinks between items, interactive publishing and sophisticated
indexing and search tools. The project will develop an interface to streamline the conversion of
digitised documents to HTML format and perform the associated hyperlinking and full text
indexing functions. Web server: adequate disk space is available on the University Library's
Web server. In general the library will provide all the infrastructure as required and absorb
related overhead costs. Bandwidth: over the last year the University has made a
substantial investment in the campus network infrastructure which provides more than
adequate bandwidth to service this project. Expertise: the Library has been an active Web
publisher for some time, both of its own information (for example
http://www.library.usyd.edu.au/) and in collaboration with other centres on campus (for
example, the publication of the University's Research Report at
http://www.library.usyd.edu.au/rrhtml/index.html).
Back to contents
From the provision of funding:
Preparation (3 months)
- Develop the eMJA World Wide Web site on the University server.
- Develop and document technical specifications and procedures.
- Train MJA staff in
HTML markup of articles and other Internet skills.
- Alert the academic and medical
community (particularly authors and peer reviewers) to the coming experiment.
Startup (2
months)
The MJA is published twice a month and contains ten or more peer-reviewed
articles (original research and subject reviews) in each issue, as well as other material. Authors
and peer reviewers will be given an informed choice of entering the open peer review project.
Relatively few articles will be placed on the Internet during the startup period while we assess
the rate of response. This experience and the response of authors is unpredictable, but at this
stage we imagine that up to 6 articles will be added to the eMJA review list per fortnight until
24 articles (equal to four issues or two months of articles) are on the list. Thereafter, six new
articles would be added each fortnight and the oldest six removed from the review list and
published in print. Running (4 months)
The computer system will automatically log data on number and origin
(Australian state/overseas) of accesses to each article. We will record other qualitative and
quantitative data on comments received, authors, readers and reviewers responses, effects on
articles and reviews, etc. Data Assessment (2 months)
Statistical and qualitative analysis.
Conference of MJA editorial staff, university staff, authors, readers and
reviewers for review and consensus recommendations. Final Report (1 month)
A report will be prepared for the Electronic Publishing Working Group and for
publication. Total project time: 12 months.
Back to contents
The project will be managed by the Project Management Group:
- Pamela Leuzinger, USL
- Linda McDonald, USL
- Ross Coleman, USL
- Kate Sexton, USL
- Martin Van Der Weyden, MJA
- Craig Bingham, MJA
This group will meet monthly during the course of the project. Day-to-day management will
be by Pamela Leuzinger, Medical Librarian, USL, and Craig Bingham, Project Coordinator,
MJA.
University of Sydney Library responsibilities
- Access to World Wide Web server.
- Initial development of World Wide Web pages with the following capabilities:
- e-mail links to MJA offices
- security protection
- statistics collection (users, articles)
- comment forms
- search tools
- Regular incorporation of articles, reviewers' comments and editorial notes.
- Regular maintenance, archiving and status reports.
- Training MJA staff
in HTML markup and related Internet skills.
- Recruiting an appropriately
qualified Project Review Group for external assessment of the methods and results.
MJA responsibilities
- Editorial assessment and peer review of articles.
- Supply of text and
graphic data in HTML format.
- Collection of qualitative data from authors and
reviewers.
- Editorial assessment of the effect on the quality of articles and reviews.
Joint responsibilities
- Promotion and evaluation of the project.
- Developing qualitative data collection tools.
- Organising a consensus conference to review the results and make
recommendations.
The project will involve the development of practical applications of
existing technologies and transfer of technological skills from the university to the journal
staff, while university staff involved in the project will gain familiarity with the workings of a
major scholarly publication.
Back to contents
How this project meets the goals of the Electronic Publishing Working Group: - to
encourage the publication of Australian research in electronic form, by means of an exemplary
journal in any field of academic endeavour;
The MJA is Australia's leading medical journal,
and one of the largest scholarly journals in the country. This project will introduce many
writers in medical research and practice to electronic publication. If successful, the model will
be applicable to many other scholarly journals. - to enhance international awareness of,
and access to, Australian research;
The MJA has an established international readership, but
the Internet promises much wider distribution of Australian medical research. The project goes
beyond anything that has been attempted by comparable research journals overseas, and the
results of this project will themselves be important to international medical journal publishers
and researchers. - to develop a closer, more productive relationship between the various
sectors involved in the scholarly publishing process - e.g. authors, librarians, information
technologists, publishers;
The project founds a new relationship between the University of
Sydney Library and the MJA, and involves a mutual transfer of skills and experience between
library and journal staff. The proposed model of peer review involves a closer dialogue
between authors and reviewers, and creates an entirely new interaction with readers that
allows them to scrutinise and contribute to the peer review process. - to provide the
Australian academic community with opportunities to test the various models for electronic
publication of scholarly information, possibly or primarily by refining an understanding of the
processes and models of electronic publication;
The project will be an educational process for
all concerned. Most MJA authors and reviewers will be having their first introduction to
electronic publishing, combined with a new opportunity to observe the processes of editorial
selection and presentation. Assessment of the outcomes will provide answers to several
modelling questions (Who in the academic community uses electronic publishing? What do
they require or expect of the medium? How effective is the interaction of readers and authors
in electronic publications of this kind? etc). - to permit a managed transition to electronic
publication, which addresses issues such as the retention of intellectual property rights and the
maintenance of appropriate quality measures like peer review;
The maintenance of appropriate
quality measures is the central concern and strength of this project. A successful outcome to
this experiment will establish a transition path for scholarly publication from print to electronic
form. The MJA is currently committed to developing an electronic journal (eMJA) as a
complementary extension of the printed publication. The model the project looks towards is
not an electronic replica of the printed journal but uses the immediacy, flexibility and
interactivity of the new medium to enhance scholarly communication. The project is designed
to show a way forward that augments quality measures like peer review. - to investigate
conventions, standards, protocols, costs, archiving and best practice applicable to electronic
publishing.
Combined, the University of Sydney Library and the MJA have wide-ranging
expertise in existing and developing standards and protocols for scholarly publishing. The
project design includes substantial preparation and evaluation periods for the documentation
of practices and outcomes. A conference of authors, reviewers, MJA and university staff is
envisaged to draw up consensus recommendations, based on the results of the study,
regarding electronic open peer review and scholarly publication.
Back to contents - A working protocol for open
online peer review that will be useful to other journals.
- A development path for scholarly
journals from printed publication to complementary printed and electronic publications. For
the MJA, for example, this may mean in future that the print MJA can publish a wider range of
short reports while the detailed reports are published electronically; other advantages of
complementary electronic and printed publication may also be developed.
- Opening peer
review to wider scrutiny should lift the standard of peer review by encouraging higher
performance from reviewers and providing new feedback on their performance.
- Open peer
review may improve the quality of published articles.
- Authors should be satisfied with
faster and wider publication to the international research community and a potentially higher
standard and broader range of peer review. This may lead to a higher standard of contributions
to the MJA.
Back to contents
It is widely imagined that the publication of research will move to electronic format, and there
are some developments in this direction. These have not yet made appreciable differences to
the academic community, who still see printed publication as the "gold standard" of academic
achievement, or to the major scholarly journals, who have not discovered a bridge to the new
medium. This project promises to create a bridge that will advance the interests of both journal
and author, because it is directed at the peer review process which is central to scholarly
publishing. If the experiment succeeds and becomes the future practice of the MJA (and other
journals), the process of peer review becomes more public, direct and interactive, electronic
publication becomes, over time, more accepted as a "citable" academic contribution, and the
best role of both print and electronic journals can develop in response to the observed
preferences of readers. The project is practical and definite, addresses "real world" issues for a
major scholarly journal, and will produce workable answers to several technical and
procedural questions in the path of electronic scholarly publishing.
|