|
Protocol for Internet Peer Review Study II
The MJA launched this study on 8 October 1998. The protocol was published online for comment and revision before the study began, and remains available for the information of participants and others.
Last revision to protocol: 8 October 1998.
To express an interest in participating in the study as a reviewer or consultant, or to make a comment, write to: Internet peer review study coordinator, Craig Bingham, by email to cbingham at ampco dotcom dotau
Postal address: The Medical Journal of Australia, Private Bag 901, North Sydney, NSW 2059, Australia
Fax: +61 2 9562 6666
For information about our previous Internet peer review study, click here
To print the protocol in full,you must print three files: iprs2int.html [the executive summary], iprs2bod.html [main protocol document] and iprs2doc.html [various associated documents].
This page is iprs2doc.html.
Associated documents
Invitation
letters |
Dear Author,
Thank you for submitting your article " " to the Journal. In an effort to
improve our peer review and editorial processes, the Journal is
currently testing a new model of peer review using the Internet and I
would like to invite you to participate in the study. Please read the
description of the Internet Peer Review Study II, which includes a
list of potential advantages and disadvantages of the new model,
before completing the attached form indicating your response to the
invitation.
Authors who participate in this study will be making an important
contribution to research on peer review. They will also have an
unprecedented opportunity to observe and participate in the peer
review and editorial decision-making process. Articles accepted
after this process will be rapidly published on the Internet (as well
as being published in print).
If you do not wish to participate in the study, your article will be peer
reviewed and (if accepted) published according to the standard
procedures of the Journal.
Whether or not you wish to participate, please return the form as soon
as possible in order to avoid delays in processing your article.
Best wishes,
Martin Van Der Weyden
Editor
_________________________________________
Dear Reviewer,
We have received a [research article / review article] entitled "..."
to consider for publication in The Medical Journal of Australia.
Would you be able to [review it/provide a statistical review] for us
within the next three weeks? We have enclosed the [abstract/first
page] to give you an idea of the content of the manuscript, and will send
the whole manuscript if you accept.
If you do wish to review the article for us, you now have a choice about
how to do so. In an effort to improve our peer review and editorial
processes, the Journal is currently testing a new model of peer review
using the Internet. Please read the attached description of the
Internet Peer Review Study II.
Reviewers who participate in this study will be making an important
contribution to research on peer review.
If you do not wish to participate in the study, we would still be pleased
to receive your review of the article in the normal way.
If you do not wish or are unable to review the article, we would greatly
appreciate a suggestion for an alternative reviewer.
Whether or not you wish to participate in the study or review the
article, please fax back the form within the next three days so that the
article can be processed without delay.
Best wishes,
Martin Van Der Weyden
Editor
|
Information about the MJA's Internet Peer Review Study-II (for
authors and reviewers)
| |
Objective
To assess the acceptability, workability and effectiveness of using
the Internet for a new model of peer review.
Outline of new model
Articles submitted for publication are circulated to reviewers via
the world wide web (using password-protected access) and the review
process is conducted as an online discussion between the journal
editors, reviewers, authors and a small panel of consultants who
represent a wider range of expertise and journal readership.
When an article is accepted for publication, both the article and the
record of its review process are rapidly published on the world wide
web for open review by the Journal's readers. After four weeks of open
review, the article is finalised and published in print.
Potential benefits
- more open processes may reduce the potential for bias or error in review and editorial decision-making
- more immediate dialogue between participants may improve feedback and opportunities to learn from the process
- faster path to publication for accepted articles
For more information about the study, see
http://www.mja.com.au/public/information/iprs2int.html
Or speak to the Study Coordinator, Craig Bingham (+61 2) 9562 6666
|
FAQs for reviewers
| |
Can I participate without using the Internet?
No. You must be able to send and receive email and access the world wide
web to participate in this study.
How do I actually submit a review under this new system?
If you agree to participate, we will send you more detailed
instructions. Basically, you access the article and its discussion
list through the world wide web. Access to the site is restricted, and
you must use your password to enter. Your review can be written
directly into the web form or (more likely) written in your usual
wordprocessor and pasted into the web form.
What else do I have to do?
You do not have to contribute anything beyond your review, but we hope
that you will follow the review discussion of the article and
contribute to it as seems appropriate.
We shall also send you a follow-up questionnaire (very short!) to seek
your opinion of the new model after you have experienced it.
Can I still do my review anonymously?
Yes. We are encouraging all contributors to sign their comments, but
you don't have to reveal your identity to the authors and other
reviewers if this will inhibit your review. Your identity will of
course be known to the Editor.
What are the potential advantages or disadvantages of this system for
me?
Through the discussion list, you will be able to communicate directly
with the authors and the other reviewers of the article. This may make
it easier for you to ask questions and explain your comments. You will
also have a clearer idea of the impact of your comments, through the
feedback provided by authors, editor and possibly other reviewers.
If you wish, you will be able to make follow-up comments, and watch the
article as it develops through revisions and copyediting into a
published paper.
This discussion list procedure could take more of your time (but this
is under your control).
|
FAQs for consultant panel members
| |
Can I participate without using the Internet?
No. You must be able to send and receive email and access the world
wide web to participate in this study.
What do you actually want me to do?
We want you to follow the review discussion of the article and
contribute to it as seems appropriate. At least two other reviewers
have been asked to provide detailed reviews of the article -- you'll
get to read these. We hope that you can add extra advice from your own
expertise. Your comments should be brief. We don't mind if you do
nothing more than observe.
We shall also send you a follow-up questionnaire (very short!) to seek
your opinion of this review process after you have experienced it.
What sort of comments do you want?
The purposes of the review process are (1) to help the editor decide
whether to publish the article and (2) to help the authors improve
their article. All comments should be directed towards serving these
purposes. This might mean commenting on the reviews of the article, if
(for instance) you thought the reviewer was incorrect or unfair in a
criticism.
Can I participate anonymously?
We are encouraging all contributors to sign their comments, but
you are not required to reveal your identity to the authors and other
reviewers if you don't want to. Your identity will of course be known to
the Editor.
Can I tell others about the article?
No. While under review the article is a confidential document. You
will be granted access only on the understanding that you will not
discuss or share the article with anyone outside the online review discussion.
|
FAQs for authors
| |
Can I participate without using the Internet?
No. You must be able to send and receive email and access the world wide
web to participate in this study. It is helpful, but not necessary, if
you have some knowledge of how html and the world wide web work.
How do I enter my article for review under this new system?
If you agree to participate, we will send you more detailed
instructions. Basically, we will prepare an html-formatted copy of
your article from your wordprocessor file. You will be able to access
the article and its discussion list through the world wide web. Access
to the site is restricted, and you must use your password to enter. Only
the authors, reviewers, journal editors and a small panel of
consultants will be able to access your article during review. They
will all have agreed to treat your article as a strictly confidential
document.
What else do I have to do?
We hope that you will follow the review discussion of the article and
contribute to it as seems appropriate. Reviewers, editor and
consultant panel members will add their comments to the discussion
list. You can respond with comments, questions or proposals for
revising the article.
If the editor asks you to revise your article, the revision will be
discussed on the web by the same group of people. If the article is
accepted for publication, the copyedited version will be prepared on
the web, approved by you, and then published. It will appear on the web
before it appears in print, and probably six to ten weeks earlier than
it would have appeared after standard peer review and editorial
processing.
We shall also send you a follow-up questionnaire (very short!) to seek
your opinion of the new model after you have experienced it.
Can I have our article reviewed anonymously?
If you wish, your identity as authors can be masked so that the
reviewers and consultants do not know who you are or where you are from
(you may think that this will increase the chances of an unbiased
review).
Are reviewers anonymous?
We are encouraging all contributors to sign their comments, but it
is possible that some reviewers will submit comments anonymously.
The identity of all participants will be known to the Editor, even when
it is not revealed to the authors.
What are the potential advantages or disadvantages of this system for
me?
Through the discussion list, you will be able to communicate directly
with the reviewers, editors and consultants who are commenting on
your article. This may make it easier for you to ask questions and
explain your position. You will also have a clearer idea of the impact
of your article, and of the processes by which the editor reaches a
decision on your article.
This discussion list procedure could take more of your time (but this
is under your control).
What happens if my article is rejected?
If your article is not accepted for publication, you will be able to
offer it for publication elsewhere if you wish. All the participants
in this Internet peer review process have promised to treat your
article in confidence.
|
The etiquette of online reviewing (notes for participants)
| |
The purpose of this online discussion is to review the article.
This is a service for the editors of the journal (who need advice
about what to publish), for the authors of the article (who may benefit
from comments designed to improve their paper), for the readers of the
Journal (who rely upon the Journal's quality control procedures),
and ultimately for the advancement of scientific medicine.
Remember that the purpose is to review the paper, not to discuss the
subject in general, so stick to the point. Avoid lengthy
contributions that reiterate arguments already made, but by all
means send a short note to signal your agreement with a point made by
someone else. Feel free to indicate whether you think the paper should
be published, but remember that the editor's decision is final.
Remember that your comments can immediately be read by others and that
your identity is known at least by the editor. Please be aware that a
written dialogue can sometimes appear more abrupt than the spoken
word: consider this when phrasing your own contributions and do not be
hasty to take offence at critical contributions made by others.
Appropriate comment at each stage
Errors of fact, plagiarism, or any evidence of research
misconduct are worthy of comment and correction at any stage.
First submission: Comments at this stage may be quite
wide-ranging. Is the article accurate? Is it original? Is it
important? Does the introduction adequately explain why the study
was done? Was the study conducted with appropriate methods, and are
these methods adequately described? Are the results accurately and
adequately described? Does the article's discussion canvass the
important issues raised by the results in an accurate and impartial
manner? Are the authors' conclusions reasonable and justified by the
data presented? Are appropriate references cited and is previous
work properly acknowledged?
Should the article be published, and if so, what revisions would
improve it?
Revised article: Comments should be more closely
focused. Have the authors addressed the issues mentioned in the
editor's request for a revision? In doing so, have they raised any new
issues requiring consideration?
Copyedited version: Comment at this stage should be
tightly focused. Has copyediting improved the readability of the
article without introducing any errors?
|
Rules for participants in open peer review
| |
- The purposes of peer review are to advise the editor on the suitability of the article for publication and to advise the authors on ways that they may be able to improve their article. All contributions to the review process should serve these purposes.
- The editor reserves the right to make a publication decision based on his/her own assessment of the best interests of the journal and its readers.
- The article under review is a confidential document not yet accepted for publication. It should not be shown to or discussed with colleagues, cited as a reference or used in your own work.
- The editor will chair the review discussion. Contributors should follow any instructions from the editor given during the discussion.
- Contributions to the review discussion should be as brief as possible. Contributors should not reiterate points that others have already made, although it is acceptable to send a note of agreement.
- References to the article under review should quote the relevant paragraph number.
- Assertions of fact should be backed with appropriate references.
- All contributors should conduct themselves with professional courtesy. Remarks that may be construed as insulting or defamatory will be removed from the record.
- Contributors who breach the required standards of the review process will be barred from further participation by the editor.
|
MJA Internet Peer Review Study II Authors' consent form
| |
Author's name:__________________________________
Manuscript
No.:___________________
Manuscript
title:_________________________________________________________________
Please indicate your age group:
__ Under 30
__30-39
__40-49
__50-59
__60 and over
Please indicate your use of Internet services: | Service | Daily
to weekly | Weekly to monthly | Occasionally | Never
|
| E-mail | | | | |
|
| World Wide Web | | | | |
|
PLEASE SIGN CONSENT TO PARTICIPATE
I, as principal author, having consulted my co-authors,
agree to participate in the study. I understand that
Our article will be made available to reviewers, journal editors and
a small panel of consultants via a secure web site on the Internet.
The review process will be conducted as an online discussion between
the reviewers, journal editors, authors and panel of consultants,
chaired by the managing editor for the article.
The authors' identities can be masked during this review process if
we wish, in which event only the editor will know who I am when I submit
comments to the discussion list.
During these proceedings, the article and the discussion are
confidential documents, not to be discussed with colleagues other
than those in the review discussion.
If the article is accepted for publication, the review discussion
will be moved to a publicly accessible web site, where it will be
available to interested readers.
If the article is not accepted for publication or is withdrawn by the
authors, the review discussion will remain confidential. It will be
removed from the web and a print-out will be stored in the MJA's
archives.
The managing editor reserves the right to remove inappropriate
comments from the discussion list or to withdraw the access rights of
any participant who does not act with professional courtesy and
responsibility.
I will be asked to complete a questionnaire seeking my impressions of
the value of the electronic publication and peer review process when
the article is prepared for print.
Signed: ____________________________________________
OR - SIGN REFUSAL OF CONSENT
I do not agree to participate in this study because
___ I do not have access to the Internet
___ Other reasons (please tell us
specifically:)___________________________________________
Signed: ____________________________________________
|
MJA Internet peer review Study II Reviewers' consent form
| |
Reviewer's name:_________________________________
Manuscript
No.:___________________
Manuscript under
review:__________________________________________________________
Please indicate your age group:
__ Under 30
__30-39
__40-49
__50-59
__60 and over
Please indicate your use of Internet services: | Service | Daily
to weekly | Weekly to monthly | Occasionally | Never
|
| E-mail | | | | |
|
| World Wide Web | | | | |
|
PLEASE SIGN CONSENT TO PARTICIPATE
I agree to participate in the study. I understand that
I will have to access the article and submit my review via a secure web
site on the Internet.
The review process will be conducted as an online discussion between
the reviewers, journal editors, authors and a small panel of
consultants, chaired by the managing editor for the article.
I can participate as an anonymous reviewer if I so wish, in which event
only the editor will know who I am when I submit comments to the
discussion list.
During these proceedings, I will treat the article and the
discussion as confidential documents, not to be discussed with
colleagues other than those in the review discussion, nor to be cited
as references or used in my own work.
If the article is accepted for publication, the review discussion
will be moved to a publicly accessible web site, where it will be
available to interested readers.
If the article is not accepted for publication or is withdrawn by the
authors, the review discussion will remain confidential. It will be
removed from the web and a print-out will be stored in the MJA's
archives.
The managing editor reserves the right to remove inappropriate
comments from the discussion list or to withdraw the access rights of
any participant who does not act with professional courtesy and
responsibility.
I will be asked to complete a questionnaire seeking my impressions of
the value of the electronic publication and peer review process when
the article is prepared for print.
Signed: ____________________________________________
OR - SIGN REFUSAL OF CONSENT
I do not agree to participate in this study because:
___ I do not have
access to the Internet
___ Other reasons (please tell us
specifically:)___________________________________________
Signed: ____________________________________________
|
Author evaluation of peer review process questionnaire
| |
Dear Author,
The MJA is evaluating its peer review and editorial procedures with
the aim of improving their quality and efficiency. Please help us by
completing this questionnaire and returning it to our office by fax or
post. If you find it more convenient to do so, you can complete the
questionnaire online at
http://www.mja.com.au/public/information/authorq.html
Name:
How many research/review articles before this one have you submitted
to peer review at the MJA or any other journal?
___ None
___ One to three
___ More than three
Please rate each item below on a scale of 1 (lowest, least
satisfactory) to five (highest, most satisfactory) and add any
comments that you would care to make.
| Fairness of the peer review process that your article has
undergone | 1 | 2 | 3 | 4 | 5
| |
Speed of the peer review and editorial decisionmaking process | 1 | 2 | 3 | 4 | 5
|
|
Usefulness of the advice given to you by the editors | 1 | 2 | 3 | 4 | 5
|
| Usefulness of the clinical reviews | 1 | 2 | 3 | 4 | 5
|
| (If relevant) Usefulness of the statistician's review | 1 | 2 | 3 | 4 | 5
|
If you have any suggestions for improving our editorial and peer
review processes, please write them here:
[Questions above are sent to study participants and controls.
Additional questions for authors in the study:]
In what ways was the new method of conducting peer review better
than the traditional method?
In what ways was the new method of conducting peer review worse than the
traditional method?
Did the new method of conducting peer review change the amount of work
required of you as an author?
___ Much more work
___ A little more work
___ No change
___ A little less work
___ Much less work
If you were submitting another paper to the Journal, how would you
prefer it to be reviewed?
___ New model
___ Traditional model
|
Reviewer evaluation questionnaire
| |
Dear Reviewer,
The MJA is evaluating its peer review and editorial procedures with
the aim of improving their quality and efficiency. Please help us by
completing this questionnaire and returning it to our office by fax or
post. If you find it more convenient to do so, you can complete the
questionnaire online at
http://www.mja.com.au/public/information/reviewerq.html
Name:
Please rate each item below on a scale of 1 (lowest, least
satisfactory) to five (highest, most satisfactory) and add any
comments that you would care to make.
|
Guidelines provided to reviewers | 1 | 2 | 3 | 4 | 5
|
|
Communications from the editorial office | 1 | 2 | 3 | 4 | 5
|
| Feedback provided to reviewers | 1 | 2 | 3 | 4 | 5
|
| Quality of the copy (ie, the appearance and readability) of the
article you were asked to review | 1 | 2 | 3 | 4 | 5
|
If you have any suggestions for improving our editorial and peer
review processes, please write them here:
[Questions above are sent to study participants and controls.
Additional questions for reviewers in the study:]
In what ways was the new method of conducting peer review better
than the traditional method?
In what ways was the new method of conducting peer review worse than the
traditional method?
Did the new method of conducting peer review change the amount of work
required of you as a reviewer?
___ Much more work
___ A little more work
___ No change
___ A little less work
___ Much less work
If you were reviewing another paper for the Journal, how would you
prefer to do it?
___ New model
___ Traditional model
|
Reviewer performance scoring form
| |
|
Timeliness Score | |
|
Review given within 3 weeks (ie, within agreed deadline): | 2 |
|
Review given within 1 week after deadline: | 1 |
|
Review given more than 1 week after deadline: | 0 |
|
Grade sheet | |
|
Grading forms completed: | 1 |
|
Not completed: | 0 |
|
Etiquette | |
|
Reviewer follows appropriate reviewing etiquette: | 1 |
|
Reviewer breaches appropriate reviewing etiquette: | 0 |
|
Completeness of review | |
|
Reviewer reviews entire article, with specific mention of all
sections, including tables, references and graphics: | 5 |
|
Review includes overview of content of article
and addresses main sections of article specifically | 4 |
|
Review includes overview of article but does not address
all sections of the article specifically | 3 |
|
Review appears to concentrate on particular features of the
article without considering it in its entirety, or considers
the article in its entirety but without adequate attention to
specific details | 2 |
|
Review addresses specific sentences in the article without
placing these in context, or discusses the article in such
general terms that it is purely superficial | 1 |
|
Review is too brief and non-specific to indicate that the
reviewer has given adequate consideration to the article | 0 |
|
References | |
|
Review offers supporting references (specific citations or
copies
of references are attached) | 2 |
|
Review mentions supporting references (incomplete citations,
eg, author name only) | 1 |
|
Review offers no supporting references | 0 |
|
Usefulness of review | |
| Reviewer has provided a clear and useful evaluation of the article,with specific recommendations for the improvement of content and analytical method and language and graphical representation and references | 5 |
| Reviewer has provided a clear and useful evaluation of the article,with many specific recommendations for the improvement of content, analytical method, language, graphical representation and references (but not for all of these categories) | 4 |
|
Reviewer has provided a clear and useful evaluation of the article, with some specific recommendations for its improvement or a reasonable explanation of why the article does not need improvement | 3 |
|
Reviewer's evaluation of the article is useful, but in parts
indecisive or vaguely worded, making it difficult to interpret
or follow the advice. Or reviewer has given insufficient detail to
support the recommendations | 2 |
|
Reviewer's evaluation is marred by errors of comprehension of
the article or the subject or by bias or by poor language and
presentation. Or review discusses the article and gives some
specific advice without providing a useful evaluation of the
whole article | 1 |
|
Review is seriously marred by errors of comprehension of
the article or the subject or by bias or by poor language
and presentation. Or review provides a cursory evaluation
of the article unjustified by proper argument and
citations | 0 |
|
Total | 16 |
|
Statistical reviewer performance scoring form
| |
|
Timeliness Score | |
|
Review given within 3 weeks (ie, within agreed deadline): | 2 |
|
Review given within 1 week after deadline: | 1 |
|
Review given more than 1 week after deadline: | 0 |
|
Grade sheet | |
|
Grading forms completed: | 1 |
|
Not completed: | 0 |
|
Etiquette | |
|
Reviewer follows appropriate reviewing etiquette: | 1 |
|
Reviewer breaches appropriate reviewing etiquette: | 0 |
|
Completeness of review | |
|
Reviewer reviews all statistical/numerical elements of article
and gives specific comments on each | 3 |
|
Review includes good overview of statistical/numerical elements
of article and addresses main features specifically | 2 |
|
Review provides only specific comments on some of the
statistical/numerical elements of the article | 1 |
|
Review is too brief and non-specific to indicate that the
reviewer has given adequate consideration to the article | 0 |
|
Usefulness of review | |
|
Reviewer has provided a clear and useful evaluation of the
statistical
methods used in the article, with specific recommendations for the
improvement of data analysis and data presentation or a
reasonable explanation of why the article does not need
improvement | 3 |
|
Reviewer has provided some specific recommendations for
improving data analysis or data presentation | 2 |
|
Reviewer's evaluation makes some useful points but is
marred by failure to comment on important
statistical/numerical aspects of the article | 1 |
|
Review is seriously marred by errors of comprehension
of the article or the subject or by bias or by poor language
and presentation. Or review provides a cursory
evaluation of statistical/numerical aspects of the article
unjustified by proper argument and citations | 0 |
|
Total | 10 |
Back to top of the protocol document ...
|
<URL:http://www.mja.com.au/public/information/iprs2doc.html>
© 1998 Medical Journal of Australia.
|