eMJA: Draft protocol for Internet Peer Review Study II

Protocol for Internet Peer Review Study II

The MJA launched this study on 8 October 1998. The protocol was published online for comment and revision before the study began, and remains available for the information of participants and others.
Last revision to protocol: 8 October 1998.

To express an interest in participating in the study as a reviewer or consultant, or to make a comment, write to: Internet peer review study coordinator, Craig Bingham, by email to cbingham at ampco dotcom dotau

Postal address: The Medical Journal of Australia, Private Bag 901, North Sydney, NSW 2059, Australia
Fax: +61 2 9562 6666

For information about our previous Internet peer review study, click here

To print the protocol in full,you must print three files: iprs2int.html [the executive summary], iprs2bod.html [main protocol document] and iprs2doc.html [various associated documents].
This page is iprs2bod.html.

    Executive summary
Objective
Outline of new peer review model
Hypotheses
Rationale
Methods
    Eligible articles
    Process
    Study size, duration and control groups
    Anonymous participation
    Enlisting consultant panels
    Technical procedures
    Outcome measures
Associated documents
    Invitation letters
    Information about the Internet Peer Review Study II
    FAQs
    The etiquette of online reviewing
    Rules for participants
    Consent forms
    Evaluation of peer review process questionnaires
    Reviewer performance scoring form
    Statistical reviewer performance scoring form

Objective

To assess the acceptability, workability and effectiveness of using the Internet for a new model of peer review.

Outline of new peer review model

Articles submitted for publication are circulated to reviewers via the world wide web (using password-protected access) and the review process is conducted as an online discussion between the journal editors, reviewers, authors and a small panel of consultants who represent a wider range of expertise and journal readership.
When an article is accepted for publication, both the article and the record of its review process are rapidly published on the world wide web for open review by the Journal's readers. After four weeks of open review, the article is finalised and published in print.

Hypotheses

This model of peer review will be preferred by authors and reviewers.
The panel of consultants will add useful information to the review process.
The open review period for accepted articles will generate a small but useful amount of comment from readers.
The editorial time and resources involved in this review process will be similar to that required for the standard review process.
Reviewers will give better quality reviews.
Articles published after this review process will have fewer errors and be of higher quality than those published after standard review.

Rationale

The editorial processes by which research articles are selected and prepared for publication play an important role in shaping the medical research literature. Yet, compared with other stages in the research process, they are relatively unscientific and unaccountable.
Editorial peer review in the past has been secretive (because only the Editor can see the whole process), narrow (because each article is usually reviewed by only one, two or three people) and somewhat arbitrary (because the choice of reviewers is limited by the Editor's knowledge and the time constraints of the potential reviewers).
The new model to be tested in this study attempts to address some of these theoretical weaknesses. The procedure integrates peer review with the electronic publication process, allowing rapid publication after rigorous peer review. It preserves the chief virtue of the traditional peer review system (expert appraisal before publication), but brings a new openness to the procedure that we hope will enhance its accuracy and fairness.
In this model, prepublication review and editorial decisionmaking are conducted under the eyes of the authors and a consultant panel. The two commissioned reviewers continue to carry the traditional responsibility of ensuring that the article is comprehensively reviewed; the extra participants may provide broader perspectives, catch other errors in the article, correct deficiencies in the reviews and smooth potential biases.
The openness of the dialogue is a more democratic model of peer review, which might improve the authors' satisfaction with the procedure. All participants are better informed about the impact of their comments and the shape of the process as a whole (no longer is the Editor the only participant with a proper overview). This requires the Editor to make decisions out in the open, which might be unnerving, but the hoped-for gains are better advice, demonstrated accountability and, ultimately, an improved service to readers through providing better articles.
The open review stage after electronic publication allows readers to observe and contribute to the review process, and provides a final "check for errors" before the article is fixed in its archival form.

Methods

Eligible articles
Eligible articles are research and review articles that have been submitted to the Journal for publication and judged by the Editorial Committee to be suitable for peer review. This excludes letters to the Editor, case reports, editorials, other articles specifically commissioned for Journal series, and all articles that the editors reject without peer review.
We plan to include 50 articles in the study. Only half of the eligible articles (randomly selected) will be considered for entry into the study -- this is simply for administrative reasons, to reduce the number of articles in the study at any one time.
Details of control groups are given in "Study size, duration and control groups".

Process
1. Article submitted for publication is considered suitable for peer review.
2. Editorial committee selects reviewers for the article according to established procedures (ie, before they know whether the article will be going into the study). Committee also selects managing editor for the article (who will moderate the online discussion if the article enters the study) and the copyeditor for the article.
3. Randomisation: The editorial committee draws a lot to determine whether authors and reviewers will be invited to participate. Randomisation is performed after the editorial committee has selected reviewers and managing editor to prevent this choice being influenced by the knowledge of whether the article is for the study.
4. Authors and reviewers invited to participate in study (see "Invitation letter", and "Information about the Internet Peer Review Study"). Each article can only enter the study if all participants agree, but we record the intention to participate (or reasons for not participating) of both authors and reviewers. Both authors and reviewers can participate anonymously if they wish, but most will probably elect to be identified (judging by the experience of our first study). See the note on "Anonymous participation" below.
5. (a) If authors and/or reviewers do not agree to participate, the article enters the standard review process. Reviewer performance, time to publication and author satisfaction with review process are monitored for comparison with outcomes in new model.
(b) If authors and reviewers agree to participate, a panel of about six consultants are also invited to participate. This panel represents a wider range of expertise and journal readership (see "Enlisting consultant panels")
Example 1: Article is about management of patients with self-poisoning; a toxicologist and an epidemiologist are chosen as reviewers; a general practitioner, psychiatrist, doctor-administrator, drug and alcohol specialist, psychologist and emergency care physician are chosen for the panel.
Example 2: Article is about liver transplantation; a surgeon and a hepatologist are chosen as reviewers; a general practitioner, medical ethicist, paediatrician, immunologist, epidemiologist and medical economist are chosen for the panel.
Example 3: Article is about general practitioners' attitudes and practices in providing nutritional advice; a general practice academic and a statistician are chosen as reviewers; a general practitioner, dietitian, general physician, psychiatrist, endocrinologist and adult education specialist are chosen for the panel.
Alternative approach is to deliberately "shadow" the expertise of the reviewers -- ie, to pick a second expert from the same field as each reviewer (providing peer review of peer review).
6. Article is placed on a closed website. Access by password is restricted to managing editor, copyeditor, authors, reviewers and consultant panel. All participants are given instructions in how to participate (see "FAQs", "Etiquette" and "Rules"). For an outline of technical procedures for handling files and web site access, see "Technical procedures".
7. Review process conducted as online discussion:
Managing editor moderates the discussion. The editor may ask participants to consider specific issues, or to stop a certain line of debate. When necessary, the managing editor can delete inappropriate remarks or block a participant's access to the discussion. The managing editor consults with the rest of the editorial committee to reach editorial decisions. The managing editor introduces and concludes the main stages of review: (1) Discussion of the first submission, concluding with the editor's decision to reject, or definitive instructions for revising the article (3-4 weeks);
[End of the line for rejected papers. Or, interval: No discussion while authors revise article]
(2) Discussion of the authors' revision, concluding with the editor's decision to accept or reject, or (rarely) to revise again (made in consultation with the Editor in Chief) (1-2 weeks);
[End of the line for rejected papers. Or, for accepted papers: Interval: No discussion while copyeditor edits article]
(3) Discussion of the copyedited article, concluded when the copyedited article is finalised and approved by the authors (one week);
See "FAQs", "Etiquette" and "Rules") for more on the bounds and purposes of discussion at each stage.
Commissioned reviewers post their reviews to the discussion list associated with the article. Although these reviews will normally be the longest contributions to the review discussion, they do not have to be the first.
Consultant panel observes and contributes (see "FAQs", "Etiquette" and "Rules")
Authors observe the discussion of their paper and can respond to comments with explanations, alternative arguments or proposals for revising their article. The authors contribute a complete revision of their article if and when this is requested by the managing editor. If the article is accepted for publication, the copyedited version is posted to the website for the authors' approval.
Copyeditor prepares accepted articles for publication (ie, edits the authors' revision to ensure clarity, readability and conformance with Journal style). The edited version is presented online for the approval of the authors.
Content review committee -- this standing committee of advisers to the Editor have a watching brief over the Journal's content, and may be invited by the Editor to join the discussion of an article.
8. After revision, acceptance and editing, the article and its associated discussion list are published for open review. Readers can contribute comments; open review is concluded when the authors and editor agree that no further changes are desirable (four weeks).
Technical note: Readers' comments are e-mailed to the editorial office for review before they appear on the web site, not automatically added to the discussion list.
9. Article allocated to a print-issue of the Journal and published in print.

Study size, duration and control groups
Because of the amount of work involved, the study will be limited to 50 participating papers and two control groups:
The 50 papers received immediately before the study period that would otherwise have met the eligibility criteria. This control group will provide baseline data about the operations of traditional peer review.
The papers received during the study period that did not go into the study because the authors or reviewers did not wish to participate. If this group exceeds 50 papers, only the first 50 will be analysed. This control group will allow us to assess differences between the papers/authors/reviewers who participated and those who did not.
Timing:
Phase 1, collecting baseline control group, 4-12 weeks. During this time also collect volunteers for consultant panels (see "Enlisting consultant panels") and make other preparations for Phase 2 (such as training staff in procedures).
Phase 2, collecting study group and non-participant control group: 12-26 weeks
Phase 3, all published articles followed up (for comment in letters to the editor) until two months after publication in print.
Endpoints:

Rejected papers: return of authors' evaluation of the peer review process.
Accepted papers: two months after printing (time to monitor response to articles expressed in letters to the editor and authors' responses).

Anonymous participation
Standard peer reviewing at the MJA is "double blind" -- author-identifying information is removed from manuscripts, and reviewers do not sign their reviews.
In our first study, reviewers of accepted articles were asked to allow their reviews to be published with the articles for an open review period on our web site. Of the 90% who agreed to participate, almost two-thirds decided to sign their reviews.
It is probably desirable that authors, reviewers and the consulting panel identify themselves to each other (it encourages trust and personal responsibility, and reveals information that may identify conflicts of interest). Our first study suggests that most will want to do so. Nevertheless, as there are few data about the impact of anonymity or naming in this context, we have decided to give participants the choice and observe the outcome.
All participants who opt for anonymity in the closed review stage will be aware that their identities are known to the editor. The editor's interface to the discussion list, unlike that of other participants, shows the full identity of each contributor (others viewing the discussion list see only the name or nickname determined by the participant).
In the open review stage (for accepted articles only, after publication), anonymous comments can be emailed to the editor, who will consider them on their merits. (In our first study, almost all readers' comments were signed.)

Enlisting consultant panels
The "consultant panel" invited to participate in the review discussion for each article will represent

similar expertise to that of the reviewers, and/or

representatives of the journal readership and/or

representatives of expertise in a field relevant to the article that is not represented by the reviewers.

Required numbers: Only invited consultants will be able to participate in the discussion of an article, and, generally speaking, at least three of the six consultants used for an article should not have been used before. This suggests that for 50 articles in the study we will require about 200 volunteers (over four months).
Recruitment: Although it would be possible to recruit all the consultants from the journal's database of reviewers, it is desirable to look further afield. To find suitable volunteers, we will:

Place ads in the Journal and on the Journal's website
Publish notices in other journals.
Write to specialty colleges, hospital and university departments in medicine and health-related disciplines.

Volunteers from outside the reviewer database will be asked to submit their names and information about their qualifications and interests. This information will be entered in the eMJA's database of registered users, to make up a searchable list.
Making up a panel: To minimise the administrative work involved, potential members will be invited to participate, but not followed up. Instead, eight potential panellists will be invited, with the intention of securing six active participants (we'll fine-tune this procedure during the study).
MJA Content Review Committee: Members of this standing committee of external advisors to the Editor may be invited to participate as extra members of consultant panels from time to time.
Technical procedures
Manuscript processing
In addition to the hard copy, authors will be asked to supply their articles, including figures, in electronic format. The wordprocessor files will be converted to rtf (rich text format) so that an automatic rtf-to-html conversion can be used.
Instructions to authors for formatting electronic files:
If possible, use Word or Wordperfect's styles to format headings, and keep it simple; eg, use only Heading 1, Heading 2, Heading 3 and Normal text.
Place tables and figures within the text at the most appropriate location (ie, not all together at the end, but close to where they are mentioned).
Also send a copy of each figure as a separate graphics file.
Provide a covering note that details the names, contents and formats of the files.
If in doubt about our requirements, please seek advice from the editorial office.
Handling of photographs or diagrams not provided in electronic format: These will be scanned in from hard copy at the editorial office.
Conversion to html: Will be done using an rtf to html converter. Copyeditor will check and correct the file for consistent style before converting.
Copyediting of accepted articles: Metadata, site links and other information required for publication will be added at this stage, figures that would normally be redrawn for publication will be redrawn, and photographs will be sent to the bureau for high-resolution scanning.
Preparation of printed copy: A conversion macro will convert the html file to a QuarkXpress style-tagged ASCII file, which will then enter the normal production routine.
Web access for prepublication review
Articles will be placed online in a site accessible only with a password. Password control is as follows:
Each editor and copyeditor will have an administrative password. This will enable them to access all articles and discussion lists, and the administrative web forms used for creating and editing discussion groups.
Each author, reviewer and consultant panel member will be entered in the eMJA registration database (see "Registration form", www.mja.com.au/registration/register.html). Registrants can access this form themselves to complete their personal details and set their own password (but they cannot access the administrative forms required to set their access rights: this is controlled by the editor).
To enter an article into online review, the editor (or study coordinator):
Places the article files in a secure directory on the server and runs the paragraph numbering utility (this numbers all the paragraphs in the article for reference).
Opens the web form used for searching the registration database and chooses reviewers and panel members. Reviewers will have already been contacted and agreed to do the review; panel members will have volunteered to be involved in the study and nominated their interest areas, but will not yet have been told about this particular article.
Opens the web form used for creating a new discussion group, creates a group for the article under review and adds the participants' names to the group. This gives the participants the right to access the article and its discussion.
Clicks on the "send message" button to inform all participants by email that the article is available for review.
Conducting the review discussion online

A link from each article leads to its associated "Discussion list". The discussion list and the article can be viewed simultaneously by opening two browser windows.
To add a comment to the discussion list, a contributor clicks on the "Add comment" link, opening a text window. Comments can be typed directly into this window or pasted from a word processor. Comments can be signed with a real name or nickname, or unsigned (but all contributors will be aware that their identity is always known to the editor). When "submit comment" is clicked, the comment is bounced back from the server and the contributor is invited to edit and confirm their words (a chance for second thoughts). When "submit comment" is clicked again, the comment is added to the discussion list and can be viewed by other participants.
Ordinary participants see comments with the signature submitted by the contributor (ie, perhaps anonymous or under a nickname).
The editor has an administrative view of the discussion list, in which contributors' real names are also shown. The editor's view also has an "Edit" link next to each comment, which allows the editor to edit or delete comments.
Comments that include the words "par x", "paragraph x", "par.s x-y" and so on will automatically link to the corresponding numbered paragraphs in the article.
The discussion list is an html document and will support all html formatting. This can be entered by savvy contributors or added later by the editor (for example, to add extra links, or to bold a certain phrase).
Handling authors' revisions
This can be done in several ways, depending on the level of revision and the authors' competence in html formatting.
Minor revisions
Author begins by noting the revisions in the discussion list. Then

the authors download the article file, make changes and email it back to the editor to be uploaded again, or
the copyeditor makes the changes for the author.

Major revisions
Author begins by noting the revisions in the discussion list. Then

the authors download the article file, make changes and email it back to the editor to be uploaded again, or
the authors submit a new wordprocessor file of the article to be converted into html.
Archiving previous versions
When a revision is received, it will replace the original, which will be renamed "[article].ori.html". Existing links will therefore now go to the revised article, which will carry the label "Revised at [date]" and include a link to the previous version.
If a second revision is required, the files will be "[article].html", "[article].1rev.html", "[article].ori.html".
Copyediting the article
When an article is accepted for publication, the copyeditor will edit the html file to produce a final version for web publication. The copyedited proof will be viewed online by the authors (and other review participants).
As with revisions, the copyedited version will take the name "[article].html", with a link back to the previous version.
Publishing the article
To publish an article that has been accepted, edited and approved by the authors, the directory that holds the article, the old versions of the article, and the discussion list will be moved from the private area of the web site to the publicly accessible area.
Outcome measures
See "Associated documents" for most of the forms used to collect and record outcomes data. Most data are also recorded for control groups to provide comparisons.
Participation (measures of the acceptability of the new model to authors and reviewers)

Number of participating authors v. number invited who declined to participate.

Number of participating reviewers v. number invited who declined to participate and number invited who declined to review at all.

Number of participating authors/reviewers/panel members who elect to be anonymous.

Number of participating panel members v. number invited who did not participate.

Number of participating panel members who contributed/did not contribute comments to review process (ie, access data compared with number of comments -- measure of silent participation).

Number of times editors/authors/reviewers/consultants contributed to each discussion (compared with access data to measure silent participation).

Proportion of reviewers among study participants who opt to be anonymous to authors, and proportion of authors who opt to be anonymous to reviewers.

Workability (measures of the efficiency of the process)

Amount of time spent:
from start of review process to editor's request for revision
from request for revision to submission of revision
from submission of revision to editor's decision to accept or reject
from acceptance to final copyediting
from acceptance to electronic publication
from acceptance to print publication

Volume of correspondence contained in the peer review process

Data from authors and reviewers about workability (elicited by questionnaire)

Effectiveness (measures of the quality of the new review process in comparison with the old)

Authors' evaluations of peer review process

Reviewers' evaluations of peer review process

Editors' assessments of peer review quality and evaluations of peer review process

Number and nature of interventions (comments, questions) by consultants, and number and nature of those that result in changes to papers.

Number and nature of interventions by readers during open review stage, and number and nature of those that result in changes to papers.

Number and nature of comments on articles received after printing as letters to the editor (may indicate failures of peer review).

Measure of dissonance between reviewer 1, reviewer 2 and Editor (control groups compared with study group) (Categorise opinions on article as strongly critical/critical/neutral/supportive/strongly supportive). In standard review, each reviewer gives one opinion uninfluenced by others; in the new model, there is the possibility for dialogue and example to influence opinions (perhaps towards a consensus?).

??-Readers' evaluations of the quality of papers before review/after traditional review/after new model review. To do this would require a large group of readers prepared to read and score several papers [can we do it?]

Measures to assess possible biases (in comparing the participating papers with the control group of non-participating papers)

Subject of paper

Age, sex, specialisation, location, computer literacy and computer access of author or reviewer

Final outcome of paper (easy reject, hard reject, hard accept, easy accept) [Easy reject = more than one reviewer and editors agree that paper is not suitable for publication. Hard reject = editors decide on rejection despite support of at least one reviewer for publication and/or strong representations from the authors that their paper merits publication. Hard accept = major revisions required to meet the requirements of editors and reviewers for publication. Easy accept = minor revisions only required to meet requirements for publication.]

Acceptance rate in new model v. acceptance rate under traditional review.

On to associated documents ...
Back to top of the protocol document ...