Democratising assessment of researchers’ track records: a simple proposal

All researchers have experienced dismay at the failings of peer review.1,2 These include cursory or ill informed reviews from those apparently unschooled in applicants’ disciplines and with superficial understanding of the quality and significance of proposals, publications and achievements being described by aspiring applicants.

Only the greenest researchers assume that those who assign papers or grants to reviewers possess Solomonic wisdom in matching the expertise of researchers and reviewers. The reality is very different. Reviewer assigners rely on reviewer databases, particularly for areas beyond their specific expertise. It is here that many problems start. Some names are in these databases by virtue of having previously submitted a grant or paper, regardless of quality or success. Self-chosen research keywords may bear little correspondence to actual expertise. Many relatively inexperienced names appear and searches suggest many who have published little.

Grant review panels seek reputable reviewers but reviewer refusal means suboptimal reviewers are often used. No published information is available on grant review refusal rates, but data obtained from the BMJ Publishing Group for 300 866 review requests sent out by 20 journals between 2002 and 2009 show that 32.4% of those asked declined to review.3 Reviewer assigners can then resort to selecting names on the basis of reviewer supplied keywords, without detailed knowledge of the reviewer. Although this can sometimes produce superb reviews, it can also produce those that are confidently written but ill informed.

The core problem with current approaches to reviewer selection is that a small number of assessors are involved, typically two or three. Although it would be impractical to involve more people in grant assessments, the rating of track record — typically attracting at least 25% of overall score — could easily break free from the reputational lottery affected by the opinions of a few, sometimes ill informed, people that may affect the fate of a grant application.

There is an easily managed, low-cost way of democratising the rating of researcher track record that would end the practice of relying on a small number of sometimes inexpert peers rating applicants’ track records.

As part of a project examining the nature of researcher influence, we invited all Australia-based researchers in six research public health fields who had published 10 or more Institute for Scientific Information-indexed publications in the past 10 years to nominate up to five Australian researchers in their respective fields whom they considered to be the most “influential” researchers. Full descriptions of the selection of participants are provided elsewhere.4,5

Participation was high: 83% of invited eligible Australia-based researchers (175/211) completed the rating task, which took about 10 minutes. One hundred and eighty-two individuals across these fields received at least one nomination, producing a distribution of “votes” on researcher influence across these researchers. In five of the six fields, there was impressive consensus on which researchers were most influential, with a long tail of researchers receiving a smaller number of votes, and many none. In all six fields, a large number of researchers (108/182; 59%) received only one vote from all those voting. Of these, 48 (26% of all nominations) were self-nominations. Peer-influence rating generally correlated strongly with four commonly used bibliometric indices of research impact6 (the h, m and q2-indices and the m-quotient).7-10

The traditional approach to track-record ranking involves a small number of researchers being selected as reviewers by an often idiosyncratic process that reflects selector knowledge and preferences and reviewer self-nomination and availability. The existing process lacks transparency and probably provides biased results from an unrepresentative sample of reviewers. Our study demonstrated that the process can be opened up by seeking votes from all research-active peers in each relevant field. Most researchers agreed to participate and found the task unproblematic. We received no critical feedback from those involved.

We asked participants to select five researchers they considered to be “most influential” in their fields. The concept of “researcher influence” is arguably wider than that of research track record, but we could have just as easily asked participants to rate the narrower concept of “research track record”. Equally, we could have asked participants to nominate and rank a larger, but non-onerous, number of influential researchers (say, 20), which would have produced a more finely grained distribution of voting, facilitating the overall ranking into quartiles or quintiles that ranged from those considered by many to be influential through to those considered influential by none of their peers.