To the Editor: Predicting research quality on the basis of past research publications is clearly imprecise, as noted by Nicol et al in their recent article on National Health and Medical Research Council (NHMRC) grant applications.1 They note that assessor ratings of applicants’ “track records” correspond poorly with the bibliometric data for authors, and that there is vast variability between discipline panels. For immunology, the correlation between track record scores and journal impact or citations was high, at over 0.7. For public health, the correlation was actually negative.

The authors consider some possible reasons for the wide discrepancies, such as poor coverage of public health publications in the journals captured by Institute for Scientific Information citation indexes. Nevertheless, they are at a loss to explain why the variation is quite so great, and conclude by suggesting that the time is right for an automated approach to assessing quality.

We need to consider the implications of this suggestion carefully. Track record within NHMRC project grants is assessed relative to opportunity, with regard to factors such as legitimate career interruptions, administrative and teaching load, and typical publication rates for the field in question. In the fellowship or program grants schemes, there appears to be less emphasis on relativity, which may explain some of the closer correspondence between actual and expected citation rates.

We need to be clear that the use of an “automated” system that uses surrogate measures of research quality will disadvantage individuals who experience a period of illness, take maternity leave, change their research area, or carry a period of heavy administrative or teaching load, as well as those who publish books, book chapters or government publications. It will also disadvantage teams in which feasibility requires fieldwork collaborators whose applied work does not readily translate into peer-reviewed journal publications. For instance, much public health research is based in the community or takes advantage of data collections in the public health system. Collaborators working in this context often have relatively limited opportunities for peer-reviewed publication. Nevertheless, their active collaboration is often critical for achieving a feasible research plan. Also disadvantaged would be teams with a new or junior investigator, particularly if the new team member was the first named investigator.

On the other hand, a move to an automated system of quality assessment would further advantage grant applicants who work in research-dedicated institutes, those engaged in basic research, and those who do not require external collaboration.

Given these reservations, I suggest further investigation, by discipline, of what makes a “good” track record, before recommending a single assessment formula.