A simple, novel and accurate method to estimate track record: a new “P” value

How much should past performance contribute to picking winners in grant competitions?

The weight that “track record” (TR) should contribute to the assessment of worth is a simple concept that is difficult to estimate. A synthesised value of TR is used to assess research grant applications. But what percentage of the overall variance does TR account for in determining success? In most human endeavours, assessors synthesise many variables to produce a single metric for TR that then becomes part of an overall scoring matrix. In assessing grant applications, the objective of this ranking is to minimise the risk that the grant given will not be productive.

In order to determine the true impact of TR, I used betting on horse races to determine the win–loss ratio when the shortest-odds favourite is backed (Pilowsky's “P” for TR). Horse races were chosen to simulate grant panels, as horses, like grants, are “marked” to some extent on the basis of their TR. A nominal “bet” of $1000 was placed on each horse at an off-track totaliser betting facility between 16 December and 20 December 2013. A total of 125 bets were placed. Losses were 83 and wins 42 (34% of total). The simulation suggests that the pooled synthesis of available information by those with an interest in estimating the best outcome (a win) gives a value of between 23% and 46% on each day (n = 5 days; mean ± SD, 38% ± 9%). I note with interest that these values are similar to those used as TR metrics by grant agencies in the past. The National Health and Medical Research Council allocated 25% to TR for project grant assessments in 2013, suggesting that a higher percentage may be appropriate.

I propose that the same strategy can be applied in many situations where TR is likely to be an important factor. With this approach, TR obtained from as many sources (peers, other workers and colleagues) as possible ought to account for about 40% in the determination. If every horse in the race had an equal chance of winning (ie, TR is not important), then a value of 0% (no contribution to outcome) would be appropriate. As an aside, the longest-odds horse is very unlikely to win — no such examples occurred here. In summary, therefore, a “good, novel and exciting idea” may be important, albeit hard to quantify, but TR accounts for 40% of the likelihood that the personnel with the best performance in the past will achieve their goal. Certainly, there appears little justification for a value lower than 30%.

Out of interest, my fiscal outcomes per day varied as follows: Day 1, + $5800; Day 2, + $17 500; Day 3, + $2100; Day 4, − $16 700; Day 5, + $16 300; total, + $25 000. A total of $125 000 was gambled to achieve this result (20% per week or 1040% annualised). Note that the placing of $1000 bets may cause significant changes to the totaliser, which is affected by the total pool of bets placed when the race starts, and thus the final payout. On-track betting, where the odds obtained are fixed at the time the bet is laid, may yield different results. I suspect that a granting agency is closer in form to a totaliser, as the available pool of funds is fixed after the bets (grants) are placed (submitted).

The findings reported here are not intended to provide a path to wealth. For this, please consult your personal turf consultant. Quantifying TR is a different issue.