Increased expenditure on Australian health and medical research and changes in numbers of publications determined using PubMed

Following the 1998 Health and Medical Research Strategic Review,1 the Australian Government decided to almost double the annual National Health and Medical Research Council (NHMRC) funding — at the beginning of 2006, this stood at $490 million, the highest in the world on a per capita basis,2 and it has been increased further in the 2006–07 budget.

In 2004, 4 years after the increase in funding started, the Grant Report was commissioned to ascertain its effect.3 The report concluded that some successes had been achieved and further long-term benefits are likely with the increased investments. The two key indicators of success were Australia’s high research productivity and the formation of 350 new biomedical start-up companies. However the report also stated that it might be “too early to attribute any improvement in Australia’s HMR [health and medical research] performance directly to investment increases”.3

Worldwide, health research funding bodies have been trying to identify reliable outcome indicators to monitor HMR expenditure. Studies undertaken to uncover the link have focused on how research results eventually lead to clinical applications.4-6 However, according to one study:7

Publications in peer-reviewed journals are the primary medium through which research findings are disseminated to the scientific community. It follows that an increase in the number of peer-reviewed publications will be an early consequence of increased funding. One study that used PubMed to assess links between funding grants from the United States National Institutes of Health (NIH) and subsequent publications supports the feasibility of this method to study the link between grant funding and research productivity.8

Building on this limited finding, we studied the association between expenditure on HMR in Australia and publications listed in PubMed between 1980 and 2004.

Data on HMR funding were obtained from the Australian Institute of Health and Welfare website. The funding comprises that provided by the Australian Government, state and local governments, and other sources. Data from 1960 to 2004 are available as interactive data cubes at <http://www.aihw.gov.au/expenditure/datacubes/>. The HMR expenditure is given by financial year (1 July to 30 June). The funding allocation to universities is available as a separate fraction only from 1995–96.

PubMed9 is the web interface of MEDLINE that is commonly used by clinicians, academics and the public. MEDLINE is the largest database that is accessed through PubMed, and it currently contains more than 14 million bibliographic citations from more than 4800 biomedical journals. (PubMed uses the term “citation” to refer to individual publications. To avoid confusion, we use the word “publication” in this article.) PubMed was used to tabulate the number of biomedical publications by primary authors linked to an Australian institution from 1980 to 2004.

A MEDLINE publication record consists of many “tags” that are abbreviated names for the different fields. The fields we used were author affiliation (AD), publication type (ptyp), medical subject heading (MeSH), title (TI) and all fields (ALL).

The AD tag includes institutional affiliation and address of the first author. To obtain publications from Australia, we searched for the word “Australia” and for all Australian state and territory names in the AD tag (Box 1). The AD tag was also used to count publications originating from universities. The ptyp tag was used to pick out a specific publication type: “clinical trials”. The MeSH tags are the main keywords of the MEDLINE database. Every year, the MeSH words are revised. MEDLINE currently has 22 997 MeSH words. The specificity of PubMed can be increased when MeSH words are used to query MEDLINE.10

In designing the queries (Box 1), our primary aim was to increase sensitivity to avoid missing publications that originated from Australia. However, this strategy tends to increase false positives — publications that did not originate from Australia, but had specific words or combinations of letters in the text that referred to other countries (eg, the US state Washington [WA], or Hong Kong [NT]). We reduced false positives by including Boolean NOT statements to exclude these countries.

Australian HMR expenditure increased from $12 million in 1960–61 to $1503 million in 2003–04. It increased at a steady rate from 1980–81 until 1998–99 (Box 2A). The next 5 years saw a steeper increase, following the NHMRC funding initiative. Over this period, total government funding increased from $603 million to $1159 million, but there was also a large increase in non-government funding, from $122 million to $344 million.

Over the same period, total PubMed publications from Australia increased from 13 in 1960–61 to 13 836 in 2003–04. The number increased steeply between 1986–87 and 1987–88, after which there was a constant upward trend until 1998–99. This was followed by a slight upward inflection in the curve in 2003–04.

For clinical trials publications from Australia (Box 2B), there were several steep increases, with the first two occurring between 1986–87 and 1987–88, and between 1993–94 and 1994–95. However, after the second occasion, the number declined in the next year before increasing again. Thereafter, the curve maintained a constant upward trend from 1996–97 to 2002–03, with a third steep increase in 2003–04. The first increase could be a result of the increased HMR expenditure that commenced in 1980–81, although the lag time is greater than might be expected. The second increase, in the mid 1990s, may be statistical noise in a relatively constant overall rate from 1988. The third increase could be the response to the new funding that commenced in 1999.

University-derived publications from Australia increased from 3681 in 1990–91 to 8169 in 2003–04 (Box 2C) at a fairly constant rate. There was an almost parallel increase over this period in Australian university HMR expenditure.

Genetic and biotechnology publications from Australia increased from 49 in 1990–91 to 277 in 2003–04 (Box 2D). The first upward inflection commenced in 1997–98, and a second occurred after 2001–02. Non-government funding increased nearly threefold, from $122 million (1998–99) to $344 million (2003–04).

To determine if local or international forces led to the increases in PubMed publications, we examined the temporal trend from 1985 to 2004 for Australia, New Zealand and the United Kingdom (Box 2E). All three showed increases, although of varying magnitudes, around 1986–1988, suggesting that changes in PubMed may have had an influence (eg, increased listing of journals from each country that were referenced in MEDLINE, or the listing of new MeSH keywords). However, the rates of increase in the number of publications from Australia and the UK have been similar since then, while the rate for NZ has been lower.

For genetic and biotechnology publications, we examined the trend for the same three countries from 1990 (Box 2F). Given the relatively small numbers, statistical noise makes detection of trend difficult. Nevertheless, Australia appears to show an increase after 1998, which may be the result of the increased NHMRC research funding and a parallel increase in non-government funding relating to genetic and biotechnology research. The greatest increase in NZ occurred between 2000 and 2001. A pronounced increase from 2002 to 2003, especially in Australia and the UK, may relate to changes in PubMed’s indexing of genetic and biotechnology journals.

Our study shows an association between expenditure on HMR in Australia and the number of Australian publications listed by PubMed. The association was more prominent for genetic and biotechnology publications. The Australian Government’s HMR investment initiative that began in 1999 may be associated with the increase in the number of publications towards the end of the analysis period, which was not evident to the same extent in publications from NZ and the UK.

Although the Grant Report suggested that the time from initiation of increased funding might be too short to observe an increase in publication output,3 the literature suggests otherwise. The NHMRC Research Evaluation and Policy Project of 2001,11 which examined journal output from NHMRC research grants for 1994–95, concluded that the turnaround time from the receipt of project grant support to the publication of results is very short. It established that two-thirds of publications resulting from a 3-year grant have been published or are in press by the end of the grant period. In a recent evaluation of NHMRC-funded research,12 number of publications was the main short-term output indicator for knowledge gain.

Bourke and Butler estimated that, in the field of basic research in the medical and health sciences, 70% of published research output is in the form of journal articles.13 Shah and Ward, reporting on the outcomes from Australian NHMRC public health research grants awarded in 1993, reported that 30 grants funded by the Public Health Research Development Committee resulted in 218 publications, with 126 appearing in peer-reviewed journals indexed in MEDLINE or EMBASE.14 A study that tracked publication outcomes of NIH grants reported that, on average, each grant produced 7.58 PubMed publications.8 Of the top grants, only 6.4% did not result in any publications.

Taking these findings into consideration, the early upward trend in clinical trials and genetic and biotechnology publications within 4 years of the commencement of the new funding initiative is consistent and encouraging.

Most Australian Government funding is directed to higher education facilities.15 Our analysis showed that increased funding was associated with an increase in university publications. The same is true for genetic and biotechnology research publications. This increase is even more impressive, but there has also been a significant increase in non-government funding for this field.

It is not surprising that genetic and biotechnology publications from Australia and the UK show similar upward trends from 2002. Our analysis is compatible with the start-up of 350 new Australian biotechnology companies following increased HMR funding, and there has been an even higher investment in medical-related research and development in the UK over the same time.16

The databases of the Institute for Scientific Information (ISI)17 have been used to track Australian research.13,18,19 The Journal Citation Index (JCI) measures how many times a published article is referenced by other publications. Higher numbers of citations normally indicate a “good” article. However, some publications can attract many citations, not because of high quality, but because the results are contested or contain errors. A related measure of quality, the impact factor, has also been criticised as a “poor measure of the worth of journals, journal articles and authors”.20 We are not aware of any studies that have compared the two major bibliographic databases — PubMed and ISI-JCI — for tracking research performance.

Although the changes in numbers of publications and funding in Australia over the past three decades show an association, and the increase in both in the past 7 or 8 years is encouraging, we cannot say that this is causal. Many factors that might influence the number of publications found using PubMed are unrelated to funding — changes in technology (word processing making preparation of manuscripts easier), in journals (more journals to publish in), in the database (new MeSH terms), in research personnel (changes in numbers and funding), and in the research community (changed imperatives about how to publish and when).

Our study has several limitations. Although MEDLINE is the largest publicly available bibliographic database, not all Australian research papers may be indexed in it. However, Australian medical research is generally regarded as being highly visible,19 so the proportion of publications not indexed in MEDLINE is likely to be low.

Our analysis depends on the accuracy of information in the “affiliation” tag of PubMed. This can sometimes incorrectly identify the country of the first author, especially when an author holds appointment in two institutions in separate countries, or may not provide the country name at all.21 Furthermore, omission of the country name (as may occur in some publications that originate in the US) with identical abbreviations of state names (WA for both Washington and Western Australia) can increase false positives.

Finally, bibliometric or literature-based analysis cannot stand in isolation. There are important qualitative aspects in the research process — particularly professional commitment, influence, reputation and scientific recognition — that are not captured by bibliometric studies.22

Nevertheless, the association between HMR expenditure and changes in PubMed publications suggests this can be another measure of the early outcome of research funding.

2 Associations between expenditure on health and medical research (HMR) and publications indexed in PubMed

---

A: Total Australian expenditure on HMR and total PubMed publications from Australia.
B: Total Australian expenditure on HMR and PubMed publications of Australian clinical trials.
C: Australian expenditure on HMR in universities and PubMed publications originating from Australian universities.
D: Total Australian expenditure on HMR and PubMed publications in genetics and biotechnology from Australia.
E: Number of PubMed publications originating from Australia, New Zealand and the United Kingdom.
F: Number of PubMed publications in genetics and biotechnology originating from Australia, New Zealand and the United Kingdom.