Effectiveness of COVID‐19 vaccines: findings from real world studies

David A Henry, Mark A Jones, Paulina Stehlik and Paul P Glasziou
Med J Aust 2021; 215 (4): 149-151.e1. || doi: 10.5694/mja2.51182
Published online: 16 August 2021

Community‐based studies in five countries show consistent strong benefits from early rollouts of COVID‐19 vaccines

By the beginning of June 2021, almost 11% of the world’s population had received at least one dose of a coronavirus disease 2019 (COVID‐19) vaccine.1 This represents an extraordinary scientific and logistic achievement — in 18 months, researchers, manufacturers and governments collaborated to produce and distribute vaccines that appear effective and acceptably safe in preventing COVID‐19 and its complications.2,3

The initial randomised trials confirmed immunological responses and generated unbiased evidence of vaccine efficacy. They were conducted in selected populations with limited numbers of participants in high risk groups, such as older people and those with serious underlying medical conditions.2,3 They provided sparse information on the impact of vaccination on transmission of severe acute respiratory syndrome coronavirus 2 (SARS‐CoV‐2), were too small to quantify rare but serious harms, and did not take account of the logistic obstacles encountered during the community‐wide rollout of new vaccines. While large cluster randomised trials could address some of these concerns,4 large observational studies have used large linked routinely collected population datasets in five countries to address important knowledge gaps.5,6,7,8,9

This article reviews findings from the initial real world studies and stresses that researchers in Australia currently do not have timely access to the linked Commonwealth and state datasets needed to perform such analyses.


Real world studies


In five countries (Israel, England, Scotland, Sweden and the United States) researchers have analysed routinely collected data to report the early outcomes of community‐wide vaccination programs with three of the first vaccines to reach market: the BNT162b2 mRNA (Pfizer–BioNTech), mRNA‐1273 (Moderna) and ChAdOx1 adenoviral vector (Oxford–AstraZeneca) vaccines.5,6,7,8,9

At the time of writing, two of the articles (from the US and Sweden ) have not yet been peer reviewed, so details reported here may change after revisions to these reports.8,9 There is a rapidly growing literature on the community impact of COVID‐19 and it has provided very consistent evidence of substantial vaccine effectiveness with the original (Wuhan) viral strain and the Alpha variant. An important focus of future work will be the effectiveness of existing vaccines against emerging viral variants.

The vaccination programs against COVID‐19 commenced in December 2020 in the study countries, so follow‐up is limited. Most of the investigators used rigorous designs and statistical methods to analyse linked routinely collected person‐level data from large community‐wide databases that tracked outcomes in vaccinated and unvaccinated individuals (Box). Importantly, allocation to vaccines was not by randomisation, and vaccinated and unvaccinated populations differed in respect of factors that were associated with both the probability of vaccination and with the severe outcomes of COVID‐19. Information that featured in most studies included demographic details, a vaccine register, results of laboratory polymerase chain reaction (PCR) testing, records of hospitalisation and death, and some geographic measures of social deprivation. In addition, the Israeli, US and Scottish studies included linkage to clinical records from which to quantify comorbidities.5,6,8 The Israeli study included information on previous adherence to influenza vaccination schedules.5

Study designs and adjustments for confounding

The studies used different approaches to adjust for confounding (Box). The most advanced design was used to analyse the linked data from members of the Clalit Health Services integrated health care organisation in Israel, which covers around 4.7 million people.5 The investigators extracted data on matched cohorts of vaccinees and non‐vaccinated controls and analysed study endpoints using rules that emulated the steps taken in a randomised trial.10 These steps minimised selection or measurement biases and controlled for potential confounders through precise 1:1 matching of vaccinated and non‐vaccinated subjects across seven domains. The investigators took the additional step of calibrating their statistical model against the results of the pivotal phase 3 randomised trial, which found no benefit during the first 2 weeks after vaccination.2 In contrast, this observational study found lower rates of infection in the first 2 weeks after vaccination, which remained after matching for age and sex — illustrating the potential for confounding. Only after full matching on seven factors was this source of bias eliminated.5

In England, investigators linked data from a national vaccine register to laboratory PCR swab results, emergency department admissions, demographic and ethnicity data, care home status, and deaths in participants aged 70 years and over (Box).7 The first part was a test‐negative case–control design, which compared vaccination status in those who received a positive PCR swab result with contemporaneous controls who returned a negative result. That both cases and controls had been tested for SARS‐CoV‐2 should have controlled for clinical and behavioural factors that influence the probability of having a test. The second part of the study followed participants aged 80 years and over with a positive PCR test result and analysed them according to vaccination status. The investigators calculated adjusted hazard ratios for death up to and beyond 14 days from the first vaccine dose.

A study in Scotland used an unmatched cohort design comparing hospital admission for COVID‐19 in people who received either the Pfizer–BioNTech or Oxford–AstraZeneca vaccines with an unvaccinated control group.6 The Oxford–AstraZeneca vaccine was given later to an older population. The study adjusted for age and sex, frequency of prior PCR tests and clinical risk groups extracted from linked health records. The statistical model generated unexpectedly strong protective effects of the vaccines on hospitalisation rates in the first 2 weeks after vaccination, indicating possible bias due to a healthy vaccinee effect.

In the US, researchers working within the Mayo Clinic health system used postcode and propensity scores (based on age, sex, race, ethnicity and records of PCR testing) to match a cohort of individuals who received the Pfizer–BioNTech or Moderna mRNA vaccine with unvaccinated controls, to measure impact on infections and hospitalisations.8

A simple unmatched cohort design using linkage of routinely collected administrative data measured infection rates in a cohort who received the Pfizer–BioNTech vaccine in a single county in Sweden.9 The unvaccinated population acted as controls (Box). Confounding adjustments in this study were limited to age and sex.

The Box summarises the results of these studies. All included at least one mRNA vaccine and the reductions in infections and hospitalisations were consistent and large. Two studies reported on mortality and the reductions were substantial, although based on small numbers of deaths in Israel.5,7 The studies did not directly compare vaccines, but the Oxford–AstraZeneca vaccine appeared to perform as well as the mRNA vaccines in reducing hospitalisations.

Other approaches to estimating vaccine effectiveness

In the UK, over 600 000 volunteers using a COVID‐19 symptom mobile phone app recorded adverse events after vaccination with either the Pfizer–BioNTech or Oxford–AstraZeneca vaccine.11 Based on post‐vaccination self‐reports of infections and after adjustment for age, sex, obesity and comorbidities, they estimated effectiveness rates of 60–70% beyond 21 days after administration of either vaccine.

Three studies measured the effectiveness of COVID‐19 vaccines in care home, health care and other frontline workers in the UK, Israel and the US.12,13,14 These projects enrolled smaller numbers of participants than the community‐based studies but used similar designs and adjustment techniques. Importantly, workers in these settings undergo routine PCR testing for SARS‐CoV‐2, which enabled detection of asymptomatic infections. These studies also found large protective effects and a potential to reduce viral transmission. The latter possibility has been investigated directly in a study conducted in Scotland that showed that 14 days or more after health care workers received a second dose of vaccine, their household members had a 54% lower rate of COVID‐19 than individuals who shared households with non‐vaccinated health care workers.15


We can draw important conclusions from these non‐randomised studies of vaccine effectiveness. Most importantly, the currently available COVID‐19 vaccines appear to be very effective in preventing severe complications and deaths from COVID‐19 in adults of all ages. Recent real world studies confirm that substantial protection extends to the Delta variant of SARS‐CoV‐2, although this requires two vaccine doses.16,17 Follow‐up periods in all studies are relatively short, and these reports do not provide information on rare but serious adverse events, such as cerebral venous thrombosis. The use of sophisticated trial emulation methods in the Israeli study5 replicated some key features of the pivotal randomised trial of the Pfizer–BioNTech vaccine,2 particularly by controlling for an early healthy cohort effect that confounded the incompletely adjusted endpoint analyses. This design should prove useful in enabling direct head‐to‐head comparisons of effectiveness and safety of vaccines, the duration of their protective effects, the degree to which vaccines prevent transmission of viral variants, and the impact of vaccines on so‐called long COVID.

These studies exemplify the value of advanced analyses of large multiply linked routinely collected community datasets. This resource is not yet readily available to researchers in Australia due to continued lack of agreement on the governance of linked state and Commonwealth datasets.18 While Australia’s current low rates of community transmission of SARS‐CoV‐2 reduce the feasibility of observational studies of vaccine effectiveness, the available data can provide important information on potential harms of vaccines. With continuing questions about the comparative safety of vaccines, the emergence of viral variants, the long term effects of COVID‐19 and the likelihood of future epidemics, it is essential that Australia urgently removes barriers to allowing prequalified researchers to safely access the linked de‐identified population datasets that are needed to expeditiously conduct the types of studies reviewed here.

Box – Characteristics of five real world community‐based studies of effectiveness of SARS‐CoV‐2 vaccines


Dagan 20215

Bernal 20217

Vasileiou 20216

Bjork 20219

Pawlowski 20218






United States


BNT162b2 (1 or 2 doses)

BNT162b2 (2 doses) or ChAdOx1 (1 dose)

BNT162b2 or ChAdOx1 (1 dose)

BNT16b2 (1 or 2 doses)

BNT162b2 or mRNA‐1273 (2 doses)

Study design

Target trial emulation using 1:1 individual matching of vaccinated and unvaccinated participants

Hybrid of test‐negative case–control followed by cohort analysis of PCR‐positive individuals

Controlled cohort study

Controlled cohort study

Controlled cohort study with 1:1 individual matching of vaccinated and unvaccinated participants

Source population

Aged ≥ 16 years: 1 503 216 vaccinated; 1 655 920 unvaccinated enrolled with single state‐mandated health care provider

Aged ≥ 70 years; > 7.5 million enrolled with NHS UK

Aged ≥ 15 years: 1 137 775 vaccinated; 3 271 836 unvaccinated enrolled with NHS UK

Aged 18‐64 years: 26 587 vaccinated;
779 154 unvaccinated enrolled with single regional health service

Aged ≥ 18 years: 249 708 enrolled with single non‐profit health care provider who had PCR test for SARS‐CoV‐2

Numbers analysed

596 618 vaccinated; 596 618 matched unvaccinated controls

44 590 cases (PCR‐positive) and 112 340 controls in case–control study;
1846 vaccinated and 8096 unvaccinated in follow‐up study

Same as source population

Same as source population

31 069 vaccinated; 31 069 unvaccinated

Analysis methods

Kaplan–Meier analysis

Logistic regression analysis

Time‐dependent Cox regression and Poisson regression adjusting for time at risk

Incidence rate ratios

Kaplan–Meier analysis

Study endpoints included in analyses (n)

Infections (10 561); hospitalisations (369); deaths (41)

Infections (32 832); hospitalisations (1859); deaths (1228)

Hospitalisations (7914)

Infections (4228); deaths (36)

Infections (924); hospitalisations (224)

Confounder adjustments

1:1 matching on day of vaccination on seven features: age, sex, place, ethnicity, past influenza vaccine, pregnancy, number of pre‐existing medical conditions

Adjusted for five features: age, sex, ethnicity, NHS region, deprivation

Adjusted for five features: age, sex, deprivation score, number of prior SARS‐CoV‐2 PCR tests, number of medical conditions

Adjusted for age and sex

Propensity‐matched based on sex, age, ethnicity, location and number of prior SARS‐CoV‐2 PCR tests

Check on bias due to healthy vaccinee effect*

Yes, calibrated to check no effect in first 14 days

Yes, used immediate post vaccination period as reference

No, and significant benefit noted before day 14

No, did not evaluate endpoints before day 14

No, and significant benefit noted before day 14

Vaccine effectiveness: selected measures (95% CI)

Days 14–20: infection, 46% (40–51%); hospitalisation, 74% (56–86%); death, 72% (19–100%)

Days 28–34 (BNT162b2): infection, 61% (51–69%); hospitalisation 43% (33–52%); death, 51% (37–62%)

Days 28–34 (BNT162b2): hospitalisation 86% (76‐91%)

Day 14+: infection, 42% (14–63%); death not calculated

Day 14+: infection, 75% (67–81%); hospitalisation 60% (14–79%)


Day 7+ after second dose: infection 92% (88–95%); hospitalisation, 87% (55–100%)

Days 28–34 (ChAdOx1): infection, 60% (41–73%); hospitalisation 37% (3–59%)

Days 28–34 (ChAdOx1): hospitalisation 94% (73–99%)

Day 7+ after second dose: infection, 86% (72–94%); death not calculated

Day 36+ (2 doses only); infection 89% (68–97%)

Viral variants of concern

Alpha variant was common during the study

Alpha variant was dominant during the study

Alpha variant was common during the study

Alpha variant was common during the study

No mention of variants

BNT162b2 =Pfizer–BioNTech mRNA vaccine; ChAdOx1 = Oxford–AstraZeneca adenoviral vector vaccine; mRNA‐1273 = Moderna mRNA vaccine; NHS = National Health Service; PCR = polymerase chain reaction.  * It is assumed that an apparent protective effect before day 14 reflects residual confounding.  † Reductions in risk of hospitalisation and death were additional to the reduction in infection risk, equivalent to an overall reduction in hospitalisation of 80% and 85% reduction for death (BNT162b2 only).  ‡ No deaths recorded in vaccinated participants.

Provenance: Not commissioned; externally peer reviewed.

  • David A Henry1,2
  • Mark A Jones1,3
  • Paulina Stehlik1,2
  • Paul P Glasziou1

  • 1 Institute for Evidence-Based Healthcare, Bond University, Gold Coast, QLD
  • 2 Gold Coast University Hospital and Health Service, Gold Coast, QLD
  • 3 University of Queensland, Brisbane, QLD


Competing interests:

No relevant disclosures.

  • 1. Our World in Data. Coronavirus (COVID-19) vaccinations. (viewed May 2021).
  • 2. Polack FP, Thomas SJ, Kitchin N, et al. Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine. N Engl J Med 2020; 383: 2603–2615.
  • 3. Voysey M, Clemens SA, Madhi SA, et al. Single-dose administration and the influence of the timing of the booster dose on immunogenicity and efficacy of ChAdOx1 nCoV-19 (AZD1222) vaccine: a pooled analysis of four randomised trials. Lancet 2021; 397: 881–891.
  • 4. Bell KJL, Glasziou P, Stanaway F, et al. Equity and evidence during vaccine rollout: stepped wedge cluster randomised trials could help. BMJ 2021; 372: n435.
  • 5. Dagan N, Barda N, Kepten E, et al. BNT162b2 mRNA Covid-19 vaccine in a nationwide mass vaccination setting. N Engl J Med 2021; 384: 1412–1423.
  • 6. Vasileiou E, Simpson CR, Shi T, et al. Interim findings from first-dose mass COVID-19 vaccination roll-out and COVID-19 hospital admissions in Scotland: a national prospective cohort study. Lancet 2021; 397: 1646–1657.
  • 7. Bernal JL, Andrews N, Gower C, et al. Effectiveness of the Pfizer-BioNTech and Oxford-AstraZeneca vaccines on covid-19 related symptoms, hospital admissions, and mortality in older adults in England: test negative case-control study. BMJ 2021; 373: n1088.
  • 8. Pawlowski C, Lenehan P, Puranik A, et al. FDA-authorized COVID-19 vaccines are effective per real-world evidence synthesized across a multi-state health system [preprint]. medRxiv 2021; 27 Feb. (viewed May 2021).
  • 9. Bjork J, Inghammar M, Moghaddassi M, et al. Effectiveness of the BNT162b2 vaccine in preventing COVID-19 in the working age population — first results from a cohort study in Southern Sweden. medRxiv 2021; 21 Apr. (viewed May 2021).
  • 10. Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available. Am J Epidemiol 2016; 183: 758–764.
  • 11. Menni C, Klaser K, May A, et al. Vaccine side-effects and SARS-CoV-2 infection after vaccination in users of the COVID Symptom Study app in the UK: a prospective observational study. Lancet Infect Dis 2021; 21: 939–949.
  • 12. Hall VJ, Foulkes S, Saei A, et al. Effectiveness of BNT162b2 mRNA vaccine against infection and COVID-19 vaccine coverage in healthcare workers in England, multicentre prospective cohort study (the SIREN Study) [preprint]. Preprints with The Lancet 2021; 22 Feb. (viewed May 2021).
  • 13. Amit S, Regev-Yochay G, Afek A, et al. Early rate reductions of SARS-CoV-2 infection and COVID-19 in BNT162b2 vaccine recipients. Lancet 2021; 397: 875–857.
  • 14. Thompson MG, Burgess JL, Naleway AL, et al. Interim estimates of vaccine effectiveness of BNT162b2 and mRNA-1273 COVID-19 vaccines in preventing SARS-CoV-2 infection among health care personnel, first responders, and other essential and frontline workers — eight US locations, December 2020–March 2021. MMWR Morb Mortal Wkly Rep 2021; 70: 495–500.
  • 15. Shah AS, Gribben C, Bishop J, et al. Effect of vaccination on transmission of COVID-19: an observational study in healthcare workers and their households [preprint]. medRxiv 2021; 21 Mar. (viewed May 2021).
  • 16. Sheikh A, McMenamin J, Taylor B, Robertson C. SARS-CoV-2 Delta VOC in Scotland: demographics, risk of hospital admission, and vaccine effectiveness. Lancet 2021; 397: 2461–2462.
  • 17. Stowe J, Andrews N, Gower C, et al. Effectiveness of COVID-19 vaccines against hospital admission with the Delta (B.1.617.2) variant [preprint]. London: Public Health England, 2021. (viewed July 2021).
  • 18. Henry D, Stehlik P, Camacho X, Pearson SA. Access to routinely collected data for population health research: experiences in Canada and Australia. Aust N Z J Public Health 2018; 42: 430–433.


remove_circle_outline Delete Author
add_circle_outline Add Author

Do you have any competing interests to declare? *

I/we agree to assign copyright to the Medical Journal of Australia and agree to the Conditions of publication *
I/we agree to the Terms of use of the Medical Journal of Australia *
Email me when people comment on this article

Online responses are no longer available. Please refer to our instructions for authors page for more information.