Objective: To assess the performance of SmartVax, a prototypic active monitoring system for adverse events following immunisation (AEFI) using short message service (SMS) text messages and clinical data extracted from commercially available medical practice management software.
Design, setting and participants: Between 11 November 2011 and 10 June 2013, adult patients and parents of paediatric patients receiving routine vaccinations in general practice were sent an SMS by SmartVax enquiring if they had experienced any AEFI and requesting a reply by SMS. Attempts were made to telephone patients who did not reply by SMS.
Main outcome measures: The proportion of patients sent an SMS who replied by SMS, and the proportion of respondents indicating possible AEFI.
Results: Of 3281 vaccinated patients, 3226 (98.3%) had a mobile telephone number on record and were sent an SMS. Of 2342 patients (72.6%; 95% CI, 70.0%–75.1%) who responded by SMS, 264 (11.3%; 95 CI, 9.9%–12.7%) reported possible AEFI. The response rate was ≥ 70% for both paediatric and adult patients. Eighty-per cent of SMS replies were received within 2 hours of transmission of the query SMS. There was no significant difference in the proportion reporting possible AEFI between patients who replied by SMS and those who did not respond by SMS but were subsequently contacted by a telephone call (P = 0.99).
Conclusions: More than 70% of patients responded by SMS to an SMS query about whether they had any vaccine reactions, with the data received in near real-time. Active surveillance of AEFI using SMS has the capacity to complement existing passive reporting systems, potentially permitting more rapid identification of emerging safety signals.
After the surge in adverse events following immunisation (AEFI) among children receiving seasonal influenza vaccine in April 2010, the limitations of existing passive vaccine safety surveillance in Australia have been widely acknowledged.1-3
In an effort to remedy the inherent constraints of passive AEFI reporting, the Illawarra Medical Centre (IMC) in Perth, Western Australia, began exploring opportunities to implement active surveillance of vaccine safety. In 2011, IMC collaborated with a software developer (Datavation) to create a tool that extracts vaccination data from the clinic’s existing, commercially available practice management software. This new tool, called SmartVax, can send a short message service (SMS) text message to patients who have been vaccinated, then automatically collate any replies received via SMS.
In this study, we aimed to assess SmartVax’s performance in terms of response rates and timeliness.
IMC administers more than 2000 vaccinations annually and documents each of these events in the patient’s electronic medical record using the standard functionality of Best Practice (Best Practice Software), the practice management software. Patients and parents of paediatric patients (considered as synonymous for this report) are routinely requested to provide a mobile telephone number and are informed at the time of vaccination that they may subsequently receive an SMS enquiring about any reactions.
Between 11 November 2011 and 10 June 2013, SmartVax was used to send an SMS to vaccinated patients who had provided a mobile telephone number. The SMS was typically sent once a week to patients vaccinated within the preceding 7 days. Two versions of the outgoing SMS were created: one tailored to parents of children aged ≤ 12 years who had received scheduled childhood vaccinations; and one for adults aged ≥ 18 years who had received influenza vaccine. The SMS asked patients if there were any reactions to the vaccination and requested a “Yes” or “No” reply by SMS.
SMS replies were automatically written back into the SmartVax tool, linked to the patients’ vaccination data and collated in real-time. Routinely, 24 hours were allowed for receipt of a response, after which clinic staff telephoned those who indicated they had experienced a reaction, as well as those who had not responded. During the phone call, a survey was administered to ascertain the nature, duration and severity of any reaction reported. The response rate was defined as the proportion of patients who responded to the clinic’s SMS with a reply SMS. Patients who did not have a mobile telephone, provided an incorrect or disconnected mobile number, or did not answer at least two subsequent attempted phone calls were classified as uncontactable.
As some individuals contributed more than one vaccination visit to the analysis, cluster analytical techniques were used to calculate proportions and confidence intervals. We also performed a secondary analysis using only information from each individual’s first vaccination visit to assess the impact of including repeat observations for some patients. Data were analysed using SAS 9.3 (SAS Institute Inc).
This activity was approved by the University of Western Australia Human Research Ethics Office in December 2010 (RA/4/1/4477).
During the assessment period, 2134 individuals contributed a total of 3281 vaccination visits for analysis. Of the patients contributing the 3281 vaccination visits, 3226 (98.3%) had a mobile number in their record and were sent an SMS enquiring about any reaction to the vaccine(s) (Box 1). Of the patients who were sent an SMS, 2342 (72.6%; 95% CI, 70.0%–75.1%) responded with a reply SMS. A further 374 patients (11.6%; 95% CI, 10.3%–12.9%) did not respond by SMS but were subsequently contacted by telephone. The remaining 510 patients (15.8%; 95% CI, 13.3%–18.3%) were uncontactable.
There was no significant difference between paediatric and adult patients in regards to the overall proportion who replied by SMS (73.9%; 95% CI, 71.4%–76.3% v 69.8%; 95% CI, 67.5%–72.1%) (Box 2). In the adult group, women were significantly more likely than men to reply by SMS (73.1% [427/584] v 65.3% [269/412]; P < 0.01).
Between patients who replied by SMS and those who did not but were subsequently contacted through a phone call, there was no significant difference in the proportion reporting a reaction (11.3%; 95% CI, 9.9%–12.7% v 11.8%; 95% CI, 9.2%–14.4%; P = 0.99).
The SMS replies were very timely. Half the patients (1212/2342, 51.8%) responded within 10 minutes of receiving the SMS, and more than 80% of responses (1944/2342) were received within 2 hours.
The analysis restricted to data from the first vaccination visit for each individual produced very similar results: the response rate remained 70% or greater for both paediatric and adult patients, and more than 80% of SMS responses were received within 2 hours.
In Australia and other countries, post-licensure monitoring of vaccine safety relies largely on passive surveillance. The constraints of passive AEFI reporting systems are well recognised and include underreporting, biased reporting and the inability to establish rates.3-5 These limitations may result in delayed detection of potential safety signals.6
The SmartVax tool permits active monitoring of vaccine safety by extracting clinical data from practice management software and using SMS technology to query vaccinated patients. Our evaluation identified several potential advantages with this approach. First, the response rate to the outgoing SMS was ≥ 70% for both paediatric and adult patients. Second, the information received was very timely, with more than 80% of all SMS replies received within 2 hours after the outgoing message was sent. Such timeliness could be valuable when an urgent investigation into potential vaccine safety issues is necessary, and it would permit ongoing safety monitoring in near real-time. Last, we found that the proportion of patients who reported an adverse reaction was similar between those who replied by SMS and those who did not but were subsequently contacted by telephone. This finding suggests that, during routine monitoring of vaccine safety, it may not be necessary to telephone people who do not respond to the SMS, substantially reducing the staff resources required to maintain the system.
We know of one other study that investigated the use of SMS text messages to monitor vaccine safety. In 2012, researchers in Cambodia testing an open-source SMS-based tool to monitor adverse events among 184 adult patients receiving vaccinations found a response rate (72%) similar to ours.7 This suggests SMS may be a viable method of conducting AEFI surveillance in a variety of settings worldwide.8
One of the strengths of the SmartVax system is that it seamlessly interfaces with the existing practice management software in our practice, reducing the amount of effort required to identify vaccinated patients for inclusion in active surveillance. The medical practice software in use at IMC is popular in Australia,9 and it is likely that SmartVax could be modified to work with other commonly used software packages. Also, because the SMS replies are linked to data extracted from the patient record, this system enables the responses to be assessed in the context of the type and number of vaccines administered, with ready access to brand and batch number. Another strength is that when more detailed information for suspected vaccine reactions is obtained through subsequent telephone interviews, this information can be directly entered into SmartVax through a data entry screen and is immediately available for aggregation and review.
The major limitation of the system, as currently configured, is that it requires clinic personnel to telephone patients who respond affirmatively to experiencing suspected AEFI, to ascertain the nature and severity of the alleged reaction. In our experience, the process of personally contacting the small subset of respondents with suspected AEFI is not particularly onerous and is certainly more efficient than attempting to interview all vaccinated patients as part of an active AEFI surveillance system. Even so, this approach necessarily offsets some of the benefits of using an otherwise very timely SMS-based reporting system. A possible solution to gathering additional clinical information on patients responding affirmatively to the initial SMS, while minimising the staff resources involved, might be to send the patient a series of sequential text message questions or a link to a web-based survey; we are exploring these options.
The ability of an AEFI monitoring system to detect potential safety signals early is predicated, in part, on having large numbers of patients included in the surveillance program. By developing an AEFI monitoring system that works with a widely used practice management software application, and that could likely be modified to work with others, there is potential to expand the number of participating practices to achieve a large representative sentinel population. If this type of AEFI surveillance were to be implemented in other practices throughout Australia, de-identified results could be aggregated so that meaningful sample sizes for individual vaccines could be attained in a timely manner. Such a system could make a valuable contribution to assuring vaccine safety when new vaccine formulations are introduced, including any future rollout of pandemic influenza vaccines.
In summary, the experiences with seasonal influenza vaccine in Australia in 2010 highlight the limitations of relying solely on passive surveillance for early detection of vaccine safety problems.3 Active postmarketing surveillance of vaccine safety using SMS technology has the capacity to complement passive reporting systems, potentially enabling more rapid identification of emerging safety signals. Further development and evaluation of systems like SmartVax are warranted.
1 Follow-up of short message service (SMS) query to patients about adverse events following immunisation, Illawarra Medical Centre, Western Australia, 11 November 2011 – 10 June 2013
2 Response rate to short message service (SMS) query about adverse events following immunisation, Illawarra Medical Centre, Western Australia, 11 November 2011 – 10 June 2013, by age group and sex of vaccinated patients*
* Response rate was defined as the percentage of patients sent an SMS who responded with a reply SMS. Bars indicate 95% confidence intervals. Paediatric patients were those aged ≤ 12 years who received one or more routine childhood vaccinations, as appropriate for their age. Adult patients were those aged ≥ 18 years who received seasonal influenza vaccine.
Received 9 September 2013, accepted 28 November 2013
- 1. Armstrong PK, Dowse GK, Effler PV, et al. Epidemiological study of severe febrile reactions in young children in Western Australia caused by a 2010 trivalent inactivated influenza vaccine. BMJ Open 2011; 1: e000016.
- 2. Horvath J. Review of the management of adverse events associated with Panvax and Fluvax: final report 10 March 2011. Canberra: Australian Government Department of Health and Ageing, 2011. http://www.health.gov.au/internet/immunise/publishing.nsf/Content/11DFBB4 FD968D072CA25789400172DA1/$File/adverse-event-march-2011.pdf (accessed Oct 2013).
- 3. Gold MS, Effler P, Kelly H, et al. Febrile convulsions after 2010 seasonal trivalent influenza vaccine: implications for vaccine safety surveillance in Australia [editorial]. Med J Aust 2010; 193: 492-493. <MJA full text>
- 4. Mahajan D, Roomiani I, Gold MS, et al. Annual report: surveillance of adverse events following immunisation in Australia, 2009. Commun Dis Intell Q Rep 2010; 34: 259-276.
- 5. Zhou W, Pool V, Iskander JK, et al. Surveillance for safety after immunization: Vaccine Adverse Event Reporting System (VAERS) — United States, 1991–2001. MMWR Surveill Summ 2003; 52: 1-24.
- 6. Rosenthal S, Chen R. The reporting sensitivities of two passive surveillance systems for vaccine adverse events. Am J Public Health 1995; 85: 1706-1709.
- 7. Baron S, Goutard F, Nguon K, Tarantola A. Use of a text message-based pharmacovigilance tool in Cambodia: pilot study. J Med Internet Res 2013; 15: e68.
- 8. Kaplan WA. Can the ubiquitous power of mobile phones be used to improve health outcomes in developing countries? Global Health 2006; 2: 9.
- 9. Best Practice Software. Media statement. 15 Apr 2013. MIMS Matters 2013; Winter Edition: 7. http://www.mims.com.au/content/MimsMatters/WinterMMJul13.pdf (accessed Mar 2014).
Publication of your online response is subject to the Medical Journal of Australia's editorial discretion. You will be notified by email within five working days should your response be accepted.