Impact of Resident Feedback on Immunization Outcomes FREE

IMMUNIZATION LEVELS in the United States have risen dramatically in recent years although many urban localities have not seen their immunization levels rise as high as others.^{1 - 2} The standards for pediatric immunization practices developed by the Centers for Disease Control and Prevention,³ Atlanta, Ga, recommend investigating each child's immunization status at every medical encounter to take full advantage of every immunization opportunity. Missed immunization opportunities are known to contribute to lower immunization levels.^{4 - 5} Feedback to clinical providers can be an effective tool in reducing missed immunization opportunities and raising immunization levels.⁶ Given historically low immunization levels in our urban hospital-based resident continuity clinic, we undertook a controlled trial of written feedback to individual residents on their missed immunization opportunities in an attempt to improve immunization levels among our patient population. Because pilot data indicated that many of our underimmunized children were lost to follow-up, we also included information on each resident's appointment failure rate in the feedback he or she received.

Thirty-two postgraduate level (PL) 1 and PL-2 residents were chosen to participate in the study. Three others were ineligible because of plans to leave the residency before the study was complete. Residents were randomized within their PL year to 1 of 2 groups: (1) the control group (no written feedback) and (2) the intervention group (receiving written feedback). Residents were informed by mail at the onset of the study that their continuity clinic patient visits were being monitored for an immunization study and were not given an opportunity to refuse participation. The Emory University School of Medicine, Atlanta, Human Investigations Committee approved the study. Before the study, all residents received formal instruction regarding the immunization schedule, medical contraindications for immunizations, and the influence of missed immunization opportunities on immunization levels. This instruction was presented as one of the routine preclinic conferences, with no mention made of the study.

The immunization level for 2-year-old children in the resident clinic (dependent variable) was the main outcome of interest. The intervention was designed to investigate the effect of written feedback (independent variable) on immunization levels. Secondary outcomes were rates of missed immunization opportunities and appointment failure.

All visits of established patients younger than 2 years scheduled to see residents in the study were monitored. Established patients were defined as those who were visiting the clinic for a second or subsequent visit. A missed immunization opportunity was defined as any visit during which an immunization was due, according to Advisory Committee on Immunization Practices guidelines,⁷ but not given. Missed immunization opportunities in appointments kept were characterized as one of the following: (1) medical contraindication,⁸ (2) documentation gaps, (3) parental refusal, or (4) other. Missed immunization opportunities in the "other" category were those for which no medical contraindication, documentation gap, or parental refusal was found. These other missed immunization opportunities were further described as child ill or child well. A documentation gap was noted in cases in which the patient's chart contained evidence that the resident indicated that the immunization had been received, but the exact date of the vaccine was lacking. All visits were monitored by the same PL-4 resident (R.M.), and all missed immunization opportunities were characterized by the same attending physician (C.T.R.). The reviewers were not blinded as to which group, intervention or control, the residents belonged. We chose the commonly used 4:3:1:3 definition for completed immunizations for 2-year-olds, meaning that a child was defined as completely immunized if by age 2 years the following had been received: 4 diphtheria and tetanus toxoids and pertussis vaccines (or diphtheria and tetanus toxoids and acellular pertussis vaccines), 3 oral poliovirus vaccines (or inactivated poliovirus vaccines), 1 measles-mumps-rubella vaccine, and 3 Haemophilus influenzae type b vaccines. Age-appropriate numbers of H influenzae type b vaccinations were also accepted for children who began the series late.

A feedback report was devised and piloted among PL-3 residents who were asked not to share information on the study with their fellow residents. The feedback form included provider- and clinic-specific information on clinic visits of children younger than 2 years, including immunization opportunities, the percentage of immunization opportunities missed, the number of appointment failures, and the number of appointment failures that included potential immunization opportunities. Additionally, each report included 2 suggestions from the English-language literature: one on reducing missed immunization opportunities, and one on reducing appointment failures. The feedback reports were mailed to the intervention group at their homes to avoid contamination since the resident's hospital mailboxes are located together in the continuity clinic. The investigators received informal confirmation that the reports were received and read. The residents' visits were monitored for 2 months prior to receipt of the first feedback report in February 1997. A total of 12 monthly reports were mailed at the end of each month detailing the previous month's relevant statistics.

Data collection consisted of chart review and query of the local immunization registry after each scheduled appointment that met these criteria. Each visit was coded as to whether a particular vaccine was due and/or given. Because of the clinic's known appointment failure and "lost to follow-up" problem, the number of visits kept by study patients was also recorded. The immunization levels for 2-year-old children were determined for 2 points during the study: prestudy and poststudy (March 1998). Immunization rate data collection was modeled after the Health Plan Employer Data Information Set 3.0⁹ and was defined as the proportion of children reaching age 2 years with complete immunizations. Children eligible for each audit were those who reached age 2 years during the calendar year prior to the audit date. All children meeting the above criteria who were also seen by a given study resident at least twice during the first 2 years of life were included. All significance tests were performed using the χ² test. Confidence limits for proportions were calculated using the exact mid-p method.

Randomization resulted in 7 PL-1 and 9 PL-2 residents in the control group, and 7 PL-1 and 9 PL-2 residents in the intervention group. The median ages of the control and intervention groups at the end of the study were 29.7 years and 30.0 years, respectively. The control group was 38% male (n=6); the intervention group, 25% (n=4).

Prestudy immunization levels for the patients as a whole (including those seen by PL-3 residents who were not in the study) were slightly higher than poststudy immunization levels for patients from both groups. The prestudy 4:3:1:3 immunization level was 127 (77.9%) of 163, compared with the intervention group patients' poststudy immunization level of 100 (71.4%) of 140, and the control group patients' poststudy immunization level of 115 (68.5%) of 168 (Table 1). These differences were not statistically significant.

Analysis of poststudy data involving the number of visits during the second year of life revealed interesting findings. Fifty (78%) of 64 children who had the recommended number of visits¹⁰ between their first and second birthdays (ie, 2 visits) were up-to-date by age 2 years. In contrast, only 59 (47%) of 125 children who had fewer than 2 visits between their first and second birthdays were up-to-date by age 2, and 105 (88.2%) of 119 children who had more than 2 visits during their second year of life were up-to-date (Table 2). These differences were significant (P<.001).

Missed immunization opportunities in appointments kept (Figure 1) for the intervention group ranged over the course of the study from 5 (13.9%) of 36 to 14 (42.4%) of 33 vs 6 (17.1%) of 35 to 16 (47%) of 34 for the control group. The 95% confidence limits for both groups during all of the months of collection overlap, indicating no significant difference between the groups and no significant before-and-after differences. During the last 10 months of the study, after residents began receiving written feedback reports, the intervention group missed 93 (25.6%) of 363 immunization opportunities in appointments kept vs the control group's rate of 112 (28.7%) of 390, not a statistically significant difference (P=.34). For both groups over the 12-month course of the study, 871 visits made by patients constituted immunization opportunities. In 243 (27.9%) of these visits, at least 1 immunization was missed, making the visit a missed immunization opportunity. Gaps in documentation were found in 223 (45.7%) of 488 missed vaccine doses, parents refused immunizations in 22 (4.5%), and medical contraindications were found in 8 (1.6%). Of the remaining 235 missed vaccine doses (48.2%), 162 (69%) were in well-child visits and 73 (31%) were in sick visits at which the child was ill but not sick enough to require admission to the hospital. Among the 243 visits during which at least 1 vaccine was missed, 131 (54%) of the missed immunization opportunities would have been corrected by the administration of 1 additional vaccine.

The level of appointments kept remained close to 50% (1520) for the duration of the study and did not vary between the 2 groups (Figure 2). In the total of 2911 visits studied, there were 1732 immunization opportunities. In 861 (49.7%) of these immunization opportunities, the patient failed to keep the appointment.

We failed to demonstrate any effect of written feedback reports used in our resident clinic on immunization levels in 2-year-old children, missed immunization opportunities, or appointment failure rates. There were no before-and-after differences in any of the outcome variables of interest. The proportion of patients with a ratio of 4:3:1:3 immunizations at age 2 years is similar to that found in population-based immunization rate surveys for the entire Atlanta metropolitan area.² Additionally, the missed immunization opportunity rate of 243 (28%) of 871 appointments kept compares favorably with the rate of 50% found in another recent urban missed immunization opportunity study¹¹ that intervened at the parent level rather than at the provider level. In this study, contamination of the effect from the intervention to control group is a concern, since the intervention was supplied to one subgroup of residents who work closely with members of the control group. The lack of a demonstrable before-after difference in the clinic's immunization levels speaks against a significant contamination effect.

Children who had adequate numbers of visits during the second year of life were significantly more likely to be completely immunized at age 2 years, illustrating the importance of consistent follow-up with the immunization provider. That half of the missed immunization opportunities for scheduled visits were owing to missed appointments highlights the influence of appointment failure on immunization opportunities. Immunization or appointment reminders are known to be effective in improving immunization levels in other locations.¹² Another beneficial strategy with demonstrated effectiveness is the use of prompts from nurses given to the resident immediately before or during the visit.¹³ This strategy will only prove beneficial during appointments kept. The Health Plan Employer Data Information Set 3.0 criteria recognize the importance of consistent follow-up during the second year of life by including only children continuously enrolled between ages 10 months and 2 years in its measure of any given provider's immunization level for 2-year-old children.⁹ Our residents were surveyed at the end of the study on their attitudes regarding appointment failures. Residents from both groups expressed a sense of powerlessness over their patients' appointment failures. The lack of control residents feel stems from the multitude of clinic and patient factors over which the residents have little control, including scheduling conflicts, third-party payer changes, and transportation difficulties. It is possible that up-to-date immunization status and adequate numbers of visits during the patient's second year are both indicators of a more generalized compliance factor of children's parents or families.

Other potential causes of missed immunization opportunities in appointments kept are overcautious interpretation of immunization contraindications^{14 - 15} and a reluctance to give multiple vaccines.^{16 - 17} At the time of our study, most residents and faculty recommended diphtheria and tetanus toxoids and acellular pertussis vaccine rather than diphtheria and tetanus toxoids and pertussis vaccine, and inactivated poliovirus vaccine rather than oral poliovirus vaccine. Neither of these vaccines was available in combination with other vaccines, nor was the newer hepatitis B and H influenzae type b vaccine combination vaccine yet available. In a setting that tries to maximize every immunization opportunity, the lack of available combination vaccines can result in up to 4 injections per immunization visit. Continued availability of newer combination vaccines will minimize this potential source of missed immunization opportunities.

In our teaching setting, few immunizations were missed owing to parental refusal or medical contraindications. However, gaps in immunization documentation played a considerable role. In nearly 223 (46%) of 488 cases in which a vaccine dose was missed, the resident believed that the child was indeed up-to-date for the relevant vaccine based on undocumented parental reports. Previous studies¹⁸ have indicated that parental reports are not a reliable indicator of immunization status. Other studies have shown reluctance on the part of providers to immunize a child for whom there was incomplete immunization documentation in the face of parental reports that the child's immunization status was up-to-date.¹⁹ State laws requiring immunizations prior to school entry are a well-known factor for improved immunization levels. Knowledge of school-entry laws may bolster the residents' case as they attempt to convince reluctant parents, or as they attempt to document immunizations received. Immunization registries are a promising new development that will help to relieve this important source of missed immunization opportunities.

In our urban teaching clinic, retrospective written feedback to residents was an ineffective strategy for reducing missed immunization opportunities or improving immunization levels. Improving documentation of immunizations would potentially reduce missed immunization opportunities to vaccinate. Adequate follow-up during the second year of life is critical in achieving high immunization levels.

Accepted for publication March 17, 1999.

This study received support from the Ambulatory Pediatrics Association, McLean, Va, and the Emory Medical Care Foundation, Atlanta, Ga.

Presented as a poster at the Ambulatory Pediatrics Association Annual Meeting, New Orleans, La, May 1, 1998.

We are indebted to the residents who so graciously participated in the study.

Editor's Note: Another seemingly good idea bites the dust. Maybe the residents forgot to put reminders in their Palm Pilots.—Catherine D. DeAngelis, MD

Corresponding author: Cynthia T. Rust, MD, Emory University School of Medicine, Department of Pediatrics, 69 Butler St SE, Atlanta, GA 30303 (e-mail: crust@emory.edu).