Measuring the Quality of Antibiotic Prescribing for Upper Respiratory Infections and Bronchitis in 5 US Health Plans FREE

Inappropriate antibiotic use is of great public health concern, both nationally and globally, because of its association with increased antibiotic resistance in the community.^1- 2 Bacterial strains that are increasingly resistant to antimicrobial agents pose a severe health threat; of particular concern is the emergence of community-acquired infections caused by multidrug-resistant Streptococcus pneumoniae,^3- 4 macrolide-resistant Streptococcus pyogenes,^5- 7 and methicillin-resistant Staphylococcus aureus.^8- 9 Community-wide antibiotic resistance reduces the effectiveness of currently available drugs to combat bacterial pathogens, and increases the individual patient’s risk of becoming infected with drug-resistant organisms.^1,10- 11

Upper respiratory infection (URI) and bronchitis are common reasons for children to visit the physician’s office and are conditions susceptible to inappropriate antibiotic use despite recommendations from clinical practice guidelines.^12- 13 Awareness of the consequences of inappropriate antibiotic prescribing has been increasing; however, the practice remains widespread. Although the overall rate of prescribing antibiotics for acute respiratory infections in children decreased during the 1990s,^14- 15 prescribing for URI remained unacceptably high at 221 antibiotic prescriptions per 1000 visits for children younger than 15 years during 1999 to 2000.¹⁵ During the 12-year period from 1989 to 2000, antibiotic prescribing rates for bronchitis did not change. In 1999 to 2000, bronchitis was diagnosed at 8.6% of all visits for infectious respiratory illness (including URI, acute otitis media, pharyngitis, sinusitis, and bronchitis) and the rate of prescribing antibiotics for bronchitis in children younger than 15 years was 773 antibiotic prescriptions per 1000 visits.¹⁵

Efforts to reduce inappropriate antibiotic use in the United States follow global efforts to combat drug-resistant pathogens. Results in other countries such as Iceland have shown correlations between low antibiotic use and low rates of penicillin-resistant pneumococci, demonstrating that a reduction in antibiotic use lowers the rate of antibiotic resistance in the community.¹⁶

One mechanism for underscoring the importance of specific medical practices is the public reporting of performance on selected quality-of-care measures. The most widely used system of performance measures, the Health Plan Employer Data and Information Set (HEDIS), allows for measurement of the degree to which evidence-based medical practices are being implemented, allows purchasers and consumers to select health plans based on such information, and raises general awareness of the need to improve measured practices.

A performance measure of antibiotic use for URI and bronchitis would shed light on the prevalence of inappropriate antibiotic prescribing by physicians and would potentially lead to decreased antibiotic use as well as diminishing antibiotic resistance in the community. The objectives of the current study were to test the feasibility of implementation, variability in health plan performance, and validity of a new quality-of-care measure that examines antibiotic prescribing rates for URI and bronchitis in children and adolescents. The results of this data analysis were ultimately used by the National Committee for Quality Assurance’s (NCQA’s) Committee on Performance Measurement to determine whether the measure met the standards required for inclusion in the HEDIS measurement set.

Our research team at the University of California, Los Angeles, collaborated with members of the quality-measure development group at the NCQA to develop this performance measure. We first convened a panel of 7 experts: 1 representative from the Centers for Disease Control and Prevention’s Respiratory Diseases Branch, 2 pediatric infectious disease specialists, 2 research experts in antibiotic overprescribing, and 2 health plan medical directors. Using NCQA’s Desirable Attributes of HEDIS measures as a guide,¹⁷ the group of experts developed a quality-of-care performance measure that examines the rate of antibiotic use for children aged 3 months to 18 years diagnosed as having URI and/or bronchitis. For the current study, we evaluated the newly developed measure with regard to 3 of these desirable attributes: the measure’s validity, the degree of variation in performance on the measure across health plans, and the feasibility of measure implementation.

To be eligible for the measure, a child member would have to (1) be covered by the plan’s commercial or Medicaid products, (2) be between 3 months and 18 years of age at the time of the encounter, (3) have both medical and pharmacy benefits, and (4) have an eligible visit for URI or bronchitis during the measurement period (July 1, 2000, through June 30, 2001).

A URI or bronchitis visit (International Classification of Diseases, Ninth Revision codes 460, acute nasopharyngitis [common cold]); 465, URI unspecified site; 466.0, acute bronchitis; and 490, bronchitis, not specified as acute or chronic) was considered eligible for inclusion in the measure if (1) no additional diagnoses for the visit were assigned that might be treated appropriately with antibiotics (bacterial diagnoses); (2) no antibiotic prescription had been issued during the 30 days before the visit date (which could represent an antibiotic refill for a previous diagnosis or visit rather than a new prescription linked to the current URI or bronchitis visit); (3) no additional outpatient visits with a bacterial diagnosis had occurred during the 7 days before or the 7 days after the visit date; and (4) the child member had been continuously enrolled in the health plan during the 30 days before and the 7 days after the visit date with no gaps in enrollment during this period.

If a child had more than 1 episode of URI or bronchitis that met all of the inclusion criteria, the first eligible episode during the measurement period was selected for inclusion in the measure. The panel chose to only include the first eligible episode of URI or bronchitis for 2 main reasons. First, they were concerned that including repeated visits by 1 child to 1 provider would result in a loss of information due to clustering of prescribing behavior within physician-patient dyads. Second, because this measure was being developed for potential inclusion in the HEDIS measurement set, the panel decided to be consistent with the majority of existing HEDIS measures where the individual patient rather than episodes of care is the unit of analysis.

We developed administrative data measure specifications to identify the eligible population and assess the management of URI and bronchitis cases. For all participating health plans, we defined the measurement period as a 12-month period starting on July 1, 2000, and ending June 30, 2001. Using administrative data, health plans identified all outpatient visits (excluding emergency department visits) during this 12-month measurement period with a qualifying diagnosis of URI or bronchitis (International Classification of Diseases, Ninth Revision codes 460, 465, 466.0, and 490). The health plan then identified the first such visit during the measurement period that met all of the inclusion criteria. Finally, using pharmacy data National Drug Codes, the health plan identified whether the member filled a prescription for antibiotics on or within the 7 days after the first eligible episode of URI or bronchitis.

The NCQA solicited health plans that participate in HEDIS to take part in feasibility and validity testing of the new performance measure. Five geographically dispersed and structurally diverse health plans agreed to test the newly developed specifications for the URI-bronchitis measure. These plans were located in the northeastern, southeastern, midwestern, northwestern, and southwestern United States. The types of health plans represented included capitated health maintenance organizations, group-model health maintenance organizations, and health maintenance organizations with preferred provider organization and independent practice association products. Three of the plans submitted commercial data only, while 2 plans submitted both commercial and Medicaid data (Table 1). With this test, we evaluated the feasibility of implementing the measure at the health plan level and the validity of using administrative data to measure performance in this clinical area.

Each plan abstracted medical records for a random sample of the eligible episodes they identified to verify and validate information obtained from administrative data. Using a medical record abstraction form designed by the research team, health plan personnel abstracted approximately 60 charts where no antibiotics were prescribed for the eligible visit and 60 for visits where antibiotics were prescribed, as indicated by the administrative data. The abstractors were asked to verify whether the medical record had evidence of the following data: the date of URI or bronchitis visit that was identified by administrative data, any exclusionary bacterial diagnoses assigned on the eligible visit date, a prescription for antibiotics issued during the 30 days before the visit date, any additional visits during the 7-day periods before and after the visit date where a bacterial diagnosis was assigned, and the date of any antibiotic prescriptions associated with the visit date, ie, on the visit date or during the 7-day period after the visit date. These data were used as a gold standard to verify the positive predictive validity of the administrative data concerning the diagnosis assigned on the eligible visit date, the presence of clinical exclusions, and sensitivity of the health plan’s pharmacy data for capturing antibiotics prescribed for the episode. Four of the 5 participating health plans agreed to perform the medical record abstractions. Only de-identified medical records data were transmitted to the research team.

We compared the rates of antibiotic prescribing for URI and bronchitis for the 5 participating health plans by means of logistic regression analysis, entering dummy variables for each health plan. We did this to test whether there was meaningful variation in performance on this measure among the 5 plans. We calculated exact binomial 95% confidence intervals for all reported proportions. In 4 of the 5 participating health plans, we used medical records data to examine the positive predictive validity of the administrative data for determining the diagnosis assigned, ie, how often the administrative data indicated that a child was diagnosed as having URI or bronchitis and the chart diagnosis was the same. We also used medical records data to assess the sensitivity of the pharmacy data for determining whether a drug was prescribed at a given visit.

After reviewing the results of the feasibility and validity testing for the measure, the expert panel made recommendations for revisions to the measure specifications. Both the study results and the revised measure specifications were reviewed by NCQA’s Committee on Performance Measurement to assess whether the measure adequately met the standards required for inclusion in the HEDIS measurement set.

All study procedures were reviewed and exempted by the University of California, Los Angeles, General Campus Institutional Review Board.

Pooling the data across the 5 participating health plans, there were 233 532 incident cases of URI and/or bronchitis during the measurement year. Of these cases, 114 404 were excluded because they met 1 or more of the 4 exclusion criteria (Figure). A total of 119 128 children aged 3 months to 18 years had at least 1 eligible episode of URI and/or bronchitis during the measurement year. Seventy-one percent (84 166) of these visits were first eligible episodes during the measurement year. Antibiotics were prescribed in 31% of these first eligible episodes (Figure).

Table 1 shows characteristics of the participating health plans. With the exception of plan C, all of the plans had similar age distributions among the child members who had an eligible episode of care.

The population-based diagnosis rates for URI and bronchitis for each health plan are shown in Table 2. Pooling data across the 5 health plans, the population-based rate of URI diagnoses was 177 cases per 1000 child members aged 3 months to 18 years and 59 cases per 1000 child members for bronchitis.

The antibiotic prescribing rates for URI and bronchitis varied widely among the participating health plans (Table 3) (range, 2%-75%; P<.001). However, antibiotic prescribing for bronchitis occurred more often than it did for URI, with 4 of the 5 health plans prescribing for more than 60% of bronchitis episodes (Table 4).

Using the medical record as the gold standard, the overall positive predictive validity of the administrative data for correctly identifying URI and bronchitis diagnoses was 89% (range, 81%-95%) (Table 5). Thus, the administrative data indicated a diagnosis of URI or bronchitis in 11% of cases where the medical record did not (false-positive rate). Medical records data also indicated a secondary (competing) bacterial diagnosis for the index episode of care in 20% of cases (range, 15%-27%) where the administrative data did not (false-negative rate).

The overall rate of agreement between the administrative pharmacy data and the medical records data for the 4 health plans for which these data were available was 88% (range, 86%-88%) (Table 5). The sensitivity of the administrative data for accurately identifying antibiotic prescriptions documented in the medical record was 86%. In 5% (21/465) of cases overall, the pharmacy data indicated antibiotics had been dispensed on or 7 days after the visit date, but there was no notation of prescribing an antibiotic in the medical record. In contrast, the medical records data indicated a prescription for antibiotics in 8% (37/465) of cases where the pharmacy data did not (false-negative rate).

The results of this feasibility and validity test indicate that there is a substantial degree of variability in performance on the measure and significant room for improvement in 4 of the 5 participating health plans. Similar to our finding that, on average, 20% of URI episodes resulted in a prescription for antibiotics (Table 4), a 1999 to 2000 analysis of the National Ambulatory Medical Care Survey found that antibiotics were prescribed in 22% of visits for children younger than 15 years diagnosed as having URI.¹⁵ Thus, although antibiotic prescribing for this condition was shown to be trending downward between 1989 and 2000,¹⁵ much room for improvement remains. For bronchitis, McCaig et al¹⁵ found no significant changes in rates of antibiotic prescribing, with 77% of visits resulting in an antibiotic prescription in 1999 to 2000. In the present study, 64% of bronchitis visits resulted in a prescription for antibiotics. Although this rate appears to be slightly better than that reported by McCaig et al, it still indicates a substantial amount of inappropriate antibiotic prescribing for this condition.

Reliance on administrative data for quality-of-care performance measurement is limited by potential inaccuracies of diagnostic coding for visits in these databases. Thus, we performed a medical records validation study to examine the positive predictive validity of the administrative data for correctly identifying URI and bronchitis diagnoses. In only 11% of cases did the administrative data indicate a diagnosis of URI or bronchitis that could not be verified in the medical record on the date of service. Of more concern was the finding that, on average, in 20% of cases, visits identified by administrative data as having no competing bacterial diagnosis assigned on the episode date in fact had a secondary bacterial diagnosis according to the medical record. This is problematic because this performance measure would count these visits as episodes of care where antibiotics were inappropriately prescribed, when in fact, according to the medical record, prescribing an antibiotic may have been appropriate. On the basis of these findings, the expert panel suggested that the benchmark for this measure should more accurately be placed around 20% rather than 0%, that is, a health plan’s performance would be considered very good on this measure if their overall rate of antibiotic prescribing for URI and bronchitis is 20% or lower. Even taking 20% as the benchmark, only 1 of the health plans participating in the study (plan B) demonstrated good performance on the measure (Table 3). If the new performance measure is adopted as a HEDIS measure, plans would likely have a strong incentive to improve the accuracy of coding for these conditions, and the benchmark could eventually be set closer to 0%.

Using pharmacy data for quality-of-care performance measurement can also undercount the extent of inappropriate prescribing, since these databases capture only antibiotic prescriptions dispensed. Thus, any prescription written but not filled will not be counted. In addition, prescriptions filled outside the health plan will not be captured. This potentially occurred in 8% of cases where we found evidence of an antibiotic being prescribed in the medical record for an episode of URI and/or bronchitis, but the pharmacy data did not capture the prescription. Thus, this measure may underestimate the true degree to which antibiotics are prescribed for these conditions. On the other hand, in 5% of cases, the pharmacy data indicated an antibiotic was dispensed but a prescription for antibiotics could not be verified in the medical record. However, these instances may not represent an overcount of prescriptions, since antibiotic prescriptions may have been called into a pharmacy after the visit occurred because the child’s case was worsening or not getting better. This type of prescribing may not be well documented in medical records.

On the basis of the results of the feasibility and validity testing of this performance measure, 3 main revisions were made to the measure by the expert panel. First, the panel concluded that comprehensively including all potentially relevant secondary bacterial diagnoses was a difficult if not impossible task. Thus, the decision was made to include only visits with a single diagnosis of URI. However, benchmark performance was still expected to be 20% rather than 0% because the medical records validation study showed that in 20% of cases the administrative data indicated a single diagnosis of URI or bronchitis while the medical records data indicated a secondary bacterial diagnosis. Second, the decision was made to drop bronchitis as a condition and focus on antibiotic prescribing only for URI. This decision was made because there was no way to recognize improved performance of a health plan resulting from decreasing the rate at which bronchitis was diagnosed in pediatric patients. The expert panel held the opinion that bronchitis should not be diagnosed more frequently than it is nationally (approximately 19% of all visits for URI and/or bronchitis in the outpatient setting in 1999-2000).¹⁵ However, consider the results reported in the last column of Table 4 for plans A and C. Plan A diagnosed bronchitis in 13% of cases (160/1225) while plan C diagnosed bronchitis in 55% of cases (5988/10 823). However, their prescribing rates for this condition were 68% and 61%, respectively. The expert panel believed that plan A had better performance than plan C for bronchitis because of their much lower rate of bronchitis diagnoses, yet this would not be reflected by the performance measure. In fact, the measure would indicate that plan C’s performance was slightly better then plan A’s. Thus, the decision was made to drop bronchitis cases from the measure and to focus on visits with a single diagnosis of URI. Finally, the window for capturing antibiotic prescriptions was narrowed from 7 days to 3 days after the eligible episode of URI. This was done because 94% of the prescriptions issued occurred during the 3 days after the episode date. This revision further avoids counting antibiotic prescriptions that may not have been linked to the eligible episode of care.

Our validation study has several limitations. Because we did not sample administrative data cases without a diagnosis of URI or bronchitis, we cannot assess the sensitivity of the administrative data for detecting all URI and bronchitis diagnoses noted in medical records. It is certainly possible that some cases of URI or bronchitis are assigned International Classification of Diseases, Ninth Revision, codes other than the ones used to identify these cases in the current study. To our knowledge, no studies have evaluated the sensitivity of administrative data for identifying URI and bronchitis diagnoses specified in the medical record. Investigations examining the sensitivity of administrative data for identifying various other diagnoses have found sensitivities in the range of 57% to 93%, with a majority reporting values greater than 80%.^18- 21 For the current study, we can comment only on the correctness or positive predictive validity of the administrative data for identifying URI and bronchitis diagnoses using the medical record as the gold standard for diagnosis. However, given that the population-based rates of these diagnoses in our administrative data (177 URI cases per 1000 enrolled members aged 3 months to 18 years and 59 bronchitis cases per 1000 child members) are similar to national population-based rates of these diagnoses (220 URI cases per 1000 population aged 0-15 years in 1999-2000 and 55 bronchitis cases per 1000),¹⁵ administrative data likely have reasonable sensitivity for identifying cases with these diagnoses.

While the administrative data analyses we have presented are well suited to assess validity of the measure and comparative performance among the participating health plans, they do not allow for investigation into why interplan variation in inappropriate antibiotic prescribing exists. Determinants of inappropriate antibiotic prescribing are best assessed by other methods of data collection, eg, patient surveys, as these factors in many cases are not available from claims databases, eg, physician perceptions of parent expectations.^22- 25

On the basis of the results of this validation study, it is clear that a significant degree of inappropriate antibiotic prescribing for URI and bronchitis continues to occur in diverse medical care settings throughout the United States. This study showed this measure to be a valid, albeit not perfect, measurement tool for inappropriate antibiotic prescribing related to pediatric URI and bronchitis. On the basis of the study findings, the NCQA Committee on Performance Measurement approved the revised version of the measure, Appropriate Antibiotic Prescribing in Children With URI, for implementation in the HEDIS measurement set in 2004.¹⁰ More than 90% of the nation’s health plans report on HEDIS measures. Implementation of this measure should help to focus health plan quality-improvement efforts nationwide on the critical public health issue of inappropriate antibiotic prescribing.

Correspondence: Rita Mangione-Smith, MD, MPH, Department of Pediatrics, UCLA, 10833 LeConte Ave, Los Angeles, CA 90095-1752 (ritams@ucla.edu).

Accepted for Publication: February 17, 2005.

Funding/Support: This study was conducted under funding from the Centers for Disease Control and Prevention, Atlanta, Ga, through a cooperative agreement with the Association of American Medical Colleges, grant MM-0163-02/02.

Acknowledgment: We thank the members of the NCQA Appropriate Antibiotic Use Technical Subgroup for their invaluable contributions to this project. We are fortunate to be able to draw on the experience, knowledge, and expertise of the following individuals: Richard Besser, MD, National Center for Infectious Diseases, Centers for Disease Control and Prevention, Atlanta, Ga; Jonathan A. Finkelstein, MD, MPH, Department of Ambulatory Care and Prevention, Harvard Medical School and Harvard Pilgrim Health Care, Boston, Mass; S. Michael Marcy, MD, Kaiser Permanente Southern California, Panorama City; Samuel Ho, MD, PacifiCare Health Systems, Santa Ana, Calif; Arch Mainous, PhD, Department of Family Medicine, Medical University of South Carolina, Charleston; Robert Scalettar, MD, MPH, Anthem Blue Cross Blue Shield, North Haven, Conn; and Ellen R. Wald, MD, Department of Pediatrics, University of Pittsburgh, Pittsburgh, Pa.