Appropriateness of Urgent Referrals by Nurses at a Hospital-Based Pediatric Call Center FREE

IN THE PAST, it was estimated that 30% of a pediatrician's weekly time was spent on the telephone giving medical advice^1- 2 and, in an era of managed care, telephone care in pediatrics is expected to increase.^3- 6 Despite the increasingly important role of telephone care, 30% to 50% of pediatricians report that it is the most difficult part of their practice,^7- 9 particularly the after-hours calls.¹⁰ In response to the problem of "physician burnout" and the necessities of increasing the consistency of telephone advice and care, improving documentation to decrease medicolegal risk, and providing telephone care in a cost-effective manner, systems for managing after-hours telephone triage and advice have been evolving. In recent years, centralized call centers, often using computer-accessed triage systems, have been established to manage after-hours calls from many physicians. Estimates suggest that at least 25% of all general pediatricians currently sign out their after-hours calls to a call center.¹¹ Despite the rapid growth of call centers using triage and advice systems, there has been little evaluation of the clinical validity of this type of after-hours care.

The after-hours program in Denver, Colo, currently provides after-hours telephone triage and advice for the patients of 91% of private practice pediatricians in the Denver metropolitan area and 90% of private practice pediatricians in the state of Colorado. Nurses at the Denver After-Hours Call Center use the Pediatric Triage and Advice System (Access Health Group, Phoenix, Ariz), a computerized system with clinical decision-making algorithms. The objectives of the present study were to evaluate after-hours telephone care given by the Denver After-Hours Call Center with respect to (1) the appropriateness of urgent after-hours referrals, (2) the clinical characteristics of urgently referred patients, and (3) parental compliance with a recommendation for urgent evaluation.

The After-Hours Call Center of the Children's Hospital in Denver is staffed by trained pediatric nurses who use the Pediatric Triage and Advice System to answer after-hours calls from patients of subscribing pediatricians. The Community-Based Pediatric Urgent/Emergent Care Network (CBPEN) consists of 5 sites in the Denver metropolitan area where approximately 90% of patients referred by the call center for urgent after-hours evaluation are seen (Table 1). The sites are staffed primarily by board-certified pediatricians or pediatric emergency medicine physicians who are salaried faculty at the University of Colorado, Denver, or the Children's Hospital, Denver, and do not depend on referrals from the Denver After-Hours Call Center as a source of income.

At the Denver After-Hours Call Center, pediatric nurses identify the presenting complaints of the caller, access the appropriate computerized algorithms, and ask a series of questions dictated by the algorithm. Based on the responses, the algorithm gives a recommended disposition. Possible dispositions include (1) the patient should be referred for an urgent after-hours evaluation, (2) the patient should be referred for evaluation the next day at his or her primary care physician's office, or (3) the patient should be seen at a later date or requires home care and advice only. In addition, if the nurse feels that the caregiver is unable to give an adequate history or if there are worrisome features to the history that are not captured by the algorithm, the nurse may override the algorithm and upgrade the disposition decision to a more urgent disposition category. Likewise, a parent who feels uncomfortable waiting for an evaluation at a later date may override a nurse's disposition and be seen for an after-hours evaluation. If an after-hours evaluation is recommended, the patient is referred to 1 of the 5 CBPEN sites and a fax including the algorithm reason for after-hours referral is automatically sent to the after-hours site and the primary care physician. Patients are referred to sites outside the CBPEN approximately 10% of the time on the basis of physician or parent preference, payer constraints, or geographic factors.

The study population included patients of subscribing Denver pediatricians who called after office hours regarding a medical problem, were triaged by the After-Hours Call Center nurses, and were referred to 1 of the 5 urgent/emergent care sites. We collected data for 2 weeks at each of the 5 sites. The 2-week data collection periods were distributed equally over a 1-year period from October 1996 to October 1997. Because we collected data at only 1 site for each 2-week period, the total volume of urgent calls for the 5 sites together was representative of the volume of urgent calls in one 2-week period for the entire Denver metropolitan area. In determining the urgent referral rate during the study period, the numerator was the total of urgent referrals at the 5 sites and the denominator was estimated by calculating the mean of the total calls in metropolitan Denver during the five 2-week periods of data collection. If a child was referred more than once during the study period, only the first referral was included. Calls in which the parents overrode the algorithm disposition were not included, as they were not representative of triage recommendations of the After-Hours Call Center.

Appropriateness of referral was determined by a questionnaire completed by the attending physician evaluating the patient at the CBPEN site. The questionnaire, given in Table 1, was developed and pilot-tested by a group of general pediatricians and pediatric emergency medicine physicians and was designed to capture factors important to practicing pediatricians in determining urgency of care. If physicians responded positively to any of the questions or if the patient was admitted to the hospital, the referral was judged to be an appropriate referral. In addition, to assess reasons for referrals judged inappropriate, we asked the question "Was the history obtained by the after-hours nurse that resulted in same-night disposition accurate (please refer to reasons on the after-hours fax)?"

Diagnoses, clinical interventions, and final dispositions were collected by the evaluating physician at the time of evaluation. Results of tests or final disposition data not available on the night of evaluation were collected from review of the medical records the morning after evaluation. Parental compliance was assessed by review of the after-hours site patient database. Reasons for noncompliance were assessed by telephone interview with the person who originally called the After-Hours Call Center 24 to 48 hours after the referral. The questionnaire gathered information about demographics, parental satisfaction, and compliance and consisted of 41 closed-ended questions and 2 open-ended questions. The survey was adapted from the Behavioral Risk Factor Survey.¹² Our study protocol was approved by the Colorado Multi-institutional Review Board, Denver.

Proportions, as well as contingency tables and χ² analysis, were employed. For proportions of particular interest, 95% confidence intervals are calculated, using the exact binomial method. All analyses were performed using commercial software (SAS software version 6.12; SAS Institute, Cary, NC).

The mean number of total calls to the After-Hours Call Center during the study period was 2223, which was representative of the volume of calls for a 2-week period for patients in the Denver metropolitan area. Of these, 509 (22.9%) were referred for urgent evaluation and 454 (89.1% of urgent referrals) were referred to 1 of the 5 CBPEN sites. A total of 409 (90.0%) were eligible for our study. Reasons for ineligibility included parental overrides of less urgent disposition categories (n=15), repeat cases (n=12), or age older than 18 years (n=1). In addition, 17 subjects were excluded because they had "walked in" to the hospital and were subsequently triaged over the phone. This latter group was excluded because of concern that the nurse's disposition decision might be affected by knowledge that the family was already at the hospital.

Telephone interviews of the person who telephoned the After-Hours Call Center were completed for 79.9% (n=362) of the referred sample. Most callers were women (87.0%) and were aged 25 to 44 years (76.5%); of this group, most had at least 1 year of vocational school or college (75.1%), were unemployed (68.8%), and had another wage earner in the household (75.1%).

Of the 409 eligible patients referred, 70 (17.1%) of the families did not comply and were not evaluated after hours. Of the 339 families that complied, evaluating physician questionnaires were completed for 332 (97.9% of those evaluated and 81.2% of all eligible cases). As shown in Table 2, the percentage of referrals judged to be appropriate varied between 81.8% and 95.1% at the sites, with a mean for all sites of 90.7%. The differences between sites were not statistically significant. Of the total evaluations, 51.2% were completed by physicians boarded in general pediatrics, 30.4% by physicians boarded in pediatric emergency medicine, 16.3% by physicians boarded in general emergency medicine, and 2.1% by family practice physicians or dentists. The percentage of referrals rated appropriate varied from 87.3% for pediatric emergency physicians to 91.8% for general pediatricians, and there were no statistically significant differences by specialty.

To assess the maximum bias that could have been introduced by noncompliant families, we also calculated the rate of appropriateness based on the assumption that all noncompliant patients were inappropriately referred. This assumption decreased the appropriateness rate for all urgent referrals to 74.9%.

Figure 1 shows the percentage of referrals judged appropriate by physicians on the basis of different factors. Overall, 295 cases were judged appropriate on the basis of history, patient discomfort, or parental anxiety, and only 5 cases were judged appropriate without any of these factors. Physicians judged that the information obtained by the nurse that resulted in urgent disposition was inaccurate in 12.7% of the cases (n=42). These cases accounted for 53.1% of inappropriate referrals.

Final diagnoses and the percentages of each diagnostic category judged appropriate are given in Table 3. Of the total sample, 37.0% (n=123) had a diagnostic test, including a laboratory test in 22.0% (n=73), a radiographic examination in 19.9% (n=66), and a subspecialty consultation in 5.0% (n=5). In 43.5% (95% confidence interval, 38.0%-48.9%; n=144) of the cases, the evaluating physician felt a therapeutic intervention was necessary that night, including suturing or other surgical interventions (22.9%), nebulized medications (11.1%), intravenous fluids (9.7%), intravenous or intramuscular medications (9.0%), orthopedic procedures (6.2%), or other therapies (steroids, removal of foreign bodies, oxygen, resuscitation, or oral antibiotics) (14.5%). Of the total sample, 93.4% (n=310) were discharged and 6.6% (n=22) were admitted to the hospital.

Of the 70 families who did not have a documented visit, 80.0% (n=56) were reached for telephone interview. Of these families, 26.7% (n=15) said they had complied despite no documentation of their visit, and 12.5% (n=7) said they had complied but had gone to a facility other than the one to which they were referred. Of families who said they did not comply (n=34), major reasons given were confusion about what was being recommended (n=16), disagreement with the need for a visit (n=12), improvement in the child's condition (n=3), weather conditions (n=2), or not wanting to go to an emergency room (n=1).

In recent years, there has been dramatic growth in after-hours call centers using triage systems to handle patient calls from multiple physicians in a community and, sometimes, from distant sites. Hospital-based call centers and service bureaus are currently providing after-hours coverage for millions of children in the United States.¹¹ Despite this profound change in the way after-hours health care is being administered, there has been little assessment of the appropriateness of referrals made by call centers using triage and advice systems. To our knowledge, our study is the first to assess the appropriateness of urgent referrals of pediatric patients triaged by pediatric nurses using a computerized triage and advice system. Our results show that evaluating physicians judged urgent referrals to have been appropriate for approximately 90% of patients who complied with the recommendation for urgent evaluation. Approximately 80% of the families complied, with major reasons given for noncompliance being lack of understanding of the recommendations or disagreement with the need for evaluation.

Previous studies have shown that without prior telephone triage, substantial numbers of patients seek care in the emergency room for nonurgent problems that would be better managed in a primary care setting.^13- 28 Estimates of the proportion of inappropriate or nonurgent pediatric visits to an emergency room range from 25% to 75%.^23- 32 The wide range in estimates of "appropriateness" is due not only to differences in the period, populations, and settings studied, but also to a lack of consensus about the definition of an appropriate urgent or emergent care visit. Some studies have defined appropriateness retrospectively based entirely on final diagnoses, patient disposition,^23,27,33 or assessment of the patient's course.^20,22,34 Purely retrospective assessments, especially those based on diagnosis alone, cannot be used to determine whether a patient should have been seen. Prospective assessments of the need for care must assess the potential for serious illness, often based on limited information, rather than the ultimate outcome after evaluation.¹⁷ Classifying an emergency visit appropriate only if, in retrospect, there was "threat to life or limb"^35- 36 is a particularly flawed approach, although a survey of medical directors of health maintenance organizations in 39 states found that 92% used "life-threatening" emergency in defining emergency room access policies.³⁷ Other studies have attempted to define appropriateness prospectively by applying criteria based on the presenting complaint and vital signs.^17,38 DeAngelis et al²⁶ used a combination of retrospective and prospective factors and included a category incorporating "legitimate, acute parental concerns," such as extreme discomfort in the child.

Our criteria incorporated prospective factors that practicing pediatricians take into account in determining need for urgent care. We also included retrospective factors, such as physical findings and the urgent need for a test or therapy or hospital admission, but these factors identified few patients who were not identified by prospective criteria. Although our methods differ slightly, our data are most comparable to the study by DeAngelis et al,²⁶ who observed an appropriateness rate of approximately 67.5% for children who were not triaged prior to arrival at the emergency department. There are limited data about rates of urgent referrals by physicians taking calls directly with which to compare our findings. A study by Caplan et al³⁹ in 1983 reported that physicians taking after-hours calls recommended after-hours evaluation for only 6.4% of their patients, although an additional 5.6% went to the emergency room without their consent. The initial referral rate of 22% by the After-Hours Call Center nurses in this study is significantly higher, although only 10% were judged by the evaluating physicians to have been overreferrals. Although direct comparisons between the findings by Caplan et al and our findings are confounded by differences in the populations, methods used, temporal factors, the differences in referral rates may indicate that our data underestimate the rate of overreferrals or that physicians taking calls directly in the study by Caplan et al underreferred in some cases.

The conditions of the patients referred by the after-hours program were moderate in severity, with approximately 7% being admitted to the hospital, more than 40% requiring a therapeutic intervention on the night of the call, and almost 40% having a diagnostic test performed. The most frequent diagnostic categories seen were similar to those reported in a recent study of pediatric visits to a general emergency department in which the prior triage status was not specified.⁴⁰

Our study has several limitations: (1) Because we were unable to make a determination of appropriateness for referred patients who did not comply, the rate of appropriateness we report most accurately pertains only to patients who came in for evaluation. It is possible that secondary triage by the patient's parents decreased the rate of inappropriate referrals in our study. The actual rate of appropriateness for all urgent referrals is probably close to 90%, but could be as low as 75%, in the extreme case that every noncompliant patient was assumed to have been inappropriately referred. (2) The method used to assess appropriateness of referral in this study has limitations. The physicians completed their forms with knowledge of the diagnosis and, in some cases, ultimate outcome of the patient. Previous studies have shown that knowledge of patient outcome affects assessments of appropriateness of care.⁴¹ However, the fact that the rates of referrals judged appropriate did not differ substantially by category of final diagnosis (Table 4) suggests that the potential confounding of determination of appropriateness by outcome was not a major factor. (3) The determinations of appropriateness in this study also represent the subjective opinions of a group of physicians without a uniform clinical approach. (4) Physician bias either for or against after-hours call centers also could have influenced decisions about appropriateness of referrals. Any bias would not be expected to be financial, however, as evaluating physicians have no financial connections to the Denver After-Hours Call Center and are all salaried. In addition, patient volumes at the CBPEN sites are high, providing little incentive to underreport overreferrals. None of the evaluating physicians were involved in the development of algorithms used in the Pediatric Triage and Advice System or in running or supervising the After-Hours Call Center. (5) Our study did not address the important issue of potential underreferral of urgent cases by after-hours call centers. (6) Finally, the socioeconomic status of our study population was relatively high and most parents were married; therefore, our results may not be generalizable to all populations.

Because many after-hours call centers do not employ nurses with pediatric training, do not have backup coverage by pediatricians, and do not use algorithms developed by pediatricians, our results may not be generalizable to all after-hours call centers. The marketing of after-hours triage systems and call centers has progressed much more rapidly than the investigation into the clinical effectiveness of this type of care or the regulation of the quality of after-hours telephone care. The newly created Provisional Section on Pediatric Telephone Care and the Committee on Practice and Ambulatory Medicine¹¹ of the American Academy of Pediatrics, Elk Grove Village, Ill, has recently developed guidelines for the safe operation of pediatric call centers. Hopefully, such guidelines will result in the development of standards of care in this area and, possibly, accreditation requirements for call centers. Further investigation is needed to examine underreferral (especially in high-risk or special-needs populations), the appropriateness of nonurgent referrals, and the effect of this type of after-hours care on the patient-physician relationship.

Accepted for publication September 9, 1999.

This study was supported by grants from the Research Institute at The Children's Hospital, Denver, Colo (Dr Kempe); and the Center for Research in Demand Management and Telephone Care, Denver (Dr Kempe).

Corresponding author: Allison Kempe, MD, MPH, Children's Hospital, 1056 E 19th Ave, B032, Denver, CO 80218.

Editor's Note: I wonder what would have happened if a parent directly accessed the computerized algorithms first, made a decision, and then did it all over again with the nurse asking the questions. Surely some money-managing mogul would pay for that study.—Catherine D. DeAngelis, MD