From the HealthPartners Research Foundation, Bloomington, Minn (Dr Rolnick and Mss Flores and Paulsen); and Minnesota Department of Health (Ms Thorson), St Paul.
To assess 2 established methods of identifying children with special health care needs (CSHCN) within a health plan population for intensified service coordination.
The tools tested were the Questionnaire for Identifying Children With Chronic Conditions (QuICCC) and the Clinical Risk Grouper (CRG) software. The QuICCC was administered by telephone to the parents of 517 children. The CRG software tool was then applied to the health plan database. The accuracy of identifying the target population was assessed by a single trained reviewer by comparison with the comprehensive medical record.
According to the QuICCC, 37.1% of the parents surveyed had CSHCN. According to the CRG, 11% of the health plan's pediatric population was categorized as CSHCN. The medical record review agreed with overall QuICCC findings in 53% to 61% of cases and overall CRG findings in 66% to 73% of cases.
Administering the QuICCC was a time- and labor-intensive endeavor with a relatively low overall level of sensitivity. The CRG was less labor intensive with slightly higher sensitivity. Identifying the target population in an effective and efficient manner remains a challenge for health plans.
APPROPRIATELY identifying children with special health care needs (CSHCN) and ensuring the provision of needed services, including intensified service coordination, to this population is an area of interest to researchers, advocates, and health care practitioners. It is critical for health plans to be awa re of the scope and magnitude of special needs of children with chronic or disabling conditions within their membership. Because the target population is variably defined, however, identifying CSHCN has been an ongoing challenge. A variety of tools have been developed to aid in this effort; some have been tested and validated, whereas others are just being released to the market.
Under the auspices of a grant from the Maternal and Child Health Bureau to the Minnesota Department of Health, HealthPartners Research Foundation became involved in the SAFE (Special Needs Assessment, Follow-up, and Education) at Home Project. The SAFE at Home Project has 2 goals: to improve the identification of CSHCN within health plans and to support primary care physicians in providing a "medical home" to this population based on the model developed by the American Academy of Pediatrics.1 This article outlines the HealthPartners Research Foundation's efforts toward the identification goal of the project.
A Definition and Identification Advisory Committee to the SAFE at Home Project, composed of community experts in the target population or in identification methods, was established. This group examined a number of available tools to assess their strengths, weaknesses, and applicability to the project goals. Tools considered included the classification of congenital and chronic health conditions, the disability payment system, the operational criteria for identifying disabled persons using MMISII (Medicaid Management Information System) data, the Questionnaire for Identifying Children With Chronic Conditions (QuICCC), and the Clinical Risk Grouper (CRG) software (3M Health Information Systems, Murray, Utah).
Based on the committee's input, the QuICCC and the CRG software were identified as viable options for identifying CSHCN within a managed care population. The committee and the project staff believed that the QuICCC survey tool could capture the broadest population and would have the benefit of incorporating parental perceptions, whereas the CRG software could capture a more targeted population based on available electronic data. Although neither tool was originally designed for the purpose of identifying children within a health plan for intensified service coordination, each tool was thought to have its own advantages, particularly in terms of yielding the most accurate and useful capture and being most feasible for use by the health plan in this type of effort. HealthPartners agreed to test both of the tools selected. The goal of this practical exercise was not to conduct a direct comparison between the 2 approaches to case identification but to assess how well each tool would individually identify CSHCN for intensified service coordination and how feasible each approach might be for ongoing implementation by the health plan.
HealthPartners is a health maintenance organization that provides comprehensive care to 73 727 children (birth through age 18 years) in its 18 owned clinics. These services include outpatient primary and specialty care, tracking of inpatient visits, laboratory procedures, and pharmacy coverage. The health plan population reflects the racial composition of the local community, which is 85% white, 6% African American, 5% Asian American, 4% Hispanic, and less than 1% American Indian. Members of the health plan are enrolled primarily (84%) through employment-based plans but also through Medicare (8%), Medicaid (5.8%), and MinnesotaCare (2%), a subsidized health care program for residents of the state who do not have access to health insurance.
All research involving HealthPartners members is closely monitored by the health plan's institutional review board, which is responsible for protecting patient rights and confidentiality. Patients are assured confidentiality and anonymity and have the option of electing not to participate in research. Data are only reported in aggregate form. All aspects of this study were reviewed and approved by the institutional review board.
The QuICCC was developed and copyrighted by Stein et al2 in 1997 to operationalize a noncategorical definition of chronic conditions. The 39-item interview tool asks questions of parents about various aspects of their child's functional ability and limitations, service use, and reliance on compensatory mechanisms. The QuICCC is a validated scale that defines CSHCN to include those with (1) a disorder that has a biological, physiologic, or cognitive basis that (2) has lasted or is virtually certain to last for at least 1 year and (3) produces limitation of function, dependence on compensatory mechanisms, and need for related ancillary services.3 The tool is not based on specific diagnoses but rather focuses on the consequences of special health conditions and was designed for epidemiologic purposes.
Permission to administer the QuICCC was obtained from its developers in September 1999. Interviews were conducted using the QuICCC survey with a sample population of parents of children between the ages of 3 and 12 years, the age group that was of most interest to our practitioners. Originally, it was thought that clinic staff could conduct 250 face-to-face surveys, with research foundation staff conducting 250 telephone surveys from a centralized site, to reach a target of 500 interviews. Because of time and staffing constraints at the clinics and despite a financial incentive, clinic staff members were unable to conduct any surveys; therefore, all surveys were ultimately conducted via telephone through the research foundation.
From the HealthPartners encounter database, a random sample of children between the ages of 3 and 12 years who had been seen at 1 of 5 of the health plan's clinics between September 1 and November 30, 1999, was generated. The clinics selected were those with the greatest number of pediatric patients. One thousand names were chosen to ensure a final sample size of 500 completed interviews. Letters were sent to the parents of all children on the list to inform them of the purpose of the study and to let them know that they might be contacted for an interview. Parents were given the option of refusing to participate and the opportunity to let the interviewers know the best time to reach them by either calling the research foundation or returning a postage-prepaid reply card that was included with the letter.
Telephone interviewers received training and practice using the QuICCC and were instructed to complete a cover sheet documenting the interview for each respondent. Interviewers were instructed to make at least 3 call attempts at various times throughout the day before declaring the subject "unreachable." Telephone interviewers documented the start and stop time of each interview and made note of any specific diagnoses mentioned during the conversation. A total of 517 interviews were conducted.
3M Health Information Systems and the National Association of Children's Hospitals and Related Institutions (NACHRI), Alexandria, Va, developed the CRG software in 1999 and released it to the public in 2000. The CRG is a population-based classification system that uses diagnostic information from claims and encounter databases and assigns each patient to a single, mutually exclusive clinical risk category and severity level.4 The CRG is based on a premise similar to diagnosis-related groups in that it provides a classification system for individual characteristics and service use; however, the CRG was designed to predict future resource use. The CRG uses an operational definition of chronic health condition that incorporates 3 elements: (1) physical, mental, emotional, behavior, or developmental disorder; (2) expected to last at least 12 months or longer or having sequelae that last at least 12 months or longer; and (3) requires ongoing treatment and/or monitoring. The CRG was designed for use by health plans and communities as they address population-level health issues.5
At the start of our project, the CRG software had not yet been released to the public. In May 2000, HealthPartners Research Foundation entered into a research agreement with 3M Health Information Systems that allowed us to test the software before marketing. Originally, HealthPartners agreed to provide a data file to 3M Health Information Systems to enable them to run the software for the health plan as part of their beta testing process, but because of timing and staff constraints within 3M, HealthPartners ultimately ran the software in-house.
HealthPartners has a claims-based data system that contains demographic, membership, and utilization information on each member. HealthPartners claims and encounter databases were used to gather the required data elements for members who were between the ages of 3 and 12 years and had as their primary clinic designation 1 of the same 5 clinics that were used for the QuICCC interviews. A combination of utilization and financial data was used to create the "provider type" and "site of service" fields. Initially, the study variables were the same 3-month time frame that was used for the QuICCC data pull. Early efforts at working with the CRG indicated that the 3-month criteria would not be sufficient for a true assessment of the capacity of the software, which had previously been tested analyzing service data from periods of at least 6 months to 1 year or more. At the suggestion of a 3M consultant, therefore, the time frame for our analysis was expanded to 9 months, encompassing 3 months of claims experience before and 3 months following the original 3-month window (June 1, 1999, to February 28, 2000).
Following the implementation of the QuICCC and the CRG, a stratified random sample (n = 217) of medical records was targeted for review to obtain clinical assessment of "special health need" status and to examine agreement between what was found in the medical records and the results produced by each tool. The sample included 45 records for children identified by QuICCC only, 104 records for children identified by CRG only (core health status groups 2-9), all 46 children identified by both tools, and 22 children identified by neither tool, which served as an internal validity check to control for potential bias in our abstractor. Of this sample, we were able to locate and review complete records for 194 children (20 records were not located, and 3 records were empty). We ultimately removed children in CRG core health status group 2 (history of significant acute disease) from our analysis. The elimination of this category was done at the suggestion of the software developers based on the goals of our project to focus on children with chronic conditions. Therefore, our sample size was reduced by 10 to a final sample of 184 complete records reviewed. Thus, 42% (38 + 43 = 81 of 192) of the QuICCC-identified cases and 12% (82 + 43 = 125 of 1050) of the CRG-identified cases were reviewed.
A specialist in developmental pediatrics with extensive experience working with CSHCN in our health plan who also served as a consultant to the SAFE at Home project assisted project staff and the advisory committee in the development of a simple record review checklist that was used to identify indicators of special health care need in the medical record. This list was reviewed by our state health department collaborators before its use. The checklist was not designed to be an exhaustive list of all possible pediatric health conditions; rather, it was an intuitive list based on what a practitioner serving CSHCN would expect to see in the medical records of the type of child who might benefit from intensified service coordination from their health plan. For most conditions listed on the checklist, any mention of the condition, diagnosis, or pattern of service use indicative of a special health condition in the medical record was considered a positive identification of special health care need. For a few conditions where severity would have an impact on assessment of special health care need, additional criteria were applied by the medical record abstractor in making a clinical assessment of CSHCN status. The checklist was designed for this practical exercise and has not been tested or validated elsewhere. The checklist is available from the authors on request.
The entire available medical record, including primary care, specialty care, mental health care, and laboratory services, was reviewed for each case by a single nurse record abstractor who was trained in the use of the checklist by our pediatrician consultant. The record abstractor assigned each case to a "yes," "no," or "possibly" CSHCN category based on what was found in the record. Cases placed in the "possibly" category were those for which the record review was considered inconclusive because a determination of CSHCN status could not be made based on what was found in the medical record. In our results, we report comparisons between medical records and the QuICCC or CRG as a range. The lower figure assumes inconclusive records were probably not indicative of CSHCN status; the higher figure treats all inconclusive records as falling within the CSHCN category.
Of the 517 children from the general pediatric population whose parents were surveyed, 192 (37%) met the QuICCC criteria of special health care need. Ninety-two members (48%) of this group answered "yes" to only 1 question on the survey, whereas 38 (20%) answered "yes" to 2 questions, 18 (9%) answered yes to 3 questions, and 44 (23%) answered "yes" to more than 3 questions.
Of those identified as CSHCN, 136 (71%) had identified compensatory needs (eg, needing equipment to eat, see, hear and communicate, walk), 116 (60%) had special service needs (eg, needing physical or occupational therapy, nursing care or medical procedures, special arrangements at school), and 54 (28%) had functional limitations (eg, physical or mental or emotional delay, restrictions in activity, difficulty in activities of daily living). Reported use of medications (55%) and visits to the physician beyond the number expected (39%) were the leading indicators of special health care need for QuICCC-identified children.
Of the sample of medical records reviewed for those identified by the QuICCC (n = 81), 53% to 61% were in agreement. Agreement between the QuICCC and the medical record review was highest (87%-96%) for children whose parents answered "yes" to more than any 3 questions on the QuICCC (Table 1).
According to the CRG, approximately 11% of the children in the HealthPartners database had a special health care need. Among the CSHCN-identified children, the CRG categorized 386 (4% of the total pediatric health plan population) in the single minor chronic disease group and 572 (nearly 6% of the total pediatric health plan population) in the single dominant or moderate chronic disease group (Table 2).
Of the sample of medical records reviewed for those identified by CRG (n = 125), overall 66% to 73% were in agreement with core health status group 5 (single dominant or moderate chronic disease), having an agreement rate of 77% to 87%. Medical record review confirmed CSHCN status of the 1 child in core health status group 6 (significant chronic disease in multiple organ systems), 1 of 2 children identified in core health status group 8 (dominant, metastatic, and complicated malignancies), and all 10 children identified in core health status group 9 (catastrophic conditions) (Table 3).
Forty-six children were identified as CSHCN by both tools, and we were able to locate and review complete records for 43 children in this group. The medical record review indicated agreement for 30 to 33 (70%-77%) of these cases. Twenty-five (58%) fell into CRG core health status group 5 (single dominant or moderate chronic disease); 11 (25%) answered "yes" to only 1 QuICCC question, whereas 16 (37%) answered "yes" to 4 or more questions.
Of the 21 complete records reviewed for children not identified as CSHCN by either tool, 18 to 19 (86%-90%) agreed with the tools' assessment of no special health care need. In 2 cases, the record review indicated a special health care need, although the tools did not, and in 1 case, the record review was inconclusive.
Identifying CSHCN continues to be a challenging task. The 2 tools selected for this project were well-researched and tested instruments that were thought to be useable and useful for health plans. In reports about the QuICCC, developers Stein and colleagues2 report that validity and reliability for epidemiologic and research purposes have been established for their tool, with further testing showing 94% sensitivity, 83% specificity, 86% positive predictive value, and 92% negative predictive value.6 The CRG is built on NACHRI's classification of congenital and chronic health conditions7,8 and a similar population-based classification system product developed by 3M Health Information Systems and has been reported to do a good job of identifying CSHCN, particularly those with conditions requiring frequent interaction with the health care system.9 Our experience, however, did not reflect similar overall sensitivity except for children with 3 or more "yes" responses to the QuICCC or for children in core health status groups 5, 6, or 9 using the CRG.
For a child who is truly in the target population of those having a special health care need who would benefit from intensified service coordination, we would expect at least some indication and/or verification of the situation in their medical record. HealthPartners has extensive and detailed medical records for each member of the health plan. All clinic visits, laboratory services, referrals, and hospitalizations are documented, as well as notations about mental, social, and educational needs. Therefore, we compared the results of medical record review to the findings of the tools, and the results were rather unexpected, with 25% to 30% of the children identified as CSHCN by both tools showing no special health care need in their medical record. Individually, each tool provided results with even less agreement with medical records. We would have expected a higher rate of agreement between the medical record and the tools to identify CSHCN.
Although the tools tested in this study are appealing conceptually, implementing them presented considerable challenges. The QuICCC was time and labor-intensive. It is designed to capture a broad segment of the population and does not provide a stratification system based on severity. This seemed to make the results somewhat unreliable. Although the developers suggest that 1 "yes" answer should trigger concern, the QuICCC seemed to be most reliable (87%-96%) for those children whose parents answered "yes" to more than 3 questions. In our study, the instrument's overall lack of accuracy in identifying CSHCN patients (53% in clear agreement and up to 61% if we assume that all "inconclusive" cases would be found to have a special health care need) makes it unlikely to be adopted by a health plan based on 1 "yes" response. We could only consider using this instrument as a vehicle for identifying children to bring to the attention of physicians for enhanced "medical home" services, such as intensified service coordination for those individuals with 3 or more "yes" responses.
Although some may have concerns about the use of interview techniques in terms of confidentiality and parental perceptions of the risks and benefits of CSHCN identification in terms of future services, our sample of parents did not seem to underreport special health care needs; in fact, the percentage with special health care needs was much greater than expected. Since parents of children with special needs often assume the role of advocate for their child, we believe it is unlikely that they would underreport their child's need for services. Furthermore, parents would have little motivation to overreport, since our system is capitated with set benefit packages; therefore, their children can be seen as often as they wish with minimal copayments.
Unfortunately, although we were able to implement the QuICCC through the health plan's research foundation under the auspices of this grant, clinical personnel were unable or unwilling to add this to their workload. Many health plans do not have a centralized research entity to facilitate implementation. It is unlikely that a health plan would be able to devote ongoing resources to this method of identification, especially when the results did not attain the desired level of sensitivity. Stein et al10 and Bethel et al11 studied whether shorter tools to identify CSHCN would yield results as accurate as the assessment of the original QuICCC. These authors found that the shorter version of the QuICCC, which is composed of 16 questions (the QuICCC-R), and the CSHCN Screener could be used as acceptable alternatives for screening purposes. Although these abbreviated tools have the potential for being more feasible for implementation in a health plan such as ours, we would need to examine the rate of accuracy for these instruments within our health plan to determine if they would be likely to be useful. Although others may now be using these shorter tools as the primary means to screen for CSHCN within a clinical setting, we believe this exercise of examining the ability of the QuICCC to identify CSHCN within a managed health plan is worthwhile because the longer tool may still be used for epidemiologic purposes.
When we tested the CRG, it required a much larger amount of up-front learning time to understand and install the software than we expected based on our conversations with the developers. Now that the product has been released to the public, some of the glitches that we encountered may have been minimized. The most reliable CRG categories were those with extremely severe conditions, such as core health status group 6 "single chronic disease in multiple organ systems" and core health status group 9 "catastrophic conditions," with core health status group 5 "single dominant or moderate chronic disease" also showing high agreement with medical record review. Nevertheless, although the CRG requires less implementation effort than the QuICCC, overall agreement with medical record review was lower than we had hoped (66% in clear agreement and up to 73% if we assume that all inconclusive cases would be found to have a special health care need) in identifying the target population for intensified service coordination. Furthermore, for the CRG to be most effective at identifying CSHCN, a minimum 6-month enrollment is needed to obtain enough clinical information for accurate assessment of service coordination needs. Some may have concerns about the ability of a tool such as the CRG to identify newly enrolled patients or patients who do not have regular interaction with the health plan. Although we initially intended to use a 3-month time frame in our study to mirror the criteria for our QuICCC sample, we ultimately used a 9-month data capture and believed that the enrollment requirement would not be a significant limitation for our use of the tool.
This study has several limitations. We chose to focus our exercise on children aged 3 to 12 years, which may affect the generalizability of overall results. Since the CRG is based on utilization data and children younger than 3 years are often high users of health care services, our narrow age range may have affected the accuracy rate for the CRG. These ages, however, were those of most interest to the practitioners in that they believed it would be best to remove very young (<3 years) children from this examination, since there are many conditions seen in this age group that resolve themselves during the first few years of life. Furthermore, within our system, children approaching their teenage years often switch to family practitioners. Therefore, the upper age limit of 12 years allowed us to focus on a population that would be seen by pediatricians. In addition, it was beyond the scope of our exercise to use the full capacity of the CRG, which is designed to give extensive descriptive information at alternate levels of aggregation. The program is sophisticated and enables the user to highly refine searches. Knowing that the use of extensive programming and/or analysis of extensive output to obtain more detailed category- or severity-level information would not be considered feasible in a sustained manner by the health plan, we limited our exercise to capture the most basic and easiest to obtain CRG core health status groups.
In addition, our samples for the 2 instruments were not directly comparable. Our QuICCC sample was based on recent appointments, whereas our CRG sample was based on primary clinic affiliation. Because of these differing approaches, only 81% of the QuICCC sample was in the total population sampled by the CRG, limiting our ability to conduct a direct comparison between the 2 instruments. However, this was not the goal of our exercise. Although direct comparisons might be of interest scientifically, our major aim was to test the accuracy and feasibility of use for each approach independently by examining its concordance with the medical record.
Finally, our medical record abstraction checklist created by our specialist in developmental pediatrics was not a standardized or validated tool. The tool was designed specifically for this exercise to identify children who should be brought to the attention of a practitioner for possible intensified service coordination. Although our medical records are of excellent quality and provide a comprehensive view of all health plan coverage, they may not capture all services provided outside the health plan. Nonetheless, we believe this potential limitation of our medical records would affect few children covered by the health plan and have minimal impact on our overall findings. Furthermore, although there may be a certain amount of subjectivity involved in making interpretations from medical records, all cases in which there was any question of CSHCN status by the abstractor were also reviewed by our pediatric expert.
Identifying CSHCN remains an important area for health care practitioners. Ensuring the provision of needed services and linking families with appropriate available resources through intensified service coordination can make a difference in the quality of life of these children. The challenge still remains, however, whether screening approaches with adequate sensitivity and specificity can be found to assist practitioners in identifying these individuals in an effective and efficient manner.
Corresponding author and reprints: Sharon J. Rolnick, PhD, MPH, HealthPartners Research Foundation, 8100 34th Ave S, PO Box 1524, Minneapolis, MN 55440-1524 (e-mail: firstname.lastname@example.org).
Accepted for publication September 26, 2002.
This study was supported by grant MCJ-27R825-01-0 from the Maternal and Child Health Bureau, Health Resources and Services Administration, Department of Health and Human Services, St Paul, Minn.
We thank Jody Jackson, BSN, for assistance with editorial comments and manuscript preparation. We also thank Brooks Donald, MD, for his guidance throughout the study and his direction of the medical record review.
Numerous tools are available to assist in identifying children with special health care needs. Although theoretically sound and well researched, the usefulness and sustainability of these tools in real-world managed health plans have not been widely investigated. This study was conducted in a managed health plan, using real members and real available data. This exercise illustrates some of the practical limitations of the 2 tools tested in terms of sustained use by a managed health plan.
Thank you for submitting a comment on this article. It will be reviewed by JAMA Pediatrics editors. You will be notified when your comment has been published. Comments should not exceed 500 words of text and 10 references.
Do not submit personal medical questions or information that could identify a specific patient, questions about a particular case, or general inquiries to an author. Only content that has not been published, posted, or submitted elsewhere should be submitted. By submitting this Comment, you and any coauthors transfer copyright to the journal if your Comment is posted.
* = Required Field
Disclosure of Any Conflicts of Interest*
Indicate all relevant conflicts of interest of each author below, including all relevant financial interests, activities, and relationships within the past 3 years including, but not limited to, employment, affiliation, grants or funding, consultancies, honoraria or payment, speakers’ bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued. If all authors have none, check "No potential conflicts or relevant financial interests" in the box below. Please also indicate any funding received in support of this work. The information will be posted with your response.
Some tools below are only available to our subscribers or users with an online account.
Download citation file:
Web of Science® Times Cited: 10
Customize your page view by dragging & repositioning the boxes below.
Enter your username and email address. We'll send you a link to reset your password.
Enter your username and email address. We'll send instructions on how to reset your password to the email address we have on record.
Athens and Shibboleth are access management services that provide single sign-on to protected resources. They replace the multiple user names and passwords necessary to access subscription-based content with a single user name and password that can be entered once per session. It operates independently of a user's location or IP address. If your institution uses Athens or Shibboleth authentication, please contact your site administrator to receive your user name and password.