Use of the Pediatric Symptom Checklist in a Low-Income, Mexican American Population FREE

MORE THAN 25 years ago, the term new morbidity was coined to describe the increasing importance of childhood psychosocial morbidity among more easily recognized and increasingly curable pediatric ailments.¹ Today, there is mounting evidence that childhood behavioral and psychosocial problems have reached staggering levels, with recent epidemiologic studies^2- 9 showing prevalence rates as high as 17% to 27% in US children. Several studies^10- 11 have shown that minority and low-income children experience even higher rates of mental health and behavioral problems, with prevalence rates in some high-risk populations approaching 30% to 50%.^12- 14

There are numerous barriers to appropriate recognition of behavioral and psychosocial problems in children.^15- 16 Pediatricians do not receive sufficient training in behavioral problems of children,¹⁷ office visits are short,¹⁸ parents often do not bring up child or family mental health issues,¹⁵ and options for referral frequently are limited.¹⁴ In addition, when working with minority or immigrant families, clinicians are faced with language or cultural obstacles to obtaining the most accurate information on a child's well-being. These combined barriers result in pediatricians recognizing as few as 4% to 7% of children with significant behavioral problems or psychiatric disorders.¹⁹ Furthermore, as few as 11% to 25% of children who have their conditions recognized and diagnosed subsequently are referred to an appropriate mental health care practitioner.^{2,10,13,15,20}

A method to improve the primary care pediatrician's ability to recognize and appropriately refer children with behavioral or psychosocial problems is to systematically screen all children with a standardized instrument designed for this purpose.^16,21 One such screening tool, developed by Jellinek and Murphy,²² is the 35-item Pediatric Symptom Checklist (PSC), designed specifically for use by the pediatrician to screen for mental health problems in children ages 4 to 16 years in the primary care setting. Due to its brevity and easy scoring, the PSC is more appropriate for use as a screener in a busy pediatric setting than are comprehensive, long, and complicated-to-score psychosocial questionnaires, such as the Child Behavior Checklist (CBCL). During the past several years, the PSC has seen widening use and was evaluated recently for use nationally.²³ However, few studies have carefully evaluated the use of the PSC in low-income minorities,^8,24 and even fewer have specifically examined the utility of the PSC in low–socioeconomic status (SES), Spanish-speaking, Hispanic populations.^14,25

The purpose of the current study was to evaluate the use of the PSC in identifying behavioral problems specifically in a population of poor, Mexican American children seen in the setting of a primary care, community-based clinic. We used the CBCL as a standard measure of behavioral problems against which we evaluated the validity of the PSC. This comparison allowed us to calculate the sensitivity and specificity of the PSC in this population. In addition, we used receiver operator characteristic (ROC) curves to investigate whether there was an optimal cutoff score, with high sensitivity and specificity, for the PSC when used in this low-income, Spanish-speaking population.

The study was conducted in East Palo Alto, Calif (population, 28 000), from March 1 through December 31, 2001. The city is composed of a diverse, middle- and low-income population with a large proportion of Mexican American immigrants. The clinic site is a small community health clinic that serves the local adult and pediatric population for both well and sick visits.

Eligible children and adolescents included all those 4 to 16 years of age who were seen at the clinic for nonemergent, well-child care with a parent or guardian during the study period. Children seen for sick or acute care visits were not included. Each eligible child was enrolled only once.

A bilingual, bicultural, Mexican American research assistant approached eligible families while they registered in the clinic and obtained consent if they agreed to participate. Questionnaires were filled out while the family was waiting to be seen. All questionnaires were available in Spanish and English. Families were asked if they would like assistance, and in nearly all cases, the research assistant then administered the questionnaires verbally. Each family received a $10 telephone card for participating. An additional questionnaire was attached to the patient's medical chart for completion by the physician after the appointment. The physician did not have access to the results of the parent-completed questionnaires. The study received the approval of the Stanford Human Studies Research Board.

The research assistant noted on each visit whether the child was being seen for a well-child examination or for vaccinations only.

Each family completed a demographic questionnaire. Information gathered included child sex, race and ethnicity, maternal education, family history of mental illness or treatment, family income, insurance status, and household structure, including parental marital status, number of children, and total number of individuals living in the home. Information on immigration was obtained by asking whether the child, parent, or grandparent was born in the United States. If the child was born outside the United States, the family was asked how many years the child had been living in this country. Information on the primary language used in the home also was collected.

The questionnaires given to each family were the CBCL²⁶ and PSC.²² The physicians filled out the Physician Psychosocial Assessment Form (PPAF⁴), attached to each study patient's medical chart.

The CBCL is a parent-completed diagnostic tool that is available in 2 forms divided by age (1½-5 years and 6-18 years). The CBCL has 118 items that describe specific behavioral and emotional problems, plus 2 open-ended items for reporting additional problems. Parents rate their children for how true each item is now or within the past 6 months using the following scale: 0, not true; 1, somewhat or sometimes true; 2, very true or often true. Raw scores for 3 broadband factors and 8 syndrome scales are generated from summed items; raw scores then are converted to normalized T scores. T scores of 60 or higher are within the clinical referral range; higher scores represent more deviant behavior.

The CBCL has been widely used and is well validated in a variety of languages and socioeconomic groups. Investigators have used it as the criterion for diagnosis of behavior disturbance against which to gauge the validity of both screening instruments²⁷ and psychological interview measures.²⁸ The published Spanish version of the CBCL used in this study has been found to have good internal consistency and concurrent validity in Spanish-speaking populations.²⁹ The CBCL was chosen as the standard of comparison to avoid basing the calculation of PSC validity on a small subset of the data as done in previous studies using structured interviews as the standard^24,30- 32 and because the CBCL has been widely used as the criterion standard in previous PSC research in both middle-class^30,33- 35 and low-income minority populations.⁸

The PSC is a 1-page, parent-completed questionnaire with 35 items that screens for childhood mental health problems. Parents rate the frequency of behaviors and symptoms listed as "never," "sometimes," or "often." These responses are given 0, 1, or 2 points, respectively, with a range of possible scores from 0 to 70 points. A total score is calculated by adding the individual values for each item. This total score then is compared with an age-specific cutoff. The published cutoff for children aged 6 to 16 years is 28 points.³⁴ In the preschool population, 4 items pertaining to school are excluded, and the cutoff for children aged 4 to 5 years is 24 points, with a possible range of 0 to 62 points.³⁶ A score above the cutoff is considered positive and indicates a need for further assessment or referral. As part of the study, a translation of the PSC to Spanish was done, followed by a back-translation to English by a different person. No significant inconsistencies were found between the original and back-translated versions of the questionnaire.

The PPAF⁴ is a 13-category checklist of developmental and psychosocial problems that is based on a World Health Organization–sponsored, primary care, child-oriented classification system.³⁷ The categories include physical growth and development, sleep, cognitive/language, school, behavior, feelings, and parent-child relationships among others. Each category lists 3 to 8 specific problems. The number of items checked on each form was summed for a single total (with a maximum score of 32). The total score represents extent or numbers of behavioral problems (as opposed to clinical level of behavioral problems).

Demographic characteristics of the study population were determined. Descriptive analyses were used to examine the association of demographic characteristics with high scores on the PSC and CBCL to determine if specific risk factors were correlated with increased behavioral problems in the study population.

Means and standard deviations for the PSC and CBCL and the number and percentage of children who fell into the positive or clinical range on both scales were calculated. Because the PSC scores were nonsymmetrically distributed, Spearman ρ correlations were calculated among the PSC scores, the PPAF scores, T scores for the CBCL Total, Internalizing, and Externalizing scales, as well as CBCL subscales. The sensitivity and specificity of the PSC were calculated using the CBCL as the criterion standard, as has been performed in previous studies.^8,30

Receiver operator characteristic curves plot all possible pairs of true-positive rates (sensitivity) against false-positive rates (1 − specificity) as the definition of a positive score (ie, the cutoff point) is changed. In keeping with the most recent literature,^8,22 we used ROC curve analysis to examine the possible need for a new, more favorable, cutoff score on the PSC for use in this population of low-SES, Mexican American children.

During the study period, 253 eligible children aged 4 to 16 years attended the clinic for routine health care. The part-time research assistant was able to approach the parents of 214 (85%) of these eligible children, and 210 (98%) of those approached agreed to participate. There was a 100% completion rate of the questionnaires for a final sample of 210 children. The clinic practitioners filled out the PPAF, with a completion rate of 92%.

The average age of the children in the study was 7.5 years (SD, 3.5 years), and 45% were female. Eighty-four percent of the children were seen for well-child physical examinations, whereas the remaining 16% were being seen for immunizations or other routine screening. Complete demographic information is listed in Table 1. Although this study used a convenience sample, 95% of the patients are of Hispanic descent (Mexican American), with 86% of the families speaking only Spanish in the home. More than one third (38%) of the children were born outside the United States. There is ample evidence of the low SES of these families with low maternal education (53% had a less than seventh grade education), low incomes (77.5% families earned <$20 000 yearly), minimal insurance coverage (41% uninsured), and large numbers of individuals living together in the home. Although 32% of the children came from single-parent families (a known risk factor for childhood behavioral problems³⁸), only 9% of the mothers had never been married. Divorce, separation, or death of a parent accounted for most of these single-parent homes.

The range of scores with the mean and standard deviation for the PSC, CBCL Total, and CBCL Internalizing and Externalizing scales are presented in Table 2. Mean scores for the PSC, CBCL Total, as well as the CBCL Internalizing and Externalizing scales, were within the normal range for child behavioral problems. The PPAF scores also are provided in Table 2. When the data were examined for male and female children separately, we found no statistically significant differences in mean values of CBCL Total, CBCL Internalizing, CBCL Externalizing, PSC, or PPAF. The PSC scores were not significantly correlated with levels of maternal education (Kruskal-Wallis test, χ² = 6.74; P>.15) or level of family income (Kruskal-Wallis test, χ² = .02; P>.99). Therefore, all subsequent analyses treated the sample as a whole in regard to these variables.

Table 3 presents the positive scores for both the CBCL and the PSC. Using a cutoff score of 60 or higher, the CBCL Total scale determined that 13% of the children had borderline clinical or clinical levels of behavioral or psychosocial problems (with 20% and 9% on the Internalizing and Externalizing scales, respectively). The PSC screened 2 (1%) of the 210 children as positive. This conclusion results from using the cutoff points (≥24 for 4- to 5-year-olds, ≥28 for 6- to 16-year-olds) established by the authors of the PSC and published in several studies.^14,23,34 If a lower cutoff point recently proposed by Simonian and Tarnowski⁸ for use in low-SES minorities is used (≥24 for both 4- to 5-year-olds and 6- to 16-year-olds), the results are the same (1%). This is because no child aged 6 to 16 years (n = 119) screened positive (either ≥24 or ≥28) on the PSC.

The CBCL previously has been used as a criterion standard in several studies that specifically evaluate the PSC.^{30,33,39- 40} Thus, we also used the CBCL as a criterion standard and calculated the sensitivity and specificity of the PSC. When used in this population, the PSC had a sensitivity of 7.4% and specificity of 100%.

Using the Spearman ρ correlations, we calculated the association between scores on the PSC to the PPAF, the CBCL Total, the CBCL Internalizing and Externalizing scales, and the specific CBCL syndrome subscale scores. We found a high correlation between the PSC and CBCL Total, Externalizing, and Internalizing scores (r = 0.71, 0.68, and 0.54, respectively, P<.01 for all). This held true for most of the CBCL syndrome subscale scores as well (r = 0.39-0.68, P<.01 for all, Table 4). The few exceptions were found for the 1½- to 5-year version of the CBCL, with no statistically significant correlations demonstrated between PSC scores and the Anxious/Depressed, Somatic Complaints, and Withdrawn subscale scores (r = 0.20, 0.20, and 0.15, respectively). The PSC scores also were significantly associated with PPAF scores (r = 0.25, P<.01). In addition, we found a correlation between the CBCL Total and PPAF scores (r = 0.33, P < .01). This correlation is similar to the PSC-PPAF correlation, underscoring the usefulness of the CBCL as a criterion standard.

Our findings regarding the PSC (ie, low sensitivity and high specificity compared with the CBCL; high correlation with the CBCL) suggest that the PSC still could be an effective screener in this population, but the cutoff point might be too high. To evaluate this possibility and to determine if a lower cutoff point could improve sensitivity while maintaining high specificity, we used ROC curves. Three ROC analyses, corresponding to the CBCL Total, Internalizing, and Externalizing scales, were undertaken using ROC4 software⁴¹ (Figure 1). These analyses computed the sensitivity and specificity of every value on the PSC to determine which value would optimally discriminate children with and without behavioral problems, as measured by the CBCL. Equal weighting was placed on sensitivity and specificity to obtain the most efficient PSC cut point. In addition to the PSC scores, child age, sex, and number of years living in the United States were entered into the analyses to explore whether these demographic variables also contributed to group differentiation.

Thirteen percent of the sample (n = 210) exhibited behavioral problems as measured by the CBCL Total scale. Of all variables entered into the analyses, including the PSC and demographic variables, the PSC at the cutoff point of 12 was the most efficient discriminator (χ² = 86.62, κ = 0.64, P<.001). Thus, of all possible values on the PSC, a cutoff of 12 most correctly classified children with and without behavioral problems (sensitivity, 0.74; specificity, 0.94; efficiency, 0.914). For children with a PSC score of 12 or higher, the proportion of value positive on the CBCL Total was 0.65 (n = 31). Within the subgroup of 31 children with PSC scores of 12 or higher, ROC indicated that age was an efficient classification variable at a cut point of 7 years or younger (χ² = 11.14, κ = 0.59, P<.001). For children older than 7 years, the proportion of value positive was 0.89 (n = 18). Child sex and number of years in the United States did not further discriminate the 2 groups (ie, PSC score <12, PSC score ≥12).

The purpose of this study was to establish the utility of the PSC for early identification of behavioral and psychosocial problems in a low-income, immigrant, Spanish-speaking population of children. Using established cutoff scores, the PSC identified 1% of these children as positive for behavioral problems. Meanwhile, the CBCL, given concurrently, determined that 13% of the children had scores in the borderline-clinical or clinical range. Using the CBCL as a criterion standard (as done in previous evaluations of the PSC^33- 34), the PSC had a sensitivity of 7.4% and a specificity of 100% for behavioral and psychosocial problems in these low-income, predominately Mexican American children. ROC analyses suggested that an adjustment of the PSC cutoff score to 12 (down from 24 or 28, depending on age) yields increased sensitivity (74%) while preserving excellent specificity (94%) for effective screening for all ages in this specific population. With the adjusted cutoff score, 31 children (14.8%) would have screened positive on the PSC.

A limited number of carefully validated studies of the PSC have been completed in Hispanic populations. On review of the literature, we found 3 studies that specifically addressed the use of the PSC in Hispanic children. Results of the first study were reported in 3 articles. Two articles^14,25 reported on behavior problem rates as determined by the PSC in 2 age groups (4- to 5-year-olds and 6- to 16-year-olds) within a population of low-income, Spanish-speaking families in northern California. No measures of the validity of the PSC in these samples were reported. Notably, the authors remarked on the "substantially" lower rates of positive PSC scores in both age groups (7% in 4- to 5-year-olds; 10.6% in 6- to 16-year-olds) compared with previously reported rates of 22% in similarly low-income, black children.²⁴ A third article⁴² addressed issues of validity and compared the PSC with an overall rating of child functioning. In that article, which represented 663 low-income, Hispanic, preschool-aged children, the PSC positive rate was reported to be 9%, the sensitivity of the PSC was 75%, and the specificity was 77%. Taking the 3 articles together, the lower rates of positive PSC scores and the lower sensitivity in these primarily Mexican American families are unlike reports of other minority groups.²⁴

In a second study conducted in a low-income, Boston, Mass, community, Navon and colleagues⁹ included a subsample of Spanish-speaking families. For the total sample, 23% of children younger than 6 years screened positive on the PSC, whereas 27% of the children aged 6 years and older screened positive. The PSC was validated in this population using clinical interviews with subsequent interviewer completion of the Child and Adolescent Functional Assessment Scale (CAFAS) for children at least 6 years old⁴³ and the Preschool and Early Childhood Functional Assessment Scale (PECFAS) for children younger than 6 years.⁴⁴ Using the CAFAS/PECFAS as a criterion standard, the PSC was found to have 91% sensitivity and 65% specificity for the total sample. Within the total sample, the lowest level of PSC accuracy was found for preschoolers in Spanish-speaking families, for whom the sensitivity and specificity were reported to be 75% and 53%, respectively. The percentage of Spanish-speaking families in the total sample is not reported in the article. Because of these results, the authors made changes in the Spanish translation of the questionnaire.

In a third study, the PSC was used in mixed-SES pediatric outpatients from a health maintenance organization in a northern California county.⁴⁵ The authors found a PSC positive rate of 14% for Hispanic children (higher than 12% reported for whites and lower than 17% reported for blacks). The Hispanic families in this study were, presumably, not low income.

Our current study and the 3 studies described herein suggest a great deal of variability in Hispanic samples with regard to the prevalence of positive PSC scores (range, 1%-23%), PSC sensitivity (range, 7.4%-75%), and PSC specificity (range, 53%-100%). Jellinek et al²³ have pointed out the variability in prevalence of positive scores among different populations and specifically have remarked that underreporting by parents due to different native language or cultural expectations might contribute. Culturally, specific health perception could play a role in such findings. Rogler⁴⁶ addressed this issue in his writings on culturally sensitive mental health research. He wrote of the need to "decenter" a translation away from the source language to achieve an "equal linguistic partnership." It also has been noted that Hispanic families are generally reluctant to seek advice or assistance from medical professionals for psychosocial problems but rather turn toward family and community contacts.⁴⁷

We found relatively lower PSC sensitivity in a sample of low-income children in Spanish-speaking, Mexican American families. This finding suggests that use of the standard cutoff values on the PSC may pose a problem in specific populations, particularly those characterized by low income, low educational achievement, use of languages other than English, and/or minority status, with possible differences even within minority groups (eg, Spanish-speaking Puerto Rican families in Boston compared with Spanish-speaking Mexican American families in California). It is important to keep in mind that the US Hispanic population is heterogeneous as to country of origin, acculturation levels, and SES, and these factors influence their perceptions of health and well-being. Our study, as well as those noted herein, shows estimates of sensitivity and specificity of the PSC in the populations that clearly differ from non-Hispanic white and black children; this may depend, in part, on what criterion standard is used. For example, the CBCL, although validated in Spanish, may be less culturally sensitive than other standards in a population of low-income, Mexican immigrants. Future studies using other gold standards, such as structured psychiatric interviews, will provide a more complete understanding of the functioning of the PSC in this population.

The role of immigration status on the function of the PSC or other screening and diagnostic questionnaires remains unclear. It is possible that the immigration variable could play a more important role than underlying culture in both the development of pediatric psychosocial problems and the validity of screening and diagnostic tools meant to measure them. It will be important to evaluate these tools in alternative populations, such as non–recent Hispanic immigrants or recent, non-Hispanic immigrants to help sort out the interaction and significance of both immigration and culture.

The low rates of positive or clinical scores found on the 2 mental health questionnaires used in this study (PSC and CBCL) raise an additional question. When previous studies^12- 13 have shown rates of pediatric psychopathological conditions as high as 12% in middle-class, white populations³⁴ and more than 30% in low-SES, minority populations, how does one explain the low prevalence in the current high-risk study population (12.9% in borderline-clinical or clinical range on the CBCL)?

If Mexican American, immigrant children have fewer behavioral problems, one intriguing explanation for the low prevalence of psychopathological conditions in this population is the idea of an epidemiological paradox,⁴⁸ a term used to describe the unexpectedly good health outcomes in Hispanic mothers and children that seem to resist the usual association between poverty and poor health.^49- 51 Our findings of low rates of psychopathological conditions in a high-risk, low-SES, immigrant community give further credence to this idea of a paradox or the protective effect of Hispanic family, community, and culture in the behavioral development of their children.

Several issues involving the use of the PSC in Hispanic patients have been raised. However, the PSC clearly is a tool that is much needed in the primary care office setting for which it is designed. We would argue that use of the PSC should be continued and that adjustments be made for its appropriate use in this population.

As a first step, we would recommend consideration of the new cutoff point determined by ROC curve analysis in our current study. This lowered cutoff score of 12 allows the current PSC to obtain a much higher sensitivity in this population while maintaining a high level of specificity. However, replication of our findings will be important, as our results are based on a single community health center. Further analysis to determine the causes of the PSC's apparently low sensitivity in the Mexican American population also will be important. A large-scale analysis of PSC data collected in a Spanish-speaking population might reveal strengths and weaknesses in the current list of questions that are not suspected. This information also might guide a need for culturally sensitive rewording or addition of specific questions to increase the questionnaire's overall sensitivity. Replication studies that use both structured psychiatric interviews and behavior checklists also would be useful, particularly if a strategy for administration can be developed that accounts for illiteracy in the participant population.

Improving the utility of the PSC in the Mexican American population is only part of the larger, more complicated issue of the evaluation of all children for behavioral and psychosocial problems. With increasing attention paid to the impact of language and culture, it becomes clear that the PSC, as well as other mental health questionnaires, must be carefully evaluated with other immigrant populations. This is obviously a daunting task that undoubtedly complicates every aspect of mental health evaluation. However, the better understanding of the wide variety of cultures also serves to enrich our understanding of human nature and potential for human development.

In addition, much work remains to be done within the Hispanic culture. As an increasingly important part of the fabric of our nation, it is imperative that we gain a more complete understanding of the effects of immigration, family, language, and culture on the functioning of Mexican American children who are in our care. Issues such as the epidemiological paradox, so well described with regard to Hispanic infant and perinatal health, may be just a hint of larger forces at work within immigrant families.

Corresponding author and reprints: Lynne C. Huffman, MD, The Children's Health Council, 650 Clark Way, Palo Alto, CA 94304 (e-mail: Lynne.Huffman@stanford.edu).

Accepted for publication July 10, 2003.

This research was supported by a grant from Stanford University's Children's Health Initiative (grant 1HNJ601).

We thank Rosana Sandoval for her outstanding research assistance and Kathleen Matusich, BA, for her steadfast and exceptional administrative support. We also appreciate the comments and helpful suggestions of David Bergman, MD.

One way to improve the primary care pediatrician's ability to recognize and appropriately refer children with behavioral problems is to systematically screen all children with a standardized instrument designed for this purpose; the PSC is one such instrument. To date, however, there are few studies that examine the usefulness of the PSC in low-income Hispanic populations. This study of the PSC screener in a group of low-income, Mexican American children demonstrated that adjustments in the PSC cutoff score for significant levels of behavioral problems are indicated. The results also strengthen the evidence for expanded efforts to understand the possible protective effects, with regard to risk for psychosocial and behavioral difficulties, which may be conferred by Hispanic culture.