Caliber of Quality-of-Life Assessments in Congenital Heart Disease: Title and subTitle BreakA Plea for More Conceptual and Methodological Rigor FREE

During the past decades, mortality in congenital heart disease has decreased significantly.^{1 - 2} However, specific social and psychological concerns that typically occur in patients as they advance through life, such as compromised employability, insurability, and declining social integration,^{3 - 5} have caused health care professionals and researchers to take notice of the effect of this change. To substantiate a differentiated view on the outcomes of this patient population, the quality of life in patients with congenital heart disease has increasingly been addressed. Information on quality of life allows us to gain a better understanding of relevant issues for these patients, which is essential for optimizing their clinical management, planning appropriate care, and evaluating specific interventions or therapeutic modes.

Although quality-of life research has grown exponentially (P.M., W.B., and S.D.G., unpublished data, 2004), major conceptual and methodological challenges have also emerged. Indeed, consensus still eludes us regarding the conceptualization, operational definition, and measurement of quality of life. Because of this, quality of life is often incorrectly used as a generic label to describe a range of physical and psychosocial variables, making it an umbrella term that covers a variety of concepts, such as functioning, health status, perceptions, life conditions, behavior, happiness, lifestyle, symptoms, etc.⁶ The lack of a uniform definition for quality of life contributes to its conceptual vagueness and obscurity (P.M., W.B., and S.D.G., unpublished data, 2004).

Obviously, interpretation of results from quality-of-life studies becomes complicated when investigators do not use a consistent conceptual basis to define quality of life or if they fail to even define quality of life. This uneven, inappropriate usage, for instance, sometimes leads researchers to conclude that patients have a good quality of life just because they are employed,⁷ do not report symptoms or require medication,⁸ or are not in need of reoperation.⁹ This problem was recognized by Gill and Feinstein¹⁰ in their landmark study that assessed the caliber of quality-of-life measurements in different populations of patients. Using 10 criteria (see “Review Criteria and Procedure” subsection of the “Methods” section) specifically developed for use in their evaluation of 75 randomly selected quality-of-life studies, they concluded that most quality-of-life measurements and studies required methodological improvement because they “aimed at the wrong target.”^{10 (p619)} The Gill and Feinstein survey was published 10 years ago.

Therefore, the purpose of this article is to critically appraise the caliber of quality-of-life assessments used to evaluate children, adolescents, and adults with congenital heart disease. In our appraisal, we used the criteria of Gill and Feinstein.¹⁰ By checking the conceptual and methodological rigor of the methods used in these studies, we can provide a basis to evaluate how quality-of-life research can be interpreted and, if necessary, strengthened.

We performed a PubMed search for studies that examined the quality of life in patients with congenital heart disease. We included in this review all empirical studies that assessed children, adolescents, and adults; were published in English, French, German, or Dutch; and appeared in print between January 1980 and October 2003. The search terms used were quality of life combined with congenital heart, heart defects, or terms referring to specific heart lesions (ie, Fallot, transposition, coarctation, Eisenmenger, septal defect, atrial septal defect, ventricular septal defect, congenital aortic stenosis, congenital pulmonary stenosis, univentricular, anomalous pulmonary venous, truncus arteriosus, ductus arteriosus, Fontan, Marfan, double outlet, double inlet, Ebstein). We excluded letters, editorials, reviews, case studies, opinion articles, studies on parents of children with congenital heart disorders, transplant-related investigations, and qualitative studies. Textbooks, proceedings, and conference abstracts were not reviewed.

Gill and Feinstein¹⁰ were mainly interested in the face validity of quality-of-life measures. Hence, they evaluated the extent to which quality-of-life assessments measured the phenomenon on face value. However, before quality-of-life measurements can be evaluated, quality of life must be defined. Gill and Feinstein therefore defined quality of life as follows: “Rather than being a description of patients’ health status, quality of life is a reflection of the way that patients perceive and react to their health status and to other, nonmedical aspects of their lives.”^{10 (p619)} Using this definition, they developed 10 criteria to be used to evaluate the caliber of quality-of-life measurements. These 10 criteria follow.

1. Did the investigators give a definition of quality of life? (15% of studies adhered to this criteria.)

Because a uniform definition of quality of life presently does not exist, investigators need to clarify their conceptualization of quality of life to ensure that readers have a good understanding of the term as they define it. Therefore, investigators must provide an explicit definition of quality of life that serves as a basis for selecting the instruments to be used in their study. Simply referring to the wide variety of definitions, describing the components of quality of life, or citing multiple definitions without unequivocally quoting the definition underpinning the measurement is inadequate.

2. Did the investigators state the domains they will measure as components of quality of life? (47% of studies adhered to this criteria.)

Quality of life is typically considered to be a multidimensional construct, comprising multiple domains. The choice of quality-of-life instrument(s) basically relies on the components included in the instrument(s). To determine whether the selected measurement suitably represents the desired target, investigators ought to stipulate explicitly which domains they consider to be significant constituents of quality of life. Just describing domains underlying a specific questionnaire is not sufficient.

3. Did the investigators give reasons for choosing the instruments they used? (36% of studies adhered to this criteria.)

Valid assessments require that the instruments used are suitable for the intended task. Since numerous quality-of-life instruments exist, investigators need to state their reasons for choosing to use a particular instrument or instruments to assess quality of life. These reasons should ensure that quality of life will be measured appropriately according to their intended goals. Just because an instrument has good psychometric properties or is widely used does not mean suitable reasons were considered for its use.

4. Did the investigators aggregate results from multiple items, domains, or instruments into a single composite score for quality of life? (14% of studies adhered to this criteria.)

Gill and Feinstein¹⁰ argued that an effective characterization of quality of life is enhanced if investigators present a composite score that summarizes the results of multiple items, domains, or instruments. Although the richness of a profile description may be lost, an aggregated score simplifies the communication of results and permits the assessment of interrelationship between quality of life and other variables.

5. Were patients asked to give their own global rating for quality of life? (17% of studies adhered to this criteria.)

Although quality of life is principally conceptualized as a multidimensional construct, a single global rating by the patient is useful. Patients’ rating of their quality of life on a 1-item scale reflects the disparate values and preferences of individual patients.¹⁰ Hence, such a rating serves as an overall estimate of quality of life that considers quality-of-life components deemed important by the respondent.

6. Was overall quality of life distinguished from health-related quality of life? (No studies adhered to this criteria.)

Health care professionals are predominantly interested in health-related factors to be components of patients’ quality of life. However, a holistic approach implies that also nonmedical phenomena emerge, such as family relationships, social networks, spirituality, pet ownership, etc. Consequently, a distinction between overall and health-related quality of life should be made clear in quality-of-life articles. In this review, we considered this criterion to be fulfilled if the authors explicitly stated the difference between overall and health-related quality of life.

7. Were patients invited to supplement the items listed in the instruments offered by the investigators that they considered relevant for their quality of life? (13% of studies adhered to this criteria.)

Since there is a growing awareness that quality of life can only be affected by components that are important for an individual,¹¹ an adequate measurement of quality of life should provide the possibility for respondents to indicate the domains that are important for their quality of life. Some argue that this approach is the only way that can lead to a valid measurement of quality of life because it explicitly includes the domains that are relevant for respondents.^{10 ,12}

8. If so, were these supplemental items incorporated into the final rating? (89% of studies adhered to this criteria.)

To take the supplemental items into consideration in the assessment of quality of life, they ought to be incorporated into the final rating. If there is no possibility of obtaining supplemental items, this criterion is not applicable.

9. Were patients allowed to indicate which items were personally important to them? (8.5% of studies adhered to this criteria.)

Patients need to have the opportunity to rate the importance of different items, either those specified by the investigators or added by the patients. This offers the possibility to individually weigh the items and precludes the assumption that all items have the same importance for all patients.

10. If so, were the importance ratings incorporated into the final rating? (50% of studies adhered to this criteria.)

As for the supplemental items, the importance rating should be incorporated into the final rating. Again, if there is no possibility of scoring the importance of different items, this criterion is not applicable.

For each criterion, the number of articles complying with that criterion was counted and the percentage was calculated. To indicate how well individual articles performed on the respective criteria, a summary score was calculated by summing the number of criteria an article fulfilled and dividing this sum by the number of criteria for which the article was eligible to be evaluated; the resulting value was then multiplied by 100.¹⁰ Summary scores could range from 0, for articles complying with none of the criteria, to 100, for articles complying with all of the criteria.

We identified 70 articles that met the inclusion criteria.^{7 - 9 ,13 - 50 51 - 79} A total of 8206 patients were surveyed in these studies, ranging from a study sample of 9³³ to 1566 patients in the Second Natural History Study.²⁶ Twenty-four studies were performed in children,^{8 ,15 - 16 ,18 - 19 ,23 ,28 - 29 ,31 ,33 ,35 ,37 ,39 ,41 - 42 ,47 - 48 ,55 ,60 - 61 ,68 ,70 ,72 ,77} 34 in adults,^{7 ,14 ,17 ,20 ,22 ,26 ,32 ,34 ,38 ,43 - 46 ,50 - 54 ,57 - 59 62 - 67 ,69 ,71 ,73 ,75 - 76 ,78 - 79} and 12 in a mixed population of children, adolescents, and adults.^{9 ,13 ,21 ,24 - 25 ,27 ,30 ,36 ,40 ,49 ,56 ,74} Most investigations were conducted using patients diagnosed with tetralogy of Fallot (11 studies),^{7 ,14 ,22 ,28 ,40 ,44 ,46 ,51 ,67 ,73 ,75} transposition of the great arteries (11 studies),^{8 ,19 ,23 ,29 - 30 ,33 ,45 ,57 ,68 ,74 ,76} or a mix of various heart defects (11 studies).^{18 ,31 - 32 ,34 ,38 ,47 ,54 ,64 ,72 ,77 ,79} The most prevalent types of patients studied were patients with more than 1 cardiac anomaly (n = 1453) or those with transposed arteries (n = 1117), ventricular septal defect (n = 1069), a right ventricle to pulmonary artery conduit (n = 910), or tetralogy of Fallot (n = 868) (Figure). Together, these patients composed more than 65% of the patients studied. Sixty-four percent of the studies were performed in the United States (n = 16),^{9 ,17 ,19 ,21 ,23 ,26 - 27 ,35 ,40 ,42 ,52 ,55 - 56 ,64 ,70 ,72} Japan (n = 10),^{7 ,13 ,33 ,37 ,46 ,49 ,62 ,66 ,78 - 79} the United Kingdom (n = 7),^{38 - 39 ,53 ,57 ,65 ,73 - 74} Germany (n = 7),^{22 ,28 - 29 ,32 ,60 - 61 ,71} and the Netherlands (n = 6).^{15 ,34 ,43 - 45 ,54}

The studies included in this review were selected because they all drew conclusions about patients’ quality of life. However, only 53 (76%) of 70 articles explicitly aimed to measure quality of life. The other 17 articles measured a range of variables, instead of quality of life per se, but drew conclusions in terms of quality of life. Moreover, 30 articles (43%) did not describe quality of life in the Methods or Results section but merely mentioned it in the Abstract or Discussion.

Our evaluation of the rigor of quality-of-life studies in patients with congenital heart disease revealed that only 1 study (1%) provided a definition of quality of life (Table 1). Seventeen studies (24%) stated explicitly the domains that constitute quality of life. Two studies (3%) argued why the specific measurement was chosen. Twenty-two studies (31%) assessed quality of life with a single-item instrument or a with composite score of multiple-item tools that provided 1 overall score. Only 1 study (1%) allowed the patients to self-rate their perceived quality of life. Another study (1%) explicitly distinguished between overall quality of life and health-related quality of life. In the 53 studies that used multiple-item instruments, none provided a way for the respondents to select supplemental items important to them. However, 2 studies (4%) used an instrument that allowed respondents to rate the importance of respective items in the instrument. In these 2 studies, the importance rates were incorporated into the overall score.

Summary scores for individual articles ranged from 0 to 56, with a median score of 0 (quartile 1 = 0; quartile 3 = 17). Indeed, more than half of the articles (51.4%) did not comply with any of the 10 criteria. Only 2 articles^{32 ,34} had a summary score of 50 or higher.

Across the 70 quality-of-life studies that we reviewed, 39 different tools, questionnaires, or variables were used to measure quality of life (Table 2). Quality of life was mostly assessed by using the New York Heart Association Classification,⁸⁰ by employment status, and by educational level. Most variables that were used referred to physical functioning, health status, or socioeconomic factors.

The evaluation of quality of life has become increasingly important in children, adolescents, and adults with congenital heart disease. This may be because of the longer life expectancy of these patients, as well as the physical, psychological, and social impediments experienced by them. Quality of life is, however, an equivocal concept (P.M., W.B., and S.D.G., unpublished data, 2004). It is often used as an umbrella term to cover an assortment of concepts,⁹³ implying that many researchers use the label quality of life to describe various variables, just because quality of life is in vogue.⁹⁴ Hence, quality of life is often used inappropriately. In this review, we aimed to evaluate the conceptual and methodological rigor of published quality-of-life assessments in patients with congenital heart disease, using previously developed criteria.¹⁰

Based on our critical appraisal of the literature, we conclude that quality-of-life assessments in patients with congenital heart disease have major conceptual and methodological drawbacks. Only 1 investigator conceptually defined quality of life. The overall failure to provide an unequivocal definition of quality of life is problematic and compromises the interpretability of past studies. The variety of measures used in these studies illustrates the different approaches used by the investigators. More specifically, we observed that 39 different measures or instruments were used as indicators of quality of life, but only a few authors gave reasons for their choice of instruments. In some studies, their choice of a specific measure seemed arbitrary because its relevance with respect to quality of life is questionable. In what sense, for instance, is the need for reoperation or hospital admission interchangeable with quality of life?

When compared with the results of Gill and Feinstein¹⁰ published 10 years earlier, in our updated review we found the caliber of quality-of-life assessments in the congenital heart disease population to be poorer. In our study, the percentage of articles complying with the criteria was lower in 7 of 10 criteria compared with that in the Gill and Feinstein study. We found a large difference for the criterion “reason for choosing the measurement used.” This discordance is owing to our stricter interpretation of this criterion. We considered good psychometric properties or widespread use of an instrument to be inadequate reasons for using it to measure quality of life, because such reasons do not guarantee that the instrument is suitable for its intended task.¹⁰ Moreover, the validity and reliability of an instrument assessed in a particular sample and setting cannot be extrapolated to other settings or populations. Our review turned up a substantially higher proportion of studies in which an overall score was given. This is because, unlike Gill and Feinstein,¹⁰ we considered also as an overall score the use of a single-item instrument or classification system such as the New York Heart Association Functional Class.⁸⁰ Recall, many studies we reviewed used the New York Heart Association Classification to assess quality of life in patients with congenital heart disease. Only for the criterion “incorporating importance ratings into the final quality of life score” did we find a 100% compliance. This is because the instruments used in the 2 studies that allowed respondents to rate the importance of respective items (Ferrans and Powers Quality of Life Index, cardiac version III⁵⁰ ; TNO-AZL Questionnaire for Adult’s Health-Related Quality of Life³⁴ ) also included these importance scores in their final ratings.

This review confirmed that the term quality of life is also often used inappropriately in studies of patients with congenital heart disease. Indeed, a quarter of the articles reviewed drew conclusions about patients’ quality of life, although they failed to measure it. Consequently, if health care professionals were to rely on the findings of the articles reviewed here in terms of outcome evaluation, the resulting medical decision making or health policy would be suspect. Thus, it is imperative that researchers use a more thorough conceptual and methodological basis in their quality-of-life studies. Furthermore, researchers need to be aware of the potential confusion that they may introduce if they conclude that patients have a good or bad quality of life when, in fact, they did not actually measure quality of life. This stresses the importance of designing quality-of-life studies based on a sound conceptualization and avoiding the incorrect use of the term quality of life.

Inspired by the results of this literature review, we conducted a thorough conceptualization of quality of life (P.M., W.B., and S.D.G., unpublished data, 2004). Based on existing concept analyses of quality of life published in the literature^{11 ,95 - 96} and relying on the evaluation of different conceptual pitfalls, we concluded that satisfaction with life is the most suitable approach for defining quality of life. However, none of the existing definitions of quality of life, reviewed in the context of life satisfaction, explicitly integrates aspects of the conceptual pitfalls identified. Therefore, after integrating the formulated critiques, we defined quality of life as

the degree of overall life satisfaction that is positively or negatively influenced by an individual’s perception of certain aspects of life that are important to them, including matters both related and unrelated to health (P.M., W.B., and S.D.G., unpublished data, 2004).

According to this definition, quality of life can, for instance, be measured using the Satisfaction With Life Scale.⁹⁷ This does not, however, dismiss the utility of generic and disease-specific instruments in measuring specific determinants or components of quality of life. While these instruments may not measure quality of life itself, they may be useful in measuring disability related to specific diseases and effectiveness of treatment. Such instruments augment insights in the self-perceived health status or functional abilities of the patients under study. In addition, we suggest to researchers planning to conduct quality-of-life studies to use the Gill and Feinstein¹⁰ criteria to strengthen their methods and thus avoid current flaws.

Two possible limitations of the present study can be identified. First, the criteria used in this review were developed in the early 1990s to critically appraise the caliber of quality-of-life studies in the medical literature.¹⁰ Whether they currently remain applicable might be questioned, because conceptual understanding of quality of life may have evolved during the last decade. Indeed, several authors have, in the meantime, published their concept analyses of quality of life.^{95 - 96 ,98 - 103} Moreover, the equal weighting of the 10 criteria may be subject to debate. Second, we did not include all quality of life articles published, because we only reviewed articles contained in the PubMed database. Although searching other databases, such as Psychinfo, CINAHL, or Embase, may have resulted in more relevant articles, nevertheless, we believe that such searches would not have dramatically affected the conclusions of this review because we focused on conceptual and methodological issues, rather than on the findings of respective studies themselves. A specific selection bias is therefore unlikely.

Quality of life is an important issue in patients with congenital heart disease. Numerous studies have investigated or drawn conclusions about quality of life in this patient population. The poor conceptual and methodological basis used in these studies implies that many results of quality-of-life studies in patients with congenital heart disease are inconclusive. We appeal for more conceptual and methodological rigor with respect to future quality-of-life studies in this group of patients.

What is already known about this topic?

Quality-of-life assessments in patients with congenital heart disease have major conceptual and methodological drawbacks. Several articles drew conclusions about patients’ quality of life, although they did not measure it.

Correspondence: Philip Moons, PhD, RN, NFESC, Center for Health Services and Nursing Research, Katholieke Universiteit Leuven, Kapucijnenvoer 35/4, B-3000 Leuven, Belgium, (philip.moons@med.kuleuven.ac.be).

Previous Presentation: This article is an update of an invited lecture given at the 22nd Conference of the European Society of Cardiology; August 30, 2000; Amsterdam, the Netherlands.

Accepted for Publication: May 27, 2004.

Funding/Support: This study was supported in part by the Belgian Foundation for Research in Pediatric Cardiology, Halle.