Usefulness of Systematic Review Search Strategies in Finding Child Health Systematic Reviews in MEDLINE FREE

Well-conducted systematic reviews provide the best evidence to guide clinical practice, are cornerstones for the recommendations of evidence-based practice guidelines, and should be an integral part of the planning of future research.^1- 2 In contrast to traditional narrative reviews, systematic reviews of the literature address a well-defined question, use an explicit search strategy to locate all relevant evidence, evaluate the retrieved studies using prospectively defined methodological criteria, and formally synthesize the results.^3- 4

Clinicians and researchers should be able to reliably and quickly find systematic reviews. Two types of systematic reviews can be identified in current bibliographic databases: (1) the Cochrane systematic reviews, which can be found in the Cochrane Database of Systematic Reviews, the Cochrane Library and (2) systematic reviews published in journals (journal reviews). Journal reviews can also be found in the Cochrane Library Database of Abstracts of Reviews of Effects (DARE) and are available from the Centre for Reviews and Dissemination Web site.⁵ The DARE database provides quality assessments of systematic reviews published in journals after 1994, identified by regular searches of important bibliographic databases such as MEDLINE, EMBASE, and CINAHL and by hand searching some of the major general journals but no specific pediatric journals. Bibliographic databases such as MEDLINE index Cochrane reviews and can be used to identify other systematic reviews, but only those indexed in the respective database. Finding systematic reviews in MEDLINE poses 2 challenges. First, only a fraction of all citations in MEDLINE are for systematic reviews and, second, the MEDLINE indexing procedures do not include “systematic review” as a “publication type.” To limit the search results from a query in MEDLINE, therefore, it is recommended that a methodological filter be used consisting of text words and Medical Subject Headings of the US National Library of Medicine (MeSH) directed to general indicators of systematic reviews in the MEDLINE record.

Several search strategies for locating systematic reviews in MEDLINE have been developed and validated^6- 9; however, to our knowledge, the performance of these filters has never been evaluated in the universe of all articles included in MEDLINE, nor have they been tested for sensitivity and precision in finding child health systematic reviews.

The objective of the present study was to assess the usefulness of existing search filters in finding child health systematic reviews in MEDLINE using the PubMed interface by applying them to a reference standard of child health systematic reviews and by determining whether these filters focus the search strategy sufficiently to be practical in the universe of MEDLINE. To this goal, the percentage of child health systematic reviews retrieved from our reference standard (sensitivity) was measured and the number of true child health systematic reviews yielded by the searches (precision) was estimated.

Herein, we use the term systematic review to refer to any literature review, meta-analysis, or other article that explicitly indicates the use of a strategy for locating evidence by mentioning at least the databases that were searched and reviewing the empirical evidence on children (aged <18 years). Excluded were systematic reviews that assessed the effects on the fetus of interventions in pregnant women.

To identify articles reporting on the development and validation of systematic review search filters in MEDLINE, we searched MEDLINE from January 1995 to January 2006 with the following MeSH terms: MEDLINE, Information Storage and Retrieval/Methods, and Review, literature. In addition, reference lists of relevant articles were reviewed and content experts were contacted to find additional studies. To improve precision, we combined the systematic review filters with a sensitive child filter developed by the Cochrane Child Health Field¹⁰ to retrieve only studies in children.

To measure the sensitivity of the search strategies, a reference standard set of systematic reviews was established by searching for child health systematic reviews in DARE⁵ and by hand searching several pediatric journals for systematic reviews. All titles and abstracts in DARE (Cochrane Library, Issue 2, 2004) were screened to find child health systematic reviews also indexed in MEDLINE. We hand searched 7 pediatric journals indexed in MEDLINE with a range of impact factors and for which full-text electronic copies were available from our medical library (Table 1). All issues of each journal were searched for the following 5 years: 1994, 1997, 2000, 2002, and 2004. In addition, we were interested as to whether DARE contained all child health systematic reviews found by our hand search of pediatric journals. Inasmuch as DARE includes only systematic reviews on prevention, intervention, or diagnosis, we selected systematic reviews covering these domains. Next, we searched DARE in July 2006 using the Cochrane Library interface to determine how many child health systematic reviews on prevention, intervention, or diagnosis found by our hand search were also included in DARE.

Because the child filter has not been validated, we measured the sensitivity of the child filter separately. The sensitivity of a search strategy was defined as the percentage of child health systematic reviews retrieved from our reference standard set of child health systematic reviews.

Sensitivity = (Number of systematic reviews retrieved from reference standard/Total number of systematic reviews in reference standard) ×100

Sensitivity was measured for all systematic review filters separately and in combination with the child filter.

The precision of a combined search strategy was defined as the percentage of true child health systematic reviews found in MEDLINE (limits 1990-2006) divided by the total number of records retrieved by the search.

Precision = (Number of true systematic reviews/All records in MEDLINE retrieved by the search) ×100

We estimated precision rather than calculated precision because the yield of each combined search strategy was large, ranging from 7861 records (Montori 2 search filter) to 618 053 records (Montori 1 search filter). To estimate precision, we selected a random sample of 100 records from the yield of each combined search strategy. The random sample was generated using R software¹¹ and stratified by year of publication to account for changes in MeSH indexing with time. Each sample was screened independently by 2 investigators (N.B. and L.T.) for true systematic reviews. We considered an article a true systematic review only if the title, abstract, author-supplied key words, or publication type terms explicitly identified the article as a systematic review or meta-analysis or if the article abstract indicated a strategy for locating the literature reviewed and included children. We classified records as unsure if the title or abstract did not specifically state whether children were included, it was stated that a literature search was performed but without stating which databases were searched, or it was stated that the article was an evidence-based overview. In all of these unsure cases, the full text of the article was retrieved and a final decision was made as to whether it was a true systematic review. Next, we applied the search strategies to 7 priority topics in child health, developed by the Cochrane Child Health Field, that necessitate systematic reviews. The number of records retrieved for the most sensitive and the most precise systematic review search strategies is reported, as is the precision for the most precise filter. The search strings for each clinical topic were created by using the MeSH browser function.¹² We reported the time it takes a pediatrician (N.B.) and a clinical librarian (L.T.) to screen 100 MEDLINE records.

We identified 4 studies reporting on the development and validation of several systematic review search strategies for MEDLINE.^6- 9 We evaluated 8 of the reported search strategies with varying sensitivity and precision and the PubMed Clinical Queries systematic review filter.¹³ Details of the systematic review search strategies (translated into PubMed format where necessary) and the child health search strategy are provided in the online supplement (eBox).

Our reference standard contained 387 child health systematic reviews indexed in MEDLINE (Table 1). A total of 431 child health systematic reviews were identified in DARE (Cochrane Library, Issue 2, 2004; N = 4645), of which only 298 were indexed in MEDLINE. Hand searching MEDLINE-indexed pediatric journals yielded 115 child health systematic reviews, of which 26 records were also included in DARE (Cochrane Library, up to Issue 2, 2004). Of our hand-searched 115 reviews, 73 reported on prevention, intervention, or diagnosis. A 2006 search of DARE revealed that 38 of these 73 systematic reviews on prevention, intervention, or diagnosis (52%) were also indexed in DARE.

Sensitivity of the child filter when run separately in MEDLINE and tested against our reference standard was 98% (380/387; 95% confidence interval, 96%-99%). Sensitivity of the systematic review filters ranged from 68% to 96% (Table 2).

Figure 1 shows the number of records retrieved with the systematic review filters in PubMed (1990-2006) with and without the child filter added. The number of records retrieved with the systematic review filters ranged from more than 38 000 (Montori 2) to more than 3 million (Montori 1) and for the combined filters from more than 8000 (Montori 2) to more than 618 000 (Montori 1).

We screened 600 titles and abstracts for systematic reviews (a random sample of 100 records retrieved with 6 systematic review filters combined with the child filter) (Table 2). We were unsure in 150 records whether the article was a true systematic review and in 15 of those whether children were included in the review. For all of these unsure results, full text was retrieved, which eventually led to a yield of 137 of the 600 reviews being true systematic reviews in children. Precision ranged from 2% to 52% in our study with the combined systematic review and child filters, compared with 3% to 69% reported in the original studies (Table 2). On average, it took the pediatrician 34 minutes and the clinical librarian 47 minutes to screen the titles and abstracts of 100 retrieved records.

Figure 2 shows the number of records retrieved when applying a sensitive and a precise systematic review filter in combination with the child filter to 7 different clinical topics. Adding a MeSH term for asthma yielded the largest number of records, and adding a MeSH term for constipation yielded the lowest number (Shojania filter, 352 vs 20; Montori 3 filter, 3178 vs 222). Precision with the combined Shojania plus child filter ranged from 49% for obesity to 93% for enuresis. On average, it took the pediatrician 18 minutes to screen the titles and abstracts of 100 topic-specific records for systematic reviews.

Although the DARE database contains quality assessments of systematic reviews published in peer-reviewed journals, DARE includes only 52% of our hand-searched child health journal reviews. However, DARE uses more stringent inclusion criteria for systematic reviews than we did.⁵ Although DARE was created in February 1995, all hand-searched child health journal reviews published in 1994 were indexed in DARE.

The Cochrane Database of Systematic Reviews and DARE are already prefiltered in their focus on systematic reviews; thus, the only task is to find topic-relevant systematic reviews in these databases. In general medical databases such as MEDLINE, systematic reviews should ideally be indexed using an appropriate and specific publication type term. Until there is an agreed definition of a systematic review, indexers will be unable to introduce or apply such a term. Pending this, we will need more precise search strategies to find systematic reviews in MEDLINE to reduce the number of irrelevant records.

The usefulness of a systematic review filter is expressed by sensitivity and precision. The trade-off between these 2 features will determine the choice of filter. Researchers conducting a new systematic review or guideline developers would best be served by the most sensitive search. This search will have the highest probability of retrieving all relevant reviews, but it will have low precision, retrieving many irrelevant articles. Those with little time on their hands (for example, clinicians looking for answers to patient care questions) will likely be best served by a more precise search strategy. The sensitivity of the systematic review filters as reported in the original studies and compared with our reference standard were similar except for the Shojania filter (93% vs 74%, respectively). Shojania's reference standard set of papers mainly included high-quality systematic reviews that probably contained consistent descriptions of systematic review criteria in the abstracts' “Methods” sections. A filter developed in such a sample will probably consist of another combination of text words and MeSH headings than would be optimal to retrieve systematic reviews in our reference standard, which includes both low- and high-quality systematic reviews.

Because the sensitivity of the child filter alone was excellent (98%), we combined the systematic review filters with the child filter to improve precision. Inasmuch as few systematic reviews were published before 1990, we limited our search to 1990 and after. The number of records retrieved with the various systematic review filters was in the hundreds of thousands. Adding the child filter reduced the number of records by a factor range of 5 to 7. Of the sensitive search filters, the Montori 3 has the best trade-off between sensitivity (94%), number of records retrieved (151 227), and precision (3%). Of the more precise filters, the Shojania filter has the best trade-off between sensitivity (74%), number of records retrieved (10 188), and precision (45%). Although we added a child filter to improve precision and used a broad definition of a systematic review, precision was lower than reported in the original studies for which we have comparative precision data. However, this is to be expected because precision changes with the prevalence of positive results (or richness of the database). The prevalence of child health systematic reviews in MEDLINE is lower than the prevalence of adult systematic reviews. In addition, most existing search strategies were developed from a small, selective subset of systematic reviews published in high-impact journals. Precision was also estimated from their subsets of journals. Clinicians searching for systematic reviews in MEDLINE using the filters search the universe of MEDLINE and not a specific set of journals. Therefore, we believe that clinicians will be interested in precision in the universe of MEDLINE. Shojania and Bero⁸ calculated precision in MEDLINE for 3 clinical topics (Table 2). As we have shown, precision varies considerably among different clinical topics and seemed to be higher than precision not limited to clinical topics. Therefore, the results will not be generalizable.

It was often difficult to determine from the title and abstract whether an article was a true systematic review. Abstracts often lacked descriptions of important criteria for systematic reviews. Instead, general terms were used, such as “A literature review was performed,” “We tried to collect all published literature,” “This is an evidence-based overview,” and “We performed a qualitative assessment of the literature.” Therefore, it is to be expected that the most sensitive systematic review filters also retrieve many irrelevant records. In 1999, the Quality of Reporting Meta-analyses (QUOROM) statement was published to improve the quality of reporting meta-analyses or systematic reviews of clinical, randomized, controlled trials.¹⁴ To be able to differentiate narrative reviews from systematic reviews, we advise that authors follow these recommendations when writing abstracts. In addition, when children are included, this should be explicitly reported in the title or abstract. To make identification using PubMed easier, there is an urgent need to set minimal criteria that any review should fulfill for it to be indexed as a systematic review. Ideally, a database of child health systematic reviews should be created. Pending this, validated child health filters are needed to be able to retrieve all child health–relevant articles. Since this study was conducted, a study has been published that determined the retrieval characteristics of age-specific terms in MEDLINE (Ovid interface; Ovid Technologies, Inc, New York, New York) for pediatric and neonatal medicine.¹⁵

We used a broad definition of a systematic review, and a number of these reviews will not fulfill some of the stringent criteria for systematic reviews used by others. Given the amount of time needed to perform a systematic review and the methodological skills required, our goal was not to exclude potentially relevant systematic reviews. The explicit mention of a strategy for locating evidence seems to be the most basic and least controversial.⁷

Our reference standard contains a subset of the MEDLINE database because it is impossible to hand search all journals indexed in MEDLINE for child health systematic reviews, and sensitivity could have been overestimated or underestimated. We tried to compose a representative reference standard of systematic reviews by hand searching 7 pediatric journals with a range of impact factors. This should help avert selection bias in our search filter accuracy study because of higher standards of study description (eg, by the use of a structured abstract with an explicit “Methods” section) in high-impact journals. In an attempt to avert possible biases occurring in any one year, we sampled records from various years within a 10-year range.

All systematic review filters except the Shojania filter were developed and tested using the Ovid interface. Because PubMed is the only MEDLINE free interface available worldwide and is widely used, we converted the Ovid search filters into PubMed format.^6- 7,9 Translating a search filter from Ovid format to PubMed format is a factor that may influence the performance of the filter.

We advise busy clinicians searching for systematic reviews on health care interventions to search the Cochrane Library (Cochrane Database of Systematic Reviews and DARE) for high-quality systematic reviews. Several studies have shown that Cochrane reviews are more rigorous than journal reviews.^16- 17 If no relevant systematic reviews are found in the Cochrane Library or the interest is other than prevention, treatment, or diagnosis, other databases such as MEDLINE should be searched next. For a specific search, the Shojania filter combined with the child filter can be used. For topics for which not many reviews are available (eg, constipation), it may be more efficient to search with the sensitive Montori 3 filter combined with the child filter. Researchers and guideline developers interested in finding nearly all systematic reviews can use the Montori 3 filter combined with the child filter. Readers can save the search filters (Figure 3) they wish to use in PubMed and use them in combination with a clinical topic.

Correspondence: Nicole Boluyt, MD, Center for Pediatric Clinical Epidemiology, Room H3-145, Emma Children's Hospital, Academic Medical Centre, PO Box 22700, 1100 DD Amsterdam, the Netherlands (n.boluyt@amc.uva.nl).

Accepted for Publication: July 13, 2007.

Author Contributions: Dr Boluyt had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Boluyt, Tjosvold, Lefebvre, Klassen, and Offringa. Acquisition of data: Boluyt and Tjosvold. Analysis and interpretation of data: Boluyt, Lefebvre, and Offringa. Drafting of the manuscript: Boluyt and Klassen. Critical revision of the manuscript for important intellectual content: Boluyt, Tjosvold, Lefebvre, and Offringa. Administrative, technical, and material support: Boluyt and Tjosvold. Study supervision: Boluyt, Klassen, and Offringa. Information science expertise: Lefebvre.

Financial Disclosure: None reported.

Additional Contributions: Marjan Loep, MLIS (deceased), Marcel van der Paardt, PhD, and Arnold Leenders, MLIS, Medical Library, Academic Medical Center, Amsterdam, the Netherlands, searched DARE and hand searched the pediatric journals; Grace Liang, PhD, Alberta Research Centre for Child Health Evidence, University of Alberta, Edmonton, Alberta, Canada, developed a program to enable random sample selection; and Liza Bialy, BSc, Alberta Research Centre for Child Health Evidence, University of Alberta, Edmonton, Alberta, screened records for true systematic reviews. None received compensation for their work.