0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Article |

Factors Influencing the Publication of Randomized Controlled Trials in Child Health Research FREE

Lisa Hartling, BScPT, MSc; William R. Craig, MDCM, FRCPC; Kelly Russell, BSc; Kelly Stevens, BSc; Terry P. Klassen, MD, MSc, FRCPC
[+] Author Affiliations

Author Affiliations: Alberta Research Centre for Child Health Evidence, Department of Pediatrics, University of Alberta, Edmonton, Canada.


Arch Pediatr Adolesc Med. 2004;158(10):983-987. doi:10.1001/archpedi.158.10.983.
Text Size: A A A
Published online

Background  Publication bias threatens the validity of clinical decisions. The root causes are relatively unknown, and there is limited investigation in child research literature.

Objectives  To identify factors associated with subsequent nonpublication of abstracts presented at the Society for Pediatric Research meetings, and to determine the relative importance of the reasons identified for nonpublication.

Design  A cross-sectional survey was used to ask researchers about their reasons for the selective publication of randomized controlled trials (RCTs). The authors of 393 RCTs presented at the Society for Pediatric Research meetings from 1992 to 1995 were surveyed. A modified Total Design Method for mail surveys was used, with a reminder sent to all potential respondents 1 week after the initial mailing and full mailings sent to nonrespondents at 3 and 10 weeks following the initial mailing.

Results  One hundred sixty-six (45%) completed surveys were returned, and 119 (72%) abstracts were published as full manuscripts. Factors significantly associated with nonpublication identified through multiple logistic regression were the respondent’s report of scientific merit and significance of results. Of the 47 studies that were not published, only 8 (17%) had been submitted for publication. Authors of unpublished studies identified the following as important reasons for not publishing: not enough time (56.4 responded important or very important); trouble with coauthors (28.9); and journal unlikely to accept (26.3).

Conclusions  Of the RCTs presented and not subsequently published, the majority (83%) were never submitted for publication. The most common reason cited by authors for nonpublication was lack of time.

Publication bias is the selective publication of studies based on the direction and strength of the study results.1 Consequently, smaller studies with nonsignificant or indefinite results are often subject to this bias. Publication bias threatens the validity of clinical decisions to the extent that these are based on the results of published literature to the exclusion of gray (ie, unpublished) literature. These decisions are vitally important, as they have the potential to affect financial, quality of life, and life and death decisions made on a daily basis. The existence of publication bias has been clearly demonstrated in many clinical areas.2 Identification of the root causes may lead to possible solutions and subsequently increase our confidence in making unbiased decisions that will ultimately improve quality of care.

Reasons for nonpublication may rest with the editors, authors, the study itself, or other external factors. Previous research39 suggests that the bias is rooted in the actions of the investigator rather than the actions of the editor. The principal reason for nonpublication that is consistently reported is lack of significant findings.35,7,912 Other major reasons cited in the literature include disinterest on the part of the authors,3,9,13 lack of time,8,9,14,15 unimportant results,5,7 and “coinvestigator or other operational problems.”9,13 There is mixed evidence linking nonpublication to a study’s methodological quality,5,7,8,10,12,16 sample size,35,10,12,17,18 and other design features such as the number of data collection sites, presence of a control group, and study type.4,7,8,13 Characteristics of the principal investigator, such as sex or academic rank, do not appear to play a role9; however, there is some suggestion that prior publication by the investigator is associated with subsequent publication.8,19 Publication has also been linked to external funding.4,7,13 Other potential factors external to the investigator and the study design include acceptance for presentation at a scientific meeting,8,10,11 the type of presentation,17,20 and the country where the study was conducted or reviewed.11,21

The purpose of this study was to build on the work already completed in this area, to further illuminate the reasons behind publication bias with an exclusive focus on child health research. The specific objectives were to identify factors associated with subsequent publication and nonpublication of abstracts that appeared in Pediatric Research between 1992 and 1995, and to determine the relative importance of the reasons identified for nonpublication.

STUDY POPULATION

The study population consisted of 1 of the authors of each of the 393 abstracts of randomized controlled trials (RCTs) presented at the Society for Pediatric Research (SPR) meetings from 1992 to 1995. As part of an earlier study, abstracts of RCTs were identified by hand-searching the proceedings from the American Pediatric Society–SPR from 1992 to 1995, inclusive.12 Abstracts were included if they reported phase III RCTs and measured pediatric outcomes; studies reporting outcomes on pregnant women were excluded. Abstracts were not included in the database if they only reported outcomes of nonrandomized treatment arms.

DATA COLLECTION INSTRUMENT

A questionnaire was designed specifically for the purposes of this study (available on request). The majority of the questions were based on factors that had been identified in the literature as potentially associated with nonpublication. These factors included statistical significance of the primary outcome as reported by the responding author (response categories were “statistically significant in favor of the test treatment,” statistically significant in favor of the control group, results favored the test treatment but did not meet statistical significance, results favored the control group but did not meet statistical significance, neither group was favored and results were not statistically significant, and other); whether the results were what the responding author had expected (yes or no); scientific merit of the study as rated by the responding author (poor, neutral, fair, good, excellent); clinical importance of the study and its findings as rated by the responding author (not important, neutral, somewhat important, important, very important); number of prior publications by the responding author; source of funding (no funding, governmental granting agency, private industry other than pharmaceutical company, pharmaceutical company, other); and reasons for nonpublication (see Table 1 for response categories; authors were also given the option to add additional reasons and comments). Other basic questions were asked regarding the responding author (eg, age at time of the abstract, sex, academic rank at the time of the study) and the study (eg, whether the abstract was published as a full manuscript, where the manuscript had been submitted and published, the year of publication, the number of publications from the same trial, and, if not published, whether it had been submitted for publication).

Table Graphic Jump LocationTable 1. Authors’ Reasons for Nonpublication*

The questionnaire was pilot tested for content and face validity among a convenience sample of 8 researchers. These individuals were asked to complete the questionnaire with respect to their own research that was reported elsewhere between 1992 and 1995. They were also asked to provide feedback regarding the format and appearance of the questionnaire, time to complete, and the comprehensibility and applicability of the questions. Based on their responses, modifications were made to the questionnaire prior to starting the study.

IMPLEMENTATION

Contact information in the form of either postal or e-mail addresses was sought for the first author mentioned in the abstract. If no information was found for the first author, an address for the next author was sought, and the next author after that if the situation repeated itself. Those authors for whom we found both a postal mailing address and an e-mail address were randomized to receive the survey by 1 of these 2 methods. The results of the randomized trial comparing responses for postal vs e-mail are the subject of another article.22 If only 1 type of mailing address was found, the survey was sent using this method of mailing.

The Total Design Method (TDM) for mail surveys was used as a guideline for the mailing procedure.23 One week after the initial questionnaire was mailed, a reminder was sent. Three weeks after the initial mailing, a second complete package was sent to those individuals who had not yet responded. Ten weeks after the initial mailing, a third and final complete package was sent to those individuals who had still not responded. No incentives were offered for completing the surveys.

DATA MANAGEMENT AND ANALYSIS

Data from the completed questionnaires were coded and entered into Microsoft Access 2000 (Microsoft Corporation, Remond, Wash). Data entry was checked for accuracy by a second researcher. All data analyses were conducted using SPSS for Windows Version 11.0.1 (SPSS Inc, Chicago, Ill). For descriptive purposes, variables were analyzed using medians and interquartile ranges (IQRs) for nonparametric data, means and SDs for normal continuous data, and percentages and 95% confidence intervals (CIs) for dichotomous/categorical data. Pearson χ2 tests were used to measure the association between different variables and publication status. Multiple logistic regression was used to simultaneously evaluate factors associated with publication status.

The study was reviewed and approved by the University of Alberta Health Research Ethics Board.

RESPONSE AND DESCRIPTION OF PUBLISHED STUDIES

One hundred sixty-six (45.5%) surveys were completed (refusals are included in the denominator but not in the numerator; 28 surveys were returned as undeliverable and were excluded from the denominator). The majority of respondents were men (77.7%) between the ages of 30 and 49 years (79.5%). According to the responding authors, 71.7% of abstracts were subsequently published as full manuscripts. Among the trials that were subsequently published, the median number of submissions prior to acceptance for publication was 1.0 (range, 1-7). The median number of publications per study was 1.0 (IQR, 1-2; range, 0-10); 26.1% of the studies generated more than 1 publication. The median number of years to publication was 2.0 (IQR, 1-2; range, 1-7). The median impact factor of the journals in which the trials were published was 3.536 (IQR, 2.089-3.708; range, 0.091-29.065). The journals in which the trials were most frequently published were Pediatrics (21.8%), Journal of Pediatrics (13.4%), New England Journal of Medicine (7.6%), Pediatric Pulmonology (5.9%), Journal of Perinatology (5.0%), Archives of Diseases of Childhood (4.2%), and Archives of Pediatrics and Adolescent Medicine (4.2%).

REASONS FOR NONPUBLICATION

Of the 47 studies that were not published, only 8 (17%) had been submitted for publication. Respondents were asked to rate the importance of possible reasons for nonpublication (Table 1). The reasons most often cited as important or somewhat important were not enough time (56.4%), too much trouble with coauthors (28.9%), authors thought that a journal was unlikely to accept the study for publication (26.3%), and results were not statistically significant (23.7%). Respondents were also asked to identify the most important reason for nonpublication:too much trouble with coauthors (32.3%), followed by not enough time (25.8%).

VARIABLES ASSOCIATED WITH PUBLICATION

Variables that were independently associated with subsequent publication are listed in Table 2. The following characteristics of respondents were tested and not found to be associated with subsequent publication: academic rank, number of prior publications, age, and sex.

Table Graphic Jump LocationTable 2. Bivariate Analyses of Trial Characteristics and Association With Publication Status

Variables significantly associated with publication identified through multiple logistic regression were respondents’ reports of scientific merit and significant results. Studies for which the respondents described the primary outcome as statistically significant were more likely to be published (adjusted odds ratio [OR], 5.0; 95% CI, 2.1-12). Studies with a higher rating of scientific merit (as judged by the respondent) were more likely to be published. This included good vs fair, neutral, or poor (adjusted OR, 4.0; 95% CI, 1.5-11) and excellent vs fair, neutral, or poor (adjusted OR, 19.0; 95% CI, 4.2-83).

We identified factors associated with publication from a large sample of RCTs presented at the SPR's annual meetings between 1992 and 1995. Our study found that the most important factors associated with subsequent publication are the authors’ reports of the scientific merits of the studies and the significance of the results. Previous research in other areas has found that the rating of scientific importance by the authors is associated with publication.5,7 We examined perceived scientific merit (methodological quality) and clinical importance (impact of results on clinical practice) as separate variables. While both were found to be significantly associated with subsequent publication, the association was stronger for perceived scientific merit. The finding that positive results are associated with publication is also consistent with previous research.7,10,14,16,24 In contrast to these findings, an evaluation of publication bias in gastroenterological research found no significant association between direction of study results and subsequent publication; however, statistically significant results did influence the likelihood of publication in high-impact journals.8

Consistent with previous research,7,8,13 we found that the reasons for nonpublication were often related to authors’ actions and perceptions. The most common reasons put forward by authors for nonpublication were lack of time, too much trouble with coauthors, authors thought the journal was unlikely to accept the study for publication, and results were not statistically significant. A recent study provided evidence to discredit the idea that studies with positive results are more likely to be published: Olson et al25 found no significant effect of the direction of study results on publication for manuscripts of controlled trials submitted to JAMA from February 1996 to August 1999.

Preventing publication bias is critical for several reasons, including ethical responsibility; accountability to funding sources; ensuring availability of information for other research, such as systematic reviews; and clinical decision-making. Many have supported the idea of a trials register to prevent publication bias, whereby all trials would be prospectively registered at their outset.1,7,13,16,2628 This would eliminate the factors associated with nonpublication that occur at the end of the research project, such as lack of time, unimportant findings, or trouble with coinvestigators. The likelihood for success of such an initiative may be optimized if it is part of the ethics approval process.

The most important limitation of the present study is the response rate; the results must be interpreted in light of this. In a previous study of this sample,12 we showed that the subsequent publication rate was approximately 60% compared with 72% in the present study. While there may have been a small degree of misclassification for various reasons, including inability to find the study in the databases searched,8 this is unlikely to account for the full discrepancy. The discrepancy suggests that authors who had published were more likely to respond to our survey; further analyses showed that the response rates were 38% vs 56% for unpublished and published studies, respectively. However, in identifying reasons for nonpublication, there is no reason to believe that the nonpublishing authors who responded would be different from those who had not. A second limitation is that the data were self-reported and may be susceptible to recall or reporting bias.3 For instance, the authors that subsequently published (or did not publish) may perceive their study and its findings differently based on their experience during manuscript submission and publication.

What This Study Adds

Publication bias threatens the validity of clinical decisions. The reasons for nonpublication appear to be related to actions of the investigator rather than those of the editor. Trials registers have been widely supported to prevent bias associated with nonpublication. These study results imply that the most important factors associated with publication among a sample of abstracts presented at a large pediatric conference between 1992 and 1995 were scientific merit, as rated by the researcher, and significance of the findings. The reasons for nonpublication were often related to authors’ actions and perceptions. The most common reasons cited for nonpublication were lack of time, trouble with coauthors, journal unlikely to accept, and results not statistically significant.

Correspondence: Lisa Hartling, BScPT, MSc, Alberta Research Centre for Child Health Evidence, Department of Pediatrics, University of Alberta, Aberhart Center One, Room 9424, 11402 University Ave, Edmonton, Alberta, Canada T6G 2J3 (lisa.hartling@ualberta.ca).

Accepted for Publication: May 27, 2004.

Funding/Support: The Alberta Research Centre for Child Health Evidence is supported by an establishment grant from the Alberta Heritage Foundation for Medical Research.

Acknowledgment: We thank Jacqueline Jubinville and Belinda Allan for administrative support. We thank Natasha Wiebe for statistical consultation. We also thank the researchers who pilot tested the data collection instrument and the authors who responded to the survey.

Dickersin  K How important is publication bias? a synthesis of available data. AIDS Educ Prev 1997;915- 21
PubMed
Scherer  RWLangenberg  P Full publication of results initially presented in abstracts [Cochrane Methodology Review on CD-ROM].  Oxford, England Cochrane Library, Update Software2003; (1)
Dickersin  KChan  SChalmers  TCSacks  HSSmith  H  Jr Publication bias and clinical trials. Control Clin Trials 1987;8343- 353
PubMed
Dickersin  KMin  YIMeinert  CL Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards. JAMA 1992;267374- 378
PubMed
Easterbrook  PJBerlin  JAGopalan  RMatthews  DR Publication bias in clinical research. Lancet 1991;337867- 872
PubMed
Riordan  FA Do presenters to pediatric meetings get their work published? Arch Dis Child 2000;83524- 526
PubMed
Stern  JMSimes  RJ Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315640- 645
PubMed
Timmer  AHilsden  RJCole  JHailey  DSutherland  LR Publication bias in gastroenterological research: a retrospective cohort study based on abstracts submitted to a scientific meeting. BMC Med Res Methodol 2002;27
PubMed
Dickersin  KMin  YI NIH clinical trials and publication bias. Online J Curr Clin Trials 1993;Doc No 50
Callaham  MLWears  RLWeber  EJBarton  CYoung  G Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. JAMA 1998;280254- 257
PubMed
Eloubeidi  MAWade  SBProvenzale  D Factors associated with acceptance and full publication of GI endoscopic research originally published in abstract form. Gastrointest Endosc 2001;53275- 282
PubMed
Klassen  TPWiebe  NRussell  K  et al.  Abstracts of randomized controlled trials presented at the Society for Pediatric Research meeting. Arch Pediatr Adolesc Med 2002;156 ((suppl 5)) 474- 479
PubMed
Dickersin  KMin  YI Publication bias: the problem that won’t go away. Ann N Y Acad Sci 1993;703135- 146
PubMed
De Bellefeuille  CMorrison  CATannock  IF The fate of abstracts submitted to a cancer meeting: factors which influence presentation and subsequent publication. Ann Oncol 1992;3187- 191
PubMed
Grzybowski  SThommasen  HVMills  JHerbert  CP Review of University of British Columbia Family Practice Resident Research Projects 1990-1997. Fam Med 1999;31353- 357
PubMed
Bardy  AH Bias in reporting clinical trials. Br J Clin Pharmacol 1998;46147- 150
PubMed
Evers  JL Publication bias in reproductive research. Hum Reprod 2000;152063- 2066
PubMed
Scherer  RWDickersin  KLangenberg  P Full publication of results initially presented in abstracts: a meta-analysis. JAMA 1994;272158- 162
PubMed
Sommer  B The file drawer effect and publication rates in menstrual cycle research. Psychol Women Q 1987;11233- 241
Juzych  MSShin  DHCoffey  JJuzych  LShin  D Whatever happened to abstracts from different sections of the Association for Research in Vision and Ophthalmology? Invest Ophthalmol Vis Sci 1993;341879- 1882
PubMed
Link  AM US and non-US submissions: an analysis of reviewer bias. JAMA 1998;280246- 247
PubMed
Craig  WRHartling  LStevens  KRussell  KKlassen  TP A randomized controlled trial of electronic versus regular mail surveys [abstract 1249]. Pediatr Res 2003;52220A
Dillman  DA Mail and internet surveys: the tailored design method.  New York, NY John Wiley & Sons Inc2000;
Kiroff  GK Publication bias in presentations to the Annual Scientific Congress. ANZ J Surg 2001;71 ((suppl 3)) 167- 171
PubMed
Olson  CMRennie  DCook  D  et al.  Publication bias in editorial decision making. JAMA 2002;2872825- 2828
PubMed
Dickersin  K The existence of publication bias and risk factors for its occurrence. JAMA 1990;2631385- 1389
PubMed
Hetherington  JDickersin  KChalmers  IMeinert  CL Retrospective and prospective identification of unpublished controlled trials: lessons from a survey of obstetricians and pediatricians. Pediatrics 1989;84374- 380
PubMed
Simes  RJ Publication bias: the case for an international registry of clinical trials. J Clin Oncol 1986;41529- 1541
PubMed

Figures

Tables

Table Graphic Jump LocationTable 1. Authors’ Reasons for Nonpublication*
Table Graphic Jump LocationTable 2. Bivariate Analyses of Trial Characteristics and Association With Publication Status

References

Dickersin  K How important is publication bias? a synthesis of available data. AIDS Educ Prev 1997;915- 21
PubMed
Scherer  RWLangenberg  P Full publication of results initially presented in abstracts [Cochrane Methodology Review on CD-ROM].  Oxford, England Cochrane Library, Update Software2003; (1)
Dickersin  KChan  SChalmers  TCSacks  HSSmith  H  Jr Publication bias and clinical trials. Control Clin Trials 1987;8343- 353
PubMed
Dickersin  KMin  YIMeinert  CL Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards. JAMA 1992;267374- 378
PubMed
Easterbrook  PJBerlin  JAGopalan  RMatthews  DR Publication bias in clinical research. Lancet 1991;337867- 872
PubMed
Riordan  FA Do presenters to pediatric meetings get their work published? Arch Dis Child 2000;83524- 526
PubMed
Stern  JMSimes  RJ Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315640- 645
PubMed
Timmer  AHilsden  RJCole  JHailey  DSutherland  LR Publication bias in gastroenterological research: a retrospective cohort study based on abstracts submitted to a scientific meeting. BMC Med Res Methodol 2002;27
PubMed
Dickersin  KMin  YI NIH clinical trials and publication bias. Online J Curr Clin Trials 1993;Doc No 50
Callaham  MLWears  RLWeber  EJBarton  CYoung  G Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. JAMA 1998;280254- 257
PubMed
Eloubeidi  MAWade  SBProvenzale  D Factors associated with acceptance and full publication of GI endoscopic research originally published in abstract form. Gastrointest Endosc 2001;53275- 282
PubMed
Klassen  TPWiebe  NRussell  K  et al.  Abstracts of randomized controlled trials presented at the Society for Pediatric Research meeting. Arch Pediatr Adolesc Med 2002;156 ((suppl 5)) 474- 479
PubMed
Dickersin  KMin  YI Publication bias: the problem that won’t go away. Ann N Y Acad Sci 1993;703135- 146
PubMed
De Bellefeuille  CMorrison  CATannock  IF The fate of abstracts submitted to a cancer meeting: factors which influence presentation and subsequent publication. Ann Oncol 1992;3187- 191
PubMed
Grzybowski  SThommasen  HVMills  JHerbert  CP Review of University of British Columbia Family Practice Resident Research Projects 1990-1997. Fam Med 1999;31353- 357
PubMed
Bardy  AH Bias in reporting clinical trials. Br J Clin Pharmacol 1998;46147- 150
PubMed
Evers  JL Publication bias in reproductive research. Hum Reprod 2000;152063- 2066
PubMed
Scherer  RWDickersin  KLangenberg  P Full publication of results initially presented in abstracts: a meta-analysis. JAMA 1994;272158- 162
PubMed
Sommer  B The file drawer effect and publication rates in menstrual cycle research. Psychol Women Q 1987;11233- 241
Juzych  MSShin  DHCoffey  JJuzych  LShin  D Whatever happened to abstracts from different sections of the Association for Research in Vision and Ophthalmology? Invest Ophthalmol Vis Sci 1993;341879- 1882
PubMed
Link  AM US and non-US submissions: an analysis of reviewer bias. JAMA 1998;280246- 247
PubMed
Craig  WRHartling  LStevens  KRussell  KKlassen  TP A randomized controlled trial of electronic versus regular mail surveys [abstract 1249]. Pediatr Res 2003;52220A
Dillman  DA Mail and internet surveys: the tailored design method.  New York, NY John Wiley & Sons Inc2000;
Kiroff  GK Publication bias in presentations to the Annual Scientific Congress. ANZ J Surg 2001;71 ((suppl 3)) 167- 171
PubMed
Olson  CMRennie  DCook  D  et al.  Publication bias in editorial decision making. JAMA 2002;2872825- 2828
PubMed
Dickersin  K The existence of publication bias and risk factors for its occurrence. JAMA 1990;2631385- 1389
PubMed
Hetherington  JDickersin  KChalmers  IMeinert  CL Retrospective and prospective identification of unpublished controlled trials: lessons from a survey of obstetricians and pediatricians. Pediatrics 1989;84374- 380
PubMed
Simes  RJ Publication bias: the case for an international registry of clinical trials. J Clin Oncol 1986;41529- 1541
PubMed

Correspondence

CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.
Submit a Comment

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 16

Related Content

Customize your page view by dragging & repositioning the boxes below.

See Also...
Articles Related By Topic
Related Collections
PubMed Articles
JAMAevidence.com

The Rational Clinical Examination
Quick Reference

The Rational Clinical Examination
Make the Diagnosis: Sexual Abuse, Child