Skip to main content

Advertisement

Download PDF

Funding source, trial outcome and reporting quality: are they related? Results of a pilot study

BMC Health Services Research volume 2, Article number: 18 (2002) Cite this article

Abstract

Background

There has been increasing concern regarding the potential effects of the commercialization of research.

Methods

In order to examine the relationships between funding source, trial outcome and reporting quality, recent issues of five peer-reviewed, high impact factor, general medical journals were hand-searched to identify a sample of 100 randomized controlled trials (20 trials/journal). Relevant data, including funding source (industry/not-for-profit/mixed/not reported) and statistical significance of primary outcome (favouring new treatment/favouring conventional treatment/neutral/unclear), were abstracted. Quality scores were assigned using the Jadad scale and the adequacy of allocation concealment.

Results

Sixty-six percent of trials received some industry funding. Trial outcome was not associated with funding source (p= .461). There was a preponderance of favourable statistical conclusions among published trials with 67% reporting results that favored a new treatment whereas 6% favoured the conventional treatment. Quality scores were not associated with funding source or trial outcome.

Conclusions

It is not known whether the absence of significant associations between funding source, trial outcome and reporting quality reflects a true absence of an association or is an artefact of inadequate statistical power, reliance on voluntary disclosure of funding information, a focus on trials recently published in the top medical journals, or some combination thereof. Continued and expanded monitoring of potential conflicts is recommended, particularly in light of new guidelines for disclosure that have been endorsed by the ICMJE.

Peer Review reports

Background

There has been increasing concern regarding the potential effects of the commercialization of research [1–5]. This concern has coincided with the reduced availability of public research funds which has, in turn, translated to scientists' increased reliance on industry support. For example, in the United States, approximately 70% of clinical drug trials are now funded by industry [2]. Industry funding of biomedical research can be viewed as a conflict of interest because the sponsor of the study has a vested financial interest in its outcome. Since even the perception of a conflict can undermine the public's trust, due diligence, in the form of recognizing and managing potential conflicts, is warranted.

This is not to suggest that industry abandon its sponsorship of biomedical research. Indeed, many medical discoveries may not have occurred without industry funding. Nevertheless, fundamental errors have been noted in the design of industry-funded trials [6]. These methodological deficiencies, perhaps operating in conjunction with the well-documented phenomenon of publication bias, have led to a preponderance of published trials that have received funding from for-profit entities and whose conclusions favour industry [7–16].

Concern over the increasing commercialization of research also extends to the quality of reporting. Low quality reports have been noted among published trials [6, 17] and reports of low quality trials have been found to exaggerate an intervention's effectiveness [16, 18]. Because relationships between funding source, trial outcome and reporting quality may bias study results, making informed decision making about the merits of an intervention more difficult for clinicians and consumers alike, it is important to examine the extent to which a trial's source of funding influences its results.

Methods

A convenience sample of 100 randomized controlled trials (RCTs) was identified by hand-searching recent issues of five peer-reviewed, high impact factor general medical journals (Annals of Internal Medicine, British Medical Journal, Journal of the American Medical Association, The Lancet, The New England Journal of Medicine). Issues published between January 1999 and October 2000 were searched until 20 RCTs/journal were identified. To be eligible for inclusion, the RCT needed to be published as a full report. Interventions were restricted to pharmaceuticals; medical devices, surgical procedures and methods of medical management (e.g., lifestyle counseling) were excluded. No attempts were made to limit the selection to any particular RCT design, number of treatment arms, comparator (e.g., placebo, active control, alternate dosing, herbal therapy), study population or disease category.

Relevant data, including funding source(s) and primary outcome, were abstracted from each eligible RCT. Reporting quality was assessed using both a composite (overall score on the Jadad scale [19]) and a component (individual items on Jadad scale and adequacy of allocation concealment [20]) approach. The Jadad scale consists of a total of five items; two items relate to blinding, two items relate to randomization and one item assesses the description of withdrawals/drop-outs. When using the Jadad scale to score the quality of a trial report, each of the five items receives a "yes" or a "no," resulting in an overall/composite quality score that can range from 0 to 5; higher scores reflect better methodological quality [19].

Allocation concealment was rated as adequate, inadequate or unclear in the manner proposed by Schulz et al [20]. Allocation concealment refers to the process that prevents foreknowledge of treatment assignment and thus, shields those who enroll participants from being influenced by this knowledge. For example, a trial was rated as having "adequate" concealment if allocations were performed using central randomization; numbered/coded bottles/containers; serially numbered, opaque, sealed envelopes or if the formulations were prepared by a pharmacy. Allocation was classified as "inadequate" if assignments were made on an alternating basis or via reference to case record number or date of birth. Trials that received an "unclear" rating would have failed to provide sufficient information regarding the allocation process on which to base our decision. In all cases, reporting quality was evaluated by two independent, experienced reviewers (DM, TC). No formal training was conducted prior to evaluating the RCTs using either of the quality assessment scales since both raters have extensive experience using these methods. Moreover, since any disagreements in quality ratings were resolved by consensus, we did not undertake assessment of inter-rater reliability.

In order to examine the relationship between trial outcome, funding source and reporting quality, SPSS-PC software was used to conduct statistical analyses in the form of Fisher's exact test or ANOVAs, as appropriate. The odds, and corresponding 95% confidence intervals, of unclear allocation concealment, by funding source and trial outcome, are also presented.

Trials were classified according to their funding source(s) in a manner similar to that used by Rochon [17], permitting comparison of trials across four levels of funding: entirely industry, entirely not-for-profit, mixed and not reported. A trial was classified as having "mixed" funding if it received support from at least one industry source and at least one not-for-profit source. Because this study was restricted to RCTs that examined pharmaceuticals, industry funding is synonymous with pharmaceutical company funding.

The primary outcome was defined as the one stated as such by the authors or, if there was no such statement, the one that was most clinically relevant (i.e., mortality over morbidity). If one outcome was not more clinically relevant than the others, the outcome contributing the most patients was used. On the basis of statistical interpretation of results, rather than reliance on authors' interpretations presented in discussion/conclusion sections, the outcome of trials was classified as favoured new treatment, favoured conventional treatment, neutral (i.e., non-significant) or unclear.

Results

Of the 100 trials reviewed, sixty-six were funded in whole or in part by industry; 6 did not disclose their source of funding. Of these same 100 trials, 67 favoured the new therapy, 6 favoured the conventional treatment, 19 reported neutral findings while, in eight cases, the outcome was classified as unclear owing to ambiguity between the defined outcome of interest and data presented in the results section. Results for reporting quality varied according to the dimensions of quality that were measured: the overall/composite score on the Jadad scale indicated that 74% of the trials were of higher quality (scores of 3–5) with a mean score of 3.31 (SD 1.19) while allocation concealment was found to be unclear in close to 60% of trials.

The results of bivariable analyses examining associations between allocation concealment and funding source and between allocation concealment and trial outcome are presented in Table 1. Odds ratios and corresponding 95% CI for these associations are also presented. Data reflecting the comparison between funding source and trial outcome are presented in Table 2. In none of these cases were the associations found to be statistically significant. A one-way ANOVA also failed to demonstrate statistical significance between the overall/composite score on the Jadad scale, measured as a continuous variable, and funding source (F = 1.853; df = 3; p = .143), trial outcome (F = 1.003; df = 3; p =.395) and allocation concealment (F = 1.319; df = 2; p = .272). The absence of a statistically significant association between reporting quality and funding source and between reporting quality and trial outcome persisted when individual components of the Jadad scale (i.e., blinding, randomization, and description of withdrawals/drop-outs) were examined.

Table 1 Allocation concealment by funding source and by trial outcome
Full size table
Table 2 Funding source by trial outcome
Full size table

Discussion

The results of this pilot study failed to document any association between funding source, trial outcome and reporting quality among a sample of RCTs that were recently published in the top five general medical journals. It is not known whether this finding (which has been observed by some [16, 21] but not by others [7, 13, 15, 22, 23]) reflects a true absence of an association or, instead, represents an artefact, due to limitations inherent in this, a pilot study.

For example, our failure to detect any significant associations may result from a type 2 error that indicates inadequate statistical power. Although our results do not even hint at a trend (and, perhaps, reflect the emphasis now placed on disclosure at the journals included in our sample [24]), the potential for type 2 error is real, as suggested by the width of the confidence intervals presented in Table 1. Our estimates of association were based on a sample size that, while relatively small, is typical of the initial phases of a program theme, such as that upon which we were embarking; future works will accrue larger samples to reduce the likelihood of this error.

Admittedly, the limitations of our study extend beyond issues of statistical power. For example, we depended on authors' disclosure of a trial's funding source(s) and the subsequent publication of this information. Recent work suggests that failure to disclose personal financial conflicts is widespread [4]. Although this is different than reporting a trial's source of funding, it supports the notion that failure to abide by journal disclosure requirements is common. Moreover, since journals, themselves, do not always abide by their own disclosure rules [1, 4], our categorization of funding source(s) may be biased.

In addition, because this study focussed on recent publications of the top five general medical journals, our results may not generalize to journals that differ in their impact factor, disclosure requirements and/or reporting policies. It is important that our results be viewed in this context, particularly since some research suggests that researchers submit their "best" work to the "best" journals [25, 26]. Future research, relying on a modified design that has been used to address similar questions [18, 20, 27] might allow more comprehensive exploration of these relationships. For example, future works, accruing a larger sample of RCTs from a wider variety of journals, should adjust for covariates (e.g., number of sites involved, number of treatment arms, sample size) in multivariable models and may find it fruitful to examine statistical outcomes in the context of effect size.

It is also possible that the discordance seen between our findings and others [7, 13, 15, 22, 23] stems, in whole or in part, from differences in the operational definition of "industry-sponsored research." More specifically, some [7, 13, 15, 22] have categorized RCTs that receive any industry funding to be "industry-sponsored" and have compared this category with those RCTs that are wholly funded by not-for-profit monies. In our study, we treated RCTs that received funding from one or more corporate sponsor and one or more not-for-profit sponsor as receiving "mixed funding" while the category "industry only" was reserved for those RCTs that did not receive any not-for-profit funds. It is important that authors acknowledge the potential for differences in operational definitions of funding source between studies to produce discordant results; this can be best accomplished by ensuring that the scheme for classifying trials according to their funding source(s) be made explicit in each report.

As a result of these limitations, we are unable to conclude, with a high degree of confidence, that the absence of an association between funding source, trial outcome and reporting quality that was documented in this study reflects a benefit of strengthened disclosure requirements at the top general medical journals. Our results do, however, suggest the benefits of one particular standardized reporting requirement, CONSORT [28]. Endorsement of CONSORT by the International Committee of Medical Journal Editors (ICMJE) [29] may have contributed to the majority (~75%) of trial reports receiving moderate-high quality composite scores on the Jadad scale. This has not always been the case, with almost uniformly poor quality found amongst trials published prior to widespread adherence to CONSORT [14, 17–20, 30]. There is still room for improvement, however. Recent revisions to CONSORT and its adoption by increasing numbers of journals should address the alarming dominance of "unclear" allocation concealment seen among trials examined in this report and elsewhere [9, 20, 30–34]. The persistence of a preponderance of trial reports favoring novel treatments [7, 8, 10, 12, 21, 35], however, remains a challenge.

Conclusion

Concerted and continued efforts to monitor the reporting quality of RCTs, to ascertain the best method(s) for its evaluation [36] and to encourage the mandatory registration of trials [8, 10, 12, 37–39] are recommended. Given that the ICMJE recently strengthened its requirement for disclosure of information as to the role(s) of study sponsor(s) in all aspects of study design, conduct and publication [24] while, at the same time, the New England Journal of Medicine has announced that they will be relaxing their longstanding rules on conflict of interest [40], the question posed by this study should be revisited to allow for more definitive determination of the impact of industry sponsorship on biomedical research. We encourage journal editors to continue to work together in order to reach consensus as to the particulars of reporting requirements.

References

  1. 1.

    Angell M, Utiger RD, Wood AJ: Disclosure of Authors' Conflicts of Interest: A Follow-up. N Engl J Med. 2000, 342: 586-587. 10.1056/NEJM200002243420812.

  2. 2.

    Bodenheimer T: Uneasy Alliance – Clinical Investigators and the Pharmaceutical Industry. N Engl J Med. 2000, 342: 1539-1544. 10.1056/NEJM200005183422024.

  3. 3.

    DeAngelis CD: Conflict of Interest and the Public Trust. JAMA. 2000, 284: 2237-2238. 10.1001/jama.284.17.2237.

  4. 4.

    Krimsky S, Rothenberg LS: Conflict of Interest Policies in Science and Medical Journals: Editorial Practices and Author Disclosures. Sci Eng Ethics. 2001, 7: 205-218.

  5. 5.

    Moses H, Martin JB: Academic Relationships with Industry: A New Model for Biomedical Research. JAMA. 2001, 285: 933-935.

  6. 6.

    Bero LA, Rennie D: Influences on the Quality of Published Drug Studies. Int J Technol Assess Health Care. 1996, 12: 209-237.

  7. 7.

    Davidson RA: Source of funding and outcome of clinical trials. J Gen Intern Med. 1986, 1: 155-158.

  8. 8.

    Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H: Publication bias and clinical trials. Control Clin Trials. 1987, 8: 343-353. 10.1016/0197-2456(87)90155-3.

  9. 9.

    Gotzsche P: Methodology and overt and hidden bias in reports of 196 double-blind trials of non-steroidal anti-inflammatory agents in rheumatoid arthritis. Control Clin Trials. 1989, 10: 31-56. 10.1016/0197-2456(89)90017-2.

  10. 10.

    Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR: Publication bias in clinical research. Lancet. 1991, 337: 867-872. 10.1016/0140-6736(91)90201-Y.

  11. 11.

    Bero LA, Galbraith A, Rennie D: The Publication of Sponsored Symposiums in Medical Journals. N Engl J Med. 1992, 327: 1135-1140.

  12. 12.

    Dickersin K, Min YI, Meinert CL: Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. JAMA. 1992, 267: 374-378. 10.1001/jama.267.3.374.

  13. 13.

    Rochon PA, Gurwitz JH, Simms RW, Fortin PR, Felson DT, Minaker KL, Chalmers TC: A Study of Manufacturer-Supported Trials of Nonsteroidal Anti-inflammatory Drugs in the Treatment of Arthritis. Arch Intern Med. 1994, 154: 157-163. 10.1001/archinte.154.2.157.

  14. 14.

    Barnes DE, Bero LA: Scientific quality of original research articles on environmental tobacco smoke. Tob Control. 1997, 6: 19-26.

  15. 15.

    Cho MK, Bero LA: The Quality of Drug Studies Published in Symposium Proceedings. Ann Intern Med. 1996, 124: 485-489.

  16. 16.

    Djulbegovic B, Lacevic M, Cantor A, Fields KK, Bennett CL, Adams JR, Kuderer NM, Lyman GH: The uncertainty principle and industry-sponsored research. Lancet. 2000, 356: 635-638. 10.1016/S0140-6736(00)02605-2.

  17. 17.

    Rochon PA, Gurwitz JH, Cheung M, Hayes JA, Chalmers TC: Evaluating the quality of articles published in journal supplements compared with the quality of those published in the parent journal. JAMA. 1994, 272: 108-113. 10.1001/jama.272.2.108.

  18. 18.

    Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP: Does quality of reports of randomized trials affect estimates of intervention efficacy reported in meta-analyses?. Lancet. 1998, 352: 609-613. 10.1016/S0140-6736(98)01085-X.

  19. 19.

    Jadad AR, Moore A, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ, McQuay HJ: Assessing the quality of reports of randomized controlled trials: is blinding necessary?. Control Clin Trials. 1996, 17: 1-12. 10.1016/0197-2456(95)00134-4.

  20. 20.

    Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995, 273: 408-412. 10.1001/jama.273.5.408.

  21. 21.

    Kjaergard LL, Nikolova D, Gluud C: Randomized clinical trials in hepatology: predictors of quality. Hepatology. 1999, 30: 1134-1138.

  22. 22.

    Friedberg M, Saffran B, Stinson TJ: Evaluation of conflict of interest in economic analyses of new drugs used in oncology. JAMA. 1999, 282: 1453-1457. 10.1001/jama.282.15.1453.

  23. 23.

    Kjaergard LL, Als-Nieslen B: Association between competing interests and authors' conclusions: epidemiological study of randomised clinical trials published in the BMJ. BMJ. 2002, 325: 249-252. 10.1136/bmj.325.7358.249.

  24. 24.

    Davidoff F, DeAngelis CD, Drazen JM, Hoey J, Hojgaard L, Horton R, Kotzin S, Nicholls MG, Nylenna M, Nylenna M, et al: Sponsorship, Authorship and Accountability. JAMA. 2001, 286: 1232-1234. 10.1001/jama.286.10.1232.

  25. 25.

    Lee KP, Schotland M, Bacchetti P, Bero LA: Association of Journal Quality Indicators with Methodological Quality of Clinical Research Articles. JAMA. 2002, 287: 2805-2808. 10.1001/jama.287.21.2805.

  26. 26.

    Juni P, Holenstein F, Sterne J, Bartlett C, Egger M: Direction and impact of language bias in meta-analyses of controlled trials: empirical study. Int J Epidemiol. 2002, 31: 115-123. 10.1093/ije/31.1.115.

  27. 27.

    McAuley L, Pham B, Tugwell P, Moher D: Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses?. Lancet. 2000, 356: 1228-1231. 10.1016/S0140-6736(00)02786-0.

  28. 28.

    Moher D, Schulz KF, Altman DF, for the CONSORT Group: The CONSORT Statement: Revised Recommendations for Improving the Quality of Reports of Parallel-Group Randomized Trials. JAMA. 2001, 285: 1987-1991. 10.1001/jama.285.15.1987.

  29. 29.

    International Council of Medical Journal Editors (ICMJE): Uniform requirements for Manuscripts Submitted to Biomedical Journals. May 2000 [http://www.icmje.org/index.html accessed August 28, 2001 at 9:36 a.m.]

  30. 30.

    Schulz KF, Chalmers I, Grimes DA, Altman DG: Assessing the quality of randomization from reports of controlled trials published in obstetrics and gynecology journals. JAMA. 1994, 272: 125-8. 10.1001/jama.272.2.125.

  31. 31.

    Chalmers TC, Celano P, Sachs HS, Smith H: Bias in treatment assignment in controlled clinical trials. N Engl J Med. 1983, 309: 1358-1361.

  32. 32.

    Nicolucci A, Grilli R, Alexanian AA, Apolone G, Torri V, Liberati A: Quality, evolution and clinical implications of randomized controlled trials on the treatment of lung cancer. A lost opportunity for meta-analysis. JAMA. 1989, 262: 2101-2107. 10.1001/jama.262.15.2101.

  33. 33.

    Altman DF, Dore CJ: Randomization and baseline comparisons in clinical trials. Lancet. 1990, 335: 149-153. 10.1016/0140-6736(90)90014-V.

  34. 34.

    Thornley B, Adams C: Content & quality of 2000 controlled trials in schizophrenia over 50 years. BMJ. 1998, 317: 1181-1184.

  35. 35.

    Colditz G, Miller J, Mosteller F: How study design affects outcomes in comparisons of treatments. 1: medical. Stat Med. 1989, 8: 441-454.

  36. 36.

    Juni P, Altman DF, Egger M: Systematic Reviews in Health Care: Assessing the Quality of Controlled Clinical Trials. BMJ. 2001, 323: 42-46. 10.1136/bmj.323.7303.42.

  37. 37.

    Simes RJ: Publication bias: the case for an international registry of clinical trials. J Clin Oncol. 1986, 4: 1529-1541.

  38. 38.

    Chalmers I, Dickersin K, Chalmers TC: Getting to grips with Archie Cochrane's agenda. All randomized controlled trials should be registered and reported. BMJ. 1992, 305: 786-787.

  39. 39.

    Horton R, Smith R: Time to register randomized trials. Lancet. 1999, 354: 1138-1139. 10.1016/S0140-6736(99)00328-1.

  40. 40.

    Gottlieb S: New England Journal loosens its rules on conflict of interest. BMJ. 2002, 324: 1474-10.1136/bmj.324.7352.1474/a.

Pre-publication history

  1. The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/2/18/prepub

Download references

Author information

Affiliations

  1. Chalmers Research Group, Ottawa, Canada
    • Tammy J Clifford
    • , Nicholas J Barrowman
    •  & David Moher
  2. Children's Hospital of Eastern Ontario Research Institute, Ottawa, Canada
    • Tammy J Clifford
    • , Nicholas J Barrowman
    •  & David Moher
  3. School of Mathematics & Statistics, Carleton University, Canada
    • Nicholas J Barrowman
  4. Department of Paediatrics, University of Ottawa, Ottawa, Canada
    • Nicholas J Barrowman
    •  & David Moher
  5. Department of Epidemiology & Community Medicine, University of Ottawa, Ottawa, Canada
    • David Moher
Authors
  1. Tammy J Clifford
    View author publications
    You can also search for this author in
  2. Nicholas J Barrowman
    View author publications
    You can also search for this author in
  3. David Moher
    View author publications
    You can also search for this author in

Corresponding author

Correspondence to Tammy J Clifford.

Additional information

Competing Interests

None declared.

Authors' Contributions

TJC and DM have participated sufficiently in the work to take public responsibility for the whole content. TJC has made substantial contributions to the intellectual content of the paper as they relate to this study's conception and design, the acquisition of data, its analysis and interpretation. TJC was responsible for the initial draft of the manuscript and all subsequent revisions. NB assisted in the analysis and interpretation of the data and provided critical revisions of the manuscript for important intellectual content. DM provided the impetus for the study's conception and design, participated in the analysis and interpretation of data, provided feedback on initial drafts of the manuscript and, via administrative supervision, also permitted TC and NB the time to carry-out this work. All authors have read and approved the final manuscript.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Clifford, T.J., Barrowman, N.J. & Moher, D. Funding source, trial outcome and reporting quality: are they related? Results of a pilot study. BMC Health Serv Res 2, 18 (2002). https://doi.org/10.1186/1472-6963-2-18

Download citation

Keywords

  • Trial Outcome
  • Funding Source
  • Allocation Concealment
  • Reporting Quality
  • Voluntary Disclosure

BMC Health Services Research

ISSN: 1472-6963

Contact us