Funding source, trial outcome and reporting quality: are they related? Results of a pilot study

Abstract

Background

There has been increasing concern regarding the potential effects of the commercialization of research.

Methods

In order to examine the relationships between funding source, trial outcome and reporting quality, recent issues of five peer-reviewed, high impact factor, general medical journals were hand-searched to identify a sample of 100 randomized controlled trials (20 trials/journal). Relevant data, including funding source (industry/not-for-profit/mixed/not reported) and statistical significance of primary outcome (favouring new treatment/favouring conventional treatment/neutral/unclear), were abstracted. Quality scores were assigned using the Jadad scale and the adequacy of allocation concealment.

Results

Sixty-six percent of trials received some industry funding. Trial outcome was not associated with funding source (p= .461). There was a preponderance of favourable statistical conclusions among published trials with 67% reporting results that favored a new treatment whereas 6% favoured the conventional treatment. Quality scores were not associated with funding source or trial outcome.

Conclusions

It is not known whether the absence of significant associations between funding source, trial outcome and reporting quality reflects a true absence of an association or is an artefact of inadequate statistical power, reliance on voluntary disclosure of funding information, a focus on trials recently published in the top medical journals, or some combination thereof. Continued and expanded monitoring of potential conflicts is recommended, particularly in light of new guidelines for disclosure that have been endorsed by the ICMJE.

Background

There has been increasing concern regarding the potential effects of the commercialization of research [1–5]. This concern has coincided with the reduced availability of public research funds which has, in turn, translated to scientists' increased reliance on industry support. For example, in the United States, approximately 70% of clinical drug trials are now funded by industry [2]. Industry funding of biomedical research can be viewed as a conflict of interest because the sponsor of the study has a vested financial interest in its outcome. Since even the perception of a conflict can undermine the public's trust, due diligence, in the form of recognizing and managing potential conflicts, is warranted.

This is not to suggest that industry abandon its sponsorship of biomedical research. Indeed, many medical discoveries may not have occurred without industry funding. Nevertheless, fundamental errors have been noted in the design of industry-funded trials [6]. These methodological deficiencies, perhaps operating in conjunction with the well-documented phenomenon of publication bias, have led to a preponderance of published trials that have received funding from for-profit entities and whose conclusions favour industry [7–16].

Concern over the increasing commercialization of research also extends to the quality of reporting. Low quality reports have been noted among published trials [6, 17] and reports of low quality trials have been found to exaggerate an intervention's effectiveness [16, 18]. Because relationships between funding source, trial outcome and reporting quality may bias study results, making informed decision making about the merits of an intervention more difficult for clinicians and consumers alike, it is important to examine the extent to which a trial's source of funding influences its results.

Methods

A convenience sample of 100 randomized controlled trials (RCTs) was identified by hand-searching recent issues of five peer-reviewed, high impact factor general medical journals (Annals of Internal Medicine, British Medical Journal, Journal of the American Medical Association, The Lancet, The New England Journal of Medicine). Issues published between January 1999 and October 2000 were searched until 20 RCTs/journal were identified. To be eligible for inclusion, the RCT needed to be published as a full report. Interventions were restricted to pharmaceuticals; medical devices, surgical procedures and methods of medical management (e.g., lifestyle counseling) were excluded. No attempts were made to limit the selection to any particular RCT design, number of treatment arms, comparator (e.g., placebo, active control, alternate dosing, herbal therapy), study population or disease category.

Relevant data, including funding source(s) and primary outcome, were abstracted from each eligible RCT. Reporting quality was assessed using both a composite (overall score on the Jadad scale [19]) and a component (individual items on Jadad scale and adequacy of allocation concealment [20]) approach. The Jadad scale consists of a total of five items; two items relate to blinding, two items relate to randomization and one item assesses the description of withdrawals/drop-outs. When using the Jadad scale to score the quality of a trial report, each of the five items receives a "yes" or a "no," resulting in an overall/composite quality score that can range from 0 to 5; higher scores reflect better methodological quality [19].

Allocation concealment was rated as adequate, inadequate or unclear in the manner proposed by Schulz et al [20]. Allocation concealment refers to the process that prevents foreknowledge of treatment assignment and thus, shields those who enroll participants from being influenced by this knowledge. For example, a trial was rated as having "adequate" concealment if allocations were performed using central randomization; numbered/coded bottles/containers; serially numbered, opaque, sealed envelopes or if the formulations were prepared by a pharmacy. Allocation was classified as "inadequate" if assignments were made on an alternating basis or via reference to case record number or date of birth. Trials that received an "unclear" rating would have failed to provide sufficient information regarding the allocation process on which to base our decision. In all cases, reporting quality was evaluated by two independent, experienced reviewers (DM, TC). No formal training was conducted prior to evaluating the RCTs using either of the quality assessment scales since both raters have extensive experience using these methods. Moreover, since any disagreements in quality ratings were resolved by consensus, we did not undertake assessment of inter-rater reliability.

In order to examine the relationship between trial outcome, funding source and reporting quality, SPSS-PC software was used to conduct statistical analyses in the form of Fisher's exact test or ANOVAs, as appropriate. The odds, and corresponding 95% confidence intervals, of unclear allocation concealment, by funding source and trial outcome, are also presented.

Trials were classified according to their funding source(s) in a manner similar to that used by Rochon [17], permitting comparison of trials across four levels of funding: entirely industry, entirely not-for-profit, mixed and not reported. A trial was classified as having "mixed" funding if it received support from at least one industry source and at least one not-for-profit source. Because this study was restricted to RCTs that examined pharmaceuticals, industry funding is synonymous with pharmaceutical company funding.

The primary outcome was defined as the one stated as such by the authors or, if there was no such statement, the one that was most clinically relevant (i.e., mortality over morbidity). If one outcome was not more clinically relevant than the others, the outcome contributing the most patients was used. On the basis of statistical interpretation of results, rather than reliance on authors' interpretations presented in discussion/conclusion sections, the outcome of trials was classified as favoured new treatment, favoured conventional treatment, neutral (i.e., non-significant) or unclear.

Results

Of the 100 trials reviewed, sixty-six were funded in whole or in part by industry; 6 did not disclose their source of funding. Of these same 100 trials, 67 favoured the new therapy, 6 favoured the conventional treatment, 19 reported neutral findings while, in eight cases, the outcome was classified as unclear owing to ambiguity between the defined outcome of interest and data presented in the results section. Results for reporting quality varied according to the dimensions of quality that were measured: the overall/composite score on the Jadad scale indicated that 74% of the trials were of higher quality (scores of 3–5) with a mean score of 3.31 (SD 1.19) while allocation concealment was found to be unclear in close to 60% of trials.

The results of bivariable analyses examining associations between allocation concealment and funding source and between allocation concealment and trial outcome are presented in Table 1. Odds ratios and corresponding 95% CI for these associations are also presented. Data reflecting the comparison between funding source and trial outcome are presented in Table 2. In none of these cases were the associations found to be statistically significant. A one-way ANOVA also failed to demonstrate statistical significance between the overall/composite score on the Jadad scale, measured as a continuous variable, and funding source (F = 1.853; df = 3; p = .143), trial outcome (F = 1.003; df = 3; p =.395) and allocation concealment (F = 1.319; df = 2; p = .272). The absence of a statistically significant association between reporting quality and funding source and between reporting quality and trial outcome persisted when individual components of the Jadad scale (i.e., blinding, randomization, and description of withdrawals/drop-outs) were examined.

Table 1 Allocation concealment by funding source and by trial outcome

Table 2 Funding source by trial outcome

Discussion

The results of this pilot study failed to document any association between funding source, trial outcome and reporting quality among a sample of RCTs that were recently published in the top five general medical journals. It is not known whether this finding (which has been observed by some [16, 21] but not by others [7, 13, 15, 22, 23]) reflects a true absence of an association or, instead, represents an artefact, due to limitations inherent in this, a pilot study.

For example, our failure to detect any significant associations may result from a type 2 error that indicates inadequate statistical power. Although our results do not even hint at a trend (and, perhaps, reflect the emphasis now placed on disclosure at the journals included in our sample [24]), the potential for type 2 error is real, as suggested by the width of the confidence intervals presented in Table 1. Our estimates of association were based on a sample size that, while relatively small, is typical of the initial phases of a program theme, such as that upon which we were embarking; future works will accrue larger samples to reduce the likelihood of this error.

Admittedly, the limitations of our study extend beyond issues of statistical power. For example, we depended on authors' disclosure of a trial's funding source(s) and the subsequent publication of this information. Recent work suggests that failure to disclose personal financial conflicts is widespread [4]. Although this is different than reporting a trial's source of funding, it supports the notion that failure to abide by journal disclosure requirements is common. Moreover, since journals, themselves, do not always abide by their own disclosure rules [1, 4], our categorization of funding source(s) may be biased.

In addition, because this study focussed on recent publications of the top five general medical journals, our results may not generalize to journals that differ in their impact factor, disclosure requirements and/or reporting policies. It is important that our results be viewed in this context, particularly since some research suggests that researchers submit their "best" work to the "best" journals [25, 26]. Future research, relying on a modified design that has been used to address similar questions [18, 20, 27] might allow more comprehensive exploration of these relationships. For example, future works, accruing a larger sample of RCTs from a wider variety of journals, should adjust for covariates (e.g., number of sites involved, number of treatment arms, sample size) in multivariable models and may find it fruitful to examine statistical outcomes in the context of effect size.

It is also possible that the discordance seen between our findings and others [7, 13, 15, 22, 23] stems, in whole or in part, from differences in the operational definition of "industry-sponsored research." More specifically, some [7, 13, 15, 22] have categorized RCTs that receive any industry funding to be "industry-sponsored" and have compared this category with those RCTs that are wholly funded by not-for-profit monies. In our study, we treated RCTs that received funding from one or more corporate sponsor and one or more not-for-profit sponsor as receiving "mixed funding" while the category "industry only" was reserved for those RCTs that did not receive any not-for-profit funds. It is important that authors acknowledge the potential for differences in operational definitions of funding source between studies to produce discordant results; this can be best accomplished by ensuring that the scheme for classifying trials according to their funding source(s) be made explicit in each report.

As a result of these limitations, we are unable to conclude, with a high degree of confidence, that the absence of an association between funding source, trial outcome and reporting quality that was documented in this study reflects a benefit of strengthened disclosure requirements at the top general medical journals. Our results do, however, suggest the benefits of one particular standardized reporting requirement, CONSORT [28]. Endorsement of CONSORT by the International Committee of Medical Journal Editors (ICMJE) [29] may have contributed to the majority (~75%) of trial reports receiving moderate-high quality composite scores on the Jadad scale. This has not always been the case, with almost uniformly poor quality found amongst trials published prior to widespread adherence to CONSORT [14, 17–20, 30]. There is still room for improvement, however. Recent revisions to CONSORT and its adoption by increasing numbers of journals should address the alarming dominance of "unclear" allocation concealment seen among trials examined in this report and elsewhere [9, 20, 30–34]. The persistence of a preponderance of trial reports favoring novel treatments [7, 8, 10, 12, 21, 35], however, remains a challenge.

Conclusion

Concerted and continued efforts to monitor the reporting quality of RCTs, to ascertain the best method(s) for its evaluation [36] and to encourage the mandatory registration of trials [8, 10, 12, 37–39] are recommended. Given that the ICMJE recently strengthened its requirement for disclosure of information as to the role(s) of study sponsor(s) in all aspects of study design, conduct and publication [24] while, at the same time, the New England Journal of Medicine has announced that they will be relaxing their longstanding rules on conflict of interest [40], the question posed by this study should be revisited to allow for more definitive determination of the impact of industry sponsorship on biomedical research. We encourage journal editors to continue to work together in order to reach consensus as to the particulars of reporting requirements.