Training practitioners in preparing systematic reviews: a cross-sectional survey of participants in the Australasian Cochrane Centre training program
© Piehl et al; licensee BioMed Central Ltd. 2002
Received: 20 February 2002
Accepted: 2 June 2002
Published: 2 June 2002
Although systematic reviews of health care interventions are an invaluable tool for health care providers and researchers, many potential authors never publish reviews. This study attempts to determine why some people with interest in performing systematic reviews do not subsequently publish a review; and what steps could possibly increase review completion.
Cross-sectional survey by email and facsimile of the 179 participants in Australasian Cochrane Centre training events between 1998 and 2000.
Ninety-two participants responded to the survey (51 percent). Response rate of deliverable surveys was 82 percent (92/112). The remainder of the participants had invalid or no contact information on file. More than 75 percent of respondents felt that the current workshops met their needs for training. The most critical barriers to completion of a Cochrane review were: lack of time (80 percent), lack of financial support (36 percent), methodological problems (23 percent) and problems with group dynamics (10 percent).
Strategies to protect reviewer time and increase the efficiency of the review process may increase the numbers of trained reviewers completing a systematic review.
Increasing emphasis is being placed on evidence-based medicine. [1, 2] The best evidence for treatment interventions comes from systematic reviews of randomised controlled trials,  however, systematic reviews with meta-analysis of published and unpublished data are time- and labour-intensive.
A major player in the evidence-based medicine movement has been the Cochrane Collaboration, an international organisation committed to 'preparing, maintaining and promoting the accessibility of systematic reviews of the effects of health care interventions.'  The Australasian Cochrane Centre is one of the component centres of the Cochrane Collaboration, providing ongoing training and support to people in the Australasian region who prepare systematic reviews that are subsequently published on the Cochrane Library.  The ultimate goal of the training program is to increase the number and quality of completed Cochrane reviews, and ensure that they are routinely updated.
determine why some people who attend Cochrane training workshop do not go on to publish a systematic review; and
what steps could increase systematic review completion.
Survey Population and Contact Information
Our survey population included the 179 participants of the Australasian Cochrane Centre's training workshops from 1998 to 2000. Contact information for the year 2000 participants was drawn from the Australasian Cochrane Centre database (initiated in January 2000) of all workshop participants and current Cochrane reviewers in Australasia. Several partial data systems provided contact information for 1998–1999 participants.
Participants were asked to respond to a cross-sectional survey of 21 'yes' or 'no' questions with a provision for open-ended comments. [See Additional File for the questionnaire used in this study.] The format of our internally developed questionnaire did not lend itself to statistical testing for internal consistency. No demographic details were collected, so that responses could be deidentified after receipt of the questionnaire.
The initial questionnaire was sent via email or fax to all participants. A second email or fax was sent four weeks later, with a third survey sent via post to those participants who had not responded to the first two requests and had a postal address on file.
Both quantitative and qualitative methods were used in data analysis. The structured questions were analysed by SPSS for Windows software, release 10.0.5, and results were presented in the form of frequencies and percentages. Content analysis was used to analyse the qualitative responses.
Publication status of workshop participants was determined by searching the Cochrane Library by each workshop participants' last name and first initial.
Approximately 40 percent of these 1998–2000 workshop participants have published a protocol or review on the Cochrane Library, as of 2001 issue 4. However, due to issues of anonymity of questionnaire response, we are unable to stratify the results based upon publication status.
Findings from Questionnaire: Factors that Interfere with Systematic Review Completion
Barriers for Training in, and Completion of, Systematic Review of the Literature
Factors Interfering with Systematic Review Completion
Lack of Time
Lack of Financial Support
Recently Completed Training and Yet to Start Review
Lack of Institutional Encouragement/Support
Working on a Systematic Review Outside of Cochrane
Difficulty Locating Trials
Problems with Group Dynamics
Loss of Interest
Training Too Infrequent
Training Too Inconvenient to Attend
Training Too Far Away
Training Assumes Too Much Prerequisite Knowledge
Training Too Long
Methodological barriers affected 23 percent of respondents. Three respondents specifically mentioned problems with translation of non-English papers, and two had no access to EMBASE (Excerpta Medica). In addition, one person was frustrated that a similar title for a Cochrane review had been registered, but not completed in a timely manner. Ten percent of respondents had problems with group dynamics, with two stating this factor as a significant delay to publication, and one complaining of contradictory editorial commentary. Thirteen respondents (22 percent) had yet to start a review. Nine respondents (16 percent) were working on a systematic review outside of the Cochrane Collaboration, with one qualitative commentator stating that the employer insisted on publication of the review prior to submission to Cochrane.
A substantial minority (38 percent) of respondents felt that follow-up communication, especially via email, would help them complete their review. From the qualitative comments, this email could occur every one to six months, with 11 of the 12 respondents suggesting a time period of every three months or less.
Findings from Questionnaire: Training Issues
Scheduling and content of the training workshops appeared to meet the needs of participants. Over 75 percent of respondents felt that the workshops were not too infrequent, too inconvenient, too far away, assumed too much knowledge, or were too long. Qualitative comments stressed the need for ongoing support after training. Three respondents requested a specific contact person, whereas another four asked for support to discuss general problems, statistical analysis issues, help selecting trials, and the creation of a bulletin board-style mentoring system. Three more respondents requested advanced training.
In January 2000, the Australasian Cochrane Centre began collecting formal immediate evaluations from workshop participants. Analysis of these data indicates that the average participant self-ranks only slightly above an absolute beginner in systematic review methodology. Mean satisfaction with content, presentations, opportunity to ask questions, and overall satisfaction range between 4.5 to 4.7, on a scale from 1 = highly dissatisfied to 5 = highly satisfied.
Finally, 83 percent of respondents stated that their Cochrane training had made them more likely to use the Cochrane Library to answer clinical questions. Only 45 percent, however, felt that their practice patterns had changed.
Lack of time is by far the most commonly reported barrier to completion of a systematic review. Financial, institutional, personal, and methodological issues are not problematic for the majority of respondents. These findings have important implications for both reviewers and entities commissioning systematic reviews. In the absence of funding for all or, at least, part of a systematic review, it will continue to be difficult for reviewers to do this research by taking more time out of their busy lives. The challenge becomes the ability to support researchers in minimising the time they spend on the process of the review, without affecting quality.
Limitations in this study are several. First of all, the subjective nature of this study may have lead reviewers to state that lack of time was a barrier to completion of the project, when another reason, such as change in professional priorities was the 'true' barrier. Second, searching publications by last name and first initial may lead to either overstatement or understatement of publication status, as reviewers may have common last names, have changed their last names, or use a different first initial to the first name they use routinely.
discussion at training events of the importance of informing the Australasian Cochrane Centre of change of address,
reminder notices on newsletters, and
liaison with other Cochrane entities.
The year 2000 also saw a change in enrolment policy for the training workshops, allowing people to attend the course only if they had a title or protocol registered with the relevant Cochrane entity. Therefore, our results are likely to be more representative of recent participants of our program, committed to the Cochrane system.
Lack of time, not skill, is the most common barrier to completion of a systematic review for authors who have attended such training given by the Australasian Cochrane Centre. One solution lies in advancements in methodology to make systematic reviews more efficient, such as automating aspects of the review or providing authors with specialised, centralised services in literature searches, project management, and the like.
We expect that these advances in systematic review methodology, increased capacity for completing systematic reviews in Australasia, and realistic expectations regarding required investment of time for valid evidence will lead to an increased number and quality of systematic reviews published on the Cochrane Library. Ultimately, consumers and practitioners of health care around the globe should benefit from this increased knowledge.
Ms. Julie Hume for her assistance with the Australasian Cochrane Centre database, and Dr. Phil Alderson of the UK Cochrane Centre and Mr. Jason Wasiak of the Australasian Cochrane Centre for their valuable comments on this manuscript.
- Sackett D, Rosenberg W, Muir Gray JA, Haynes RB, Richardson WS: Evidence based medicine: what it is and what it isn't. BMJ. 1996, 312: 71-2.View ArticlePubMedPubMed CentralGoogle Scholar
- Moynihan R: Too Much Medicine? The Business of Health and its Risks for You. Sydney: ABC Books. 1998Google Scholar
- NHMRC: A Guide to the Development, Implementation and Evaluation of Clinical Practice Guidelines. Canberra: NHMRC.
- Mission statement of the Cochrane Collaboration. 2002, [http://www.Cochrane.de/]
- The Cochrane Library, Issue 4, 2001. Oxford: Update Software.
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/2/11/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article: verbatim copying and redistribution of this article are permitted in all media for any purpose, provided this notice is preserved along with the article's original URL.