This article has Open Peer Review reports available.
Credible knowledge: A pilot evaluation of a modified GRADE method using parent-implemented interventions for children with autism
- J Michael Van Adel1Email author,
- Jennifer Dunn Geier†2,
- Adrienne Perry†3 and
- Jo-Ann M Reitzel†4
https://doi.org/10.1186/1472-6963-11-60
© Michael Van Adel et al; licensee BioMed Central Ltd. 2011
Received: 23 December 2009
Accepted: 22 March 2011
Published: 22 March 2011
Abstract
Background
Decision-making in child and youth mental health (CYMH) care requires recommendations that are developed through an efficient and effective method and are based on credible knowledge. Credible knowledge is informed by two sources: scientific evidence, and practice-based evidence, that reflects the "real world" experience of service providers. Current approaches to developing these recommendations in relation to CYMH will typically include evidence from one source or the other but do not have an objective method to combine the two. To this end, a modified version of the Grading Recommendations Assessment, Development and Evaluation (GRADE) approach was pilot-tested, a novel method for the CYMH field.
Methods
GRADE has an explicit methodology that relies on input from scientific evidence as well as a panel of experts. The panel established the quality of evidence and derived detailed recommendations regarding the organization and delivery of mental health care for children and youth or their caregivers. In this study a modified GRADE method was used to provide precise recommendations based on a specific CYMH question (i.e. What is the current credible knowledge concerning the effects of parent-implemented, early intervention with their autistic children?).
Results
Overall, it appeared that early, parent-implemented interventions for autism result in positive effects that outweigh any undesirable effects. However, as opposed to overall recommendations, the heterogeneity of the evidence required that recommendations be specific to particular interventions, based on the questions of whether the benefits of a particular intervention outweighs its harms.
Conclusions
This pilot project provided evidence that a modified GRADE method may be an effective and practical approach to making recommendations in CYMH, based on credible knowledge. Key strengths of the process included separating the assessments of the quality of the evidence and the strength of recommendations, transparency in decision-making, and the objectivity of the methods. Most importantly, this method combined the evidence and clinical experience in a more timely, explicit and simple process as compared to previous approaches. The strengths, limitations and modifications of the approach as they pertain to CYMH, are discussed.
Keywords
Background
An essential ingredient in effective clinical practice is that clinicians benefit, in a timely way, from current research knowledge in their fields. Those responsible for the organization and delivery of child and youth mental health (CYMH) care require such knowledge, which is rapidly changing and ever increasing. Moreover, clinicians benefit from knowledge that is credible. This is especially critical in the development of clinical recommendations, which are intended to achieve the linked objectives of improvements in patient or client outcome and the efficient use of resources. To be credible, these recommendations must consider input from two sources: scientific evidence and the practice-based evidence, reflecting the "real world" expertise and experience of service providers.
Clearly, it is less than practical to expect that individual clinicians can continuously survey and evaluate available knowledge and adapt clinical decisions accordingly. Despite this recognition, however, there are still obstacles that need to be overcome in improving the access to and practical application of credible knowledge in CYMH. Primarily, there is a lack of agreement about the best methods for organizations to identify, assess, and synthesize relevant studies [1–7]. Without a useful methodology for identification and evaluation, published research results may not be adequately assessed for their scientific validity. Poor research, which can suffer from serious methodological flaws, may then be relied upon to make critical treatment decisions. On the other hand, when recommendations are based on rigorous evaluation of only high quality evidence, this evidence may not be assessed for its relevance and practical application to specific care and may not be suitable for generalization to clinicians' practice. These flaws are present in the current approaches to synthesizing evidence-based practice in CYMH.
Numerous professional bodies and scientific organizations have taken the approach of developing clinical practice guidelines. These guidelines include recommendations to aid patient and practitioner decision-making in specific clinical situations. A strength of this approach is that a panel of experts typically develops these recommendations. Panel members have access to the evidence, an understanding of the clinical problems and research methods, and sufficient time for deliberation. However, the recommendations are often formulated without systematically identifying and summarizing the evidence [8]. A proper evaluation of the evidence is critical in ensuring the validity of guidelines. Otherwise, relevant studies may be missed and recommendations made which contradict the current best available evidence. Furthermore, when more rigorous and transparent methods are used, guideline development can take two years or more. As a result, these guidelines can require considerable financial and human resources and the timeline may result in a delay in getting pertinent information to health care providers, thereby having a potential impact on clinical practice [4, 9, 10].
Systematic reviews take a different approach by using explicit search strategies and inclusion criteria to identify and judge evidence of the effectiveness of clinical interventions [11]. The relative comprehensiveness of these reviews is a key strength. Null or even negative findings are often included, whereas in other approaches such research has been largely ignored in the accumulation of supportive findings [12]. However, health professionals, physicians, and policy makers infrequently refer to these reviews in decision-making [13–15]. So, despite the increasing summaries of the evidence in systematic reviews and clinical practice guidelines, as many as 30 to 45 per cent of patients fail to receive evidence-based treatment, leading to less than optimal outcomes [16–18]. This is due to a number of limitations in these reviews [11]; perhaps the most significant one being the lack of a well-defined, transparent process for including the input of relevant evidence users that have practice-based knowledge. This limitation results in the omission of input from those intended to implement and benefit from the recommendations.
Input from practicing health professionals can make recommendations more practical and clinically- relevant. This can be accomplished through an approach that integrates scientific evidence with the knowledge of patients' values and preferences as well as the implications for treatment planning and decision-making in clinical practice.
A New Approach
The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach [19] incorporates the strengths of previous approaches and accounts for their limitations. It may be the more effective method for providing credible knowledge that is essential to CYMH.
GRADE employs a sequential assessment of 1) the quality of the evidence, followed by 2) an evaluation of the balance between benefits and harms and 3) a subsequent judgment about the strength of recommendations [20]. The separation of judgments on the quality of evidence and the strength of recommendations is a critical and defining feature of this method.
GRADE was developed based on a systematic review that identified the fundamental elements of an effective evidence grading system. These elements include: the rating of the quality of individual studies; the quantity in terms of magnitude of effect, number of studies and sample size; and the consistency of the findings between similar and different study designs [21].
The current approaches to synthesizing evidence-based practice research in the CYMH field should be more appropriately described as clinical treatment efficacy research [22]. In the past, these approaches have relied on a one-to-one fit between the research on effective practices and the implementation of these practices in routine care. One clear solution to this problem would see research conducted that attends to service delivery issues at the outset. However, until that sort of evidence becomes available existing evidence-based practice research would be left collecting dust on academic shelves. Instead, the GRADE method may provide a method that can make use of the wealth of existing research. The GRADE approach attempts to merge research-based knowledge of efficacy with practice-based knowledge of effectiveness through the input of the expert panel.
The process begins with the selection of a focused question that can lead to action and the establishment of an expert panel with knowledge related to the question. Ideally, and for the sake of efficiency, relevant published and unpublished scientific evidence relating to the question is gathered from an existing systematic review. A list of potentially important outcomes is then created which panel members must independently rate according to importance ranging from unimportant to critical. The improvement of outcomes with high patient importance (i.e., considered critical) should lead to stronger recommendations [23].
Following the rating of the importance of outcomes, the quality of the evidence is then rated. A number of factors can increase or decrease the quality rating and this should reflect the confidence in the studies' estimates of benefits and harms.
Finally, the evidence, its quality rating and the outcome information is synthesized succinctly and transparently in evidence profiles. These profiles guide the panel in deciding on the strength of recommendation according to the tradeoffs between benefits and harms of the various alternatives.
Based on conceptual arguments and past experience with the process, many developers and users of GRADE believe that this process can be applied across all forms of intervention [24]. Others have argued that a single system cannot adequately address evidence across a wide range of contexts without being too complex to be useful [25, 26]. Until now, GRADE has been used primarily to address topics in the field of medicine [e.g., [23, 27, 28]]. CYMH is an area in need of a more efficient and effective approach to producing credible knowledge. Furthermore, it is an area in which the potential of the GRADE method has not been evaluated.
Current Study
Against this backdrop, the Provincial Centre of Excellence for Child and Youth Mental Health at the Children's Hospital of Eastern Ontario completed a pilot project to investigate the use of a streamlined version of the GRADE approach. Autism intervention research was identified as a suitable and interesting area in which to pilot test this methodology. Specifically, this streamlined method was used to determine current credible knowledge concerning the effects of parent-implemented, early intervention for children with autism. Implementing a methodology such as GRADE has considerable implications for improving efficiency, communication, and clinical decision-making. GRADE may facilitate the progression of the CYMH field toward greater use of evidence-based practice by service providers. This pilot study is the first step in evaluating the feasibility of this methodology as applied to CYMH research and determining whether any modifications are necessary in its application to this field. The findings related to autism as well as the strengths and limitations of the approach will be discussed.
Autism Research
The history of controversy surrounding autism makes this area suitable for this pilot project and is certainly illustrative of the importance of an established methodology for determining credible knowledge in the field of CYMH. Autism research has, for example, seen the persistence of a dispute surrounding evidence concerning causal linkages associated with the measles-mumps-rubella (MMR) vaccine or the vaccine preservative thimerosal. Notwithstanding, there has been no evidence to support an association between an MMR vaccine and autism, nor is there a plausible biological mechanism [as reviewed by 29]. Valuable time and resources were used to conduct as many as twenty studies to definitively respond to a study [30], whose arguments had many flaws that need not be explored here [29, 31, 32].
There has also been intense debate surrounding the evidence base and suitability of the variety of educational strategies and interventions that have been developed for individuals with autism [33–37]. For instance, controversy exists regarding the necessary intensity of intervention to achieve positive outcomes, the use of certain outcomes and the efficacy of one program compared to another [38]. Some approaches have also been popular with parents, but many professionals have not embraced them enthusiastically, again leading to considerable controversy [39]. While debate is clearly healthy in furthering the scientific research in relation to these practices, the level of uncertainty in this area has resulted in professionals, parents, and policy makers receiving confusing and sometimes unreliable information about the relative effectiveness of interventions [6].
Narrative reviews of interventions in this area have sought to address this problem by identifying evidence-based treatments. Unfortunately, many of these reviews have not taken a systematic approach, limiting their comprehensiveness and validity [33, 35, 40–44]. Alternatively, systematic reviews have included a wider range of evidence but most have also been found to suffer from methodological weaknesses that make them susceptible to bias [45]. Other reports have taken a systematic approach to the scientific evidence along with expert clinical opinion and took many years to complete. For instance, the National Autism Centre [46] recently completed a review of effective treatments for autism. This review took four years and required substantial resources. Still, others using this approach could not provide precise recommendations regarding specific interventions.
The National Research Council [36] and New York State Department of Health [47] each conducted a systematic review and developed clinical practice guidelines intended to identify effective practices. These reports did not recommend specific interventions. The reports cited a lack of well-controlled studies and an inability to compare interventions between studies. This was stated to be a result of varying descriptions of interventions, participants, treatment fidelity and other methodological considerations. Instead, both reports identified several key features of successful approaches to the education of children with autism, including early intervention after diagnosis, intensive instructional programming, as well as parental involvement in educating their child.
Parent-implemented interventions have been a particular focus in recent reviews, as increased parental skills allow for continual opportunities for children's learning across a range of situations [48, 49]. Even in using a more narrowly defined scope of interventions for children with autism, the authors of these reviews were unable to provide recommendations as to specific interventions and offered few implications for practice. This study aims to show that a modified GRADE approach can improve on these attempts at developing recommendations when applied to a CYMH topic.
Methods
The Grade Approach
The process used for this pilot project was adapted from that described by the GRADE working group and is more fully described elsewhere [19, 24, 50–53]. The approach represents an explicit assessment of the quality of evidence, the balance between benefits and risks and the strength of recommendations. Separating the judgments of the quality of the evidence and the strength of recommendations is a critical and defining aspect of GRADE.
Adjustments to the process were made throughout the course of the pilot project to adapt the method in answering a CYMH question. Any changes to the process are noted. The question selected for this pilot project was: What is the current credible knowledge concerning the effects of parent-implemented, early intervention for their children with autism?
- (1)
Families had to have at least one child who received a formal diagnosis of autism.
- (2)
Parent-implemented: At least one parent was formally trained to deliver an intervention for their child with autism, which was different in some way from any intervention(s) delivered by service providers.
- (3)
Early intervention: At the time of study initiation, within each study group at least one recipient of a parent-implemented intervention had been under eight years of age.
- (4)
Research designs: prospective and controlled.
- (5)
All publication dates considered.
- (6)
Effects: any outcomes (e.g., child, interaction, parent, system-related), which includes benefits and harms.
Based on these criteria, an independent information specialist searched the following databases: ERIC, The Cochrane Library, MEDLINE, EMBASE, PsycInfo, CINAHL, and Dissertation Abstracts International. Other sources of information were examined including the bibliographies of systematic and narrative reviews and reference lists of key articles identified through the search strategy. Screening by three independent screeners of the results yielded 11 primary studies to be included [55–65]. This deviates from the original GRADE method in that the evidence-base is typically independently established by an existing systematic review in a GRADE review. Given the issues with the existing systematic reviews on this topic (noted above), the procedure was modified in the current study to have an independent information specialist search for individual studies. This subsequently modifies the procedure for appraising the evidence, as described below.
A panel, which included three clinical and methodological experts in autism research, was then formed. In this pilot project, one role of the panel was to serve as a proxy for end users in determining the values and preferences of those end users in rating the importance of outcomes. The inclusion of additional members was beyond the resources of this work.
Outcome Ratings
In addition to using the panel's experience, an extensive literature search was performed to identify preference estimates from population-based studies in order to gain additional, evidence-based, perspectives in rating the importance of outcomes. These studies are intended to provide information that reflects the values, beliefs and preferences of those affected by autism. The panel could use these studies in rating the importance of outcomes. The panel then created a list of potentially important outcomes to rate, independent of the primary studies. Outcomes are rated on a scale from 1-9: a rating of 7-9 indicating that the outcome is critical for a decision or recommendation; a rating of 4-6 indicating it is important; and, a rating of 1-3 indicating it is not important.
Quality of Evidence
Quality of the evidence was then assessed using the GRADE four-category system (high, moderate, low and very low quality). The quality of the evidence reflects the extent to which one can be confident that the estimates of the benefits and harms of a researched intervention are accurate. Evidence based on randomized controlled trials begins with a high quality rating, observational studies with a low rating and any other evidence with a very low rating. However, studies may then be upgraded or downgraded. Five limitations--related to study quality, consistency, directness, precision, and reporting bias--may lead to downgrading. Upgrading the quality of the evidence may result from large effects, dose-response gradient and if all plausible confounding variables or biases would decrease the demonstrated effect, such as only sicker patients receiving an experimental intervention, yet still faring better. It is important to note that evidence that is downgraded for any reason cannot be upgraded. To ensure transparency and objectivity, the panel provided explicit reasons for downgrading and upgrading by including footnotes in the evidence profiles; this is a requirement of the GRADE system.
Factors assessed in critically appraising the evidence
Factors that might decrease quality of evidence | Factors that might increase quality of evidence |
|---|---|
• Lack of allocation concealment • Lack of blinding (particularly for subjective outcomes) • Failure to adhere to an intention to treat analysis • Stopping early for benefit • Selective outcome reporting • Poor matching of groups in nonrandomized studies or inappropriate comparison group (e.g., children with more serious behavioural problems placed in control group) • Absence of treatment fidelity assessment • Inadequate description of intervention and those delivering intervention • Insufficient follow-up period or lack of reporting on losses to follow-up | • Large magnitude of effect • Plausible confounding, which would reduce a demonstrated effect • Dose-response gradient (increased intensity of treatment leads to enhanced benefits) |
Strength of Recommendations
The strength of recommendations was then determined according to the tradeoffs between benefits and harms of the various alternatives. The strength of a recommendation reflects the extent to which one can be confident that the desirable effects of an intervention outweigh the undesirable effects. Although the level of confidence is a continuum, the GRADE system offers two levels of recommendations for or against treatments (i.e., strong and weak), according to the extent of difference between benefits and harms. The strength of a recommendation is also influenced by the values and preferences of end users, in that critical outcomes are the primary factors for establishing a recommendation. This allows the GRADE system to emphasize that weak recommendations in the face of high quality evidence and vice versa, can occur. This is because other factors, beyond the quality of the evidence, including the values and preferences of the relevant population and the importance of the outcomes, impact the strength of a recommendation [20]. Given that the heterogeneity of the evidence did not allow for a synthesis of the evidence, the labels strong and weak were applied to individual interventions, as opposed to the body of the evidence as in the original GRADE method.
Recommendations included details about the population and intervention. This should aid in the interpretation of the strength of the recommendations and help clinical decision-makers deal with individual patients. Given that some of the judgments leading to the recommendations are subjective, the decisions made are explicit and transparent in the remarks that go along with the recommendations. Specific suggestions for new studies to be undertaken were also noted (see example in Additional file 1).
Results
Descriptions of the studied interventions
Study | Intervention | Sample Size |
|---|---|---|
Aldred et al. (2005) | Therapist delivered, manual-based program for training and educating parents in adapted communication. | Treatment = 14 Control = 14 |
Drew et al. (2003) | Speech and language therapist delivered program aimed to train parents to develop their child's joint attention skills and joint action routines. | Treatment = 12 Control = 12 |
Howlin & Rutter (1987) | A behavioural, linguistic and social training program used to teach parents language modification techniques, management training and behavioural procedures. | Treatment = 16 Control = 16 |
Koegel et al. (1996) | Pivotal Response Training, a naturalistic paradigm emphasizing motivational procedures and responsivity to multiple cues to improve parent-child interaction. | Treatment = 7 Control = 10 |
McConachie et al. (2005) | "More Than Words" course aimed at improving parents' understanding of autism and facilitating communication. | Treatment = 26 Control = 25 |
Neef (1997) | A pyramidal model of parent training by peers to instruct parents in making use of skill training opportunities during daily routines. | Treatment = 14 Control = 12 |
Ozonoff & Cathcart (1998) | Individualized TEACCH-based home program focusing on cognitive, academic, and prevocational skills. | Treatment = 11 Control = 11 |
Rickards et al. (2000) | Specialist pre-school teacher delivered, home-based program using discussion, direct skills training and adaptation of the home environment to the child. | Treatment = 18 Control = 21 |
Schreibman et al. (1995) | Pivotal Response Training, a naturalistic paradigm emphasizing motivational procedures and responsivity to multiple cues to improve parent-child interaction. | Treatment = 10 Control = 7 |
Tonge et al. (2001) | Special educator or psychologist delivered, manual-based, parent education and behaviour management intervention. | Treatment = 35 Control = 35 |
Wang (2008) | The author delivered a program aimed at improving the interactive skills of parents using applied behaviour analysis and naturalistic teaching principles | Treatment = 14 Control = 11 |
Quality Ratings. Frequency count of the quality of the evidence as rated by the panel
Summary of the Evidence
Study | Quality Rating | Factors affecting qualityStrengths (+)/Weaknesses(-) | Recommendation* | Best Estimates |
|---|---|---|---|---|
Aldred et al. (2005) | High | + RCT + Independent randomization + Allocation Concealment/blinding + Clear outcome reporting + Low risk of bias + Valid outcome measures | Weak + | Improved symptom severity, social communication skills and parent perception of child's communication skills |
Drew et al. (2002) | Moderate | + RCT + Appropriate randomization + Clear outcome reporting + Low risk of bias +Valid outcome measures - poorly matched groups (gender) - lack of blinding | Weak + | Improved Communication Skills |
Howlin & Rutter (1987) | Low | + Appropriately matched control group + Allocation Concealment/blinding + Clear outcome reporting + Low risk of bias + Valid outcome measures - Quasi-Experimental design | Weak + | Improved mother/child speech and interaction |
Koegel et al. (1996) | High | + RCT + Appropriate randomization + Allocation Concealment/blinding + Clear outcome reporting + Low risk of bias + Valid outcome measures | Weak + | Improved parents' global style of interaction with their child |
McConachie et al. (2005) | Low | + RCT + Appropriately matched control group + Allocation Concealment/blinding + Clear outcome reporting + Low risk of bias + Valid outcome measures - Quasi-Experimental design | Weak + | Improved children's vocabulary and parents' use of facilitative strategies |
Neef (1997) | Very Low | + Low risk of bias - Quasi-Experimental design - Inappropriately matched control group - Allocation Concealment/blinding - Incomplete outcome reporting - Undetermined validity of outcome measures | Weak + | Improved parent teaching skills and child skills development |
Ozonoff & Cathcart (1998) | Very Low | + Appropriately matched control group + Low risk of bias + Valid outcome measures - Quasi-Experimental design - lack of blinding - Incomplete outcome reporting | Weak + | Improved overall development, imitation, fine and gross motor skills and nonverbal conceptual skills |
Rickards et al. (2000) | High | + RCT + Appropriate randomization + Allocation Concealment/blinding + Clear outcome reporting + Low risk of bias + Valid outcome measures | Weak + | Improved children's IQ and behaviour problems |
Schreibman et al. (1995) | Moderate | + RCT + Appropriate randomization + Allocation Concealment/blinding + Low risk of bias + Valid outcome measures - Incomplete outcome reporting | Weak + | Improved parental affect while interacting with their child |
Tonge et al. (2001) | High | + RCT + Independent randomization + Allocation Concealment/blinding + Clear outcome reporting + Low risk of bias + Valid outcome measures | Weak + | Improved self-reported parent mental health |
Wang (2008) | High | + RCT + Appropriate randomization + Allocation Concealment/blinding + Clear outcome reporting + Low risk of bias + Valid outcome measures | Weak + | Improved the quality of parent's interactions with their child |
Discussion
This pilot project provided evidence of the potential utility and feasibility of the streamlined GRADE process in the field of CYMH. Being the first implementation of this method in this area, it provides a unique contribution, though modifications to the process were required and certain limitations will have to be addressed in future studies in order for the GRADE method to achieve its potential in this field. A synthesis of the evidence relating to parent-implemented interventions for their child with autism was completed, along with an objective evaluation of this evidence. Following the evaluation of the evidence, recommendations were developed in detail to facilitate interpretation by decision makers for whom the recommendations are intended. This reporting should ensure that decision makers have all of the information required to judge the quality of a recommendation, determine its applicability, and adapt it, if necessary. Overall, it appeared that early, parent-implemented interventions for autism result in positive effects that outweigh any undesirable effects (though publication bias was not assessed and could have impacted the results of this study). However, as opposed to an overall recommendation for these interventions as a whole, specific recommendations had to be made based on the question of whether a particular intervention's benefits outweighed its harms, a necessary departure from the original GRADE method.
Limitations
The first limitation may also be considered one of the strengths of this study. The GRADE approach was being implemented in a new field of research. Yet, as a result, there were no previous detailed applications or standards to draw from in the literature and modifications to the process were required. Some of the lessons learned from the current application of GRADE should inform future implementations of this approach in CYMH. This may also suggest that another panel may not reproduce the same results. In fact, others may disagree with aspects of our application to GRADE in this area of research and the subsequent grading of the quality of evidence and strength of recommendations. However, the transparency maintained by this process should allow others to understand the decisions made by this panel. For example, studies may be downgraded due to a lack of precision when there are very few patients or events, but there are no defined standards for these terms. Consequently, this panel followed the GRADE principle that a study may require greater precision in support of a recommendation when there is a close balance between the advantages and disadvantages of an intervention [66]. Given the lack of harmful events in the studied interventions, this was not the case here. Furthermore, the current panel consistently recommended the use of interventions for which there was no evidence of harm and some demonstrated benefit on one of the important outcomes. This is a particular judgment on the part of the current panel that has impacted the recommendations. Taken together, future iterations of this process may use panels with different values and judgments that would raise the threshold for demonstrated benefit as well as decide on a minimum sample size, which, if not met, would be downgraded for a lack of precision. Regardless of these decisions, they should continue to be made explicit and applied consistently.
The lack of variability in the composition of the panel may also be considered a limitation in the present application of GRADE. This panel was composed of clinical and methodological experts in the area of autism research. This may have reduced the variability of opinion in establishing the values and judgments and created a conflict of interest in reviewing and appraising evidence in their own field. This became a more prominent factor when the panel was required to appraise individual studies and provide individual recommendations for each intervention as part of the modified GRADE approach. However, these experts also had unique insight into the values and preferences of end-users, for which they were used as a proxy, as well as previous experience in this area of research that facilitated the timely completion of this pilot project. Moreover, these modifications to the process were necessary due to the evidence base and, again, all decisions were made transparent.
Examining the cause of the somewhat unsatisfactory finding, it is seemingly a product of the evidence and not due to any shortcomings in the GRADE method itself. The results of the studies could not be combined or compared due to the high degree of heterogeneity in the evidence base. Two interventions were rarely comparable in terms of the content of the intervention, outcomes measured, specific population studied or those implementing the interventions. Therefore, it was not always possible to construct a consistent account of the effects of an intervention. This is a common finding in mental health, and specifically autism research [33, 36, 45, 48, 49, 67, 68].
Research in this area has been identified as containing numerous gaps and being too heterogeneous to allow for direct comparisons of interventions [36, 68]. A number of directions need to be pursued in order to allow for these comparisons. Specific areas to be pursued include: studying a wider range of outcomes; lengthening the time to follow-up assessments; assessing moderators and mediators of treatment outcomes; and, testing varied approaches to treatment delivery or even replicating past research [12, 68, 69]. An in-depth search of studies addressing these factors in relation to autism interventions was conducted for this pilot project and returned few results [e.g., [70, 71]]. Studies addressing these factors would allow for mental healthcare providers and users to make the most informed decisions possible for each individual. Furthermore, the use of common, agreed upon outcome measures including child, parent, family and interactions variables, would facilitate the comparison of studies. Moreover, this research would produce more valuable information by using minimal standards in design and reporting on the interventions and participants. To this end, guidelines could be followed to improve the reporting of how studies were planned, conducted and analyzed [e.g., [72, 73]].
The current approaches to synthesizing and evaluating this evidence cannot overcome the limitations in the research. An assumption of these approaches is that the development of the evidence-base has considered the fit between the intervention and its use in clinical practice. In reality, this fit has rarely been taken into account. Moreover, many make the assumption that all randomized control trials are of a high quality since they are generally considered the gold standard in research. The difficulty with this assumption is that there are situations where randomized control trials are impractical or unethical [31, 74]. This is particularly true in some areas of CYMH research, where for a variety of logical reasons these designs are infrequently applied, such as autism [6]
In effect, the evidence-base in CYMH is primarily comprised of efficacy studies for which development is based on theory, methods and models to evaluate treatments that do not correspond to the demands of clinic and community-based care [75]. As a result, clinical research, and notably randomized controlled trials, does not provide practitioners with definitive answers for dealing with the individual-at-hand [76]. This may account for the discrepancy between the scientific support of interventions and the lack of their use in these settings [22, 77–79].
Strengths
The GRADE approach explicitly addresses the fit between the research and its use in practice through the functions of the panel, a clear strength. While research should still attend to the factors noted above, GRADE can focus on the clinical relevance of the considerable body of evidence already in existence. Another strength of this approach is that it does not reduce the evidence underlying an intervention to a simple Yes/No conclusion, but it includes information about key variables from the studies in the recommendations. This should aid in the interpretation of the strength of the recommendations and may affect clinical decision-making due to diverse values and preferences of individuals implementing the interventions. The short amount of time required to prepare the recommendations was also a strength of this process. It took four months to complete the process, whereas guideline development and systematic reviews can take up to two years or more [4, 9, 10].
This approach also benefits from an explicit methodology, transparency, consistency in judgments about the quality of evidence, and linking evidence to recommendations. Though evidence is linked to recommendations, the judgments about the quality of the evidence and the balance of benefits and harms leading to the recommendations are made separately. Not all grading systems separate decisions regarding the quality of the evidence from the strength of recommendations [19]. Those that fail to do so may create confusion. High quality evidence does not necessarily imply strong recommendations, and strong recommendations can arise from low quality evidence. GRADE also compels the panel to consider all relevant outcomes and to adhere to a consistent and comprehensive process in evaluating the evidence. This is a key advantage of this process, especially when considering that guideline developers have been found to lack objectivity due to excessive influence from industry and experts participating in the guideline panels, and to rarely abide by guidelines for the preparation of guidelines [8, 80]
It should also be noted that GRADE has now been adopted by a number of organizations including the Cochrane Collaboration and the World Health Organization [28, 53]. The increased usage of this methodology and presentation of its results, typically in summary of findings tables, may result in more rapid, clear and consistent translation of evidence. Furthermore, the results of one implementation of GRADE could be used in subsequent reviews quickly and transparently. For example, evidence from a GRADE review of interventions for other disorders on the autism spectrum could potentially be added to this review, while being downgraded for a lack of directness.
These strengths contribute to our experience that GRADE is a feasible approach in CYMH. Its methodology promotes reconciliation between clinicians', methodologists', administrators' and patients' perspectives along with the scientific evidence. Eliminating this traditional separation, seen as a significant limitation in clinical practice guidelines and systematic reviews, will help the evolution of the CYMH field toward the use of evidence-based practice by service providers.
Future Directions
Future implementations of the GRADE approach should include a broader representation of stakeholders in the panel than was used in this pilot project. Other members of the panel could include those of varying levels of service administration such as policy makers, parents, and other practitioners. They would provide more varied perspectives and reduce the gap between the evidence and practice. For example, considering practical barriers to implementation, such as funding restrictions or gaps in clinical expertise could be especially helpful in making the most useful recommendations. In addition, including stakeholders of varying levels would help to drive the knowledge exchange of the findings. Knowledge exchange is an area in and of itself that should be explored to provide guidance on the best procedure for translating results in order to produce meaningful change when using the GRADE method [81]. Other considerations in future uses of GRADE include having the details of the manualization of the intervention as part of the evaluation of methodological quality. Manuals serve the important objective of making the recommended treatments available for clinical training and practice. The specification of sample characteristics in the study could also be evaluated in the GRADE process. These details reflect the recognition that specific treatments may be efficacious only within a limited range of individuals depending on such factors as their age, sex, problem severity, socioeconomic status, and ethnicity [82].
Conclusions
In conclusion, it was determined that it is feasible to develop more timely recommendations in CYMH, specifically early parent-implemented autism interventions, using a modified version of the systematic and transparent GRADE approach. A definitive answer as to which is the best intervention was not produced. However, the GRADE approach appeared capable of providing a specific answer had the evidence been less heterogeneous and directly comparable. Due to the heterogeneity of the evidence a modified GRADE approach was presently evaluated. As research in this area increases and fills in the gaps identified above, it is important to have a method to evaluate the benefits and risks of new interventions, in an efficient manner, as they arise. The GRADE process meets this need and establishes credible knowledge that leads directly to recommendations for effective organization and delivery of CYMH care. Most importantly, GRADE and the recommendations developed from it, will help CYMH service providers and users to make treatment decisions that are informed by the scientific and practice-based evidence.
Notes
Declarations
Acknowledgements
The Provincial Centre of Excellence for Child and Youth Mental Health at CHEO is funded by the Ontario Ministry of Children and Youth Services. The views expressed in this paper are those of the author and do not necessarily represent the views of the Ministry of Child and Youth Services. We would like to thank Dr. Ian Manion for his review and assistance in preparation of the manuscript and Dr. Howard Schachter for his work in developing this project. Finally, thanks to the GRADE Working Group for free access to their software.
Authors’ Affiliations
References
- Barwick MA, Boydell KM, Stasiulis E, Ferguson HB, Blasé K, Fixsen D: Research utilization among children's mental health providers. Implement Sci. 2008, 3: 19-10.1186/1748-5908-3-19.View ArticlePubMedPubMed CentralGoogle Scholar
- Harbour R, Miller J: A new system for grading recommendations in evidence based guidelines. BMJ. 2001, 323: 334-6. 10.1136/bmj.323.7308.334.View ArticlePubMedPubMed CentralGoogle Scholar
- McAlister FA, van Diepen S, Padwal RS, Johnson JA, Majumdar SR: How Evidence-Based Are the Recommendations in Evidence-Based Guidelines?. PLoS Med. 2007, 4: e250-10.1371/journal.pmed.0040250.View ArticlePubMedPubMed CentralGoogle Scholar
- National Institute for Health and Clinical Excellence: The Guideline Manual. 2007, LondonGoogle Scholar
- Scottish Intercollegiate Guidelines Network: A Guideline Developer's Handbook. 2001, EdinburghGoogle Scholar
- Simpson RL: Evidence-based practices and students with autism spectrum disorders. Focus Autism Other Dev Disabl. 2005, 20: 140-149. 10.1177/10883576050200030201.View ArticleGoogle Scholar
- U.S Department of Education: Identifying and implementing educational practices supported by rigorous evidence: A user friendly guide. 2003, Washington, DCGoogle Scholar
- Shaneyfelt TM, Mayo-Smith MF, Rothwangl J: Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA. 1999, 281: 1900-1905. 10.1001/jama.281.20.1900.View ArticlePubMedGoogle Scholar
- Kunz R: What's in the Black Box?. Chest. 2006, 129: 7-10. 10.1378/chest.129.1.7.View ArticlePubMedGoogle Scholar
- Raine R, Sanderson C, Black N: Developing clinical guidelines: A challenge to current methods. BMJ. 2005, 331: 631-633. 10.1136/bmj.331.7517.631.View ArticlePubMedPubMed CentralGoogle Scholar
- Grimshaw JM, Hutchinson A: Clinical practice guidelines-do they enhance value for money in health care?. Br Med Bull. 1996, 51: 927-940.Google Scholar
- Weisz JR, Hawley KM: Finding, evaluating, refining and applying empirically supported treatments for children and adolescents. J Clin Child Psychol. 1998, 27: 206-216. 10.1207/s15374424jccp2702_7.View ArticlePubMedGoogle Scholar
- Dawes M, Sampson U: Knowledge management in clinical practice: a systematic review of information seeking behavior in physicians. Int J Med Inform. 2003, 71: 9-15. 10.1016/S1386-5056(03)00023-6.View ArticlePubMedGoogle Scholar
- Lavis JN, Ross SE, Hurley JE, Hohenadel JM, Stoddart GL, Woodward CA, Abelson J: Examining the role of health services research in public policymaking. Milbank Q. 2002, 80: 125-54. 10.1111/1468-0009.00005.View ArticlePubMedPubMed CentralGoogle Scholar
- Oxman AD, Lavis JN, Fretheim A: Use of evidence in WHO recommdendations. Lancet. 2007, 369: 1883-1889. 10.1016/S0140-6736(07)60675-8.View ArticlePubMedGoogle Scholar
- McGlyn EA, Asch SM, Adams J, Keesey BA, Hicks J, DeCristofaro A, Kerr EA: The quality of health care delivered to adults in the United States. N Engl J Med. 2003, 348: 2635-2645. 10.1056/NEJMsa022615.View ArticleGoogle Scholar
- Leatherman S, Sutherland K: Quality of care in the NHS of England. BMJ. 2004, 328: E288-290. 10.1136/bmj.328.7445.E288.View ArticlePubMedGoogle Scholar
- Seddon M, Marshall M, Campbell S, Roland MO: Systematic reviews of studies of quality of clinical care in general practice in the UK, Austrialia and New Zealand. Qual Health Care. 2001, 103: 152-158. 10.1136/qhc.0100152.View ArticleGoogle Scholar
- Atkins DD, Best D, Briss PA, Eccles M, Falck Ytter Y, Flottorp S, Guyatt GH, Harbour RT, Haugh MC, Henry D, Hill S, Jaeschke R, Leng G, Liberati A, Magrini N, Mason J, Middleton P, Mrukowicz J, O'Connell D, Oxman AD, Phillips B, Schunemann HJ, Edejer TT, Varonen H, Vist GE, Williams JW, Zaza S: Grading quality of evidence and strength of recommendations. BMJ. 2004, 328: 1490-1498. 10.1136/bmj.328.7454.1490.View ArticlePubMedGoogle Scholar
- Swiglo BA, Murad MH, Schunemann HJ, Kunz R, Vigersky RA, Guyatt GH, Montori VM: A Case for Clarity, Consistency, and Helpfulness: State-of-the-Art Clinical Practice Guidelines in Endocrinology Using the Grading of Recommendations, Assessment, Development, and Evaluation System. J Clin Endocrino Metab. 2008, 93: 666-673. 10.1210/jc.2007-1907.View ArticleGoogle Scholar
- West S, King V, Carey TS, Lohr KN, McKoy N, Sutton SF, Lux L, Systems to Rate the Strength of Scientific Evidence: In Evidence Report/Technology Assessment Volume AHRQ Publication No. 02-E016. Rockville, MD: Agency for Healthcare Research and Quality. 2002, 1-199.Google Scholar
- Weisz JR, Weiss B, Donenberg GR: The lab versus the clinic: effects of child and adolescent psychotherapy. Am Psychol. 1992, 47: 1578-1585. 10.1037/0003-066X.47.12.1578.View ArticlePubMedGoogle Scholar
- Guyatt G, Schünemann HJ, Cook D, Jaeschke R, Pauker S, Bucher H: Grades of Recommendation for Antithrombic Agents. Chest. 2004, 119 (Suppl 1): 3-7.Google Scholar
- Schünemann HJ, Fretheim A, Oxman AD: Improving the use of research evidence in guideline development: 1. Review of guidelines for guidelines. Health Res Pol Sys. 2006, 4: 13-21.View ArticleGoogle Scholar
- Roddy E, Zhang W, Doherty M, Arden NK, Barlow J, Birrell F, Carr A, Chakravarty K, Dickson J, Hay E, Hosie G, Hurley M, Jordan KM, McCarthy C, McMurdo M, Mockett S, O'Reilly S, Peat G, Pendleton A, Richards S: Evidence-based clinical guidelines: A new system to better determine true strength of recommendation. J Eval Clin Pract. 2004, 12: 347-352. 10.1111/j.1365-2753.2006.00629.x.View ArticleGoogle Scholar
- Baumer JH: Guideline Review: Autism spectrum disorders, SIGN. Arch Dis Child Educ Pract Ed. 2008, 93: 163-166. 10.1136/adc.2008.145938.View ArticlePubMedGoogle Scholar
- Brozek JL, Baena-Cagnani CE, Bonini S, Canonica GW, Rasi G, van Wijk RG, Zuberbier T, Guyatt G, Bousquet J, Schunemann HJ: Methodology for development of the Allergic Rhinitis and its Impact on Asthma Guideline 2008 update. Allergy. 2008, 63: 38-46. 10.1111/j.1398-9995.2007.01560.x.View ArticlePubMedGoogle Scholar
- De Palma R, Liberati A, Ciccone G, Bandieri E, Belfiglio M, Ceccarelli M, Leoni M, Longo G, Magrini N, Marangolo M, Roila F: Developing Clinical Recommendations for Breast, Colorectal, and Lung Cancer Adjuvant Treatments Using the GRADE System: A Study From the Programma Ricerca e Innovazione Emilia Romagna Oncology Research Group. J Clin Oncol. 2008, 26: 1033-1039. 10.1200/JCO.2007.12.1608.View ArticlePubMedGoogle Scholar
- Gerber JS, Offit PA: Vaccines and Autism: A Tale of Shifting Hypotheses. Clin Infect Dis. 2009, 48: 456-461. 10.1086/596476.View ArticlePubMedPubMed CentralGoogle Scholar
- Wakefield AJ, Murch SH, Anthony A, Linnell J, Casson DM, Malik M, Dhillon AP, Thomson MA, Harvey P, Valentine A, Davies SE, Walker-Smith JA: Ileal-lymphold-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. Lancet. 1998, 351: 637-641. 10.1016/S0140-6736(97)11096-0.View ArticlePubMedGoogle Scholar
- Davidson KW, Trudeau KJ, Ockene JK, Orleans CT, Kaplan RM: A Primer on Current Evidence-Based Review Systems and Their Implications for Behavioral Medicine. Ann Behav Med. 2004, 28: 226-238.Google Scholar
- Offit PA, Quarles J, Gerber MA, Hackett CH, Marcuse EK, Kollman RT, Gellin BG, Landry S: Addressing parents' concerns: do multiple vaccines overwhelm or weaken the infant's immune system? Pediatrics. 2002, 109: 124-9.Google Scholar
- Dawson G, Osterling J: Early intervention in autism. The Effectiveness of Early Intervention. Edited by: Guralnick MJ. 1997, Baltimore, MD: Paul H Brookes, 307-324.Google Scholar
- Eikeseth S: Outcome of comprehensive psycho-educational interventions for young children with autism. Res Dev Disabil. 2009, 30: 158-178. 10.1016/j.ridd.2008.02.003.View ArticlePubMedGoogle Scholar
- Gresham FM, Beebe-Frankenberger ME, MacMillan DL: A selective review of treatments for children with autism: Description and methodological considerations. School Psychol Rev. 1999, 28: 559-575.Google Scholar
- National Research Council: Educating Children with Autism. 2001, Washington, DC: National Academy PressGoogle Scholar
- Smith T: Outcome of early intervention for children with autism. Clin Psycho: Res-Pract. 1999, 6: 33-49. 10.1093/clipsy/6.1.33.Google Scholar
- Ospina MB, Seida JK, Clark B, Karkhaneh M, Hartling L, Tjosvold L, Vandermeer B, Smith V: Behavioural and Developmental Interventions for Autism Spectrum Disorder: A Clinical Systematic Review. Plos One. 2008, 3: e3755-10.1371/journal.pone.0003755.View ArticlePubMedPubMed CentralGoogle Scholar
- McWilliam RA: Controversial Practices: The Need for a Reacculturation of Early Intervention Fields. Top Early Child Spec. 1999, 19: 177-188. 10.1177/027112149901900310.View ArticleGoogle Scholar
- Connor M: A Review of Behavioural Early Intervention Programmes for Children with Autism. Educ Psychol Pract. 1998, 14: 109-117. 10.1080/0266736980140206.View ArticleGoogle Scholar
- Green G: Early behavioral intervention for autism. What does research tell us? Behavioral intervention for young children with autism. A manual for parents and professionals. Edited by: Maurice C, Green G, Luce S. Austin TX. 1996, Pro-Ed, 29-44.Google Scholar
- Harris SL: Autism and Pervasive Developmental Disorders. 1998, Cambridge: Cambridge University PressGoogle Scholar
- Perry A, Condillac RA: Evidence-based practices for children and adolescents with Autism Spectrum Disorders: Reviews of the literature and practice guide. 2003, Toronto, ON: Children's Mental Health OntarioGoogle Scholar
- Probst P: Parent Training in a rehabilitation setting for autistic children: Concepts and outcomes. Z Klin Psychol Psychiatr Psychother. 2001, 49: 1-32.Google Scholar
- Seida JK, Ospina MB, Karkhaneh M, Hartling L, Smith V, Clark B: Systematic reviews of psychosocial interventions for autism: an umbrella review. Dev Med Child Neurol. 2009, 51: 95-104. 10.1111/j.1469-8749.2008.03211.x.View ArticlePubMedGoogle Scholar
- National Autism Centre: National Standards Report. 2009, Randolph, MassGoogle Scholar
- New York State Department of Health Early Intervention Program: Clinical practice guideline: The guideline technical report autism/pervasive developmental disorders. 1999, New York: AuthorGoogle Scholar
- Diggle T, McConachie H, Randle V: Parent implemented early intervention for young children with autism spectrum disorder. Cochrane Database Syst Rev. 2003Google Scholar
- McConachie H, Diggle T: Parent implemented early intervention for young children with autism spectrum disorder: a systematic review. J Eval Clin Pract. 2007, 13: 120-129. 10.1111/j.1365-2753.2006.00674.x.View ArticlePubMedGoogle Scholar
- Atkins D, Eccles M, Flottorp S, Guyatt GH, Henry D, Hill S, Liberati A, O'Connell D, Oxman AD, Phillips B, Schünemann HJ, Edejer TT, Vist , Williams JW, the GRADE Working Group: Systems for grading the quality of evidence and the strength of recommendations I: Critical appraisal of existing approaches. BMC Health Serv Res. 2004, 4: 38-10.1186/1472-6963-4-38.View ArticlePubMedPubMed CentralGoogle Scholar
- Atkins D, Briss PA, Eccles M, Flottorp S, Guyatt GH, Harbour RT, Hill S, Jaeschke R, Liberati A, Mangrini N, Mason J, O'Connell D, Oxman AD, Phillips B, Schünemann HJ, Edejer TT, Vist GE, Willams JW, the GRADE Working Group: Systems for grading the quality of evidence and the strength of recommendations II: Pilot study of a new system. BMC Health Serv Res. 2005, 5: 25-10.1186/1472-6963-5-25.View ArticlePubMedPubMed CentralGoogle Scholar
- Guyatt G, Gutterman D, Baumann MH, Addrizzo-Harris D, Hylek EM, Phillips B, Raskob G, Lewis SZ, Schünemann HJ: Grading strength of recommendations and quality of evidence in clinical guidelines: report from an American College of Chest Physicians Task Force. Chest. 2006, 129: 174-181. 10.1378/chest.129.1.174.View ArticlePubMedGoogle Scholar
- Guyatt G, Vist G, Falck-Ytter Y, Kunz R, Magrini N, Schunemann H: An emerging consensus on grading recommendations?. Evid Based Med. 2006, 11: 2-4. 10.1136/ebm.11.1.2-a.View ArticlePubMedGoogle Scholar
- Wallen M, Stagnitti S: There was insufficient evidence to conclude whether parent-mediated early intervention was effective for children with autism. Aust Occup Therap J. 2006, 53: 137-139. 10.1111/j.1440-1630.2006.00557.x.View ArticleGoogle Scholar
- Aldred C, Green J, Adams C: A new social communication intervention for children with autism: a pilot randomised controlled treatment study suggesting effectiveness. J Child Psychol Psyc. 2004, 45: 1420-1430. 10.1111/j.1469-7610.2004.00338.x.View ArticleGoogle Scholar
- Drew A, Baird G, Baron-Cohen S, Cox A, Slonims V, Wheelwright S, Swettenham J, Berry B, Charman T: A pilot randomised controlled trial of a parent training intervention for preschool children with autism. Eur Child Adoles Psy. 2002, 11: 266-272. 10.1007/s00787-002-0299-6.View ArticleGoogle Scholar
- Howlin P, Rutter M, Berger M, Hemsley R, Hersov L, Yule W: Treatment of Autistic Children. 1987, New York: John Wiley & SonsGoogle Scholar
- Koegel RL, Bimbela A, Schreibman L: Collateral effects of parent training on family interactions. J Autism Dev Disord. 1996, 26: 347-359. 10.1007/BF02172479.View ArticlePubMedGoogle Scholar
- McConachie H, Randle V, Hammal D, Le Couteur A: A Controlled Trial of a Training Course for Parents of Children with Suspected Autism Spectrum Disorder. J Pediatr. 2005, 147: 335-340. 10.1016/j.jpeds.2005.03.056.View ArticlePubMedGoogle Scholar
- Neef NA: Pyramidal Parent Training by Peers. J Appl Behav Anal. 1995, 28: 333-337. 10.1901/jaba.1995.28-333.View ArticlePubMedPubMed CentralGoogle Scholar
- Ozonoff S, Cathcart K: Effectiveness of a home program intervention for young children with autism. J Autism Dev Disord. 1998, 28: 25-32. 10.1023/A:1026006818310.View ArticlePubMedGoogle Scholar
- Ricards AL, Walstab JE, Wright-Rossi A, Simpson J, Reddihough DS: A Randomized, Controlled Trial of a Home-Based Intervention Program for Children with Autism and Developmental Delay. J Dev Behav Pediatr. 2007, 28: 308-316. 10.1097/DBP.0b013e318032792e.View ArticleGoogle Scholar
- Schreibman L, Kaneko WM, Koegel RL: Positive affect of parents of autistic children: a comparison across two teaching techniques. Behav Ther. 1991, 22: 479-490. 10.1016/S0005-7894(05)80340-5.View ArticleGoogle Scholar
- Tonge B, Brereton A, Kiomall M, MacKinnon A, King N, Rinehart N: Effects on Parental Mental Health of an Education and Skills Training Program for Parents of Young Children with Autism: A Randomized Controlled Trial. J Am Acad Child Psy. 2006, 45: 561-569. 10.1097/01.chi.0000205701.48324.26.View ArticleGoogle Scholar
- Wang P: Effects of a Parent Training Program on the Interactive Skills of Parents of Children with Autism in China. J Pol Pract Intel Disabil. 2008, 5: 96-104. 10.1111/j.1741-1130.2008.00154.x.View ArticleGoogle Scholar
- Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ: Rating Quality of evidence and strength of recommendations. What is "quality of evidence" and why is it important to clinicians?. BMJ. 2008, 336: 995-998. 10.1136/bmj.39490.551019.BE.View ArticlePubMedPubMed CentralGoogle Scholar
- Canadian Paediatric Society: Early intervention for children with autism. Paed Child Health. 2004, 9: 267-270.Google Scholar
- Jensen PS, Weersing R, Hoagwood KE, Goldman E: What is the Evidence for Evidence-Based Treatments? A Hard Look at Our Soft Underbellly. Ment Health Serv Res. 2005, 7: 53-74. 10.1007/s11020-005-1965-3.View ArticlePubMedGoogle Scholar
- Weisz JR, Jensen AL: Child and adolescent psychotherapy in research and practice contexts: Review of the evidence and suggestions for improving the field. Eur Child Adolesc Psychiatry. 2001, 10: 12-18. 10.1007/s007870170003.View ArticleGoogle Scholar
- Lechago SA, Carr JE: Recommendations for Reporting Independent Variables in Outcome Studies of Early and Intensive Behavioral Intervention for Autism. Behav Modif. 2008, 32: 489-503. 10.1177/0145445507309034.View ArticlePubMedGoogle Scholar
- Rogers S, Vismara L: Evidence-Based Comprehensive Treatments for Early Autism. J Clin Child Adoles Psychol. 2008, 37: 8-38. 10.1080/15374410701817808.View ArticleGoogle Scholar
- von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP: Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ. 2007, 335: 806-808. 10.1136/bmj.39335.541782.AD.View ArticlePubMedPubMed CentralGoogle Scholar
- Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup DF: Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA. 1996, 276: 637-639. 10.1001/jama.276.8.637.View ArticlePubMedGoogle Scholar
- Cartwright N: Are RCTs the Gold Standard?. Biosocieties. 2007, 2: 11-20. 10.1017/S1745855207005029.View ArticleGoogle Scholar
- Hoagwood K, Burns BJ, Kiser L, Ringeisen H, Schoenwald SK: Evidence-based practice in child and adolescent mental health services. Psychiatr Serv. 2001, 52: 1179-1189. 10.1176/appi.ps.52.9.1179.View ArticlePubMedGoogle Scholar
- Feinstein AR, Horwitz RI: Problems in the 'evidence' of 'evidence-based medicine'. Am J Med. 1997, 103: 529-535. 10.1016/S0002-9343(97)00244-1.View ArticlePubMedGoogle Scholar
- Schoenwald SK, Hoagwood K: Effectiveness, transportability, and dissemination of interventions: What matters when?. Psychiatr Serv. 2001, 52: 1190-1197. 10.1176/appi.ps.52.9.1190.View ArticlePubMedGoogle Scholar
- Southam-Gerow MA: Some reasons that mental health treatments are not technologies: Toward treatment development and adaptation outside labs. Clin Psychol Sci Pract. 2004, 11: 186-189. 10.1093/clipsy/bph070.View ArticleGoogle Scholar
- Weisz JR: Lab-clinic differences and what we can do about them: I. The clinic-based treatment development model. Clin Child Psychol News. 2000, 15: 1-3.Google Scholar
- Grilli R, Magrini N, Penna A, Mura G, Liberati A: Practice guidelines developed by specialty societies: The need for a critical appraisal. Lancet. 2000, 355: 103-106. 10.1016/S0140-6736(99)02171-6.View ArticlePubMedGoogle Scholar
- Grol R, Grimshaw J: From best evidence to best practice; about effective implementation of change in patient care. Lancet. 2003, 362: 1225-30. 10.1016/S0140-6736(03)14546-1.View ArticlePubMedGoogle Scholar
- Kraemer HC, Frank E, Kupfer DJ: Moderators of treatment outcomes: clinical, research, and policy importance. JAMA. 2006, 296: 1286-1289. 10.1001/jama.296.10.1286.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/11/60/prepub
Pre-publication history
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
