Quality and methods of developing practice guidelines
© Cruse et al; licensee BioMed Central Ltd. 2002
Received: 20 October 2001
Accepted: 11 January 2002
Published: 11 January 2002
It is not known whether there are differences in the quality and recommendations between evidence-based (EB) and consensus-based (CB) guidelines. We used breast cancer guidelines as a case study to assess for these differences.
Five different instruments to evaluate the quality of guidelines were identified by a literature search. We also searched MEDLINE and the Internet to locate 8 breast cancer guidelines. These guidelines were classified in three categories: evidence based, consensus based and consensus based with no explicit consideration of evidence (CB-EB). Each guideline was evaluated by three of the authors using each of the instruments. For each guideline we assessed the agreement among 14 decision points which were selected from the NCCN (National Cancer Comprehensive Network) guidelines algorithm. For each decision point we recorded the level of the quality of the information used to support it. A regression analysis was performed to assess if the percentage of high quality evidence used in the guidelines development was related to the overall quality of the guidelines.
Three guidelines were classified as EB, three as CB-EB and two as CB. The EB guidelines scored better than CB, with the CB-EB scoring in the middle among all instruments for guidelines quality assessment. No major disagreement in recommendations was detected among the guidelines regardless of the method used for development, but the EB guidelines had a better agreement with the benchmark guideline for any decision point. When the source of evidence used to support decision were of high quality, we found a higher level of full agreement among the guidelines' recommendations. Up to 94% of variation in the quality score among guidelines could be explained by the quality of evidence used for guidelines development.
EB guidelines have a better quality than CB guidelines and CB-EB guidelines. Explicit use of high quality evidence can lead to a better agreement among recommendations. However, no major disagreement among guidelines was noted regardless of the method for their development.
The objective of guidelines development is to assist physicians and patients in making optimal health care decisions, which in turn should improve the quality of clinical practice .
Different methods are used to develop guidelines. Some are developed by a consensus of experts while others also use a formal way to appraise the literature and create evidence-based (EB) guidelines. In general, evidence-based guidelines are considered to provide better recommendations for practice than consensus-based guidelines but are time consuming and expensive to create [2, 3]. This belief that EB guidelines are superior to other types of guideline is based on our normative views of methods for guidelines development  and not on empirical comparison of practice recommendations using different methods for development of guidelines. To date no formal evaluation has been performed to detect if there are differences in the quality and recommendations between evidence-based and consensus-based (CB) guidelines.
If guidelines developed by using consensus or evidence-based methods have the same quality and agree in the recommendations, then obviously resources spent on the laborious and time-consuming process of locating and appraising evidence can be used elsewhere. Otherwise, if evidence based guidelines have a better quality and their recommendations differ from those guidelines produced by consensus, then creation of evidence based guidelines may become the only acceptable method of guideline development.
In this paper, we explore if there are differences in the quality and recommendations between EB and CB guidelines.
Identification and assessment of instruments for measurement of the quality of guidelines
Interobserver agreement of instruments for assessment of the guidelines quality
Guidelines with K > 0.4 among all four evaluators
ACCC (19), CMA (7)
ACCC (19), NHMRC (16), MPS (18)
Petrie (SIGN) (10)
NCCN (6), ACCC (19), SIGN (17), ICSI (15), MPS (18), SSO (5)
Shaneyfelt 1999 (12)
ACCC (19), CMA (7), NHMRC (16), ICSI (15), MPS (18), SSO (5)
Identification and classification of breast cancer guidelines
A literature search was conducted for published breast cancer guidelines using MEDLINE for the years 1996 – April 2000. The following keywords were used in combination: Guidelines, Practice Guidelines, recommendations, breast neoplasms. An Internet search was also performed, using the method described by Sanders et al . 131 articles were retrieved, and reviewed for their content. We considered any article that fit the definition of the National Library of Medicine for practice guidelines: directions or principles presenting current or future rules of policy for the health care practitioner to assist him in patient care decisions regarding diagnosis, therapy, or related clinical circumstances . Eight papers referred to breast cancer guidelines [5–7],[15–19] and were selected for the analysis.
Classification of Breast Cancer Guidelines according to the method of development.
Consensus Based with no explicit consideration of evidence
Evaluation of guidelines
Evaluation of agreement among guidelines
Using instruments to evaluate practice guidelines yields conclusions regarding normative aspects of the guidelines development , but does not necessarily mean that recommendations provided by guidelines using different methods will produce different management advice to our patients. To assess if recommendations among various guidelines differ, we need to determine the level of agreement among guidelines for each specific decision point.
Since NCCN (National Comprehensive Cancer Network) guidelines  were presented in explicit, algorithmic format, we used this one to identify the decision points for matched comparison with other guidelines. These guidelines have been developed by the leading 18 cancer institutions in the US and have been constantly updated and re-evaluated. They have also been developed to closely mimic clinical practice. Therefore, we feel that selection of decision points based on the NCCN guidelines were appropriate. We identified fourteen decision points in the management of stage I and II breast cancer that were linked to specific recommendations in the other guidelines for our comparison. Comparison of recommendations for advanced stages of breast cancer has not been performed since there was only one guideline that included it .
Subsequently, four of us evaluated each of these decision points in each guideline examining level of agreement among various guidelines. Since matching between recommendations in the guidelines that were presented in non-algorithmic format was poor, we decided to use NCCN guidelines as a benchmark. We classified agreement of each guideline with the NCCN guidelines as having full agreement, partial agreement and disagreement. It was considered that guidelines agree with the NCCN if the management recommendation was the same; the guidelines were considered to disagree if they provided different recommendations. A partial agreement was judged to exist if the guideline recommended the same management but in a broadly defined sense and not in explicit, clear manner.
Each of these decision points was also classified as supported by high quality evidence or not. High quality evidence was considered to be based on randomized trials (RCT) or systematic reviews (SR)/meta-analysis (MA). If the quality evidence was not based on RCT or SR/MA or was not stated, it was classified as low quality evidence.
Subsequently, we performed a regression analysis to assess the contribution of the quality of evidence to the total score obtained by each instrument for the evaluation of the guidelines quality. Independent variable was the proportion of decisions supported by high quality evidence while dependent variable was score obtained by each instrument. A regression analysis was performed after it has assessed that the distribution of the variables was normal by Wilks-Shapiro test.
Evaluation of the quality of guidelines
Quality of breast cancer guidelines
Instruments for assessment of guidelines quality
Petrie (SIGN) (10)
Evaluation of agreement among guidelines
Level of agreement between NCCN guideline and other breast cancer guidelines.
Statistical significance (p)
N. of decision points with high quality evidence
Guidelines have been increasingly used in medical decision-making. Different methods have been used in guideline development. Does it matter how guidelines were produced? Most authors believe that it matters very much  and that guidelines produced using evidence-based methods are superior to other methodologies of development [2, 4, 9]. However, empirical investigations to assess if guidelines produced by different methods have different quality and result in different recommendations have not been performed. Here, we report such a study.
Using formal instruments for evaluation of the quality of guidelines we found that EB-guidelines had substantially higher score than CB-guidelines or guideline that considered evidence in a less formal way (CB-EB). As discussed above (see Results), this is not a surprising result, since the instruments for the guidelines evaluation measure the quality based on the number of desired normative characteristics in a particular guideline. Since appraisal of evidence is considered inherently important for the development of a good guideline, one would then expect that the guidelines that pay more attention to its evidence basis (i.e., those that are evidence-based) would receive higher quality score than other types of the guidelines (i.e. guidelines developed solely by a consensus process) (see Fig 1). This is also evident in our finding that variation in the total quality score can be up to 94% explained by the quality of evidence (see Fig 2).
Not all instruments for evaluation of guidelines performed equally well. Only two of the instruments available to address the quality of guidelines had a good level of agreement among evaluators (k > 0.4) in most of guidelines. This result raises concern about the reproducibility of results using the other instruments reported in the literature. In general, a few studies have been done to evaluate reproducibility of the instruments for assessment of the guidelines quality. Any future study attempting to address the quality of guidelines should take this finding into account.
A more interesting question is to assess if the recommendations among guidelines produced by different methods actually differ. We found no instance of total disagreement among guidelines regardless of the method of development. We also found that EB and CB-EB guidelines had more points of agreement with our benchmark guidelines (NCCN) than guidelines developed using exclusively consensus method. We also found that when high-quality evidence existed in the literature (see Results) less disagreement was found among various guidelines. This is not completely surprising because formulation of guidelines does not happen in a vacuum. Most guideline developers are experts in the field who have knowledge of the literature. When evidence is unequivocal, less disagreement may be expected. Consequently, less practice variation may be found when high-quality evidence exists.
In conclusion, EB guidelines have a better quality than CB guidelines as measured by the quality assessment instruments used in this study. The explicit use of high quality evidence is desirable and can lead to a better agreement among recommendations. However, no major disagreement among guidelines was noted regardless of the method for their development.
We thank Dr.Stephen Edge for reviewing our paper and his helpful comments and constructive critique.
- Institute of Medicine. Guidelines for clinical practice: from development to use. Washigton DC: National Academic Press;. 1992Google Scholar
- Woolf SH: Evidence-based medicine and practice guidelines: an overview. Cancer Control. 2000, 7 (4): 362-7.PubMedGoogle Scholar
- Miller J, Petrie J: Development of practice guidelines. Lancet. 2000, 355 (9198): 82-3. 10.1016/S0140-6736(99)90326-4.View ArticlePubMedGoogle Scholar
- Eddy DM: Clinical decision making: from theory to practice. Practice policies-guidelines for methods. JAMA. 1990, 263 (13): 1839-41. 10.1001/jama.263.13.1839.View ArticlePubMedGoogle Scholar
- Morrow M, Bland KI, Foster R: Breast cancer surgical practice guidelines. Society of Surgical Oncology practice guidelines. Oncology (Huntingt). 1997, 11 (6): 877-81.Google Scholar
- Update: NCCN practice guidelines for the treatment of breast cancer. National Comprehensive Cancer Network. Oncology (Huntingt). 1999, 13 (11A): 187-212.Google Scholar
- The Steering Committee on Clinical Practice Guidelines for the Care and Treatment of Breast Cancer. CMAJ. 1998, 158 (Suppl 3): S1-2.Google Scholar
- Cluzeau F, Littlejohns P, Grimshaw J, Feder G: Appraisal Instrument for Clinical Guidelines. London: St. George's Hospital Medical School;. 1997, Available from: St. George's Hospital Medical School web site http://www.sghms.ac.uk/depts/phs/hceu/clinguid.htm. Accessed 11 June 2001.Google Scholar
- Grilli R, Magrini N, Penna A, Mura G, Liberati A: Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet. 2000, 355 (9198): 103-6. 10.1016/S0140-6736(99)02171-6.View ArticlePubMedGoogle Scholar
- Petrie J, Barnwell E, Grimshaw J: Criteria for Appraisal for National Use – Scottish Intercollegiate Guidelines Network (SIGN). SIGN Publication Number 39, 1995. Edinburgh: Scottish Intercollegiate Guidelines Network (SIGN);. 1995, Available from SIGN web site http://www.sign.ac.uk/guidelines/fulltext/50/index.html (Version 2001). Accessed 11 June 2001.Google Scholar
- Sanders GD, Nease RF, Owens DK: Design and pilot evaluation of a system to develop computer-based site-specific practice guidelines from decision models. Med Decis Making. 2000, 20 (2): 145-59.View ArticlePubMedGoogle Scholar
- Shaneyfelt TM, Mayo-Smith MF, Rothwangl J: Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA. 1999, 281 (20): 1900-5. 10.1001/jama.281.20.1900.View ArticlePubMedGoogle Scholar
- Landis J, Koch G: A measurement of observer agreement for categorical data. Biometrics. 1977, 33: 159-174.View ArticlePubMedGoogle Scholar
- NLM PubMed resources page. 2001, Available via internet http://www.ncbi.nlm.nih.gov/entrez/meshbrowser.cgi?term=Practice+Guidelines&retrievestring=&mbdetail=n Accessed 11 June 2001.
- ICSI Institute for Clinical Systems Improvement. ICSI Health Care Guideline: Breast Cancer Treatment. ISCI,. 2000, Available from: ICSI web site http://www.icsi.org/guidelst.htm. Accessed 11 June 2001
- NHMRC-AU. NHMRC National Breast Cancer Centre – Clinical Practice Guidelines For The Management of Early Breast Cancer 1999. NHMRC-AU. 1999, Available from: NHMRC-AU web site http://www.health.gov.au/nhmrc/advice/pdf/earlybrs.pdf (Version 2000). Accessed 11 June 2001.
- SIGN – Scottish Intercollegiate Guidelines Network. Breast Cancer in Women – A National Clinical Guideline. SIGN Publication Number 29,1998. Edinburgh: Scottish Intercollegiate Guidelines Network (SIGN);. 1998, Available from SIGN web site http://www.sign.ac.uk/pdf/sign29.pdf. Accessed 11 June 2001.
- Winchester DP, Cox JD: Standards for diagnosis and management of invasive breast carcinoma. American College of Radiology. American College of Surgeons. College of American Pathologists. Society of Surgical Oncology. CA Cancer J Clin. 1998, 48 (2): 83-107.View ArticlePubMedGoogle Scholar
- ACCC – Association of Community Cancer Centers. Oncology Patient Management Guidelines. Breast Carcinoma version 3.0. ACCC. 1999Google Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/2/1/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article: verbatim copying and redistribution of this article are permitted in all media for any purpose, provided this notice is preserved along with the article's original URL.