Gaining insight into the Clinical Practice Guideline development processes: qualitative study in a workshop to implement the GRADE proposal in Spain

a) Context, participants and data collection

The ethnographic study [16] was carried out in the same premises as used for the two technical consensus meetings, with the attendance of 19 of the 20 professionals who were initially asked to participate (Table 1).

The first stage of the study involved non-participant observation, whereby two researchers attended the technical consensus meetings (TCM1 and TCM2) as observers; these meetings were recorded and the behavior, attitudes and communications lines generated were noted down. The researchers were previously introduced by one of the promoters and all the participants agreed to the researcher participation in the study. The research material also included reports, forms and the questionnaire model used.

Once the technical consensus meetings had finalized, two focus groups (FG1 and FG2) were created with the 9 and 10 participants from each of the sub-groups into which the 19 experts participating in the study had been divided, considering the group technique to be ideal for promoting debate and interaction. These focus group meetings lasted approximately 90 minutes until all the contributions from within each group had been exhausted. One of the researchers with experience in qualitative studies acted as a moderator, requesting prior authorization to record the meetings whilst guaranteeing that the information would be treated confidentially. A second researcher acted as an observer, noting down the most relevant attitudes. These notes were then completed by both researchers at the end of the meetings.

b) Analysis and validation

An analysis was carried out on the transcriptions of the technical meetings and focus groups, on the actual live recordings and on the observations noted down and the rest of the material compiled by the researchers. Cognitive mapping [17] was used to select and classify the different categories and the sociological discourse analysis model [18] was used as a reference model for the text analysis, as it was considered important to relate the contents with the framework in which they were generated. The ethnographic approach was particularly useful in articulating the descriptive and analytical dimensions of the research and in configuring concepts and categories.

All the researchers took part in the analytical process and discussed the results. The repeated comparison with the empirical material compiled in the texts (recordings, transcriptions and field notes), in addition to the search for negative cases, led to some modifications to the initial approach. The results were subsequently sent to the participants for any possible further input, which was also assessed and included in the analysis, all of which was aimed at improving its validity. The text of the report translated into English was revised by the authors participating in the meetings and focus groups in order to ensure the accuracy of the translation of the verbatim.

1. Experiences in the use of the GRADE method

The analysis of the participants' perceptions helped understand certain responses to the quantitative questionnaire that were previously commented in the Introduction (Table 2) and that would otherwise be difficult to explain.

2. Beyond GRADE

The GRADE method assessment process revealed the existence of certain factors conditioning the experts' perceptions and which are not directly related to the particularities of the new method.

I. The GRADE method and the tensions in the CPG preparation process: adaptation to the complexities of clinical practice

The pilot study to assess this new GRADE method confirms the inevitable need to relate the CPG discussion and preparation processes with the contexts in which they are generated [20, 21].

The contents of the discourses and participant interaction reflect a reality marked by different fields of tension. Firstly, the perception of a certain divergence between the evidence derived from the experimental studies on the one hand and clinical practice on the other, underlining the non-linear nature of the relationship between the study design requirements and the complexity, dynamism and individuality present in health care work [22]. From this perspective, some of the concerns expressed in the previous section become understandable, such as the fact that certain "outcomes" considered important from a professional's point of view have not been the subject of a study or do not have the "quality" provided by experimental design; that the guidelines derived from a clinical test do not correspond to the specific characteristics of the patient requiring care; or the importance given to aspects such as external validity and implementability.

Consequently, and to the point that GRADE is perceived as being closer to those issues that professionals have to deal with in practice (outcome grading, process systematics, explicitness in the grading of recommendations), to a certain extent this new method is viewed favorably and appears to promote reflection. However, together with this greater closeness to reality, the need to express value judgments in order to finally reach a consensus with regard to the greater or lesser importance of the outcomes, provokes a marked concern over subjectivity and over the difficulty in expressing the said judgments in quantifiable tables.

The participants' desire to base professional practice on scientific criteria largely explains the opposition expressed. However, in turn, these fears and concerns also reflect the perplexity generated by the inevitable presence of values in a medium – such as the one represented by the CPGs – that is apparently exclusively reserved for the so-called objectivity of facts.

The forced separation of both dimensions (value judgments/facts that can be reduced to numbers) is also present in each of the different "repertoires" or facets (scientific, practical, political and procedural) present in the CPG preparation process [15].

In fact, the artificiality derived from concealing the value judgments and the frequent reduction of the evidence to something that is numerically quantifiable has already been questioned in the past in the light of what actually happens in clinical practice [23, 24] and, in particular, in the numerous attempts to gain an insight into the barriers to CPG uptake and implementation in professional practice [25–27].

It therefore follows that the solutions to the said tensions should be sought not so much in the simplification of the instruments used to develop and evaluate the CPGs but in a more complete and thorough knowledge of the reality on which the CPGs are based and in a dynamic and open adaptation of their design to the requirements of this reality. In this respect, the decisions and assessments that professionals need to make in their daily practice should not be overlooked, so some excessively simplified response models ("do it" "probably do it") may be of limited benefit.

II. By whom and for whom

A new area of tension is delimited by the role corresponding to the different actors involved in the CPG design and grading processes. In principle, both the GRADE method and those instruments already in place are based on the necessary participation of the clinician in the CPG development and dissemination process [28, 29].

However, clinicians interested in the preparation and promotion of CPGs are first faced with a considerable number of studies published with insufficient guarantees of rigor and impartiality. They are thus forced to carry out the arduous task of systematically reviewing the biomedical literature in order to "separate the grain from the straw" [3]. This work, apart from requiring training and experience, tends to be given to "methodologists" or specialist technicians; the documentation prepared on this by NICE [29] possibly provides the greatest details of the tasks corresponding to each member of the CPG preparation group.

In our case, the display of mistrust for secondary data and the demand to access the original studies (the "bricks") by some of the participants in the research study, and the references to the "burden" that involvement in the mentioned critical reading tasks represents for many clinicians, would corroborate the importance of the tensions derived from the distribution of tasks in the CPG development process.

Certain authors have interpreted this tendency to divide the work as a possible paradigm shift according to which medical practice is moving from a more autonomous and individualized model to a more standardized model that depends on the criteria established by non-clinical professionals such as epidemiologists or bio-statisticians [30]. The concerns arising in this respect in this study paint a reality that is less dichotomic although it is undoubtedly sensitive to the said issues.

On the one hand, the majority of participants had worked in multidisciplinary groups to prepare CPGs and were aware of the advantages of doing so, both with regard to the distribution of tasks as well as the input from different professional perspectives. However, in turn, and probably as a result of their experience in critical reading, a marked prejudice towards the risks of misrepresenting scientific information can be inferred from their discourses. These risks are perceived to be greater as access to the original sources becomes more distant.

The greater degree of "scientific" authority that direct access to the data constituting the "evidence" confers on the "methodologists" [15] would help explain the concerns and mistrust put forward in the debate on the GRADE method by the clinicians in the face of possible irregularities derived from the "division of tasks". Clinicians, who in this case, had experience as authors of CPGs and were also witnesses to the difficult incorporation of their colleagues into this type of work.

Consequently, in the CPG development, in addition to the necessary multidisciplinary collaboration, it is also necessary to guarantee that the process is carried out on the basis of some common criteria of rigor and transparency right from the initial selection stages and secondary preparation of information up to the final dissemination of the recommendations.

The detailed description of the functions and tasks corresponding to the different members (epidemiologists, doctors, nurses, patients) of the CPG preparation groups [29] constitutes an unquestionable aid in this respect. However, greater attention should also be paid to the specific requirements and risks of the intercommunications process based on the type of participant. In our case, the "methodologists" and "clinicians" were the ones to reveal the "tensions" that appear to affect their inter-relations. Therefore, the debate on the different roles and task distribution dynamics of the CPG development process still remains open.

III. Rigor and reflexivity: the internal environment

As indicated in the Results section, the different categories and dimensions analyzed interact through intra-group dynamics, an aspect which should be considered with particular interest from the point of view of CPG rigor and quality.

The "non neutral" component present in the attitudes shown by the different participants in the GRADE method pilot test is in line with earlier findings [14] and, in fact, the role played by the debate and consensus process in the final quality of the CPGs has not been overlooked by the experts in their attempts to standardize this process to achieve greater control over it [28, 29, 31].

Subsequent to our data collection and after starting the results analysis for this present research work, some new proposals have been published for possible techniques oriented in this same direction [32]. These proposals underline the frequent error of identifying the "it's like this" derived from experimental studies with "it ought to be done like this" characteristic of CPGs, overlooking the plurality and importance of "internal" conditioning factors not made explicit in the present grading systems.

In general, the proposals published come within the sphere of the so-called "consensus methods" (nominal groups, Delphi techniques etc), more or less modified, and are directed at "controlling" the mentioned "internal dynamics" and making this process more visible. The demands for greater explicitness by the experts participating in the research work could be interpreted along these same lines. However, in addition, the influence of these factors is conditioned not only by the interests, priorities and future expectations regarding the issues subject to debate but also those of the guideline developers themselves, and this leads us to the concept of reflexivity, a term common to the field of social science and related to making clear the subject of knowledge. This reflexivity means the expert's acceptance of his inevitable active role, not only in the application of a specific procedure, but also in the choice of a particular method to incorporate techniques and instruments directed at selecting "evidence" for its subsequent development into CPG recommendations, and even in identifying the social and health conditions that delimit the context in which these choices are made.

As a result of this reflexivity, some standardizable procedures of consensus are unquestionably required to make the dynamics intervening in the CPG development transparent and explicit. These procedures should also address the higher levels of knowledge (methodological and epistemological) [33] responsible for classifying, giving priority to and processing the information on which the CPG elaboration processes are based.