Skip to main content
  • Research article
  • Open access
  • Published:

A process evaluation of the improving wisely intervention: a peer-to-peer data intervention to reduce overuse in surgery

Abstract

Background

The Improving Wisely intervention is a peer-to-peer audit and feedback intervention to reduce overuse of Mohs Micrographic Surgery (MMS). The objective of this study was to conduct a process evaluation to evaluate Mohs surgeons’ perceptions of the implementation quality and perceived impact of the Improving Wisely intervention.

Methods

Surgeons in the Improving Wisely intervention arm, comprised of members of the American College of Mohs Surgeons (ACMS) who co-led the intervention, were invited to complete surveys and key informant interviews. Participants described perceptions of implementation quality (evaluated via dose, quality of implementation, reach and participant responsiveness), perceived impact of the Improving Wisely intervention (evaluated on a 1–5 Likert and qualitatively), and barriers and facilitators to changing surgeons’ clinical practice patterns to reduce Mohs overuse.

Results

Seven hundred thirty-seven surgeons participated in the survey. 89% were supportive of the intervention. Participants agreed that the intervention would improve patient care and reduce the annual costs of Mohs surgery. Thirty surgeons participated in key informant interviews. 93% were interested in receiving additional data reports in the future. Participants recommended the reports be disseminated annually, that the reports be expanded to include appropriateness data, and that the intervention be extended to non ACMS members. Six themes identifying factors impacting potential MMS overuse were identified.

Conclusions

Participants were strongly supportive of the intervention. We present the template used to design and implement the Improving Wisely intervention and provide suggestions for specialty societies interested in leading similar quality improvement interventions among their members.

Peer Review reports

Background

Mohs Micrographic Surgery (MMS) is a procedure where dermatologic surgeons aim to obtain a cancer-free margin while preserving healthy tissue by removing a lesion in small slices called stages. This approach is typically used to treat skin cancers with aggressive pathology in areas at high risk for recurrence and/or with functional or cosmetic implications, including tumors on the head and neck, genitals, hands and feet. In the past two decades, the prevalence of MMS use has increased by over 700%, with over 2000 physicians performing Mohs surgery annually in the United States (Chen et al. 2016).

A 2017 study by our research team showed marked variation in the number of stages per case performed for MMS procedures. Approximately 6% of physicians who billed Medicare for MMS between 2012 and 2014 were found to be outliers, defined as surgeons performing a stage per case rate greater than 2 standard deviations from the national mean (Chen et al. 2016; Krishnan et al. 2017).

To address this variation, we developed the Improving Wisely (IW) intervention, a quality improvement initiative that utilizes national data to identify overuse patterns for various medical and surgical procedures. The intervention is composed of personalized, confidential data reports coupled with use of opinion leaders and opportunities for confidential mentoring and education. In a non-randomized controlled trial, 84% of outlier surgeons in the intervention group showed a reduction in mean stages compared to 69% of outliers in the control group.

We posit that the IW intervention can be sustainability implemented to mitigate overuse, underuse or misuse both in Mohs surgery and other areas of medicine. Prior to sustained implementation, we wished to conduct a process evaluation to canvass surgeons’ perceptions on the perceived appropriateness, impact and generalizability of the IW intervention (Moore et al. 2015). Therefore, the purpose of this study was to use mixed methods to evaluate Mohs surgeons’ perceptions regarding the implementation quality and perceived impact of the IW intervention and to identify barriers and facilitators to changing Mohs surgeons’ clinical practice patterns to reduce Mohs overuse.

Methods

The study design included surveys and key informant interviews to inform a process evaluation for the IW intervention. In Part I, we describe the methods used to design and conduct the survey. In Part II, we use the Consolidated Criteria for Reporting Qualitative Research (COREQ) checklist to report the methods used to design and conduct the key informant interviews (Tong et al. 2007).

IW Mohs intervention

The Mohs intervention was delivered to 1045 Mohs surgeons who were members of the American College of Mohs Surgery (ACMS), the largest specialty society for MMS. Surgeons received a confidential, personalized report that presented each surgeon’s mean number of stages per case for head and neck lesions, accompanied with the national average of stages per case and a consensus metric definition of acceptable stages per case, defined by an expert panel. Surgeons were categorized as inliers or outliers (i.e., fell outside of the accepted range of variation, defined as > 2.0 standard deviations above the national mean) (Albertini et al. 2019). The confidential reports were endorsed by national opinion leaders at the annual ACMS meeting. The data reports were accompanied with a cover letter which was signed by national opinion leaders. The intervention group were offered opportunities for confidential mentoring and education (Albertini et al. 2019).

In a non-randomized controlled trial, an additional 14% of outlier surgeons in the intervention group decreased their mean stages per case as compared to the control group, which resulted in a statistically significant decrease in mean stages per case across groups. Notified inliers also significantly decreased their stages per case, as compared to non-notified inliers. This study and findings are described in detail elsewhere (Krishnan et al. 2017; Albertini et al. 2019).

Part I – surveys

Setting and participants

The surveys were distributed on May 5, 2018 to ACMS members attending the annual scientific meeting in Chicago. Surveys were distributed during the morning plenary session to facilitate maximum reach of potential participants. All participants who attended the plenary session were invited to complete a survey (regardless of whether or not they had received an IW report). Prior to distribution of the survey, ACMS leaders led a short presentation describing the IW intervention, including a depiction of a sample report. This provided an opportunity for ACMS members who had not received a personalized data report to also provide feedback on their perceptions of the intervention.

Outcomes

Outcomes included 1) Perceptions of the IW data; 2) Perceptions of intervention impact; and 3) Perceptions of intervention generalizability and acceptability.

Variables

The survey questions were informed by the Theoretical Domains Framework (TDF), which is a framework rooted in psychology theory (Michie and Prestwich 2010; Cane et al. 2012). The TDF includes 14 domains used to elicit perceptions of the factors(barriers and/or facilitators) that can impact physician behavior and can also be used to evaluate a behavioral intervention (Michie and Prestwich 2010; Cane et al. 2012). An initial 35-item questionnaire (2–4 items per TDF domain) was generated. The survey items were iteratively refined and reviewed by knowledge experts (surgeons, ACMS leadership), an implementation scientist and IW team researchers.

The final survey included three categories and 29 questions (Additional File 1). Part 1, Perceptions of the Report Data, included 12 questions. Part 2, Perceptions of Impact, included 9 questions. Part 3, Generalizability and Acceptability, included 8 questions. Data were evaluated using a 5 point, anchored Likert scale (5 = Strongly Agree; 4 = Agree; 3 = Neither Agree nor Disagree; 2 = Disagree; 1 = Strongly Disagree). Negatively and positively worded questions for the same item were included to identify potential response bias (e.g., I am supportive vs. I am not supportive of the IW intervention). The survey included screening questions to determine the proportion of respondents who were aware of the IW initiative and who received/saw an individualized data report. An open ended question invited participants to highlight other feedback/comments. Demographics collected via the survey included participant name (optional), sex), age range, and years of practice as a Mohs surgeon.

Data collection

Surveys were distributed in paper format and were available for online access via RedCap software for a 6 month period (May–October 2018). The survey included a statement indicating that completion of the survey would serve as consent to participation.

Data analysis

Descriptive statistics were generated using Excel software. Frequencies and proportions were provided for demographic data. Median, mode and range were reported for Likert data. Missing data were not included in the analysis, total responses per survey question were reported. Positively or negatively framed questions for similar items were examined for response bias (Paulhus 1991). Open-ended data were collated by theme by a single researcher (CF). Frequencies for each theme were generated; individual themes within each comment were counted independently. Themes reported by fewer than < 5 surgeons were excluded, in order to protect confidentiality. Quantitative and qualitative data were triangulated to provide an overall summary of the survey findings (Oleinik 2011).

Sample size

The survey was distributed to approximately 900 ACMS Mohs Surgeons attending the annual meeting.

Part II – key informant interviews

Research team and reflexivity

The primary author (CF) conducted the interviews as a postdoctoral fellow on the Improving Wisely research team. The interviewer holds extensive experience conducting key informant interviews. CF did not hold a relationship with the study participants.

Study design

Theory

The interviews were theoretically rooted in the Theoretical Domains Framework and analyzed using a content analysis approach.

Sampling

We aimed to purposively and representatively recruit ACMS members who had received an IW report and were classified as an outlier or an inlier.

Recruitment

ACMS Mohs surgeons were emailed a recruitment script inviting them to participate in a 20–30 min telephone interview to describe their overall feedback and perceptions of the IW report and project. Interested participants were asked to respond to the email.

Sample Size

We aimed to recruit n = 13 outliers and n = 13 inliers for a total of n = 26 participants. This sample size was informed by the guidance of Francis et al. which suggests using an initial sample size of n = 10 and a stopping criteria of n= 3 when conducting theoretically-rooted key informant interviews (Francis et al. 2010). We reached out to 267 inliers and 24 outliers, for whom we had an email address. Each surgeon received a personalized email that outlined the study objectives and invited them to participate in a confidential interview. Given the limited sample of outliers, each outlier was sent two recruitment reminders to participate in the key informant interview (Hoddinott and Bass 1986).

Data collection

A researcher corresponded with interested participants to schedule the semi-structured interviews. Following verbal informed consent, the interview was recorded and transcribed verbatim.

Presence of non-participants

Only the interviewer was present in the room during the interviews.

Interview guide

We designed an 11 item semi-structured interview guide (Additional File 2) informed by the TDF and usability questionnaires (Cane et al. 2012; Chisnell and Rubin 2008).

Repeat interviews

Key informants participated in a single study interview; transcripts were not returned to the participants.

Field notes

The interviewer (CF) took detailed field notes during and following each key informant interview, noting potential themes.

Analysis and findings

Data Analysis:Summary statistics were generated for demographic variables. Interview recordings were transcribed by an independent agency. Two researchers (CF, WEB) each independently read 3 transcripts to gain familiarity with the interview content. The researchers then developed a codebook based on the TDF (Cane et al. 2012).Additionally, codes were developed to evaluate the implementation quality and ways to improve the IW intervention. Implementation quality, sometimes referred to as a process evaluation, can be used to determine whether the IW intervention was appropriately implemented, or if there were potential barriers that hindered successful implementation (Moore et al. 2015). Process evaluations can explain why an intervention was/was not successful at impacting target outcomes. Implementation quality codes were rooted in the Durlak and Dupre process evaluation framework (Durlak and DuPre 2008) and includeddose (i.e., how often surgeons want to view the report), quality of delivery (i.e., was the report helpful? was it easy to understand/view?), reach (i.e., did the right audience receive the report), and participant responsiveness (i.e., were surgeons engaged in/approving of the IW intervention?).

Definitions and coding rules were developed for each TDF domain and process measure. The researchers double coded 35% of the interviews using NVivo software (Nvivo 1999). Any discrepancies were resolved viaconsensus discussion. The remainder of the interviews were divided and each half was single coded by one researcher. The codes were reviewed by the researchers to develop a list of overarching themes.

Results

Survey results

Demographics

A total of 737 ACMS Mohs surgeons participated in the survey (response rate = 82%). Participants were mostly male (62%) and between the ages of 30–49 (68%). Ninety-six percent of those surveyed were aware of the IW intervention and 76% had received an individualized feedback report. Among those who received a report, 88% reported their mean number of stages were ‘as they had expected’; 7 and 6% reported their mean number of stages were lower and greater than expected, respectively (Table 1). 89% were supportive of the intervention.

Table 1 Participant demographics and Summary Responses

Part 1: perceptions of intervention

The majority of participants found the IW intervention simple to interpret and expressed strong interest in receiving another utilization report in the future (median, mode = 5). Participants strongly agreed that it was valuable to see their data compared to national benchmarks and that it was personally important to know that their stages fell within the ACMS-established boundaries of variation. Despite support for the intervention, seeing the data reports did not change surgeon’s perceptions of their practice patterns (Table 2).

Table 2 Survey Results

Part 2: perceptions of impact

Participants agreed or strongly agreed that the IW intervention would improve patient care and reduce annual costs of Mohs surgery (Table 2). Participants agreed that improvements in surgeon practice would be sustained with repeated, annual reports. Participants’ gave neutral responses regarding beliefs about personal consequences, intentions, and decision making processes (e.g., the report has improved the quality of my surgical practice; I intend to be more mindful of my stages per case rate).

Part 3: overall impressions and generalizability of the intervention

The majority of participants believed that the principles of the IW intervention would also be valuable for other areas of medicine (median, mode = 5). Participants agreed that their peers would support the IW project and suggested that the intervention goals be communicated to Mohs trainees. Participants did not feel threatened by the project (median, mode = 2,1) or experience undue pressure from seeing their report (median, mode = 2).

Open ended feedback data

Analysis of the open-ended survey data revealed three overarching categories and 7 themes, outlined alongside participant quotes in Additional File 3. The first category (3 themes) presented feedback on the data used to create the report. N = 23 participants (3%) expressed concern that complex cases may be driving up their average. Commonly, case complexity was attributed to working in a tertiary referral center or in a rural area. Less than 1% of participants (n = 6) suggested that specific appropriateness criteria, rather than statistical variation, should be used to determine outlier status in order to ensure a sustainable quality improvement intervention. N = 35 participants (5%) suggested additional metrics for IW reports; common suggestions included flaps versus grafts and next day closures.

The second category (3 themes) of feedback was categorized as general support for the IW intervention. Five participants (< 1%) reported that the data report made them more mindful of stages per case, the cost of health care, and highlighted that they share this metric with their patients. Seven participants (1%) suggested the reach of the intervention should be extended to include every provider that bills Mohs codes (i.e., do not limit to ACMS members). Finally, 11% of participants provided positive feedback on the intervention and praise for the ACMS for leading this intervention.

The third category (1 theme) highlighted concerns about the potential unintended consequences of the IW intervention. N = 29 (4%) were concerned that these data would no longer be confidential (i.e., might get ‘leaked’ to policymakers, funders or patients). Some participants expressed concerns that surgeons might change their practice to cater to the report rather than patient care (e.g., take unnecessarily large margins to reduce their mean score). Others suggested that outliers, once aware that this metric was being tracked, would continue to ‘game’ the system in other ways.

Triangulation

The responses did not indicate evidence of a response bias. Polarized scores were demonstrated for negatively and positively worded questions for the same item (see Table 2, items 11 and 12, 23 and 24). The quantitative and open-ended survey data were convergent.

Part 2 – key informant interviews

Thirty Mohs surgeons participated in the key informant interviews. The average length of the interviews was 21 min (range: 10–50 min.). Interviews were conducted between September 12, 2018-January 24, 2019. Sixty-three percent of interview participants were male and 43% had 6–10 years of practice as a Mohs surgeon. A total of 26 notified inliers (87%) and 4 notified outliers (13%) participated in the interviews. Saturation was reached for the inlier, but not the outlier group.

Process evaluation of IW intervention

Dose

93% of those interviewed were interested in seeing their data report in the future. Consistent with the surveys, the majority agreed that the report should be disseminated annually, suggesting that this frequency would be sufficient to nudge outliers without creating too much burden (See Table 3 for sample quotes).

Table 3 Implementation Quality – Key Informant Data, Sample Quotes

Quality of Delivery

All interviewed participants found the report straight forward, easy to interpret and clear (Table 3). Suggestions on how to improve the report included expanding the data report to include non-CMS databases (e.g., private insurance) and incorporating appropriate use criteria to establish recommended targets. Participants believed that the incorporation of appropriateness would detect differences in geographic location or case complexity.

Participant responsiveness

Participants, particularly notified inliers, were generally supportive of the IW intervention. The most common identified theme was the sense of reassurance generated by the report. Participants were supportive of ACMS leading the IW intervention, highlighting the importance of ‘policing your own’ before another regulatory body did so (Table 3).

Reach

Many participants suggested that the intervention should be extended to non-ACMS members.

Perceptions of Impact

The most common theme regarding perceived impact of the intervention was that it improved surgeon awareness of Mohs overuse (Table 3). Some participants, particularly those classified as outliers, were concerned that the intervention might have unintended consequences, such as gaming the system, adding undue pressure, or compromising patient quality.

Factors impacting Mohs overuse

Six main themes, representing six TDF domains that impact Mohs overuse were identified. The most common theme was the perception that financial incentives drive Mohs overuse (Table 4). Tied closely to the facilitator of financial incentives was the perception that there was a lack of regulation. Both inlier and outlier participants highlighted that lack of monitoring led to either the conscious or unconscious justification of taking more stages per case. Another common theme, particularly highlighted by participants classified as outliers, is that geographic variation or clinic type impacts the types of cases presenting for Mohs surgery. Because these cases tend to be more complex, or because there is less access to Mohs care in rural areas, more stages per case are required in order to provide an adequate treatment. Participants who highlighted this theme were supportive of incorporating appropriate use criteria into the IW reports. Finally, the last two themes pertained to surgeon experience, beliefs about capabilities, and surgical skill. Both inlier and outlier surgeons reported that skills ‘slip’ once surgeons leave fellowship training. Additionally, participants hypothesized that inexperienced or less confident surgeons may err on the side of caution, opting to take additional stages to ensure patient safety.

Table 4 Factors impacting potential Mohs overuse

Most interview participants, including outliers interviewed, were supportive of the IW intervention. Nearly all participants highlighted that the data report made them more aware of their stages per case. Of the four outliers interviewed, three attributed their outlier status to having more complex cases or being in an underserviced geographic location. One believed that financial gain was driving up overuse of Mohs procedures.

The most common suggestion to reduce Mohs overuse was the use of ‘champions’. The suggested role of these champions is to support decision making for complex cases (e.g., peer review) and to reach out to ‘coach’ identified outliers in a supportive manner.

Discussion

We report a process evaluation for the IW intervention, aimed at reducing potential overuse of Mohs surgery. Surveys with Mohs surgeons revealed strong majority support for the intervention, and provided useful suggestions on additional metrics. Interviews with Mohs surgeons provided feedback on the preferred delivery of the data reports and confirmed the acceptability and utility of the IW intervention. Participants most commonly believed that Mohs overuse was driven by financial incentives and surgeon inexperience. Interview participants also stressed the importance of incorporating Mohs appropriate use criteria to generate future data reports and to account for potential geographic variation or case complexity.

This study demonstrates the value of conducting process evaluations for implementation interventions. Increasingly, implementation researchers have called for improved standards for tracking and reporting process evaluations to support widespread implementation and sustainability (Lewis et al. 2018). Process evaluations provide researchers and implementers with insight on why an intervention may have worked, or not; illuminates potentially critical barriers and facilitators that can aid in further implementation; and supports an integrated approach of involving the user in the evaluation of the implementation, which may lead to improved uptake and sustainability (Limbani et al. 2019).

The IW intervention is a relatively low cost intervention that has resulted in significant clinical and positive process outcomes (Albertini et al. 2019). We posit the success of the IW intervention can be attributed to three main factors. First, the data reports are rooted in best practices for audit and feedback, including recommending specific actions, providing individualized data, using comparators to reinforce suggestions, presenting data both visually and in text while reducing cognitive overload, and by preventing negative reactions to the feedback by emphasizing the non-punitive, collaborative approach of the intervention (Brehaut et al. 2016). Second, we use an integrated knowledge translation approach to developing the datareports, meaning that content experts in the field lead, create and endorse the metrics used to generate the data reports (Gagliardi et al. 2015; Graham et al. 2018). This approach is believed to improve both the buy-in for and impact of implementation strategies (Gagliardi et al. 2015; Graham et al. 2018). Finally, the intervention is endorsed by respected leadership and opinion leaders (Flodgren et al. 2019). We hypothesize that the feasibility and the impact of the IW intervention would have been compromised without the support of ACMS leadership.

The IW intervention was first piloted among Mohs surgeons. Recently, our team has expanded the intervention to address other areas of overuse, underuse and misuse in medicine and surgery (ClinicalTrials.gov. 2019). Interestingly, one of our most significant barriers to spread of the IW intervention is our ability to engage specialty societies to endorse, design and co-deliver the intervention. We are optimistic that the positive feedback presented in this study may promote society engagement and provide an example of how specialty societies can take a leadership role in leading quality improvement initiatives among their members. We provide in Fig.1 a summary of key recommendations based on our experience that may be generalizable to other specialty societies interested in implementing similar interventions.

Fig. 1
figure 1

Summary of recommendations

Finally, this study is not without limitations. First, we were unable to successfully recruit our target sample size of outlier surgeons, despite significant recruitment efforts. Given that outliers are the primary target of the IW intervention, not having their voice fully represented presents a significant gap in knowledge, and engaging this population presents an area for additional research. We posit the low recruitment rate may be attributed to a limited population sample, which may not have been sufficient to recruit our target of n = 13. Our team contacted n = 24 outliers who received an IW report, meaning we required a > 50% recruitment rate to reach the n = 13 target. Additionally, it is possible that outliers did not agree with the data report findings. 75% of the outliers we interviewed suggested that geographic variation or case complexity were the factors influencing their increased Mohs utilization. Further integration of Mohs appropriate use criteria into the data metrics may result in increased buy-in, particularly among outlier surgeons. Second, it is likely that our sample population, namely Mohs surgeons who are members of a leading specialty society and who have attended the ACMS annual meeting, represent a sample of physicians who are potentially more supportive of society-led quality improvement interventions. We are currently in the process of conducting another IW trial that will deliver feedback interventions to both specialty society members and non-members. Collation of participant feedback to this iteration will provide an interesting comparison to this study.

Conclusions

The majority of study participants supported the Improving Wisely peer-to-peer, audit and feedback intervention; participants were supportive of receiving additional data reports in future. Participants provided informative feedback on the content and structure of data reports that should be incorporated in future iterations of the intervention. Six themes identifying factors impacting potential MMS overuse were identified; additional interventions can be used to address these factors. Specialty societies can play an important role to ensure appropriate use of surgical, clinical and diagnostic procedures. We present the template used to design and implement the Improving Wisely intervention as an example for societies who are interested in leading initiatives to improve appropriate use of medications, procedures or diagnostic testing among their members. Finally, this study demonstrates the value of conducting process evaluations for implementation research and supports the use of an integrated approach to involving the target user in implementation evaluation.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

ACMS:

American College of Mohs Surgery

IW:

Improving Wisely Intervention

MMS:

Mohs Micrographic Surgery

TDF:

Theoretical Domains Framework

References

  • Chen JT, Kempton SJ, Rao VK. The economics of skin Cancer: an analysis of Medicare payment data. Plast Reconstr Surg Glob Open. 2016;4(9):e868.

    Article  Google Scholar 

  • Krishnan A, Xu T, Hutfless S, Park A, et al. Outlier practice patterns in Mohs micrographic surgery: defining the problem and a proposed solution. JAMA Dermatol. 2017;153(6):565–70.

    Article  Google Scholar 

  • Moore GF, Audrey S, Barker M, Bond L, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.

    Article  Google Scholar 

  • Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.

    Article  Google Scholar 

  • Albertini JG, Wang P, Fahim C, Hutfless S, et al. Evaluation of a Peer-to-Peer Data Transparency Intervention for Mohs Micrographic Surgery Overuse. JAMA Dermatol. 2019;155:906–13 [Epub ahead of print].

    Article  Google Scholar 

  • Michie S, Prestwich A. Are interventions theory-based? Development of a theory coding scheme. Health Psychol. 2010;29:1–8.

    Article  Google Scholar 

  • Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7:37.

    Article  Google Scholar 

  • Paulhus DL. Measurement and control of response bias. In: Robinson JP, Shaver PR, Wrightsman LS, editors. Measures of social psychological attitudes, Vol. 1. Measures of personality and social psychological attitudes: San Diego, Academic Press 1991. p. 17–59.

  • Oleinik A. Mixing quantitative and qualitative content analysis: triangulation at work. Qual Quant. 2011;45:859–73.

    Article  Google Scholar 

  • Francis JJ, Johnston M, Robertson C, Glidewell L, et al. What is an adequate sample size? Operationalising data saturation for theory-driven interview studies. Psychol Health. 2010;25:1229–45.

    Article  Google Scholar 

  • Hoddinott SN, Bass MJ. The Dillman Total design survey method. Can Fam Physician. 1986;32:2366–8.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Chisnell D, Rubin J. Handbook of usability testing: how to plan, design and conduct effective tests. 2nd ed. New York: Wiley; 2008.

    Google Scholar 

  • Durlak JA, DuPre EP. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol. 2008 Jun;41(3–4):327–50.

    Article  Google Scholar 

  • NVivo qualitative data analysis software; QSR International Pty Ltd. Version 12, 2018

  • Lewis CC, Klasnja P, Powell BJ, Lyon AR, et al. From classification to causality: advancing understanding of mechanisms of change in implementation science. Front Public Health. 2018;6:136.

    Article  Google Scholar 

  • Limbani F, Goudge J, Joshi R, Maar MA, et al. Process evaluation in the field: global learnings from seven implementation research hypertension projects in low and middle-income countries. BMC Public Health. 2019;19:953.

    Article  Google Scholar 

  • Brehaut JC, Colquhoun HL, Eva KW, Carroll K, et al. Practice feedback interventions: 15 suggestions for optimizing effectiveness. Annals Intern Med. 2016;164(6):435–41.

    Article  Google Scholar 

  • Gagliardi AR, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci. 2015;11:38.

    Article  Google Scholar 

  • Graham ID, Kothari A, McCutcheon C, on behalf of the Integrated Knowledge Translation Research Network Project Leads. Moving knowledge into action for more effective practice, programmes and policy: protocol for a research programme on integrated knowledge translation. Implementation Sci. 2018;13:22.

    Article  Google Scholar 

  • Flodgren G, O'Brien, Parmelli E, Grimshaw JM. Local opinion leaders: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2019;6:CD000125.

    PubMed  Google Scholar 

  • ClinicalTrials.gov. Bethesda (MD): National Library of Medicine (US). 2019 February 27. Identifier NCT03857100, Improving Wisely: An Audit and Feedback Intervention to Improve High Risk Prescribing Among Primary Care Physicians – A randomized Controlled Trial. Available from: https://clinicaltrials.gov/ct2/show/NCT03857100.

Download references

Acknowledgements

Not applicable.

Funding

The Improving Wisely intervention was funded by the Robert Wood Johnson Foundation. This study was not funded and the Robert Wood Johnson Foundation had no role in the design and conduct of the study, collection, management, analysis, interpretation of the data, preparation, review or approval of the manuscript or decision to submit the manuscript for publication.

Author information

Authors and Affiliations

Authors

Contributions

CF, WEB, MAM, JGA designed the study and materials. CF collected all study data. CF and WEB analyzed and interpreted the data. CF, WEB, MAM, JGA were major contributors in writing the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Christine Fahim.

Ethics declarations

Ethics approval and consent to participate

This study qualified as exempt research under the Johns Hopkins Office of Human Subject Research Institutional Review Board (acknowledged July 6, 2018). All participants provided verbal consent to participate in key informant interviews. This study was considered ‘low-risk’ and was thereby approved for oral consent to participate. The study information sheet was presented to the participants by the interviewer (CF), the tape recorder was then turned on and used to document verbal consent.

Consent for publication

Not applicable.

Competing interests

CF, WEB and MAM have no conflicts of interest. JGA has received personal fees from Novascan for serving on a scientific advisory board and serves on the board of directors, as Chief Medical Officer and is a shareholder of QualDerm Partners.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

Survey

Additional file 2.

Interview Guide

Additional file 3.

Open-ended Survey Data

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fahim, C., Bruhn, W.E., Albertini, J.G. et al. A process evaluation of the improving wisely intervention: a peer-to-peer data intervention to reduce overuse in surgery. BMC Health Serv Res 21, 100 (2021). https://doi.org/10.1186/s12913-020-06017-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12913-020-06017-4

Keywords