Table 2 Evaluation and results found in the included literature

Chinman

• The trial assessed three sets of variables: quality of performance in conducting key programming practices (e.g., goal setting, planning, evaluation), fidelity of MPC (e.g., adherence, classroom delivery, dosage), and the sexual health outcomes of participating middle school youth.

• In typical community-based settings, manuals, and training common to structured EBPs may be sufficient to yield low performance levels and moderate fidelity levels, but that more systematic implementation support is needed to achieve high levels of performance and fidelity.

• In each of the 2 years, BGC sites that received MPC training plus GTO (intervention group) were found to have higher ratings of performance than sites just receiving MPC training (control group). Regarding the adherence dimension of fidelity, in year 1, sites receiving GTO were observed to have fewer instances where they did not conduct an activity of the MPC program at all compared to sites without GTO. However, both groups of sites implemented MPC activities fully only approximately half the time (55–57%). In year 2, the intervention group significantly improved their adherence, implementing MPC activities fully 92% of the time, while the control group remained similar to year 1 (55%). Overall, the second year showed more GTO impact.

Redmond

• To evaluate the effectiveness of WellConnect programs, the research team developed a secure, web-based platform that enabled WellConnect EBP leaders to enter key data elements. Each participant’s name and date of birth was used to locate their medical records from the REP.

• Of the 737 EBP workshop signups researchers were able to link medical records from the REP for 572 (77.6%) cases. There were no statistically significant differences in health outcomes between WellConnect program participants and matched controls. Falls prevention EBP participants demonstrated a 34% decreased likelihood of being admitted to the ED or hospital at 1 year of follow-up and chronic disease/pain management EBP participants demonstrated a 19% decreased likelihood compared to matched controls. This is similar to published reductions in the likelihood of ED (32%) and hospital (28%) admittance found for chronic disease management program participants.

Williams

• In-depth key-informant interviews.

• Staff from six recipients reported increased knowledge of local communities and audiences, improved efficacy, and skills to conduct media interviews, enhanced capacity to identify and train champions and influencers, and greater community partner investments. With marketing support, 90% of recipients reported increased enrolment, of which 40% exceeded self-set targets and another 40% doubled or tripled their enrolment numbers.

Goldzweig

• Process evaluation included characterization of MSFA’s involvement in each state as high (including all states selected as primary targets); low (defined as an average of less than one week (40 hours) of effort per year for the 7.42 years of evaluation or less than 297 hours); or intermediate.

• From January 2003 to May 2010, passage of primary legislation was 4.5 times as likely (95% CI 1.90, 10.68) in states with high versus low alliance involvement.

Cloke

• Conducted Focus Groups following the final workshop sessions. We explored our facilitated interactive group learning approach to enhancing the equity-sensitive evaluation of local healthcare services. To do this we asked the following: (1) What was the experience of participants of the PPP and the CIGs? (2) How was knowledge mobilisation achieved? What, why and by whom? (3) What are the key elements that enhance the process of coproducing equity-sensitive evaluations?

• Four themes were identified to illustrate how the CIG approach to delivering intensive and facilitated training supported equity-sensitive evaluations of local healthcare services that informed local decision-making by (1) creating the setting (2); establishing a common purpose (3); making connections and (4) challenging and transforming the role of evaluation.

Morais

• The facilitation team designed the evaluation and 12 months after the workshop the team reached out to participants to take part in interviews. The assessment’s goal was to elicit participants’ perceptions of the workshop’s influence on their thinking one year after the activity. Analysis of their responses provides a lens through which to view the potential contributions of CBSD as a tool for facilitating the link between knowledge production and policy-making and implementation.

• Qualitative analysis revealed five major themes in participants’ responses that indicate how the workshop experience influenced their understanding of food and transport systems and enabled their ability to incorporate this understanding into practice to support policy processes: (1) stakeholder engagement, (2) shared language and understanding of the problem, (3) interconnections, (4) dialogue across sectors, (5) use of systems thinking.

Cooke

• A mixed method evaluation of the OPEN GVP activities was conducted. Data sources included surveys and qualitative interviews of OPEN members 18 months after OPEN’s launch, an OPEN member end-of-grant survey, and GVP team meeting notes and members’ critical reflections with respect to the creation and implementation of the GVP team model.

• The 2016 end-of-project survey of OPEN teams collected data on how the OPEN teams had used GVP resources, as well as how sex, gender and vulnerable populations had been included in their research activities; six out of six teams responded. All six OPEN project teams reported using the GVP team’s recommended survey questions in their research. Four project teams had used the online sex, gender, and intersectionality learning module. Two project teams reported using individual consultations with the GVP team, however, other project teams reported that arranging formal consultations with the GVP team was not necessary as they had a team member who was part of the GVP team and able to provide the necessary support. None of the OPEN teams mentioned using the modified HEIA tool in their research.

Springs

• The project evaluation included an online survey and debriefing sessions after each phase of the process and as indicated by the CRPs. The survey measured indicators of engagement in research as described in the Patient-Centered Outcomes Research Institute (PCORI) Engagement Strategy Rubric (i.e., reciprocal relationships, partnership, co-learning, transparency, honesty, and trust).

• The survey revealed that the approach to evidence synthesis measured high on all aspects of engagement. Participation also improved CRPs confidence in engaging with the health care system, developed greater empathy, and understanding of others in the community and increased interest in using science or research in advocacy efforts. The researchers measured the tangible skills developed as part of the training, with most participants indicating that they felt confident in their abilities to develop a research question, search the medical literature, read a journal article, and identify a population, intervention, or outcome in a journal article.

Maar

• Participants were invited to contribute through semi-structured telephone interviews using open-ended questions to discuss their perspectives and experiences about the program. Three different, but complementary, interview guides were developed, interviewing 1) Community contact, 2) Medical students, and 3) Faculty Supervisors.

• The research showed that the CETR program had the potential to maintain positive trust-based relationships between medical learners and Indigenous communities. Furthermore, medical students experienced the importance of relationship building in Indigenous research. Yet, to ensure the sustainability of the CETR program several strategies are required including, (1) Formalize the supervisory relationship between faculty and student, (2) Initiate ethics application early, (3) Identify sustainable funding, (4) Track and evaluate research output.

Stajic

• At the end of each Masterclass, participants were encouraged to complete a short paper-based evaluation form.

• A formal evaluation to answer the research question ‘Can a short educational intervention strengthen research capacity in the Aboriginal Community-Controlled Organisation sector?’ was undertaken to better understand the reach and impact of these Masterclasses.

• This evaluation found that the foundation-level Masterclass, ‘Understanding Research’, was requested most often by the ACCHO sector, providing an indication of the sector’s current research literacy needs and that the priority of health services is service delivery and not undertaking or leading research. Most survey and interview participants had established careers in Indigenous health, suggesting that their developed research capacity is likely to be sustained in the sector. Participants described a developed understanding and confidence in research, leading to increased willingness to participate in research and, importantly, a greater sense of empowerment in interactions with external research partners.

Pullyblank

• RE-AIM framework was helpful in evaluating public health interventions, including dissemination and scaling of projects in a real-world setting. “Reach” was assessed using survey data as well as the REDCap database. Patient characteristics were used descriptively to assess Reach. “Effectiveness” was evaluated through completion status. “Adoption” was assessed from internal documents to summarize engagement of partner organizations and peer leaders. We evaluated “Implementation” using internal documents to assess fidelity, as well as capturing the Plan-Do-Study-Act (PDSA) changes that occurred throughout the implementation process. “Maintenance” included sustainable strategies we implemented which were gleaned from reviewing internal documents.

• REACH: 474 individuals enrolled in a DSMP workshop (34 workshops offered), and 306 individuals enrolled in CDSMP workshop (29 workshops offered).

• EFFECTIVENESS: Completion percentages among those enrolled in the DSMP or CDSMP were 74.7 and 79.4%, respectively.

• ADOPTION: A total of 617 referrals to the program had been made by clinicians within the health care network between 2017 and 2019, with 15% of enrolees recruited through the EHR in 2019.

• IMPLEMENTATION: By the end of 2019, participants could learn about the program through traditional media, social media, bulk communication sent through the patient portal, their provider, a call from Living Well, or from partner organizations.

• MAINTENANCE: Living Well continuously sought funding to sustain the program until a payment model could be implemented either by the health care system or third-party payers.

Sharma

• Cross-sectional quantitative survey of 140 village health sanitation and nutrition committees (VHSNCs) designed to assess six parameters of VHSNCs, including their formation, composition, meeting frequency, activities, supervisory mechanisms, and funds’ receipt and expenditures.

• Researchers developed a semi-structured questionnaire in the local language consisting of questions related to each of the six parameters, questions on supervisory mechanisms (visits by officials, e.g., medical officers and program managers, and visit frequency), funds’ receipt and expenditures. Additionally, we asked about the role of accredited social health activists (ASHA) in the VHSNC meetings, and the issues and challenges faced.

• The number of members in VHSNCs in most of the studies, including our study, were in accordance with the guidelines, except Odisha. The VHSNC guidelines recommended the presence of representatives from the marginalized classes in the committee. However, most studies echoed that marginalized classes are frequently poorly represented or not taken seriously. This poor representation of marginalized classes in the meetings connotes the caste and power dynamics in villages. Unlike most of the studies, we reported that VHSNC meetings occurred regularly. In our study, most of the VHSNCs had a fixed date for holding the meetings which may help prompt the members to attend it regularly. Our study echoed the previous findings that funds were limited, and there was a delay in payments to VSHNCs. We found inadequate supportive supervision and monitoring visits by the government officials, (e.g., medical officers, child development project officer).

Buchwald

• One-on-one exit interviews with the scholars and mentors, as well as open-ended questions on surveys administered to scholars and representatives from community organization partners about their overall experience and satisfaction with the program. The quantitative evaluation was based on the results of the Clinical Research Appraisal Inventory (CRAI), augmented by a comprehensive module on PCOR and CER skills and competencies developed by PCORP faculty.

• Of the 22 scholars who completed the post-evaluation, 23% completed their projects before training ended, 64% were still working on it, and 9% did not plan to complete their projects. Some barriers identified to completing the project included institutional issues, EHR data extraction issues, patient attrition, Institutional Review Board issues, shifts in organizational priorities, change of roles, and lack of a home institution mentor due to turnover or role transitions. Overall, 32% of scholars reported that PCORP improved their skills significantly and 59% reported moderate improvement. PCORP’s overall usefulness to the scholars was rated as very good by 36%, good by 46%, and fair by 18%. Satisfaction with the overall experience in PCORP was rated as very satisfied by 32%, satisfied by 59%, and dissatisfied by 9%.

Bertone

• Analyses of online discussion forums.

• Performance Based Financing (PBF) is expanding rapidly in sub-Saharan Africa; while the PBF Community of Practice’s own contribution is difficult to ascertain, it has established itself as the main platform for knowledge exchange and development on PBF. Some early analyses of the online forum discussions have confirmed the focus on a specific policy domain, the collective sharing of a technical repertoire and the emergence of an identity and community spirit, all key features of a de facto CoP.

Driedger

• Participants were asked a series of questions to address perceptions around important aspects that modellers and public health practitioners faced during the 2009 H1N1 outbreaks regarding mathematical modelling and pandemic responses.

• Three main challenges to developing model-informed public health decisions emerged through the following themes: (1) models need to be relevant to public health priorities (2); clear communication and plain language about what models can (and cannot) do is needed; and (3) the importance of developing strong working relationships through collaboration and integration.

Guinaudie

ACCESS OM was assessing the innovative ways in which SDM strategies might foster effective integrated knowledge translation (IKT) in a youth mental health research setting. A working group with representatives from these two councils and each participating site was created to provide feedback on key outcome domains and measures that should form part of the ACCESS OM’s quantitative assessment protocol.

• The inclusion of representatives highlighted important criteria for choice of assessments tools (e.g., short tools with youth-friendly language, domains that go beyond symptoms), and item response options: the demographic questionnaire has an expanded set of options for sexual orientation and gender identity. Informal feedback from site service providers indicated that young service users across sites appreciated the range of options provided for these two questions and seen this as an indicator of ACCESS OM sites being “safe spaces”. A notable SDM activity involved youth and family/carers in working groups and consultations to influence the redesign of research consent forms, which had been reported as being too lengthy and difficult to understand.

Malcolm

• Post seminar face-to-face interviews were conducted individually with several participants who were asked to talk about the experience of completing the EBAT seminar. For each EBP, clinicians were asked about their knowledge in that particular evidence-based practice, their perceived skill in implementing that practice, and how often they use that practice when it is clinically appropriate.

• Looking across EBP, results suggest that clinicians consistently grew in their knowledge, skill, and ability to implement EBP when clinically appropriate.

Smith

• Publicly available policy materials were studied in both case studies, with a focus on contributions to the European Commission consultation on smokefree policies and the English consultation for the Marmot Review. Team attempted to investigate relationships among network members (‘network’ referring to organisations that provided a customised policy response). Semi-structured narrative interviews with politicians, researchers, advocacy groups, and other individuals involved in policy discussions related to each case study were used to collect qualitative data.

• The analysis suggested that the ways in which actors organised themselves to employ (and deploy) evidence in policy debates was crucial to understanding its impact.

• Case Study 1 illustrated how researchers, health professionals, advocates and policymakers could actively collaborate in public health policy development, with public health advocates drawing on scientific evidence to strengthen their arguments; researchers working with advocates to better understand the strategic policy context and provide the kinds of research required to advance policy goals; and policymakers seeking contact with researchers and advocates to develop policies which were backed by available evidence.

• Case Study 2, in contrast, demonstrated the difficulties of employing evidence in policy contexts for which advocates, and advocacy coalitions were lacking. In the context of a lack of leadership and an unfavourable political climate, no organised network emerged and levels of trust between those involved in trying to effect policy change was low.