Guideline evaluation and implementation mechanisms in school health services (GuideMe): protocol for a hybrid randomized factorial trial
BMC Health Services Research volume 23, Article number: 1259 (2023)
Norwegian school health services received a national best-practice guideline in 2017. To promote healthy life skills and identify adolescents needing support, the guideline includes strong recommendations for individual consultations with all 8th graders and increased collaboration with schools. To help implement the recommendations, a blended implementation strategy (SchoolHealth) was co-created with school nurses, students, and stakeholders. SchoolHealth consists of three implementation elements: Digital dialog and administration tool (audit and feedback +), Dialog support (external consultation), and Collaboration materials (targeted dissemination). This hybrid study will test the main and combined effects of the elements on guideline fidelity and effectiveness.
The GuideMe study is a factorial cluster randomized controlled trial examining SchoolHealth's effectiveness on guideline fidelity and guideline effectiveness goals. Forty Norwegian secondary schools will be randomized to eight different combinations of the elements in SchoolHealth. Participants will include school nurses and school personnel from these schools, and 8th grade students (n = 1200). Primary outcomes are school nurses' fidelity to the guidelines and student's ability to cope with their life (i.e., health literacy, positive health behaviors and self-efficacy). Quantitative methods will be used to test effects and mechanisms, while mixed- and qualitative methods will be used to explore mechanisms, experiences, and other phenomena in depth. Participants will complete digital questionnaires at the start and end of the schoolyear, and after the consultation during the schoolyear. The study will run in two waves, each lasting for one school year. The multifactorial design allows testing of interactions and main effects due to equal distribution of all factors within each main effect. Sustainment and scale-up of optimized SchoolHealth elements using national infrastructure are simultaneously prepared.
The study will investigate possible effects of the implementation elements in isolation and in combination, and hypothesized implementation mechanisms. In-depth study of user experiences will inform improvements to elements in SchoolHealth. The results will yield causal knowledge about implementation strategies and the mechanisms through which they assert effects. Mixed-methods will provide insights into how and when the elements work. Optimizing guideline implementation elements can support adolescents in a crucial life phase.
ISRCTN24173836. Registration date 8 August 2022.
The Norwegian Directorate of Health launched a new national guideline for school health services in 2017. The guideline's aim is to promote service quality and sustainability, with less unwanted variation in practices and more coherent service pathways for students. The guideline strongly recommends individual health-promoting consultations with all 8th graders, aiming to improve the students' ability to cope with life and to thrive by increasing health literacy, promoting positive health behaviors and self-efficacy, and identifying students needing follow-up. The recommendations emphasize empowering students in consultations and focusing on their needs. The guideline also strongly recommends interprofessional collaboration with schools to promote quantity and quality of care, and increase student attendance . Although the guidelines are based on evidence and professional consensus , the effects of adhering to the guidelines have not been evaluated.
Adolescence is a crucial phase in which future life opportunities and patterns of adult health develop . Therefore, adolescence is important for concurrent and prospective well-being and the economic development of nations . Attending secondary school is free and obligatory in Nordic countries. Thus, schools provide opportunities to promote positive relationships, healthy behaviors, and resilience to cope with stressful events regardless of social background. The school health service is a mandatory part of the municipal health services in Norway. They are located at schools, free of cost for all students, and have health promotion and prevention as core aims .
The guideline recommendations are professionally normative. Any service choosing to deviate must document and justify their choice. However, the guideline is not explicit regarding how the school health services should implement the recommendations and reach their intended goals.
Implementation of national guidelines is a struggle across public service sectors . Successful implementation and sustainment rely upon effective strategies appropriately addressing key implementation determinants and mechanisms across service levels [7, 8]. These mechanisms may be caused by dynamic connections between different elements of implementation (e.g., discrete implementation strategies, processes, and contextual circumstances; ), the guideline being implemented (e.g., their compatibility and relevance for practice), and the people doing and receiving implementation (e.g., the self-efficacy and capacities of practitioners). Empirical evidence about the most effective and efficient implementation strategies is scarce . Also, implementation strategies are typically evaluated in packages of several discrete strategies, such as multi-element and blended strategies [9, 10]. Thus, it remains uncertain what different discrete strategies and elements contribute to effectiveness, how they contribute, and which are likely superfluous .
Through a human-centered co-creation approach, we developed a guideline implementation tool called SchoolHealth. The first version, inspired by a Danish equivalent named BørnUngeLiv.dk, has been found feasible and user-friendly in pilot testing . Subsequently, SchoolHealth has been improved based on pilot results and re-designed into three elements representing discrete implementation strategies: (1) Digital dialog and administration tool (audit and feedback +), (2) Dialog support (external consultation), and (3) Collaboration materials (targeted dissemination). The elements represent complementary implementation strategies tailored to facilitate the implementation of the guideline with fidelity and help services reach the guidelines' intended goals. An important aspect of achieving the guideline goals is ensuring appropriate user pathways for adolescents in health services. However, how adolescents with health vulnerabilities are handled in the healthcare system is largely unknown , including the role of school health services in identifying follow-up needs.
The current study
The overall objectives of the GuideMe study are to help the school health services implement the guideline recommendations and reach their goals, and simultaneously increase scientific knowledge about effective implementation strategies and health service use among students.
We will conduct a hybrid cluster randomized factorial experiment to evaluate and optimize the effectiveness of SchoolHealth. Quantitative-, qualitative-, and mixed- methods will be used to evaluate the main and combined effects of the three implementation elements on fidelity to the guideline, school and student outcomes (guideline goals), and investigate mechanisms of change and user experiences. Baseline data will be complemented with epidemiological studies and registry data to study students’ health service use in Norway. Additionally, we will prepare for system-wide scale-up of the optimized version of SchoolHealth by developing solution designs for national infrastructure.
The study will investigate the following research questions:
What are the main and combined effects of the implementation elements in SchoolHealth on fidelity to the guideline recommendations for:
The individual 8th-grade consultations with students.
School health services collaboration with schools.
i) What are the main and combined effects of the implementation elements in SchoolHealth on:
Identification of vulnerable students in need of follow-up.
Students' health literacy, health behaviors, self-efficacy, quality of life, school environment, and attendance?
Students' involvement in the 8th-grade consultations?
ii) How are effects associated with school nurses' fidelity to recommendations?
i) What are the main and combined effects of the implementation elements in SchoolHealth on:
School nurses' work-related self-efficacy and relation with students?
ii) How are effects associated with school nurses' fidelity to recommendations?
Through what mechanisms do implementation elements assert their influence on implementation outcomes, and how?
How do individual and contextual implementation determinants influence fidelity and effects?
What are the participants' experiences with SchoolHealth?
School nurses' experiences with the elements in SchoolHealth, 8th-grade consultations, and collaboration with schools.
Teachers' experiences with interprofessional collaboration.
Students' experiences with the 8th-grade consultation and perspectives on health literacy and quality of life.
Experiences in Norway compared to the Danish equivalent.
What are the associations between self-reported health status in adolescence and user pathways in health- and welfare services?
The study setting is Norwegian lower-secondary schools and school health services. The Norwegian school system is mainly public. The first ten years are compulsory, and all who have completed compulsory schooling are granted the right to three to four years of free upper-secondary education.
The school health service in Norway is part of the primary municipal health care services. The service aims to promote good health and prevent disease. They work on individual, group, and universal levels. The resources and structures of the services, as well as what they offer, vary substantially between municipalities [13, 14].
This multicenter study will collect data from 8th-grade students, school nurses, their leaders, and school personnel from several municipalities in southeast and central Norway, representing both rural and urban areas. Participating schools choose which 8th-grade classes (1–3 classes) to include. All students in the participating classes will then be invited. The data-collection period will last for two school years (2022/23 and 2023/24, see Fig. 1).
Intervention: evidence-based guideline
8th grade consultations
The recommendations for individual consultations with 8th graders are built on legislation, evidence, and professional consensus . The consultation should be health-promoting based on the students' needs, include weighing and height measuring, and address topics related to health behaviors (sleep, diet, physical activity), physical and mental health, social relationships, family, sexuality, dental health, drugs, violence, abuse, and neglect. The estimated consultation timeframe is 30 min. Before the consultation, the school nurses should familiarize themselves with the students' health records and must document the consultation in this record. The school nurse is expected to conduct follow-up consultations, initiate interprofessional collaboration, or refer students to other professionals when necessary.
Collaboration between school health services and schools
The guideline encompasses twelve recommendations on collaboration between school health services and schools, all marked as strong recommendations or legislative requirements . The recommendations include system-oriented collaboration, monitoring students' health status, contributing to health education in groups and school classes, facilitating visits to adolescent health centers, providing health information in parent meetings, and follow-up of students' school absences. The collaboration should be systematically planned and organized.
Table 1 specifies the experimental implementation elements per recommendations for reporting implementation strategies using the Template for Intervention Description and Replication . The following describes SchoolHealth and the content of each strategy and its target functions.
Audit and feedback + (Digital Feedback and Administration Tool [DFA])
DFA is the only element that involves the students directly. School nurses/teachers administer a digital health information form to students before their 8th−grade consultation. The topics in the form are based on the recommended topics in the guideline. Filling out a health form may empower students through preparing topics they can bring up in the consultation and the opportunity to reflect upon topics important to them. The school nurse will receive an individual feedback report, with the main aim to support school nurses in tailoring consultations to individual needs, thus increasing nurses' capability, opportunity, and motivation to use the 8th-grade consultation recommendations. It may also help identify students in need of additional support and follow-up. After the 8th-grade consultation, the students and the school nurses answer questions regarding the consultation. The students report user satisfaction, including relation with the school nurse and involvement in consultation, generating the user satisfaction report. The school nurse and the students answer questions about the content of the consultation (fidelity to guideline), generating the 8th-grade consultation report. Both reports will be available for the school nurse and their service leaders on the DFA platform. Additionally, the DFA platform include a school report, an aggregated summary based on all the students' pre-consultation health information forms. Here, school health nurses and leaders can compare their schools/districts aggregated answers with that of others in the study. The school report covers topics highlighted in the guideline recommendations on which the school health service and schools should collaborate.
Ongoing consultation (Dialog Support)
Dialog Support aims to increase school nurses' capability, opportunity, and motivation to adhere to the guideline recommendation for 8th-grade consultations with fidelity and to increase their self-efficacy toward this consultation. The strategy includes an e-learning module and ongoing consultations. The e-learning has two main sections on theory and instructional videos about conducting health-promoting 8th-grade consultations. The first section provides health-promoting theories (salutogenesis, health literacy, empowerment, and user participation). The next section encompasses communication (starting, conducting, and ending conversations, ). The ongoing consultations will include training in conducting and reflections about the 8th-grade consultations. The consultations will be modeled after the Reflexive teams approach , aiming to provide a space for case reflection, problem-solving, and collegial support. They will be conducted in-person and digitally (hybrid format) and consist of four meetings.
Targeted dissemination (Collaboration Materials)
Collaboration Materials is the only element that involves school personnel directly. The element is labeled an active dissemination strategy because a web-based package with educational and organizational materials and resources is actively provided to teachers, head teachers, and school nurses. The main aim of the dissemination is to improve and structure interprofessional collaboration.
The element consists of four digital modules with short educational videos, reflection tasks, and a final summary module. The element will be introduced at a kick-off meeting, and participants will be encouraged to plan how to implement the modules during one school year. The project staff does not engage in this.
Each module is estimated to last approximately 60 min, and the topics are based on what is considered important to promote interprofessional collaboration and support . The topics are: (1) Conditions for systematic and interprofessional collaboration at different levels of intervention (2), Overlapping topics in the National curriculum for schools and the Guideline for school health services, (3) Available resources and supports for evidence-based interventions and utilization of existing data, and (4) Interprofessional communication – identification of barriers and facilitators, (5) exchange experiences and plan further collaboration.
The study is a hybrid type 2 trial, studying both fidelity to guidelines and guideline effects using: (i) a cluster randomized factorial experiment, ii) hermeneutic phenomenological qualitative methods, and (iii) convergent and sequential mixed-methods. To study health service use in Norwegian adolescent, survey data will be linked with national registers (iv).
The randomized factorial experiment (i)
We will employ a stratified, randomized cluster factorial design to evaluate the effects of the three implementation elements separately and in different combinations. The schools will be randomized to one of eight different experimental conditions.
The factors in Table 2 reflect the three previously described elements. The three elements are complementary, and together (yes/yes/yes) represent a blended implementation strategy that, in theory, should elicit the strongest outcomes based on an additivity or ecology principle (i.e., effects of implementation strategies are the sum of their parts or more than the sum of their parts, . However, these implementation elements have rarely been evaluated independently or together. The multifactorial design allows for the testing of interactions and main effects due to the equal distribution of all factors within each main effect.
Statistical methods (i)
Primarily, linear mixed-effects models will be used to investigate the implementation elements' main- and interaction effects on fidelity to the guideline recommendations and guideline goals. In addition, a stepwise theory-informed strategy will be used to explore the effect of implementation determinants.
To investigate psychometric properties, exploratory and confirmatory factor analyses will be performed on instruments with sufficient respondents (primarily instruments administered to students). For all instruments, correlations between subscales will be computed using Pearson's r and other relevant statistics. Internal consistency for the scales and subscales will be investigated using Cronbach's alpha and other relevant statistics.
Hermeneutic phenomenological qualitative methods (ii)
We will use a hermeneutic-phenomenological qualitative approach  to explore the experiences of students, school nurses, and school personnel. We use hermeneutic phenomenology to explore and interpret phenomena as understood and formulated by the participants [20, 21].
Qualitative individual and focus group interviews will be used to gain in-depth knowledge of central phenomena and user experiences [21, 22]. Qualitative analyses of meaning content will be carried out by informed models for qualitative analyses as described by e.g., Kvale and Brinkmann , Van Manen , and Braun and Clarke .
All interviews will be semi-structured. Guides for individual interviews with students and school nurses contain 4–5 predetermined themes with follow-up questions, and permit an open dialog. Students are interviewed about their experiences with the 8th-grade consultation, relation to the school nurse, health literacy, quality of life, and coping. School nurses' individual interviews touch upon their experiences with coping during the 8th-grade consultation, student relations, and how to identify and follow up students with additional needs.
The guide for focus group interviews with school nurses is made with the opportunity to add themes related to implementation after preliminary quantitative data analyses (see mixed-methods). The complete interview guide covers themes such as experiences with the 8th-grade consultation, collaboration with school, national guidelines, implementation of the elements, and determinants for implementation. School personnel are interviewed on three main topics regarding the nature, content, and quality of collaboration with school health services. In addition, they are asked about experiences with implementing SchoolHealth and relevant implementation determinants.
A mixed-methods experimental design (convergent and sequential ), from a pluralistic and meta-paradigmatic perspective , will be used to investigate experiences with the different implementation elements and the complexity of implementation mechanisms across conditions. We will corroborate quantitative and qualitative data on the value of, and experiences with, the implementation elements and the guideline recommendations. Qualitative data will provide a more in-depth understanding of findings from different viewpoints. According to Teddlie and Tashakkori , mixed methods research involves seeing qualitative and quantitative data as two different ends of a continuum, where one moves seamlessly across it to pursue optimal answers to the different research questions of the study. This means that the different data sources will be given different weights and priorities throughout the analytical process to best answer the research question. Thus, the sequential dimension of the design will include conducting preliminary analyses of quantitative data about implementation determinants, fidelity, and collaboration after each data collection wave to inform themes and questions for the qualitative interviews.
Student data and national data registers (iv)
We will compare baseline data from GuideMe on health and health service use with cross-sectional data from wave 4 of the adolescent part of the Nord-Trøndelag Health Study (Young-HUNT Study) , a Norwegian population-based study in Central Norway [28, 29]. In the Young-HUNT wave 4 survey there were 8066 (76% of the invited) 13–19-year-old participants. Data from GuideMe and Young-HUNT4 will also be linked with national registry data on healthcare service use, such as The Norwegian Registry for Primary Health Care (KPR) and the Norwegian Patient Registry (NPR) .
Health and health service use among participants in both studies will be assessed and compared. Subgroup analyses will be conducted to test whether demographics (e.g., gender, socioeconomic status, schools) affect associations between health measures (e.g., mental health, medical conditions, health behavior).
The study uses the Exploration, Preparation, Implementation, and Sustainment Framework (EPIS)  as a theoretical framework for investigating implementation as multilevel processes influenced by innovation factors, the outer and inner implementation context, and the interplay between factors (i.e., bridging factors) across four phases of implementation . Due to the highly autonomous and individual nature of school nurses' practice settings, we complement EPIS with the Capabilities, Opportunities, and Motivation model of Behavior change (COM-B) ) to inform explorations of how and why implementation strategies influence school nurses' fidelity to guidelines. EPIS and COM-B have informed the development of studies' theories of change and hypotheses for how the implementation strategies influence fidelity to guidelines and guideline effects (see Fig. 2 for logic model). EPIS has informed measurements of organizational and individual-level implementation determinants and focus group interviews. COM-B has informed measures of individual-level determinants of behavior change and individual interviews.
The outcomes are operationalized and described in Table 3 by measurements, data collection method, informant, and timepoint. Details about the measurement instruments, including psychometric properties and validations, are in Supplementary file 1.
Implementation outcomes (proximal)
Fidelity to guidelines will be assessed through items encompassing the three constructs adherence to guidelines, adaptations to guidelines, and quality in using guidelines. Adherence to guidelines is conceptualized as adhering to specific key recommendations for how to carry out the 8th-grade consultation and collaboration with schools. Measurement of 8th-grade consultation adherence include items about whether health information was adapted to the student, and focus on habits important to promote good health. Additionally, checklist items about themes addressed and registering height and weight also index adherence. Collaboration adherence is measured with questions about how the collaboration is organized (formal and informal meetings), whether school nurses participated in any of the schools' planning hours or meetings, and topics on which the school and school nurses are supposed to collaborate and how they collaborate.
The dosage of 8th grade consultation will be measured by the time used on the consultation, and the dosage of collaboration by the number of scheduled meetings.
Quality in using the 8th-grade consultation guidelines is indexed by post-consultation measures of school nurses’ and students’ perceptions of their alliance and achievement of the guidelines' core functions, such as empowerment, reinforcement of positive health behavior, and identification of follow-up needs. Quality in using collaboration guidelines is indexed by measuring perceptions of achievement of the core functions of collaboration guidelines, such as common values and understanding, role and responsibility clarification, ease of contact with each other, knowledge about each other's competence and regulations, mutual respect, and structure.
School nurses report adaptations to consultation through open-ended questions in the post-consultation questionnaire (T2). Adaptations will be retrospectively coded by using the Framework for Reporting Adaptations and Modifications to Evidence-based Implementation Strategies , labeled fidelity-consistent (positive) or fidelity-inconsistent (negative). The labeling will be based on a qualitative judgment of whether the adaptation was likely to maintain the core function of the recommendation in our theory of change (fidelity-consistent) or not . The qualitative judgment will also be informed by the measures of quality. Qualitative interviews with school nurses and school personnel will explore adaptation to collaboration guidelines.
Health and service outcomes (medial)
Identification of vulnerable students in need of follow-up will be captured qualitatively and assessed quantitatively through school nurses' evaluation of students' physical, psychological, and social functioning, registration of follow-up group, the number of follow-ups during the school year, and the student's self-reported mental health [The Strength and Difficulties Questionnaire (SDQ)] .
Students' health outcomes will be assessed by somatic symptoms (The Children’s Somatic Symptoms Inventory) , quality of life (Kidscreen-27) [41, 42], general self-efficacy [General Self-Efficacy scale (GSE-5)] [43, 44], and health literacy (Health Literacy for School-Aged Children)  at the start (T1) and end of the school year (T3). Health literacy and self-efficacy will also be measured post-consultation (T2), and health literacy will be explored qualitatively. Students will assess their health behaviors through items on behaviors of sleep, physical activity, nutrition, and screen time activities.
Students' assessments of School environment and attendance will be measured through a mix of self-developed questions and questions used in similar studies (, see Supplementary file 1 for details).
User satisfaction is an overall assessment of students' experiences (qualitatively) and degree of user satisfaction and empowerment in consultation (quantitatively). It includes items of involvement in consultations (like being heard and talking about what matters to them) and student-school nurse alliance, informed by both students and school nurses. The items are partly self-developed, inspired by similar scales [47, 48].
School nurses and school personnel will assess interprofessional collaboration between the school and school nurse . School nurses will complete an assessment on their work-related self-efficacy using an adjusted version  of the GSE-5 .
Implementation determinants will be measured to investigate their influence on fidelity to guidelines and guideline effects. These include school nurses' and leaders' assessments of implementation climate (Implementation Climate Scale) [36,37,38, 51], implementation leadership (Implementation leadership Scale) [52, 53], implementability of guidelines (Feasibility of Intervention Measure, Acceptability of Intervention Measure, Intervention Appropriateness Measure) , fidelity to implementation elements, implementation capacity (qualitative interviews), and school nurses' work-related self-efficacy .
Background variables will be collected from school nurses and teachers regarding age, gender, education, and years of work experience. Additionally, context characteristics will be assessed by school nurses. Students' assessment of demographics includes items on socioeconomic status, gender, and ethnicity. Other student-determinants will be assessed by mental health (SDQ) , self-efficacy [44, 55], and user satisfaction as described under Health and Service Outcomes.
Health data and linkage with national registers
Student questionnaires in GuideMe and the Young-HUNT4 Survey cover overlapping topics and identical instruments, subscales, or items. Both include for example the SDQ, items about general health and quality of life, health care use, and health behavior.
From the national registers, data on socioeconomic status, along with use of the school health services (KPR), general practitioners (KPR), physiotherapists (KPR), and specialized healthcare services (including psychiatric care) (NPR) will be linked to GuideMe data.
The schools and school health services will be invited mainly through a convenience sampling approach.
School health services
Recruitment of school health services will be done through oral and written information and meetings with the leaders of the services. Additionally, written information will be provided to administrative leaders of the local municipalities.
Two different approaches will be used to invite schools: (1) After the school health services have agreed to participate or (2) simultaneously. A brief description of the study will be sent to the school leaders, with an invitation to attend an information meeting. The study will then be presented to the school principals and the school health services in each location. The interested schools will be asked to nominate a key contact person. School health services and schools agreeing to participate sign a cooperation agreement.
8th grade students
Students in the participating classes and their parents will be introduced to the study via class visits by school nurses and parent meetings. The schools will provide parents with written information and a link to a digital informed consent form, including a voluntary option for providing the second parents' e-mail so that s/he can get information that the parent has consented to the student participating in the study. For students to participate in the study, at least one of the parents must complete an electronic consent form. The students will be given age-appropriate written and animated information at school. The students will consent to participate by filling in the questionnaire. A project webpage (https://guideme.rbup.no/en) is developed to enhance communication with all participants.
The recruitment of participants will be reported per the Consolidated Standards of Reporting Trials (CONSORT) guidelines for clustered randomized trials.
Inclusion and exclusion criteria
Inclusion criteria are students who agree to participate, have informed consent from one of their parents, and are able to answer the web-based questionnaires.
The main exclusion criteria are intellectual disability or language problems, defined as not being able to complete the questionnaires. In addition, long-term school absenteeism may also be an exclusion criterion but will be considered individually. The reasons for exclusion will be documented in the CONSORT flowchart.
Randomization in the factorial experiment (i)
The schools will be randomly assigned to test different combinations of the three implementation elements in SchoolHealth. The school randomization procedure will be carried out in R using a function specifically written for the GuideMe study. The function is developed by a statistician in collaboration with key personnel in the project and will be witnessed by an objective third party. The schools will be randomized to one of the eight experimental conditions (see Table 2).
Power analysis and sample size in the factorial experiment (i)
An R-package called MOST developed for power analyses in factorial trials will be used (see supplementary file 2 for R-script). When conducting a factorial trial, one option for specifying effect size for power calculation is deciding the smallest effect of practical interest . This can be decided using Cohen's rule of thumb .
We selected the following statistical attributes: α = 0.05, an effect size of d = 0.30, and statistical power of 0.80 (β = 0.20). Being a cluster trial, the design effect may affect our power calculation. Thus, an intraclass correlation coefficient (ICC) of 0.05 and an average size of clusters = 30 (SD = 15) was also accounted for . The results from the power calculations indicated that 36 schools and 1080 students were needed in the study. To account for possible dropout and the need for subgroup analyses, we aim to recruit approximately 40 schools and 1200 students.
Participants in qualitative interviews (ii and iii)
The qualitative data will be collected in both waves (Fig. 1). We will conduct individual interviews with 24 students and 12 school nurses, and focus group interviews with 12–24 school nurses, 12–24 teachers, and 6–12 school leaders. Variations in the experimental condition and geographic region will be emphasized when inviting participants to facilitate representativeness. The selection of students for qualitative interviews will be stratified . When schools are selected, school nurses will provide names for students that fit pre-defined criteria regarding gender (boys/girls), quality of conversation in 8th-grade consultation (good/difficult), and cultural background (Norwegian/second culture).
School nurses, school personnel, and school leaders will be recruited through purposive availability sampling, emphasizing the participants` ability to elucidate a specific theme . All participating nurses will be invited due to the limited number of participants and the large number of conditions. School personnel and leaders will be recruited to ensure representativeness to different experimental conditions, particularly element 3, Collaboration materials, due to their active role in this condition.
In Denmark, interviews with school nurses and teams implementing BørnUngeLiv.dk will be conducted. The main aim is to compare SchoolHealth with the Danish equivalent.
All interviews will be digitally audio-recorded and transcribed verbatim.
Implementation of schoolhealth
We will monitor implementation quality by measuring implementation fidelity to ensure validity in experimental conditions. We conceptualize implementation fidelity similarly to guideline fidelity . Measures of implementation fidelity are designed to index whether implementation in each condition is conducted as planned (e.g., content, structure, dosage, materials, absentees, turnover), whether any adaptations are fidelity consistent (done to maintain core functions in our theory of change) or fidelity inconsistent (drifting away in a manner unlikely to maintain core functions), and whether proximal functions of the implementation (e.g., increased self-efficacy related to using guidelines).
Measures of fidelity to implementation elements
To index fidelity to implementation elements, school nurses and school personnel will answer questions about the completion and quality of each element:
Adherence and adaptations will be assessed using questions at T3 about training and support received during the study. Satisfaction will be assessed by asking how satisfied s/he was with the elements in SchoolHealth, and whether s/he would recommend them to a colleague. To assess functions, we will analyze the change in self-efficacy and collaboration adherence from pre to post. The school nurses will also be asked whether and how the elements helped them carry out the 8th-grade consultation and cooperate with schools. School personnel in element 3 will be asked whether and how the material helped them cooperate with the school health services and how many collaboration meetings they completed.
In addition, project coordinators register information about implementation in all experimental conditions. For training and consultations, the following will be registered: attendance, time spent, content completed, significant events, adaptations to plans, adherence. For technical assistance requested during the study that is of relevance to experimental conditions, the following will be registered: participant, time spent, content/issue, significant events, turnover/sick leaves, and other adaptations.
Sustainment and scaling
Planning and preparing for sustainment and scale-up have been part of the co-creation process from the start of the exploration phase of the study. The projects' collaboration with key stakeholders, institutions educating health nurses, and authorities lays the foundation for using national infrastructure and regional competence centers (RBUP and RKBU) in scaling up.
The Norwegian Healthnet serves as a hub for developing a plan for sustainment and scale of functions in the DFA. This partnership provides a fruitful platform for designing, establishing, and testing secure data collection directly from users by means of Helsenorge.no, the digital platform for user interaction between citizens and patients with health services and registries.
Should the ongoing consultation (Dialog support) be a significant contributor to important implementation mechanisms and effects, we will plan for further improvements, sustainment, and scale by establishing an implementation group at the national competence centers involved in the study. Also, a protocol describing the structure, methods, and content of the ongoing consultations will be developed and made nationally available for other institutions to adopt. The e-learning module will be made accessible for educational purposes to the master's programs in public health nursing and will serve as a resource for the clinical practice of public health nursing.
If the results indicate that Collaboration material provides value, the material will be further improved based on participant feedback. RBUP and RKBU will offer schools and school health services an introduction and access to the revised material, which will be included as part of RBUP and RKBU Central Norway's ordinary teaching- and service provision.
In summary, each element in SchoolHealth can be sustained and scaled independently of the other, or in more ecological combinations. The results of the study will inform decisions regarding plans and recommendations for sustainment and scale.
Dissemination of results
Results will be disseminated through scientific publications, the study's and collaborating institutions' webpages, seminars with school health services and schools, popular science publications, and press releases. Research fellows, who are part of the project team, will publish and publicly defend dissertations related to the study. Master students will also publish results from the study. Planned scientific publications include reporting results on primary outcomes, secondary outcomes, psychometrics, and implementation mechanisms. The project team determines authorship of scientific publications in line with the Vancouver Protocol.
This hybrid type 2 study can optimize large-scale strategies for implementing evidence-based guideline recommendations in school health services to improve students’ health literacy, positive health behaviors, identify students needing follow-up, and improve interprofessional collaboration. The study "deconstructs" a blended implementation strategy that has been co-created with a wide array of relevant stakeholders and partners into its smaller meaningful parts (i.e., implementation elements), which represents three human-centered discrete implementation strategies (audit and feedback + , ongoing consultations, and active dissemination). The multifactorial design allows testing the effects of the elements in isolation and all possible combinations, as well as testing hypothesized implementation mechanisms informed by theory. By combining methods from multiple paradigms (i.e., factorial design, pluralistic mixed-methods, phenomenology), we can investigate cause and effects, mechanisms, and value from the perspectives of complementary causal theories and the lived experience of participants. This will allow us also to explore narratives about how, when, and for whom value do or do not occur or emerge from the implementation strategies and use of guideline recommendations. The study also addresses the degree of guideline fidelity needed for intended effects to occur. Investigations as outlined above have been extensively called for to advance implementation science [7, 9, 61, 62].
The study evaluates an innovative digitalization effort co-developed to meet expressed needs of users and services. It will also extend knowledge on adolescents' service use and user-pathways important for developing youth-friendly human-centered models of primary care.
Availability of data and materials
Capabilities, Opportunities, and Motivation model of Behavior change
Consolidated Standards of Reporting Trials
Digital Feedback and Administration Tool
Exploration, Preparation, Implementation, and Sustainment Framework
General Self-Efficacy, 5-item
The Norwegian Registry for Primary Health Care
Norwegian Patient Registry
Norwegian University of Science and Technology
The Centre for Child and Adolescent Mental Health, Eastern and Southern
Regional Centre for Child and Youth Mental Health and Child Welfare
The Strength and Difficulties Questionnaire
VID Specialized University. The letters in VID are a Norwegian acronym for Specialized (Vitenskapelig), International (Internasjonal) and diaconal (diakonial)
- Young-HUNT Study:
The adolescent part of the Nord-Trøndelag Health Study
The Norwegain Directorate of Health. Nasjonal faglig retningslinje for det helsefremmende og forebyggende arbeidet i helsestasjon, skolehelsetjeneste og helsestasjon for ungdom 2017 [updated 05.02.2018. Available from: https://helsedirektoratet.no/retningslinjer/helsestasjons-og-skolehelsetjenesten. Norwegian.
Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94.
Patel V, Flisher AJ, Hetrick S, McGorry P. Mental health of young people: a global public-health challenge. Lancet. 2007;369(9569):1302–13.
Viner RM, Ozer EM, Denny S, Marmot M, Resnick M, Fatusi A, et al. Adolescence and the social determinants of health. Lancet. 2012;379(9826):1641–52.
The Norwegian Ministry of Health and Welfare. Lov om kommunale helse- og omsorgstjenester m.m. (helse- og omsorgstjenesteloven). Oslo: Helse- og omsorgsdepartementet; 2011. Norwegian
Gagliardi AR, Alhabib S, Group atmotGINIW. Trends in guideline implementation: a scoping systematic review. Implement Sci. 2015;10(1):54.
Lewis CC, Boyd MR, Walsh-Bailey C, Lyon AR, Beidas R, Mittman B, et al. A systematic review of empirical studies examining mechanisms of implementation in health. Implement Sci. 2020;15(1):21.
Williams NJ. Multilevel mechanisms of implementation strategies in mental health: integrating theory, research, and practice. Adm Policy Ment Health. 2016;43(5):783–98.
Engell T, Stadnick NA, Aarons GA, Barnett ML. Common elements approaches to implementation research and practice: methods and integration with intervention science. Global Implement Res Appl. 2023;3(1):1–15.
Powell BJ, Fernandez ME, Williams NJ, Aarons GA, Beidas RS, Lewis CC, et al. Enhancing the Impact of Implementation Strategies in Healthcare: A Research Agenda. Frontiers in Public Health. 2019;7(3).
Sagatun Å, Kvarme LG, Misvær N, Myhre M, Valla L, Holen S. Evaluating a web-based health-promoting dialogue tool in school health services: feasibility and user experiences. J Sch Nurs. 2019;37(5):363–73.
Tylee A, Haller DM, Graham T, Churchill R, Sanci LA. Youth-friendly primary-care services: how are we doing and what more needs to be done? Lancet. 2007;369(9572):1565–73.
Waldum-Grevbo KS. Helsesøster: Følger du de nye, nasjonale anbefalingene? Sykepl Fag. 2018. Norwegian.
Waldum-Grevbo KS, Haugland T. A survey of school nurse staffing in the school health services. Sykepleien Forskning. 2015;4(10):352–60.
Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8(1):139.
Tveiten S. Helsepedagogikk : helsekompetanse og brukermedvirkning. 2nd ed. Bergen: Fagbokforlaget; 2020. Norwegian
Calvert FL, Crowe TP, Grenyer BFS. Dialogical reflexivity in supervision: An experiential learning process for enhancing reflective and relational competencies. Clin Superv. 2016;35(1):1–21.
Borg E, Christensen H, Fossestøl K, Pålshaugen Ø. Hva lærerne ikke kan! Et kunnskapsgrunnlag for satsning på bruk av flerfaglig kompetanse i skolen. Work Research Institute (AFI); 2015. Report No.: 8276093566. Norwegian
Laverty SM. Hermeneutic phenomenology and phenomenology: a comparison of historical and methodological considerations. Int J Qual Methods. 2003;2(3):21–35.
Van Manen M. Researching lived experience: Human science for an action sensitive pedagogy: Routledge; 2016.
Kvale S. Det kvalitative forskningsintervju. 2. utg. ed. Brinkmann S, Anderssen TM, Rygge J, editors. Oslo: Gyldendal akademisk; 2009. Norwegian
Morgan DL. Focus groups as qualitative research: Sage publications; 1996.
Braun V, Clarke V. Reflecting on reflexive thematic analysis. Qual Res Sport Exerc Health. 2019;11(4):589–97.
Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. 3rd ed. Los Angeles: Sage; 2018.
Engell T, Løvstad AM, Kirkøen B, Ogden T, Amlund HK. Exploring how intervention characteristics affect implementability: A mixed methods case study of common elements-based academic support in child welfare services. Child Youth Serv Rev. 2021;129: 106180.
Teddlie C, Tashakkori A. Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences: Sage Publications; 2009.
Technology NUoSa. The HUNT Study - a longitudinal population health study in Norway [cited 06.07.2023]. Available from: https://www.ntnu.edu/hunt .
Krokstad S, Langhammer A, Hveem K, Holmen TL, Midthjell K, Stene TR, et al. Cohort Profile: the HUNT Study. Norway Int J Epidemiol. 2013;42(4):968–77.
Åsvold BO, Langhammer A, Rehn TA, Kjelvik G, Grøntvedt TV, Sørgjerd EP, et al. Cohort Profile Update: The HUNT Study. Norway Int J Epidemiol. 2022;52(1):e80–91.
Bakken IJ, Ariansen AMS, Knudsen GP, Johansen KI, Vollset SE. The Norwegian patient registry and the norwegian registry for primary health care: research potential of two nationwide health-care registries. Scand J Public Health. 2020;48(1):49–55.
Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Admin Policy Mental Health Mental Health Serv Res. 2011;38(1):4–23.
Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implement Sci. 2019;14(1):1.
West R, Michie S. A brief introduction to the COM-B Model of behaviour and the PRIME Theory of motivation. Qeios. 2020.
Miller CJ, Barnett ML, Baumann AA, Gutner CA, Wiltsey-Stirman S. The FRAME-IS: a framework for documenting modifications to implementation strategies in healthcare. Implement Sci. 2021;16(1):36.
Engell T. Co-design and implementation of common elements-based academic support in Norwegian Child Welfare Services [dissertation]. University of Oslo; 2021.
Ehrhart MG, Aarons GA, Farahnak LR. Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS). Implement Sci. 2014;9(1):157.
Engell T, Kirkøen B, Aarons GA, Hagen KA. Individual level predictors of implementation climate in child welfare services. Child Youth Serv Rev. 2020;119: 105509.
Peters N, Borge RH, Skar A-MS, Egeland KM. Measuring implementation climate: psychometric properties of the Implementation Climate Scale (ICS) in Norwegian mental health care services. BMC Health Services Research. 2022;22(1):23.
Goodman R, Meltzer H, Bailey V. The Strengths and Difficulties Questionnaire: a pilot study on the validity of the self-report version. Eur Child Adolesc Psychiatry. 1998;7(3):125–30.
Walker LS, Garber J. Manual for the Children’s Somatic Symptoms Inventory (CSSI). 2018.
Ravens-Sieberer U, Auquier P, Erhart M, Gosch A, Rajmil L, Bruil J, et al. The KIDSCREEN-27 quality of life measure for children and adolescents: psychometric results from a cross-cultural survey in 13 European countries. Qual Life Res. 2007;16(8):1347–56.
Haraldstad K, Richter J. Måleegenskaper ved den norske versjonen av KIDSCREEN. PsykTestBarn. 2014; 2:1. Norwegian.
Schwarzer R, Jerusalem M. Optimistic self-beliefs as a resource factor in coping with stress. Extreme stress and communities: Impact and intervention. New York, NY: Kluwer Academic/Plenum Publishers; US; 1995. 159–77.
Tambs K, Røysamb E. Selection of questions to short-form versions of original psychometric instruments in MoBa. Norsk Epidemiologi. 2014;24:1–2.
Paakkari O, Torppa M, Kannas L, Paakkari L. Subjective health literacy: Development of a brief instrument for school-aged children. Scand J Public Health. 2016;44(8):751–7.
Aasen AM, Nordahl T, Mælan EN, Drugli MB, Myhr L. Relasjonsbasert klasseledelse: et komplekst fenomen. Oppdragsrapport nr. 13 – 2014. Eleverum: Høgskolen i Hedmark; 2014. Report No.: 8276719562. Norwegian
Haugum M, Danielsen K, Iversen HKH. Utvikling av spørreskjema for å måle barn og unges erfaringer med barne-og ungdomspsykiatriske poliklinikker. [Development of a questionnaire to measure children’s and adolescents’ experiences with outpatient child and adolescent mental health services]. Oslo: Folkehelseinstituttet 2019. Report No.: 8284060302.
Sjetne IS. Pasienterfaringer i spesialisthelsetjenesten : et generisk, kort spørreskjema. Oslo: Nasjonalt kunnskapssenter for helsetjenesten; 2009. Norwegian
Martinussen M, Adolfsen F, Lauritzen C, Richardsen AM. Improving interprofessional collaboration in a community setting: Relationships with burnout, engagement and service quality. J Interprof Care. 2012;26(3):219–25.
Svare H, Klemsdal L. Hvordan skape økt mestring blant frontlinjeansatte i servicenæringen : rapport fra et FoU-prosjekt støttet av NHOs arbeidsmiljøfond. Oslo: Work Reserch Institute (AFI); 2011. Norwegian
Ehrhart MG, Torres EM, Wright LA, Martinez SY, Aarons GA. Validating the Implementation Climate Scale (ICS) in child welfare organizations. Child Abuse Negl. 2016;53:17–26.
Aarons GA, Ehrhart MG, Farahnak LR. The implementation leadership scale (ILS): development of a brief measure of unit level implementation leadership. Implement Sci. 2014;9(1):45.
Braathu N, Laukvik EH, Egeland KM, Skar A-MS. Validation of the Norwegian versions of the Implementation Leadership Scale (ILS) and Multifactor Leadership Questionnaire (MLQ) in a mental health care setting. BMC Psychol. 2022;10(1):25.
Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):108.
Schwarzer R. Measurement of perceived self-efficacy : psychometric scales for crosscultural research. Berlin: Freie Universität Berlin; 1993.
Guastaferro K, Shenk CE, Collins LM. The multiphase optimization strategy for developing and evaluating behavioral interventions. In: Wright A, Hallquist M, editors. The Cambridge Handbook of Research Methods in Clinical Psychology Cambridge Cambridge University Press; 2020. 267–78.
Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, N. J: Laurence Erlbaum; 1988.
Peugh JL. A practical guide to multilevel modeling. J Sch Psychol. 2010;48(1):85–112.
Etikan I, Musa SA, Alkassim RS. Comparison of convenience sampling and purposive sampling. Am J Theor Appl Stat. 2016;5(1):1–4.
Robinson OC. Purposive Sampling. In: Michalos, A.C. (eds) Encyclopedia of Quality of Life and Well-Being Research. Springer, Dordrecht; 2014.
Beidas RS, Dorsey S, Lewis CC, Lyon AR, Powell BJ, Purtle J, et al. Promises and pitfalls in implementation science from the perspective of US-based researchers: learning from a pre-mortem. Implement Sci. 2022;17(1):55.
Brownson RC, Shelton RC, Geng EH, Glasgow RE. Revisiting concepts of evidence in implementation science. Implement Sci. 2022;17(1):26.
We would like to thank all adolescents, school nurses, leaders and school personnel who have co-created with us or provided invaluable feedback throughout the development of SchoolHealth and assessments used in the study. At RBUP we would like to thank the development team for Research and Innovation who developed the data collection platform for all quantitative data as well as the technical solution in the Digital Feedback and Administration Tool, our statistician Tore Wentzel Larsen for his help and support, and Ida Svantorp for her involvement in the development of Dialog Support and evaluation measures. At VID Specialized University we would like to thank Grethe Savosnick and her team for their contribution in the development of Dialog Support. We would also like to thank Sanne Angel at Århus University for developing interview guides and organizing interviews with practitioners of the Danish equivalent BørnUngeLiv.dk, to compare it with SchoolHealth. Finally, we would like to offer special thanks to Professor Dr. Med Carsten Obel, who was crucial in the first phase of the study. Although he is no longer with us, his enthusiasm and innovative thoughts continues to inspire us.
The study is primarily funded by the Norwegian Research Council (grant number 320097). The participating research environments: RBUP, NTNU and VID have also provided funding, primarily through personnel resources.
Ethics approval and consent to participate
The study was reported to the Norwegian Regional Committees for Medical and Health Research Ethics for approval. They concluded that the project falls outside the scope of the Norwegian Health Research Act, cf. § 2, and can be carried out without their approval. The study follows Norwegian procedures for ethical evaluation, and will be performed in line with the Norwegian ethical guidelines for research (https://www.forskningsetikk.no/en/guidelines/general-guidelines/). The data protection is evaluated by Sikt – The Norwegian Agency for Shared Services in Education and Research. Informed consent will be obtained from all participants. For participants below 16 years, informed consent to participate will be obtained from their parents or legal guardians. We anticipate a low risk of harm for participants, as SchoolHealth primarily aims to support school nurses and school personnel in their ordinary practice.
RBUP East and South and the collaborating partners have developed and signed an agreement on joint data processing responsibility.
Consent for publication
The authors declare no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Sagatun, Å., Engell, T., Brekke, M. et al. Guideline evaluation and implementation mechanisms in school health services (GuideMe): protocol for a hybrid randomized factorial trial. BMC Health Serv Res 23, 1259 (2023). https://doi.org/10.1186/s12913-023-10179-2