Table 2 Evaluation overview
TOOL | STAGE OF INTERVENTION | DATA SOURCE & COLLECTION | WHO PARTICIPATES | EXPECTED No. OF PARTICIPANTS | TIME COMMITMENT | INFORMATION NEEDED & JUSTIFICATION |
|---|---|---|---|---|---|---|
Baseline Survey* (detail below table) | PRE & DURING | Online anonymous survey implemented via the ComPrac by the Social Scientist | Current or emerging executives or managers from all sites who enrol in the CompPrac and make decisions to improve the value of the programs, services, initiatives or technology for which they are responsible | 80–120 across all sites | 15 min | Baseline measure of individual values and confidence and organisational support and barriers around economic evaluation; as well as expectations about the embedded Economist online learning support. This survey will inform the learning support and the remaining embedded components of the program |
Non-participant observations | PRE & DURING | Observation notes collected by Social Scientist | Economists and staff who planned or participated in the embedded Economist component of the intervention | Maximum 50 per site | PRE: 5 h maximum: 3 × 1 h scoping meetings 2 × 1 h co-production meeting DURING: a maximum of two hours per interaction for no more than four (4) meetings (minimum commitment 2 h maximum commitment 8 h) | Naturally occurring in situ group data about the process of the intervention (what was done by whom?), the relational aspects (how did the participants interact with one another?) and broader organisational context (i.e. what actions occurred and what decisions were taken in this environment) Provides a feel for organisational environments and the roles of actors within them that cannot be achieved through an interview situation. |
Documentary analysis | PRE, DURING & POST | PRE: Meeting agendas and minutes Emails and telephone notes Draft operational plan Site decision making frameworks, policies, guides, tools, documentation developed by economist and sites including evidence of application DURING: Meeting notes, agendas, minutes, and associated documentation developed by the economist and sites including economic evaluations drafted, evidence of application and practice change from meeting agendas, minutes and site generated documentation will be collected POST: During the semi-structured interviews, participants will be asked to provide any documentation that demonstrate a change in processes and work practices as a result of the embedded Economist Project. Documentation may include meeting notes, agendas, minutes, policies, tools and guidelines | Site Leads and Economists and staff who planned or participated in the embedded Economist intervention will be asked to provide documents | Maximum 10 per site | 2 h maximum (sourcing and providing documents) | Documentary evidence of how each stage of the intervention was formally enacted by the site and does this accord with what was observed and reported? Existing decision-making frameworks will also be analysed to ascertain the use of economic evaluation. Documentary analysis provides additional insights about the practices and around decision making and any practice outcomes from the intervention, thereby supporting any reported changes from interview data with tangible evidence |
Semi-structured interviews (face-to-face or telephone) | PRE, DURING & POST | Interview transcripts and recordings | Economists and staff who planned or participated in the program | PRE: 5 staff 2 economists DURING: 25 staff 2 economists POST: 5 staff 2 economists | maximum 1 h per interview (N.B staff participants may be interviewed once during each phase of the intervention & economists may be interviewed up to 6 times during each phase) | Face to face anonymous inquiry providing in-depth descriptions and interpretations that are constructed and separated from the time (and place) in which the events take place about the process, context and relational aspects of each stage of the embedded component of intervention, including planning, how the intervention was enacted and the outcomes and impact in the post phase; as well as the learning support components at the time of the semi-structured interview. Participants will have the opportunity to review their transcript prior to data analysis. |
Field diary | PRE, DURING & POST | Field diary from economists & staff | Economists and staff who planned or participated in the program | PRE: Lead economist DURING: Lead and embedded Economist 25 staff | Staff: Minimum commitment 15 min per week × 12 weeks maximum commitment 1 h per week × 12 weeks) Economist: Minimum commitment 15 min per week × 15 weeks) | Reflections on how participants perceived and experienced each phase of the intervention. What did they do (process)? Who did they interact with (relational)? What broader organisational factors impacted each phase? |
Education survey ** (DETAIL BELOW TABLE) | POST Completion of the university course | Online anonymous survey | Maximum of 60 across all sites | How are participants applying the knowledge gained from the university course in practice? Do they feel supported by their leadership to use evidence from economic evaluations? How are the learning supports contributing to practice change? i.e. have they increased capacity? |