Study protocol | Open | Open Peer Review | Published:
Using a realist approach to evaluate smoking cessation interventions targeting pregnant women and young people
BMC Health Services Researchvolume 10, Article number: 49 (2010)
This paper describes a study protocol designed to evaluate a programme of smoking cessation interventions targeting pregnant women and young people living in urban and rural locations in Northeast Scotland. The study design was developed on so-called 'realist' evaluation principles, which are concerned with the implementation of interventions as well as their outcomes.
A two-phased study was designed based on the Theory of Change (TOC) using mixed methods to assess both process and outcome factors. The study was designed with input from the relevant stakeholders. The mixed-methods approach consists of semi-structured interviews with planners, service providers, service users and non-users. These qualitative interviews will be analysed using a thematic framework approach. The quantitative element of the study will include the analysis of routinely collected data and specific project monitoring data, such as data on service engagement, service use, quit rates and changes in smoking status.
The process of involving key stakeholders was conducted using logic modelling and TOC tools. Engaging stakeholders, including those responsible for funding, developing and delivering, and those intended to benefit from interventions aimed at them, in their evaluation design, are considered by many to increase the validity and rigour of the subsequent evidence generated. This study is intended to determine not only the components and processes, but also the possible effectiveness of this set of health interventions, and contribute to the evidence base about smoking cessation interventions aimed at priority groups in Scotland. It is also anticipated that this study will contribute to the ongoing debate about the role and challenges of 'realist' evaluation approaches in general, and the utility of logic modelling and TOC approaches in particular, for evaluation of complex health interventions.
In 2007, a team of researchers from the University of Aberdeen was commissioned by NHS Grampian to conduct an external evaluation of six local smoking cessation pilot projects (SCPPs) targeting pregnant women and young people in the Northeast of Scotland. The SCPPs were awarded Scottish Government funding which had been made available to support the newly introduced ban on smoking in public places . This national funding was distributed to regional health boards who, in turn, allocated the funding to local Community Health Partnerships (CHPs) on the basis of their peer-reviewed intervention proposals. This paper describes a study protocol that has been designed to evaluate those SCPP proposals that received funding.
Interventions of this type are generally complex and dynamic; often evolving in response to local circumstances, target group engagement and other events beyond the control of the implementers  which can adversely (or otherwise) affect the impact of the intervention . Pawson and Tilley have advocated the use of evaluation study designs that are capable of dealing with these issues .
For those reasons, this protocol and study design based on three core principles: (a) the need to verify the characteristics and quality of the intervention at the outset of the evaluation; (b) assess the intervention fairly against process and outcome indicators that; and are (c) 'ideally' identified and defined through a process of consultation with all stakeholders i.e. funders, intervention planners and implementers, and those intended to benefit from the intervention [5, 6]. Therefore, this evaluation will use an integrated process and outcomes study design, using so-called 'realist' principles which are concerned with illuminating the context in which an intervention is implemented; the mechanisms of that intervention, as well as its outcomes [7, 4].
Smoking - the public health problem
Pregnant women and young people
Prevalence and trends in smoking behaviour
In Scotland, approximately 20.9% of women smoke during the first trimester of pregnancy. Whilst there has been a reduction in the level of smoking in recent years, rates are still significantly higher in women from particular social backgrounds and age groups. Smoking in pregnancy rates for women living in the most deprived areas Scottish Index of Multiple Deprivation 5 (SIMD 5) of Scotland are 33.4% compared to 7.4% for women who live in the least deprived areas (SIMD 1). Rates are also considerably higher in the under 20 and 20-24 age groups, 40% and 31.0% respectively .
Although there has been a decline in the smoking rates of young people, levels of smoking among young people, girls in particular, remains high. The Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS) 2006 reported that 4% of 13 year olds; (3% boys and 5% girls) and 15% percent of 15 year olds; (12% boys and 18% girls) were regular smokers .
Regional policy and practice
Government funding was increased to enable the expansion of regional smoking cessation services to support targeted efforts to engage pregnant women, children and young people, particularly those living in deprived areas. The ultimate aim of this work is to improve access to services and reducing smoking rates in these groups [10, 11].
Local smoking cessation service provision
Current service provision in NHS Grampian (Northeast Scotland) is fairly comprehensive and operates across a diverse geographical area. The Smoking Advice Service (SAS) typically offers its clients an intensive support programme of 1 hour group sessions over a 6-7 week period, led by a trained smoking cessation advisor.
Studies suggest that pregnant women would benefit from sustained cessation support throughout pregnancy and may engage more successfully with a support service if home visits are offered. However there is still some debate concerning the most appropriate person to deliver the service e.g. midwife versus other health professional [12, 13].
National guidance on smoking cessation services for pregnant women recommended two key implementation priorities. These include,  visiting pregnant women at home if they find it difficult to attend specialist services, and , monitoring smoking status and offering smoking cessation advice, encouragement and support throughout the pregnancy and beyond .
Existing smoking cessation service provision may not be meeting the preferences and needs of young people resulting in poor service uptake and engagement of young people . Young people tend not to contact formal services for smoking advice; more commonly seeking advice from family or friends . Other evidence suggests that adolescents and young people would prefer a more informal, non-school based service which offers flexible support and guidance [16, 17].
Clearly, there is limited evidence about the most effective methods of engaging and supporting these two groups of smokers to quit, particularly for individuals living in disadvantaged circumstances [18–20]. Nevertheless, reducing smoking in these groups remains a public health goal in Scotland.
The local interventions
NHS Grampian funded its three regional community health partnerships with the twin aims of:
Developing and piloting of a modified version of the existing Smoking Advice Service (SAS) to meet the needs of people living in disadvantaged areas;
Creating supportive environments that would increase the availability of a tailored SAS for young people and pregnant smokers living in disadvantaged circumstances.
The services were at an early stage of development at the outset of the evaluation and details of the interventions were lacking at that point. Consequently, the evaluators were expected to map the components of the interventions in addition to identifying outcomes and outcome measures using a Logic Modelling process.
Evaluating complex health interventions
Smoking cessation programmes, typically aim to positively change a health behaviour, which in turn, is determined by a diverse range of individual (e.g. cognitive and affective) and environmental (social and cultural) factors. Moreover, such interventions set out to achieve positive outcomes in ever changing social, organisational, economic and political contexts [21–23].
These types of interventions are not easily evaluated by traditional experimental designs [24–27]. Health researchers are increasingly urged to use evaluation designs which not only capture outcomes but which also elucidates implementation processes, and contextual factors (internal and external) that may influence the success (or otherwise) of an intervention. Collectively, these data can provide a clearer link (or otherwise) between the intervention and observed outcomes; thus making any claims of attribution more valid [28, 29, 2, 30, 4, 31, 7]. Furthermore, it is important to ensure 'implementation failure' is eliminated as an explanation for the failure of some programmes to reach their desired outcomes [32–34].
It is also crucial to identify relevant process and outcome indicators that accurately reflect the nature and intensity of the intervention under review [32, 35]. Ideally, the process of identifying relevant indicators and measures should be done as a participatory and collaborative process with all key stakeholders [36, 6, 37].
Phased study designs are posited as an evaluation approach that can: (a) map the context, components and complexity of emergent health promotion interventions; and (b) identify and select plausible and realistic process and outcome measures on which to judge the merit or worth of the intervention at a later point in time. The MRC (Medical Research Council) framework for example, recommends - as a first stage - that the 'active ingredients' of an intervention, and its likely impact on health outcomes are identified, before attempting a larger scale intervention study .
The Theory of Change (ToC) approach, which has been championed by realist methodologists, [4, 3] was developed in the US in an effort to find ways of evaluating processes and outcomes in community-based programmes that were not adequately addressed by existing approaches . The ToC approach is defined as 'a systematic and cumulative study of the links between activities, outcomes and contexts of the initiative' . In generating a programme ToC, steps are taken to link the original problem; the context in which the programmes operates; the planned activities with its intended medium and longer-term outcomes.
Previous research suggests that evaluators should engage with key stakeholders at the early planning and development stage to identify a programme ToC that will link outcomes and activities and explain how and why the desired change is expected to come about [21, 10]. Proponents of ToC often advocate the use of logic models in this process, as a means of identifying an intervention's inputs and activities and its intended outcomes .
Aims and objectives
The overall aim of this study is to evaluate six SCPPs offering smoking cessation support services; three are aimed at young people and three at pregnant women living in Grampian. As this protocol is based on realist evaluation , there is no hypothesis, but the two overarching research questions were: "How did the interventions operate?" and "Did they reach their own aims and objectives?"
The objectives of the evaluation are to establish:
the intervention processes;
any changes made to the six interventions in response to local conditions;
the uptake of the services by the intended target group;
the degree to which project services become established and stable;
the acceptability of the interventions by intended clients and service providers;
to what extent the SCPPs met their short and medium term-objectives;
the feasibility of further roll out of promising interventions to other areas.
Given the ambiguity of the proposed interventions, the research team proposed a ToC approach to develop an evaluation framework.
The evaluation will proceed in two Phases commencing in April 2007 and ending in June 2010. (See Table 1 for research procedure and timeline).
Phase One (Exploratory Stage)
The main focus of Phase One was to address evaluation objectives 1 and 2. Researchers worked in close consultation with relevant stakeholders through a series of group meetings and individual discussions to (a) establish individual project baseline data; (b) map and understand the nature of each intervention's processes; (c) identify and define appropriate short, medium and long term outcome measures; and (d) identify any known external or internal factors that might influence implementation and outcomes. This understanding was developed using a logic modelling process, [31, 39] which is a technique which helps practitioners, health planners and evaluators to identify the things that need to be done within an intervention to help identify short and medium-term impact and longer term outcomes. The strength of logic modelling is that it links the desired impacts and outputs to the actual activities (processes) of the intervention, to ensure that these activities help contribute towards achieving the (measurable) final outputs. In this study, it was intended that the logic modelling process would produce six individual project logic models and their associated outcome indicators at the end of Phase One.
Phase Two (Explanatory stage)
Phase Two will focus on evaluation objectives 3 to 7. Mixed-research methods will be used for this part of the study  to consider: (a) the extent to which the SCPPs are implemented as intended; (b) the extent they reach their target groups; (c) the acceptability of the SCPPs from the perspective of the service providers and target populations, and their experiences of service delivery and use (or non-use); (d) whether (or not) they are able to meet the short, medium and long-term objectives identified in Phase One.
Across all six SCPPs, a purposive sample of approximately 40 managers and health care professionals who were involved in planning or delivering the interventions will be invited to participate in semi-structured face-to-face interviews. Approximately 30 young people and 30 pregnant women who have engaged with the SCPPs will also be interviewed. An additional sample of five young people and five pregnant women who have not fully engaged with the SCPPs will also be invited to take part in the study. To become eligible for the study, all service or non- service users must have registered to receive smoking cessation advice and support (Table 2). Young people are defined as being aged between 11 to 18 years attending secondary school.
Details of every individual who registers with the SCPPs are recorded (with consent) by SCPP staff as part of the Minimum Dataset (MDS) of core data required for anonymous national reporting of clients who access Scottish NHS Board specialist smoking cessation services . Additional, anonymised project specific monitoring data is also collected by SCPP staff (with consent from service users) for analysis by the evaluation team. As all six interventions are each fairly small scale projects and the collection of the MDS is compulsory, it was decided to also collect the additional project specific data from of all service users; hence no sample size calculation was needed for this total sample.
This dataset collates information about those who engage with the services; set a quit date; quit smoking at 4 weeks; and their self-reported smoking behaviour at follow-up points of 3 and 12 months. Individual projects are collecting additional data related to aspects of engagement their target, e.g. how many people expressed an interest in their project over time, methods used to contact prospective clients. A combined anonymised datafile containing the SAS and the project engagement information will then be sent to the evaluation team for quantitative analysis.
Semi-structured interviews will be conducted with individual service users, non-users and service providers using interview guides that will be derived from the evaluation objectives listed in Table 3 below.
Data management and analysis
A descriptive statistical analysis will be conducted of routine SAS and project monitoring data using SPSS (version 16). The qualitative data will be analysed using thematic framework analysis  incorporating aspects of grounded theory [43, 44]. The latter will include reading and re-reading the transcripts of the qualitative interviews to distil the key themes. NVivo (version 7) and Excel software will be used to manage and analyse the data. Using quantitative methods in conjunction with qualitative approaches can help in triangulation of data. [45, 45]
This study has been approved by the regional research ethics committee. All the participants will be invited to opt-in to the study and sign an informed consent prior to enrolment in the study. The preferences of study participants will be considered by offering them a choice of a face-to-face or telephone interview at a place and time at their convenience. All information sheets and consent forms are designed following the guidelines published by the NHS National Patient Safety Agency (2007) .
Our study team were commissioned to evaluate a diverse range of pilot projects; which were being implemented with varying levels of funding and resources, using different intervention strategies. Increasing calls for health researchers to provide more comprehensive accounts of interventions under review, coupled with the demand for evidence-based policy and practice, has led to a greater expectation that evaluations will illuminate what works (or not), with which target groups, and why [24, 4].
Therefore, we argue that the phased, realist evaluation approach described in this paper, will not only provide more insight into the precise nature of the interventions implemented in this case, but also help link the activities more closely to the final outcomes (if any). Policy makers and health professionals often come under pressure from both politicians and the media, to develop new ways of tackling long-standing health issues and to demonstrate their effectiveness (or otherwise) within relatively short timeframes. This presents an enormous challenge, given that complex health interventions take time to implement and establish before they become fully operational, and show any detectable impact on mortality or morbidity figures.
This study aims to provide the evaluators, and the SCPP staff with the means to ascertain and map the components of their interventions, and identify plausible project outcome measures that fairly reflect their nature and intensity. There is also potential to provide all SCPP stakeholders with an earlier indication of the likely effectiveness of these interventions, and more evidence about the degree to which the intervention is associated with the observed outcomes, because of the use of logic modelling.
Scottish Executive: The Smoking, Health and Social Care (Scotland) Act 2005, Part 1. Edinburgh. 2005
Abma TA: Responsive evaluation in health promotion: its value for ambiguous contexts. Health Promotion International. 2005, 20 (4): 391-397. 10.1093/heapro/dai013.
Judge K, Bauld L: Strong theory, flexible methods: evaluating complex community-based initiatives. Critical Public Health. 2001, 11 (1): 20-38.
Pawson R, Tilley N: Realistic Evaluation. 1997, Sage Publications Inc
Rootman I, Goodstadt M, Potvin L, Springett J: A framework for health promotion evaluation. Edited by: Rootman I, Goodstadt M, Hyndman B, McQueen DV, Potvin L, Springett J, et al. 2001, Evaluation in health promotion: Principles and prespectives: World Health Organisation, 7-38.
World Health Organisation: Health Promotion Evaluation: Recommendations to Policymakers. 1998
Sridharan S, Campbell B, Zinzow H: Developing a Stakeholder-Driven Anticipated Timeline of Impact for Evaluation of Social Programs. American Journal of Evaluation. 2006, 27 (2): 148-162. 10.1177/1098214006287990.
ISD Scotland: Smoking and Pregnancy: smoking at booking by NHS board. 2008, Accessed February 12th, 2009, [http://www.isdscotland.org/isd/2911.html]
BMRB Social Research: Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS) - National Report 2006. 2006
Scottish Executive: A Breath of Fresh Air for Scotland Improving Scotland's Health: The Challenge Tobacco Control Action Plan. 2004, Scottish Executive
NHS Health Scotland and ASH Scotland: Reducing Smoking and Tobacco-Related Harm - A Key to Transforming Scotland's Health. 2003
Connelly H, Finnie A, Rumbles A: Stop for Life: Final Project Report. 2007
McCurry N, Thompson K, Parahoo K, O'Doherty E, Doherty A: Pregnant women's perception of the implementation of smoking cessation advice. Health Educ J. 2002, 61 (1): 20-31. 10.1177/001789690206100103.
NHS National Institute for Health and Clinical Excellence: Smoking cessation services in primary care, pharmacies, local authorities and workplaces, particularly for manual working groups, pregnant women and hard to reach communities. 2008
Gnich W, Sheehy C, Amos A, Bitel M, Platt S: A Scotland-wide pilot programme of smoking cessation services for young people: process and outcome evaluation. Addiction. 2008, 103 (11): 1866-1874. 10.1111/j.1360-0443.2008.02316.x.
MacDonald S, Rothwell H, Moore L: Getting it right: designing adolescent-centred smoking cessation services. Addiction. 2007, 102 (7): 1147-1150. 10.1111/j.1360-0443.2007.01851.x.
NHS Health Scotland and ASH Scotland: Designing and delivering smoking cessation services for young people: Lessons from the pilot programme in Scotland. 2007
Murray RL, Bauld L, Hackshaw LE, McNeill A: Improving access to smoking cessation services for disadvantaged groups: a systematic review. Journal of Public Health. 2009, 31 (2): 258-277. 10.1093/pubmed/fdp008.
Roddy E, Antoniak M, Britton J, Molyneux A, Lewis S: Barriers and motivators to gaining access to smoking cessation services amongst deprived smokers - a qualitative study. BMC Health Services Research. 2006, 6 (1): 147-10.1186/1472-6963-6-147.
Pound E, Coleman T, Adams C, Bauld L, Ferguson J: Targeting smokers in priority groups: the influence of government targets and policy statements. Addiction. 2005, 100 (s2): 28-35. 10.1111/j.1360-0443.2005.01025.x.
Coote A, Allen J, Woodhead D: Finding Out what Works: Building Knowledge about Complex, Community-based Initiatives. 2004, London: The King's Fund
Rychetnik L, Frommer M, Hawe P, Shiell A: Criteria for evaluating evidence on public health interventions. J.Epidemiol.Community Health. 2002, 56 (2): 119-127. 10.1136/jech.56.2.119.
Nutbeam D: Evaluating health promotion - progress, problems and solutions. Health Promot Int. 1998, 13 (1): 27-44. 10.1093/heapro/13.1.27.
Kelly MP, Stewart E, Morgan A, Killoran A, Fischer A, Threlfall A, et al: A conceptual framework for public health: NICE's emerging approach. Public Health. 2009, 123 (1): e14-e20. 10.1016/j.puhe.2008.10.031.
NHS National Institute for Health and Clinical Excellence: Behaviour change at population, community and individual levels. 2007
Bonner L: Using theory-based evaluation to build evidence-based health and social care policy and practice. Critical Public Health. 2003, 13 (1): 77-92. 10.1080/0958159031000100224.
Tones K: Beyond the randomized controlled trial: a case for 'judicial review'. Health Educ Res. 1997, 12 (2): 161-10.1093/her/12.2.161.
Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M: Developing and evaluating complex interventions: the new Medical Research Council guidance. Br Med J. 2008, 337: a1655-10.1136/bmj.a1655.
Tannahill A: Beyond evidence--to ethics: a decision-making framework for health promotion, public health and health improvement. Health Promot Int. 2008, 23 (4): 380-390. 10.1093/heapro/dan032.
Glasgow RE, Vogt TM, Boles SM: Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999, 89 (9): 1322-1327. 10.2105/AJPH.89.9.1322.
Haw SJ, Gruer L, Amos A, Currie C, Fischbacher C, Fong GT, et al: Legislation on smoking in enclosed public places in Scotland: how will we evaluate the impact?. Journal of Public Health. 2006, 28 (1): 24-30. 10.1093/pubmed/fdi080.
Green J, South J: Evaluation. 2006, Open University Press
Tones K: Evaluating health promotion: a tale of three errors. Patient Educ Couns. 2000, 39 (2-3): 227-236. 10.1016/S0738-3991(99)00035-X.
Owen J, Rogers P: Program evaluation: Forms and approaches. 1999, London: Sage
Wimbush E, Watson J: An evaluation framework for health promotion: Theory, quality and effectiveness. Evaluation. 2000, 6 (3): 301-321.
Wadsworth Y: Everyday evaluation on the run. 1997, Australia: Allen & Unwin
Kubisch A, Brown P, Chaskin R, Hirota J, Joseph M, Richman H, et al: Voices from the field: Learning from comprehensive community initiatives. 1997
Connell JP, Kubisch AC: Applying a theory of change approach to the evaluation of comprehensive community initiatives: Progress, prospects and problems. Theory, Measurement, and Analysis. 1998, Washington, DC: Aspen Institute
Kellogg WK: Logic Model Development Guide. 2004, Michigan: WK Kellogg Foundation
Cresswell JW: Chapter 11: Mixed methods procedures. Research design: Qualitative, quantitative and mixed methods approaches. 2003, Thousand Oaks: Sage, 208-227. 2
Ash Scotland: The Minimum Dataset for Scottish Smoking Cessation Services. 2009, Accessed 02/10, 2010, [http://www.ashscotland.org.uk/ash/files/MinimumDataset_Jan2009.doc]
Ritchie J, Lewis J: Qualitative Research Practice: A Guide for Social Science Students and Researchers. 2003, London: Sage Publications Inc
Pope C, Ziebland S, Mays N: Qualitative research in health care: Analysing qualitative data. BMJ. 2000, 320 (7227): 114-116. 10.1136/bmj.320.7227.114.
Strauss AL, Corbin JM: Basics of qualitative research: Techniques and procedures for developing grounded theory. 1998, CA: Sage Publications Inc
Van Teijlingen E, Forrest K: The range of qualitative research methods in family planning and reproductive health care. Journal of Family Planning and Reproductive Health Care. 2004, 30: 171-173. 10.1783/1471189041261519.
NHS National Patient Safety Agency: Guidance on information sheets and consent forms. Version 3.2 National Research Ethics Service. 2007
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/10/49/prepub
This study is funded by NHS Grampian.
The authors declare that they have no competing interests.
FD conceived of the study, participated in its design and drafting of the manuscript. DG participated in study design, its coordination and drafting of the manuscript. EvT participated in the study design and drafting of the manuscript. All authors read and approved the final manuscript and are willing to take the responsibility for all content.