- Research article
- Open Access
- Open Peer Review
Describing the impact of health research: a Research Impact Framework
- Shyama Kuruvilla1,
- Nicholas Mays2Email author,
- Andrew Pleasant3 and
- Gill Walt4
https://doi.org/10.1186/1472-6963-6-134
© Kuruvilla et al; licensee BioMed Central Ltd. 2006
- Received: 22 July 2006
- Accepted: 18 October 2006
- Published: 18 October 2006
Abstract
Background
Researchers are increasingly required to describe the impact of their work, e.g. in grant proposals, project reports, press releases and research assessment exercises. Specialised impact assessment studies can be difficult to replicate and may require resources and skills not available to individual researchers. Researchers are often hard-pressed to identify and describe research impacts and ad hoc accounts do not facilitate comparison across time or projects.
Methods
The Research Impact Framework was developed by identifying potential areas of health research impact from the research impact assessment literature and based on research assessment criteria, for example, as set out by the UK Research Assessment Exercise panels. A prototype of the framework was used to guide an analysis of the impact of selected research projects at the London School of Hygiene and Tropical Medicine. Additional areas of impact were identified in the process and researchers also provided feedback on which descriptive categories they thought were useful and valid vis-à-vis the nature and impact of their work.
Results
We identified four broad areas of impact:
I. Research-related impacts;
II. Policy impacts;
III. Service impacts: health and intersectoral and
IV. Societal impacts.
Within each of these areas, further descriptive categories were identified. For example, the nature of research impact on policy can be described using the following categorisation, put forward by Weiss:
Instrumental use where research findings drive policy-making;
Mobilisation of support where research provides support for policy proposals;
Conceptual use where research influences the concepts and language of policy deliberations and
Redefining/wider influence where research leads to rethinking and changing established practices and beliefs.
Conclusion
Researchers, while initially sceptical, found that the Research Impact Framework provided prompts and descriptive categories that helped them systematically identify a range of specific and verifiable impacts related to their work (compared to ad hoc approaches they had previously used). The framework could also help researchers think through implementation strategies and identify unintended or harmful effects. The standardised structure of the framework facilitates comparison of research impacts across projects and time, which is useful from analytical, management and assessment perspectives.
Keywords
- Social Capital
- Health Research
- Health Literacy
- Health Information System
- Research Impact
Background
Researchers are increasingly, and regularly, requested to describe the impact of their work in:
-
Grant proposals, especially sections on research users, beneficiaries, communication and expected impact;
-
Research project evaluations and reports, particularly to provide accountability for funds invested in research;
-
Publicity information e.g. for institutional annual reports and press releases that highlight "success stories";
-
Research dissemination and implementation strategies;
-
Assessments of the impact of research on policy, practice and public opinion;
-
Individual and institutional research assessment exercise (RAE) submissions.
However, researchers are often hard-pressed to come up with coherent and comprehensive narratives of actual or potential research impacts and produce ad hoc accounts that do not facilitate comparison across time or learning across cases. More specialised research impact assessment studies can be difficult to replicate and may require resources and skills not available to individual researchers [1, 2]. Further, the specialised focus of these models, for example on publications and economic benefits, may not cover a range of other areas where research could have impact. For example with regards to health research, environmental effects and social capital are increasingly associated with health outcomes and are areas where research can have impact [3–5], but are often neglected in existing health research impact assessment models and frameworks. A research impact scoping exercise at the London School of Hygiene and Tropical Medicine (LSHTM) identified some additional areas in which health research can have impact, such as on policy networks and on other sectors such as energy, that are also often missing in existing frameworks and models of health research impact [6].
In this paper we take a practical approach to help researchers describe the impact of their work. We present the Research Impact Framework, which is designed as a 'DIY' approach with descriptive categories that prompt researchers to systematically think through and describe the impact of their work. The framework was also designed with some additional goals in mind: to develop a standardised framework to help describe impacts across research topics and methods and facilitate comparison across time and cases; to guide researchers in planning research implementation and evaluation strategies; to facilitate researchers in looking at the broader influences and effects on and of their work in society; to promote research accountability in relation to the use of resources and the consequences of research; to help in the attribution of effects to health research given the range of other determinants of health and societal impacts; and to contribute to more extensive or specialised evaluations of research impact.
Researchers' narratives of the impact of their work could also serve as building blocks for a variety of more specialised analytical purposes. For example, researchers' accounts are considered a starting point for more specialised research impact assessments and can provide similar findings, though the level of detail and analysis may differ [7]. In analysing the role of research in diffusion of innovations, Greenhalgh et al. [8] found that "researchers in different traditions had conceptualised, explained and investigated diffusion of innovations differently and had used different criteria for judging the quality of empirical work. Moreover, they told very different over-arching stories of the progress of their research." Thus researchers' narratives also facilitate analysis of differences in the 'doing', diffusion and impact of research within and across research fields [8].
While the Research Impact Framework provides a simple, practical approach to developing research impact narratives, it is important to recognise that these narratives are generated and assessed in the context of historically rich and complex, often contending, views on the role of science and its relationship with society. Some may align with Bacon's [9] utopian vision wherein wise and benevolent rulers coordinate and disseminate research for the good of the state. Others take a 'Republic of Science' view, espoused by Polyani [10] and Vannevar Bush [11], where science is idealised as an independent enterprise, separate from societal concerns and having intrinsic value in and of itself. However, researchers on the sociology of scientific knowledge posit that science and society are interlinked and mutually influential [12, 13]. Further, Kuhn [14] and Callon [15] among others, discuss how science itself is a social enterprise with researchers interacting within specific scientific cultures and communities.
There may also be different types of impact expected from different types of research, for example basic, applied, action, clinical, user driven, translational or curiosity-driven research. Then there are contesting and complementary theories and models of causal pathways of research impacts associated with different levels and types of impact on research, policy and practice [16–19]. Several academic disciplines including communication, diffusion of innovation studies, policy science, sociology of scientific knowledge and organisational research all offer valuable perspectives on different aspects of assessing the production, communication, utilisation and impact of research [20]
Further considerations in describing research impacts include questions of accountability, for example, whether there is a different standard of accountability for researchers compared with practitioners with regards to impact [21]. There are also concerns about whether research impacts are positive or negative (and for whom) and what individual or institutional biases and incentives may operate in describing impact. Prioritisation of impact is another assessment issue, though priorities may vary in accordance with different funders' priorities, in different research fields and in different socio-political contexts.
Researchers may variously ascribe to different worldviews and may take into account various assessment considerations, either knowingly or unknowingly, when describing the impact of their work. However, the Research Impact Framework presented in this paper is not aligned with any particular philosophy, is not in itself evaluative and does not prioritise impacts or propose causal pathways. The Research Impact Framework was primarily designed as a practical tool to help researchers think through and describe the impact of their work; this could then serve a range of practical and analytical purposes as required.
While much of the framework presented in this paper could apply to research impact narratives in general, specific considerations warrant a topical focus on health research, which is defined as "the generation of new knowledge using the scientific method to identify and deal with health problems" [22]. While health research is nested within the larger science and technology enterprise, there are a multitude of social, governmental, academic, service, manufacturing and legal institutions specifically in place to deliver the products of health research to society [23]. Further, a systematic review on the diffusion of innovations, including research, emphasized the need for "building up a rich picture of process and impact" in health services [24].
To address the gaps in existing approaches to and models of health research impact assessments, we first drew from the literature on health research impact assessment models and criteria to map out a framework of health research impacts. This framework was then tested and modified against impacts identified by LSHTM researchers working on a wide range of health research topics. Using the Research Impact Framework allowed individual researchers to identify and select impacts relevant to their work and develop impact narratives without requiring specialised skill in the field of research impact assessment. The standardised descriptive categories also facilitated analysis across the narratives. The methods used to develop the Research Impact Framework are described in the following section.
Methods
Our objective was to develop a conceptual framework to help health researchers think through and describe the possible outcomes of their work. A framework is a way of setting out a range of possible variables related to the issue of interest, but does not necessarily identify the relationships between them [25] or provide an evaluative judgement of the variables. For example, the fallout from the controversy of research linking the MMR vaccine with autism received substantial coverage in scientific journals and in the media and had various adverse effects such as a drop in immunisation rates [21]; a framework would include these areas of impact, but would not evaluate the nature of this impact. We developed a conceptual framework that covered a wide range of potential areas of health research impact, and standardised ways of describing them, so that individual researchers without any specific training in research impact assessment could use the framework to describe the impact of their work, selecting descriptive categories as per relevant purposes, priorities and assessment criteria.
To develop the Research Impact Framework, we first mapped out potential research impact areas based on a review of the main health research impact assessment approaches. The goal was not to conduct an exhaustive review of the literature, or to analyse the causal claims of different impact assessment models, but only to identify and map out the scope and coverage of potential areas of health research impact across a range of models, frameworks and criteria. The approaches we drew on to map the health research impact areas, and there were was overlap between them, were the Payback Model of health research benefits [18], a framework to analyse Health Research Systems [23], a 'knowledge transfer' approach to assessing the impact of research [26], a model of the path from evidence generation to clinical application [27] as well as economic approaches to assessing health research impact, such as the Funding First approach, which includes calculations of the economic value of increased life expectancy resulting from investments in research [28]. We also drew on frameworks that did not explicitly focus on health research, but explicated the links between globalization and health [3], pathways of communication and social change [29] outcomes of health promotion [30] and the non-financial constraints in health systems [31] as these are all areas where health research potentially could have impact. Research assessment criteria, for example, as proposed in the OECD's Oslo Manual [32], criteria for the initiation and evaluation of National Institutes for Health (NIH) Extramural Center Programs [33] and the UK Research Assessment Exercise [34], further informed the development of the Research Impact Framework by highlighting contemporaneous criteria against which the significance and impact of research are judged.
Guide to help structure research impact narratives
Narrative areas | RESEARCH PROJECT/PROGRAMME DESCRIPTION | Key dates (mm/yy) |
|---|---|---|
Description of research project/programme | Topics/research area: Geopolitical contexts: Funders and budget: Research management, influencing events and challenges: | |
1. Research-related impact | 1.1 Type of problem/knowledge 1.2 Research methods used 1.3 Publications and papers 1.4 Products, patents and translatability potential 1.5 Research networks 1.6 Leadership and awards 1.7 Research management 1.8 Communication | |
2. Policy impact | 2.1 Level of policy-making 2.2 Type of policy 2.3 Nature of policy impact 2.4. Policy networks 2.5 Political capital | |
3. Service impact | 3.1 Type of services: health/intersectoral 3.2 Evidence-based practice 3.3 Quality of care 3.4 Information systems 3.5 Services management 3.6 Cost-containment and cost-effectiveness | |
4. Societal impact | 4.1 Knowledge, attitudes and behaviour 4.2 Health literacy 4.3 Health status 4.4 Equity and human rights 4.5 Macroeconomic/related to the economy 4.6 Social capital and empowerment 4.7 Culture and art 4.8 Sustainable development outcomes |
The principal investigators (or LSHTM focal points in collaborative projects) of the selected research projects were interviewed using the semi-structured interview guide that was structured according to the mapping of research impact areas (See Table 1). Principal investigators were selected as the main interviewees because they were primarily responsible for drawing up descriptions of research impact in grant proposals as well as in final project reports. In addition, LSHTM colleagues whose expertise included research impact assessment as well as the heads of the research units were interviewed, not on the substance of the individual projects per se, but on the development of the Research Impact Framework. One investigator conducted all the interviews that took around 45 minutes each, and took contemporaneous notes. Information was also gathered from research project documents including reports, published papers and correspondence. The data were analysed based on a thematic analysis using the descriptive categories of the framework as the main themes.
Research Impact Framework
Research-related impacts | Policy impacts | Service impacts | Societal impacts |
|---|---|---|---|
• Type of problem/knowledge • Research methods • Publications and papers • Products, patents and translatability potential • Research networks •Leadership and awards • Research management • Communication | • Level of policy-making • Type of policy • Nature of policy impact • Policy networks • Political capital | • Type of services: health/intersectoral • Evidence-based practice • Quality of care • Information systems • Services management • Cost-containment and cost-effectiveness | • Knowledge, attitudes and behaviour • Health literacy • Health status • Equity and human rights • Macroeconomic/related to the economy • Social capital and empowerment • Culture and art • Sustainable development outcomes |
Within each impact area in the Research Impact Framework, we also identified descriptive categories in the literature, discussed in following sections, to support and help standardise the development of research impact narratives to enable comparative analysis across projects. Through an iterative and consultative process with LSHTM researchers involved in the selected projects, we determined which terms and categories were most useful to help researchers structure narratives of research impact.
The Research Impact Framework was thus validated through congruence with the literature on health research impact assessment, as well as through empirical analysis of research projects. Researchers in the LSHTM study found the approach more systematic than the more usual ad hoc and guess work-based process. The framework also provided new perspectives on potential and actual health research impacts and allowed for comparison of impact and assessment-related issues across the selected projects [6].
In this paper, the Research Impact Framework is presented, including the main health research impact areas and the descriptive categories within them. Examples from the LSHTM study and from the research impact assessment literature are used to illustrate key points.
Results: the Research Impact Framework
- I.
Research-related impacts
- II.
Policy impacts
- III.
Services impacts: health and intersectoral
- IV.
Societal impacts
Within each of these areas, key descriptive categories were identified (See Table 2). By listing this broad range of health research impact areas, we do not propose that all these impacts should be expected or targeted all the time. Neither do we suggest that these are necessarily mutually exclusive categories. Rather, the framework offers an overview of potential, and sometimes overlapping, research impact areas that can serve as a series of prompts or a checklist that researchers can select from or modify, as appropriate to the research being described and taking into account, for example, funders' priorities or research assessment criteria.
I. Research-related impacts
II. Policy impacts
Increasingly, one of the main objectives of health research is to inform and influence policy and there are several potential impacts in this area:
-
Level of policy-making
-
Type of policy
-
Nature of policy impact
-
Policy networks
-
Political capital
Level of policy-making
Research can have an impact on policy-making at different levels and including different groups, for example, national and local politicians, health services administrators and managers/directors, representatives of local, national and international professional groups, NGOs and business leaders. In describing impact, the geopolitical level of policy impact, i.e. whether at international, national, or sub-national level, can be described as well as if the changes were localised to a particular organisation or network of organisations within those levels.
Type of policy
The type of policy influenced by research is another way that impact can be described. For example, Black [62] endorses a distinction made between: practice policies, service policies and governance policies. Type of policies may also be described in relation to particular types of policy institutions, for example, researchers in the LSHTM study discussed how findings on home temperature-related morbidity and mortality [63] were used in the UK parliament in support of the Housing Bill (2004). The publication of WHO guidelines on ethics related to research on women and adolescents who had been trafficked [43] that had been developed on the basis of a research project was cited another example of policy impact in the LSHTM study [6].
Nature of policy influence
There are several ways in which research can influence policy and Weiss (1998) identified four main modes of influence listed below [64]. There are a wide range of other categorisations of research to policy impact [65], including in Weiss's earlier work [17], describing rational, tactical and symbolic use of research in policy-making. In the LSHTM study, the following categories were found to be conceptually clear and relevant to policy impacts identified by researchers:
-
Instrumental use where research findings directly drive or define policy.
-
Mobilisation of support [supportive evidence] where research findings provide persuasive evidence to back ongoing and proposed policy activities or raise awareness and support for new policy-making, for example through writing policy briefs. As one LSHTM researcher noted, "Writing policy briefs gives us the opportunity to discuss the wider policy implications of our research and to anticipate and rehearse policy arguments ... Our work adds weight to policy deliberations." [6]
-
Conceptual use where research leads to new ideas and language that influence the nature and substance of policy discourse, for example the concept of equity is increasingly used in considerations of health care financing [66, 67].
-
Redefining/wider influence refers to research impact that leads to a wide change or transformation of accepted beliefs and practices. This may involve overturning orthodoxies, as in the revolutionary shift in eighteenth century medical science from the concepts of vapours and humours to the constructs of anatomy and pathology [68]. More contemporary examples also indicate how research can lead to the changes in existing beliefs and practices. For instance, a multi-country randomised control trial found that widely accepted practice of administering corticosteroids after head injury could be harmful [69] with implications for both policy and practice on this topic.
Policy networks
In relation to a specific policy problem, networks of actors and institutions with particular interests and perspectives on that issue may pre-exist or form and interact in policy-making [70]. At times, these networks are referred to as epistemic communities and extend across normally disparate elements of research, policy, practice and the public, but are based on a common research-based understanding of the issue [63]. As with the dissemination of research through research networks, policy ideas and interests are often developed through policy networks. The extent to which researchers are part of, or inform, policy networks is, therefore, an important aspect of research impact. Collaborations with policy advocacy groups, think tanks and government institutions can be listed as part of this description. In the LSHTM study, researchers emphasised the importance of establishing networks that extended beyond the formal bureaucracy as "knowledge users include policy-makers as well as the range of civil society organisations." [6]
Political capital
Research impact on political capital takes into account the value of research evidence and researchers themselves in policy negotiations, in reaching in high quality agreements and improvements to the policy-making process, in the ability in societies to achieve agreed-upon ends, in the socio-political prestige resulting from innovations, and in increased national security [71]. For example, research on negotiation and consensus building methods on health issues could contribute to improving the quality of deliberations and reaching agreements that are reliable and satisfactory to the parties involved [72, 73].
III. Service impacts
The following categories help describe the impact of research on both health and intersectoral services.
-
Type of services: health/intersectoral
-
Evidence-based practice
-
Quality of care
-
Information systems
-
Services management
-
Cost-containment and cost-effectiveness
Type of services
Health research, often primarily targeted at improving clinical and public health services, can have a range of impacts in this area that are detailed in the following sections on evidence based practice, quality of care, health information systems and on health services and systems management, including cost-effectiveness and cost-containment. Research on prevention and control measures for newly emerging diseases such as avian flu is another example of an area in which research can have an impact on health services [74].
In addition, health research can influence a range of other service sectors that contribute to public health. For example, in the LSHTM study researchers described how the WHO ethical guidelines, developed on the basis of a research project, for research on women and adolescents who were trafficked [43], were now being used to train journalists who conduct interviews on this topic and were also used in police training courses in several countries [6].
Living and work environments are associated with negative or positive effects on human health and are, therefore, critical areas for health research impact assessment [75]. For example, research on the health benefits of well-lit and safe places to work and exercise can contribute to reducing injury and promoting physical activity [76]. Impacts can also be described in the areas of measures undertaken, as a result of research, to reduce indoor and outdoor air pollution as well as in the use of more energy efficient devices [77].
Road traffic injuries are increasingly recognised as a major public health problem. Research could influence road safety and health, for example by highlighting issues for the regulation of vehicular use to facilitate safe walking and cycling, particularly for children [42] or by showing how the growth in sales of SUVs and pick-up trucks in the US contributed to a slower decline in road fatality rates there than in other industrialised countries [78].
Health research may also have an impact on vector control services, for example as related to studies on the cost-effectiveness of different types of insecticides to prevent malaria as well as on their safety for humans [39]. Additionally, clean drinking water and soil fertility are critically important to human health [79]. Thus, if applicable, research impact narratives could include descriptions of research impacts on food production and safety, water access and quality.
The impact of climate change and weather events on public health is another intersectoral area that is receiving increasing attention in health research. For example, in 2003, a heat wave resulted in numerous deaths across Europe, in France alone there were around 14,000 deaths [41]. This event raised the profile of climate change as a public health issue in Europe. Changes in forecasting and early warning systems as well as on related response mechanisms and guidelines, informed by research findings, are examples possible health research impacts in this area.
Evidence-based practice
The use of research evidence by different groups involved in clinical diagnosis and decision-making is an indicator of research impact in health services [27]. Studies also indicate that in clinical decision-making and interviews between patients, their families and health professionals, the use of decision aids that are based on research evidence, can have significant effects on treatment decisions, choices and costs [80, 81].
Research can also influence clinical practice and service delivery through [19, 24, 59]:
-
Adoption of research findings and health technologies by health service providers
-
Adherence to research-informed policies and guidelines and
-
Addressing barriers to the use of research-informed interventions in the health system. Potential barriers include inflexible organisational workflows, inadequate resources and staff training as well as staff and patient attitudes and beliefs.
Quality of care
Research-influenced changes in health services can lead to improved information on quality of care and improvements in quality of care itself. Quality of health care addresses [82]:
-
Efficacy of health interventions. For example, one of the studies included in the LSHTM analysis was a clinical audit of tonsillectomy surgical techniques where the findings were that that 'hot' surgical techniques – diathermy or coblation – had a higher risk of complication than cold steel tonsillectomy methods [83].
-
Availability of services, for example in different geographical regions.
-
Accessibility of services, especially for disadvantaged and vulnerable individuals and groups.
-
Acceptability of services provided, for example in terms of quality and cultural appropriateness.
-
Utilisation and coverage of health services. Research can inform strategies to improve the coverage of health services and can also have impact through the development of methods to assess coverage. For example, availability and accessibility of services or interventions are often used as a proxy of health services coverage. However, it has been proposed that coverage should be assessed by estimating the probability that those who need a particular intervention will actually receive it, the probability of their requiring the intervention in the future and the probability of receiving effective health services based on previous experience [84].
-
Responsiveness of health services to population health needs is a further area that can have impact vis-à-vis the quality of services.
Information systems
A portion of health research is also directed toward improving the quality and efficiency of public health monitoring, reporting and evaluation. Possible indicators of the effects of research in this area include the accuracy, completeness, efficiency, relevance and timeliness of health monitoring and reporting systems, e.g. health information systems and GIS tools. For example, research in Turkey compared health information systems across 729 hospitals run by either the Ministry of Labor and Social Security or the Ministry of Health [85]. The study highlighted several differences in the use of health information systems across the two ministries that would need to be addressed given a proposed merger of the hospitals. This type of research could have an impact on the way health information systems are developed and used.
Health systems and services management
The extent to which research contributes to changes in health systems management and administration is therefore another important area of potential impacts. For example, management of health services procurement and provisioning in both private and public sectors can also be influenced by research [86]. In assessing the impact of research on the management of health systems and markets, there is a broad range of considerations including health care financing and insurance. In addition, Hanson et al. [31] provide a conceptual framework to analyse non-financial constraints in health systems including sociopolitical changes, intersectoral issues, such as transport to health facilities and planning for long-term outcomes such as improving health by promoting female education.
Cost-containment and cost-effectiveness
Cost-containment, while obviously related to health systems management is often a particular focus for assessing the impact of research on health services provision and health system management. This can have far-reaching implications for how health systems resources are allocated and used [87].
Cost-effectiveness of health services is another common focus in health research impact assessment and can be studied by analysing research-related changes in health systems in terms of both expenditure and related health outcomes [87]. In the LSHTM study, some research projects specifically included economic evaluations of the interventions being researched, for example the cost-effectiveness of social marketing of insecticide treated bed nets [88].
IV. Societal impacts
Finally, research impacts can be described in terms of impacts at a societal level with reference to:
-
Knowledge, attitudes and behaviour
-
Health literacy
-
Health status
-
Equity and human rights
-
Macroeconomic/related to the economy
-
Social capital and empowerment
-
Culture and art
-
Sustainable development outcomes
Knowledge, attitude and behaviour impacts
As a result of research dissemination and implementation strategies, various research impacts on knowledge, attitudes and behaviour could be expected or targeted. For example, research information could lead to changes in:
-
Knowledge about and attitudes toward health risks and resources including the development of health self-efficacy. For example, in a study that was part of the LSHTM analysis, researchers had found that bilingual young people, while previously mainly viewed as an at risk group also served as a resource, often playing the role of translators and mediators in promoting their families' access to health care services [89]. These findings also led to changes in service institutions working with bilingual young people, to focus on both risks and resources in strategies aimed at improving access to health care [6].
-
Healthy behaviours e.g. nutrition and diet, physical activity, safe sex, immunisations and improved hygiene practices can be promoted and adopted as a result of health research being disseminated, for example through health education and health promotion activities.
However, care must be taken in predicting or describing direct effects from research-based information being communicated and the desired effects occurring. There are a host of intermediary effects and influences that complicate that process, including culture, local and socio-historical contexts, the framing and relevance of the issues as well as perceived benefits and risks related to the research, so these should be noted if possible [29, 90].
Health literacy
Health literacy encompasses the wide range of skills and competencies required to find, understand, evaluate and use health information and concepts to make informed choices, reduce health risks and increase quality of life [91]. As such, health literacy is a key component of the complex relationship between knowledge, attitudes, decision-making, behaviour and health outcomes [30, 92]. Thus, health literacy is a potential impact of health research. In addition health literacy can facilitate successful communication of health research or can be a potential roadblock in situations when health research is communicated in a manner not commensurate with the health literacy skills of the intended audience.
As one example of research that can have an impact on health literacy, traditionally malaria control strategies rely on the free distribution of insecticide treated bed nets through public health and donor agencies. However, research shows that approaches to malaria control approaches that raise awareness in communities about the causes and prevention of malaria and promote community skills and capacities to purchase and treat bed nets on their own, provide a more sustainable and equitable method of malaria control and significantly improve health outcomes [39, 40].
Health status
Research can contribute to improvements in health status by contributing to interventions that reduce morbidity, mortality and disability and promote health as well as by developing methods to measure and monitor health status. However, the attribution of health status effects to research can be challenging given the range of other influencing factors and there are ongoing efforts to address this challenge [18, 28]. There are various measures for assessing health outcomes such as Quality Adjusted Life Years (QALYs); Disability Adjusted Life Years (DALYs); and Health Adjusted Life Expectancy (HALE) and a host of specific measures related to particular diseases/conditions. As is the case for assessment in general, the choice of measure usually reflects researchers' preferences, decision-makers' needs and the focus of assessment. Research can therefore also have impact by elucidating the values and assumptions underlying health status assessment measures so that these can be deliberated on by a broad range of stakeholders [93]. Research impact on health outcomes can thus be assessed both in terms of changes in health outcomes as well as in methods of measuring health outcomes.
Equity and human rights
Health equity and the realisation of human rights are increasingly discussed as desired health research outcomes and need be taken into account at different levels of assessment, be they related to structure, process or outcome [94, 95]. Research can also influence health equity outcomes by explicitly focusing on the needs of disadvantaged and vulnerable populations [96] as well as through the application of research to facilitate more equitable access to health care [67]. For example, there is concern that by focussing on mass interventions and targets set at the overall population level, the Millennium Development Goals may fail to address the needs of the most vulnerable and marginalised groups such as indigenous communities [97].
Human rights also include the right of individuals to participate in decisions that influence their lives and the right to information on the same. Additionally, the impact of participation on health outcomes is increasingly recognised. For example a randomised control trial showed that birth outcomes in a poor rural population in Nepal improved greatly through a low cost, potentially sustainable community-based participatory intervention with women's groups [98]. This type of research potentially has equity, human rights, methodological and health impacts.
Macroeconomic/related to the economy
There are several areas in which the economic outcomes of health research can be assessed [87]. The term 'macroeconomic' is used here to distinguish between impact on the overall economy (though the scale of impact may vary) and impact in terms of cost-savings and cost-effectiveness in the health care system, which was described in an earlier section.
There are various ways in which health research can have impact on the economy. The extent to which individual researchers have access to this level of economic analysis may vary, but some multidisciplinary research collaborations explicitly include an economic analysis of the proposed research-based intervention or technology and others may rely on post hoc economic evaluations.
-
Commercial outcomes can result from research in terms of monetary returns on investments in product development and marketing as well as from new or more efficient industrial processes [32, 87].
-
The impact of research on private markets, particularly with regard to the distribution of health interventions, needs to be further understood and assessed. For example, research evidence indicating that the distribution of soap to promote hygiene and nets to prevent malaria is more effective and as equitable when done through local markets than through the public health system, including to poor populations [38, 99]. These findings have influenced how some public health programmes implement related strategies.
-
Healthy workforce outcomes can also impact the economy. For example, analyses of the economic impact of health services research combine cost savings in health care provision with the economic value of a research intervention in terms of working days lost and produce a single metric of economic impact [100].
-
The value of health gain (or loss) from research to societies in monetary terms. For example, the Funding First approach calculates the economic value of improved health and life expectancy resulting from investment in and application of research-based health interventions, e.g. to treat cardiovascular disease [28]. Evaluation of health outcomes in economic terms, however, is subject to criticism even from those who use the approach, as evaluating human life in monetary terms is ethically controversial. Other potential problems include the equity implications of such approaches, for instance, evaluating the impact of health research in terms of productivity could favour research primarily focused on those of working age and/or those with higher incomes [101].
Social capital and empowerment
Social capital is increasingly viewed as a desired resource in societies and studies also find that there are links between social capital and health outcomes [4, 5]. Therefore the contribution of research to the development of social capital and its assessment is a key area of impact. Social capital is measured through a variety of social research techniques and the context of the assessment is an important consideration, but many authors seem to agree on the importance of the following as measures of social capital [4, 5]:
-
Civic engagement and social cohesion (through social interaction, mobilisation and connections as well as the development of shared knowledge and concepts).
-
Self-efficacy and collective efficacy (the ability and resources for problem solving at individual and societal levels).
-
Trust (in other individuals and groups in society). For example, in a series of surveys in the UK on who people generally trusted to tell them the truth, doctors and teachers were listed first, scientists and the 'ordinary man/woman in the street' were near the middle and politicians and journalists were at the bottom of the list [102]. Similar surveys have been carried out in other countries [103]. Research can generate information that affects peoples' trust of different groups. In addition, trust in researchers themselves is associated with the support and use of science.
Research that empowers communities to access, understand and use health research informed interventions, as earlier discussed with reference to the malaria control example, and to take ownership of health promotion efforts [30] is another key area of health research impact.
Culture and art
Culture refers to patterns of human learning and activity that lead to shared communication, artefacts and characteristics in societies; including language, patterns of behaviour, beliefs, identity, customs, traditions and other modes of expression [104]. These learned characteristics enable group members to hold and communicate shared meanings. The ability to influence culture is the also the ability to influence conceptions and categories that guide behaviour [105], including health behaviour. Health research generates new concepts and knowledge that influences societal culture (though the relationship works both ways). Health researchers are also constantly renegotiating the boundaries between what is considered traditional medicinal practice and research-based evidence as well as on what constitutes health and ill health. For instance, Maori researchers in New Zealand found that they were able to apply the methods and values of both indigenous and scientific knowledge in order to reach more comprehensive understandings of health and illness [106].
Arts and entertainment is a particular area where the influence of and on health research has generated interest in 'lay' society [107]. Some scientists serve as advisors on movies and TV shows. For example, the Journal of Public Health Policy reviewed the film The Constant Gardener. The film was based on John le Carre's book of the same title, in which the author mentions talking with Peter Godfrey-Faussett, a senior researcher at LSHTM, among others, in doing background research for the book. The review concludes with an observation of the potential impact of such work noting that,
This one production will be seen by millions of people around the world [and] will have done more to present the pharmaceutical industry's obstacles to improving health in developing countries ... than all health and science journals can in a year [108].
The influence of science on art and of art on science as well as on health and wellbeing certainly bears further research [109]. However, the interpretation and communication of science through art is increasingly considered an important means of science communication, not only in the mass media and movies, but also in museum collections on art and health as well as the art collections and programmes of major health research funders, such as the Wellcome Trust [110].
Sustainable development
Sustainable development outcomes depend on a combination of several of the earlier described impacts. One commonly encountered conceptualization of sustainable development is that it enables all people throughout the world to satisfy their basic needs and enjoy a better quality of life without compromising the quality of life of future generations, for example as defined in the Rio Declaration on Environment and Development [111].
The UK was one of the first countries to establish a set of indicators to review sustainable development in 1996 and these have been reported on regularly and reviewed in annual reports [77]. Key elements of the UK sustainable development strategy are:
-
Living within environmental limits
-
Ensuring a strong, healthy and just society
-
Achieving a sustainable economy
-
Promoting good governance
-
Using sound science responsibly.
Additionally, the continued ability to learn and adapt to new knowledge as it is produced is a key aspect of sustainable development. Thus, research may have a direct impact on sustainable development outcomes through the production of new knowledge on a range of topics, but can also help further develop concepts and assessment methods and help collect, validate and analyse data to guide sustainable development.
Discussion
Researchers are often hard pressed to come up with coherent accounts of the significance and impact of their work as regularly required in a range of situations, from writing grant proposals to designing implementation strategies and in submissions to research assessment exercises. Increasingly health research funders expect measurable outcomes from their investment in research. While some may decry this as neglecting the inherent value of science, others point out to the underutilisation or misutilisation of health research or to the need to prioritise research funding based on identified societal needs [16–19]. There is also evidence that researchers' accounts of the impact of their work provide a reliable starting point for more in-depth analyses and can provide similar information to more specialised assessments, although the scope and level of analysis may vary [7].
In this light, we pragmatically set out to identify the areas where health research could be expected to have an impact and then organized these areas into a framework. The validity and usefulness of the Research Impact Framework was tested at the London School of Hygiene and Tropical Medicine. The framework categories were finalised based on feedback from researchers working on a wider range of health research topics as to which areas they thought provided a valid indication of the impact of their work and which categories they found most useful to describe the same.
While initially sceptical, LSHTM researchers found that using the Research Impact Framework prompted them to identify a wide range of impacts related to their work in a relatively systematic manner (compared to the ad hoc approaches they had previously used) [6]. Researchers' narratives contained specific and verifiable evidence and the standardized structure of the narratives facilitated analysis across projects that could inform research management, practice and assessment [6]. One caution we offer is to be aware of various biases that may influence identification and description of research impact. For instance there may be strong historical and institutional biases and incentives to identify only positive impacts from research. However, researchers interviewed in the LSHTM study were aware of potential negative impacts of their research, for example, stigma that could arise with the publication of research findings related to a particular group or community. We will continue to learn as much, if not more, about the relationship of science and society and research impacts by maintaining focus on scientific controversies as well as scientific developments.
Conclusion
The Research Impact Framework provides a useful set of descriptive categories to help researchers identify and describe the impact of their work. It could also help researchers think through strategies to enhance the use of research-informed interventions as well as to identify unintended or harmful effects. The standardised structure of the framework also facilitates comparison of impact across projects, which is useful from an analytical and research management perspective. In addition, the framework should help health researchers systematically present the value of the work they have done as well as the work they wish to conduct in the future. This approach will further enable researchers to better explain what it is that they do and why they do it, not only to funders and governments, but also to review boards, colleagues, journalists and family and friends. This paper contributes to ongoing efforts seeking to understand, assess and explain the role, relevance and impact of health research.
Declarations
Acknowledgements
We acknowledge that this paper has been influenced by many discussions with colleagues over the years and insights from collaborative projects on the topic of assessing the impact of research, for example in consultations on a research utilisation project at WHO. We thank the two peer reviewers, Mark Petticrew and Liz Farmer for valuable comments and suggestions that have strengthened the paper. Special thanks also to Steve Hanney from the Health Economics Research Group (HERG) at Brunel University and to Andy Haines, Nick Black and Jo Lines at the London School of Hygiene and Tropical Medicine for their ongoing advice and support. We also thank John Browne, Val Curtis, Hazel Dockrell, Pat Doyle, Lucy Gilson, Judith Green, Sari Kovats, Anne Mills, Ian Roberts, Charlotte Watts, John Whittaker, Paul Wilkinson, Cathy Zimmerman, the LSHTM Senior Management Team and other colleagues who contributed to the development and testing of the Research Impact Framework at the London School of Hygiene and Tropical Medicine.
This project received no external funds and was supported by the authors' institutions in that the authors worked and consulted with colleagues there: the London School of Hygiene and Tropical Medicine for SK, NM and GW and Rutgers University for AP. The research ethics committee at the London School of Hygiene and Tropical Medicine approved the study that informed the development of the Research Impact Framework.
Authors’ Affiliations
References
- Cozzens S: The knowledge pool: measurement challenges in evaluating fundamental research programs. Evaluation and Program Planning. 1997, 20: 77-89. 10.1016/S0149-7189(96)00038-9.Google Scholar
- Jordan GB, Malone EL: Performance Assessment. Management Benchmarking Study, Washington Research Evaluation Network. 2002, Washington DC, U.S. Department of Energy, Office of ScienceGoogle Scholar
- Woodward D, Drager N, Beaglehole R, Lipson D: Globalization and health: a framework for analysis and action. Bull World Health Organ. 2001, 79 (9): 875-881.PubMedPubMed CentralGoogle Scholar
- Campbell C, Wood R, Kelly M: Social capital and health. 1999, London, UK , Health Education AuthorityGoogle Scholar
- Kawachi I: Social capital for health and human development. Development. 2001, 44 (1): 31-35. 10.1057/palgrave.development.1110211.Google Scholar
- Kuruvilla S, Mays N, Walt G: Describing the impacts of health services and policy research. J Health Serv Res Policy. 2007.Google Scholar
- Hanney S, Davies A, Buxton M: Assessing the benefits from health research projects: can we use questionnaires instead of case studies?. Research Evaluation. 1999, 8 (3): 189-199.Google Scholar
- Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, Peacock R: Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Soc Sci Med. 2005, 61 (2): 417-430. 10.1016/j.socscimed.2004.12.001.PubMedGoogle Scholar
- Bacon F: The Philosophical Works of Francis Bacon. Edited by: Robertson MJ. 1905, London, UK , Routledge & Kegan Paul, 1626.Google Scholar
- Polyani M: The Republic of Science: Its political and economic theory. Minerva. 1962, 1 (54): 73.Google Scholar
- Bush V: The Endless Frontier. A report to the President on a program for post-war scientific research. 1945, Washington DC, USA , US Government Printing OfficeGoogle Scholar
- Collins HM, Evans R: The third wave of science studies: Studies of expertise and experience. Social Studies of Science. 2002, 32 (2): 235-296.Google Scholar
- Gibbons M, Limoges C, Nowotny H, Schwartzman S, Scott P, Trow M: The new production of knowledge. 1994, London, UK , SageGoogle Scholar
- Kuhn TS: The structure of scientific revolutions. 1962, Chicago, US , University of Chicago PressGoogle Scholar
- Callon M: Is science a public good? Fifth Mullins lecture, Virginia Polytechnic Institute. Science, Technology & Human Values. 1993, 19 (4): 395-424.Google Scholar
- Nutley S, Walter I, Davies HT: From knowing to doing: a framework for understanding the evidence-into-practice agenda. Evaluation. 2003, 9 (2): 125-148. 10.1177/1356389003009002002.Google Scholar
- Weiss C: The many meanings of research utilization. Public Administration Review. 1979, 39: 426-431. 10.2307/3109916.Google Scholar
- Buxton M, Hanney S: How can payback from health research be assessed?. J Health Serv Res Policy. 1996, 1: 35-43.PubMedGoogle Scholar
- Haines A, Kuruvilla S, Borchert M: Bridging the implementation gap between knowledge and action for health. Bull World Health Organ. 2004, 82 (10): 724-731.PubMedPubMed CentralGoogle Scholar
- Kuruvilla S, Mays N: Reorienting health-research communication. Lancet. 2005, 366: 1416-1418. 10.1016/S0140-6736(05)67580-0.PubMedGoogle Scholar
- Black N, Carter S: Public accountability: one rule for practitioners, one for scientists?. J Health Serv Res Policy. 2001, 6 (3): 130-132. 10.1258/1355819011927369.PubMedGoogle Scholar
- Report of the Commission on Health Research for Development: Health research - essential link to equity in development. 1990, New York , Oxford University PressGoogle Scholar
- Pang T, Sadana R, Hanney S, Bhutta Z, Hyder A, Simon J: Knowledge for better health - a conceptual framework and foundation for health research systems. Bull World Health Organ. 2003, 81: 815-820.PubMedGoogle Scholar
- Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O: Diffusion of innovations in service organizations: systematic review and recommendations . Milbank Quarterly. 2004, 82 (4): 581-629. 10.1111/j.0887-378X.2004.00325.x.PubMedPubMed CentralGoogle Scholar
- Ostrom E: Institutional rational choice: an assessment of the institutional analysis and development framework. Theories of the Policy Process. Edited by: Sabatier PA. 1999, Boulder, Colorado , Westview Press, 35-71.Google Scholar
- Lavis JN, Ross SE, McLeod CB, Gildiner A: Measuring the impact of health research. J Health Serv Res Policy. 2003, 8 (3): 165-170. 10.1258/135581903322029520.PubMedGoogle Scholar
- Haynes BHA: Barriers and bridges to evidence based clinical practice. BMJ. 1999, 319 (1497).Google Scholar
- McClellan M, Heidenrich P: Biomedical research and then some: the causes of technological change for heart disease. 1999, New York, USA , Lasker Charitable Trust: Funding FirstGoogle Scholar
- Figueroa ME, Kincaid DL, Manju R, Lewis G: Communication for social change: an integrated model for measuring the process and its outcomes. The Communication for Social Change Working Paper Series. Edited by: Rockefeller Foundation and Johns Hopkins University Center for Communication Programs. 2002, New York , Rockefeller Foundation, No.1.Google Scholar
- Nutbeam D: Evaluating health promotion - progress, problems, and solutions. Health Promotion International. 1998, 13 (1): 27-44. 10.1093/heapro/13.1.27.Google Scholar
- Hanson K, Ranson K, Oliveira-Cruz V, Mills A: Constraints to scaling up health interventions: a conceptual framework and empirical analysis. CMH Working Paper Series Paper No WG: 14. 2001, Commission on Macroeconomics and HealthGoogle Scholar
- OECD - Organization for Economic Cooperation and Development: The measurement of scientific and technical activities: Proposed guidelines for collecting and interpreting technological innovation data (Oslo Manual). 1997, Paris , OECDGoogle Scholar
- Manning FJ, McGreary M, Estabrook R: NIH Extramural Center Programs: criteria for initiation and evaluation. Committee for Assessment of NIH Centers of Excellence Programs: Board of Health Sciences Policy. 2004, Washington DC , Institute of Medicine of the National Academies; The National Academies PressGoogle Scholar
- Higher Education Council for England: RAE 2008 Panel criteria and working methods. Research Assessment Exercise. 2006, Bristol , HEFCEGoogle Scholar
- Patton MQ: Qualitative research & evaluation methods. 2001, Thousand Oaks, USA , SageGoogle Scholar
- Carson C, Hajat S, Armstrong B, Wilkinson P: Declining vulnerability to temperature-related mortality in London over the 20th century. Am J Epidemiol. 2006, 164 (1): 77-84. 10.1093/aje/kwj147.PubMedGoogle Scholar
- Curtis V, Cairncross S: Effect of washing hands with soap on diarrhoea risk in the community: a systematic review. Lancet Infect Dis. 2003, 3: 275-281. 10.1016/S1473-3099(03)00606-6.PubMedGoogle Scholar
- Webster J, Lines J, Bruce J, Armstrong Scellenberg JR, Hanson K: Which delivery systems reach the poor? A review of equity of coverage of ever-treated nets, never-treated nets, and immunisation to reduce child mortality in Africa. Lancet Infect Dis. 2005, 5 (11): 709-717. 10.1016/S1473-3099(05)70269-3.PubMedGoogle Scholar
- DFID Malaria Knowledge Programme: DFID Malaria Knowledge Programme. [http://www.lshtm.ac.uk/dfid/malaria/]
- Oommen J, Henry VK, Pidikaka M: Prevention and control of malaria through community effort: experiences of a tribal community in rural Orissa, India. Regional Health Forum, WHO South-East Asia Region. 1999, 3 (1): 9-12.Google Scholar
- World Health Organization, London School of Hygiene and Tropical Medicine, European Commission - Energy Environment and Sustainable Development, German Weather Service (DWD): Heat waves: risks and responses. Health and Global Environmental Change Series, No 2. 2004, Copenhagen , WHO/ EuropeGoogle Scholar
- Sonkin B, Edwards P, Roberts I, Green J: Walking, cycling and transport safety: an analysis of child road deaths. J R Soc Med. 2006, 99 (8): 402-405. 10.1258/jrsm.99.8.402.PubMedPubMed CentralGoogle Scholar
- World Health Organisation, London School of Hygiene and Tropical Medicine, Daphne Programme of the European Commission: WHO Ethical and Safety Guidelines for Interviewing Trafficked Women. 2003, Geneva , WHOGoogle Scholar
- Verzilli CJ, Stallard N, Whittaker JC: Bayesian graphical models for genomewide association studies. Am J Hum Genet. 2006, 79 (1): 100-112. 10.1086/505313.PubMedPubMed CentralGoogle Scholar
- The Cochrane Collaboration: The Cochrane Collaboration. Evidence based healthcare. [http://www.cochrane.org/docs/ebm.htm]
- The Campbell Collaboration: The Campbell Collaboration. What helps? What harms? Based on what evidence?. [http://www.campbellcollaboration.org]
- Mays N, Pope C, Popay J: Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Policy. 2005, 10 (Suppl 1): 6-20. 10.1258/1355819054308576.PubMedGoogle Scholar
- Price DJS: Citation measures of hard science, soft science, technology and nonscience. Communication among scientists and engineers. Edited by: Nelson CE, Pollock DK. 1970, Lexington, Massachusetts , Heath Lexington BooksGoogle Scholar
- Narin F, Hamilton KS, Olivastro D: The increasing linkage between U.S. technology and public science. Research Policy. 1997, 26 (3): 317-330. 10.1016/S0048-7333(97)00013-9.Google Scholar
- Moran M: A breakthrough in R&D for neglected diseases: new ways to get the drugs we need. PLoS Medicine. 2 (9): e32.Google Scholar
- Oliver S, Clarke-Jones L, Rees R, Milne R, Buchanan P, Gabbay J, Gyte G, Oakley A, Stein K: Involving consumers in research and development agenda setting for the NHS: Developing an evidence-based approach. Health Technology Assessment. 2004, 8 (15): 1-148.PubMedGoogle Scholar
- Platt J: Research dissemination: A case study. The Quarterly Journal of Social Affairs. 1987, 3 (3): 181-198.Google Scholar
- Lomas J: Using 'Linkage and Exchange' to move research into policy at a Canadian foundation. Health Affairs. 2000, 19 (3): 236-240. 10.1377/hlthaff.19.3.236.PubMedGoogle Scholar
- Wright J, Parry JM, Scully E: Institutionalizing policy-level health impact assessment in Europe: is coupling health impact assessment with strategic environmental assessment the next step forward? . Bull World Health Organ. 2005, 83 (6): 472-477.PubMedPubMed CentralGoogle Scholar
- Global Forum for Health Research: The 10/90 Report on Health Research. 2000, Switzerland , Global Forum for Health Research, 1-155.Google Scholar
- Feller I, Gamota G, Valdez W: Assessing appropriateness: developing science indicators for basic science offices within mission agencies. Research Evaluation. 2003, 12 (1): 71-79.Google Scholar
- Merton R: The normative structure of science. The sociology of science: theoretical and empirical investigations. Edited by: Storer N. 1973, Chicago , Chicago University Press, 267-278.Google Scholar
- Haas PM: Introduction: epistemic communities and international policy coordination. International Organisation. 1992, 46 (1): 1-36.Google Scholar
- Grimshaw J, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, Grilli R, Harvey E, Oxman A, O'Brien MA: Changing provider behaviour: an overview of systematic reviews of interventions. Medical Care. 2001, 39 (Supplement 2): 12-45.Google Scholar
- Pleasant A, Kuruvilla S, Zarcadoolas C, Shanahan J, Lewenstein B: A framework for assessing public engagement with health research. Technical Report to WHO. 2003, Geneva , World Health OrganizationGoogle Scholar
- Wallack L: Mass media and health promotion: promise, problem, and challenge. Mass communication and public health: complexities and conflicts. Edited by: Atkins C, Wallack L. 1990, Newbury Park, US , Sage Publications, 13-40.Google Scholar
- Black N: Evidence based policy: proceed with care. BMJ. 2001, 323: 275-279. 10.1136/bmj.323.7307.275.PubMedPubMed CentralGoogle Scholar
- University of Warwick, London School of Hygiene & Tropical Medicine, Office of the Deputy Prime Minister: London: Statistical Evidence to Support the Housing Health and Safety Rating System. 2003, London , Office of the Deputy Prime Minister (ODPM)Google Scholar
- Weiss CH: Have we learned anything new about the use of evaluation. American Journal of Evaluation. 1998, 5 (4): 468-486.Google Scholar
- Hanney S, Gonzalez-Block M, Buxton M, Kogan M: The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Research Policy and Systems. 2003, 1:2.Google Scholar
- Gilson L: Key note address: Confronting inequity in South Africa: the interface between research and policy: Toronto. 2002Google Scholar
- Judge K, Mays N: Equity in the NHS: allocating resources for health and social care in England. BMJ. 1994, 308: 1363-1366.PubMedPubMed CentralGoogle Scholar
- Foucault M: The birth of the clinic. Trans. Sheridan A. London , Tavistock: 1973/1963Google Scholar
- Edwards P, Arango M, Balica L, Cottingham R, El-Sayed H, Farrell B, Fernandes J, Gogichaisvili T, Golden N, Hartzenberg B, Husain M, Ulloa MI, Jerbi Z, Khamis H, Komolafe E, Laloe V, Lomas G, Ludwig S, Mazairac G, Munoz SML, Nasi L, Olldashi F, Plunkett P, Roberts I, Sandercock P, Shakur H, Soler C, Stocker R, Svoboda P, Trenkler S, Venkataramana NK, Wasserberg J, Yates D, Yutthakasemsunt S, CRASH trial collaborators: Final results of MRC CRASH, a randomised placebo-controlled trial of intravenous corticosteroid in adults with head injury-outcomes at 6 months. The Lancet. 2005, 365 (9475): 1957-1959. 10.1016/S0140-6736(05)66552-X.Google Scholar
- Buse K, Mays N, Walt G: Making health policy. 2005, Maidenhead, UK , Open University PressGoogle Scholar
- Weiss C: Knowledge creep and decision accretion. Knowledge: Creation, Diffusion, Utilization. 1980, 1: 381-404.Google Scholar
- Vella K, Goldfrad C, Rowan K, Bion J, Black N: Use of consensus development to establish national research priorities in critical care. BMJ. 2000, 320: 976-980. 10.1136/bmj.320.7240.976.PubMedPubMed CentralGoogle Scholar
- Drager N, McClintock E, Moffitt M: Negotiating health development: A guide for practitioners. 2000, Conflict Management Group and World Health Organization, [http://www.cct-freiburg.de/who/publications.htm]Google Scholar
- World Health Organization: Avian influenza: Epidemic and pandemic alert and response. [http://www.who.int/csr/disease/avian_influenza/en/index.html]
- World Health Organisation: The World Health Report 2002. 2002, Geneva, Switzerland , World Health OrganizationGoogle Scholar
- Kickbusch I: Approaches to an ecological base for public health. Health Promotion. 1989, 4: 265-268.Google Scholar
- Department for Environment Food and Rural Affairs: Securing the future. 2005, London, UK , DEFRAGoogle Scholar
- Robertson LS: Motor vehicle deaths: failed policy analysis and neglected policy. J Public Health Policy. 2006, 27 (2): 182-189. 10.1057/palgrave.jphp.3200074.PubMedGoogle Scholar
- Daily G, Ellison K: The new economy of nature: the quest to make conservation profitable. 2002, Washington DC , Island PressGoogle Scholar
- Kennedy AD, Sculpher MJ, Coulter A, Dwyer N, Rees M, Abrams KR, Horsley S, Cowley D, Kidson C, Kirwin C, Naish C, Stirrat G: Effects of decision aids for menorrhagia on treatment choices, health outcomes, and costs: a randomized controlled trial. JAMA. 2002, 288 (21): 2701-2708. 10.1001/jama.288.21.2701.PubMedGoogle Scholar
- O'Connor AM, Rostom A, Fiset V, Tetroe JM, Entwistle V, Llewellyn-Thomas H, et al: Decision aids for patients facing health treatment or screening decisions: Systematic review. BMJ. 1999, 319: 731-734.PubMedPubMed CentralGoogle Scholar
- Lohr KN: Ensuring Quality Cancer Care. 1990, Washington, DC , National Academy PressGoogle Scholar
- The Royal College of Surgeons of England, The British Association of Otorhinolaryngologists – Head and Neck Surgeons Comparative Audit Group, The Clinical Effectiveness Unit (Health Services Research Unit of the London School of Hygiene and Tropical Medicine): National prospective tonsillectomy audit: final report of an audit carried out in England and Northern Ireland between July 2003 and September 2004. 2005, London , The Royal College of SurgeonsGoogle Scholar
- Shengelia B, Tandon A, Adams OB, Murray CJ: Access, utilization, quality, and effective coverage: an integrated conceptual framework and measurement strategy. Soc Sci Med. 2005, 61 (1): 97-109. 10.1016/j.socscimed.2004.11.055.PubMedGoogle Scholar
- Tengilimoglu D, Celik Y, Ulgu M: Comparison of computing capability and information system abilities of state hospitals owned by Ministry of Labor and Social Security and Ministry of Health. J Med Syst. 2006, 30 (4): 269-275. 10.1007/s10916-005-9008-6.PubMedGoogle Scholar
- Mays N, Wyke S, Evans D: The evaluation of complex health policy: lessons from the UK Total Purchasing Experiment. Evaluation. 2001, 7 (4): 405-426. 10.1177/13563890122209801.Google Scholar
- Buxton M, Hanney S, Jones T: Estimating the economic value to societies of the impact of health research: a critical review. Bull World Health Organ. 2004, 82 (10): 733-739.PubMedPubMed CentralGoogle Scholar
- Hanson K, Kikumbih N, Armstrong SJRM, Mponda H, Nathan R, Lake S, Mills A, Tanner M, Lengeler C: Cost-effectiveness of social marketing of insecticide-treated nets to prevent malaria in the United Republic of Tanzania. Bull World Health Organ. 2003, 81 (4): 269-276.PubMedPubMed CentralGoogle Scholar
- Green J, Free C, Bhavani V, Newman T: Translators and mediators: bilingual young people’s accounts of their interpreting work in health care. Soc Sci Med. 2005, 60: 2097-2110. 10.1016/j.socscimed.2004.08.067.PubMedGoogle Scholar
- Lee CJ, Scheufele DA, Lewenstein BV: Public attitudes toward emerging technologies: examining the interactive effects of cognitions and affect on public attitutes toward nanotechnology. Science Communication. 2005, 27 (2): 240-267. 10.1177/1075547005281474.Google Scholar
- Zarcadoolas C, Pleasant A, Greer D: Health literacy: an expanded model. Health Promotion International. 2005, 20: 195 -1203. 10.1093/heapro/dah609.PubMedGoogle Scholar
- Nielsen-Bohlman L, Panzer AM, Kindig DA: Health literacy: a prescription to end confusion. 2004, Washington DC, Institute of Medicine and the National Academies, [http://www.iom.edu/report.asp?id=19723]Google Scholar
- Coast J: Is economic evaluation in touch with society's health values?. BMJ. 2004, 329: 1233-1236. 10.1136/bmj.329.7476.1233.PubMedPubMed CentralGoogle Scholar
- Hunt P: Interim report of the Special Rapporteur of the Commission on Human Rights on the right of everyone to enjoy the highest attainable standard of physical and mental health. United Nations General Assembly fifty-eighth session, Agenda item 117 (c). 2003Google Scholar
- World Health Organization: Comparative evaluation of indicators for gender equity and health. WHO/WKC/TechSer/032. Kobe , WHO Kobe Centre. 2003, 320.Google Scholar
- Global Forum for Health Research: Monitoring financial flows for health research. 2001, Geneva, Switzerland , World Health OrganizationGoogle Scholar
- Stephens C, Porter J, Nettleton C, Willis R: Disappearing, displaced, and undervalued: a call to action for Indigenous health worldwide. Lancet. 2006, 17 (9527): 2019-2028. 10.1016/S0140-6736(06)68892-2.Google Scholar
- Manandhar DS, Osrin D, Shrestha BP, Mesko N, Morrison J, Tumbahangphe KM, Tamang S, Thapa S, Shrestha D, Thapa B, Shrestha JR, Wade A, Borghi J, Standing H, Manandhar M, Costello AML, members of the MIRA Makwanpur trial team: Effect of a participatory intervention with women’s groups on birth outcomes in Nepal: cluster-randomised controlled trial . Lancet. 2004, 364: 970-979. 10.1016/S0140-6736(04)17021-9.PubMedGoogle Scholar
- Water and Sanitation Program, World Bank / Bank-Netherlands Water Partnership, World Bank Water Anchor, London School of Hygiene and Tropical Medicine, Academy for Educational Development, USAID/Hygiene Improvement Project, UNICEF, USAID, Colgate-Palmolive, Unilever, Procter & Gamble, Centers for Disease Control and Prevention: The Global Public-Private Partnership for Handwashing with Soap. [http://www.globalhandwashing.org/Aboutus.htm]
- Weisbrod B: Economics and medical research. 1983, Washington DC, US , American Enterprise InstituteGoogle Scholar
- Brouwer WBF, Koopmanschap MA: On the economic foundations of CEA. Ladies and gentlemen, take your positions! . Journal of Health Economics. 2000, 19: 439–459-10.1016/S0167-6296(99)00038-7.PubMedGoogle Scholar
- Worcester R: Whom do we trust? Neither politicians nor journalists!. [http://www.ipsos-mori.com/publications/rmw/whomdowetrust.shtml]
- Dierkes M, von Grote C: Between understanding and trust: the public, science and technology. 2000, Amsterdam, Netherlands , Harwood Academic Publishers, i-373.Google Scholar
- Boas F: Methods of ethnology. American Anthropologist. 1920, 22: 311-322. 10.1525/aa.1920.22.4.02a00020.Google Scholar
- Hall S: Gramsci's relevance for the study of race and ethnicity. Critical dialogues in cultural studies. Edited by: Morley D, Chew K. 1996, New York , RoutledgeGoogle Scholar
- Durie M: Understanding health and illness: research at the interface between science and indigenous knowledge. Int J Epidemiol. 2004, 33 (5): 1138-1143. 10.1093/ije/dyh250.PubMedGoogle Scholar
- Kirby D: Science advisors, representation, and Hollywood films. Molecular Interventions. 2003, 3 (2): 54-60. 10.1124/mi.3.2.54.PubMedGoogle Scholar
- Robbins A: Film review: The Constant Gardener. J Public Health Policy. 2006, 27 (2): 211-212. 10.1057/palgrave.jphp.3200072.Google Scholar
- Hamilton CM, Petticrew M: Spending (slightly) less on health and more on the arts. Response: Robust research is needed. BMJ. 2003, 326: 660-10.1136/bmj.326.7390.660/a.PubMed CentralGoogle Scholar
- The Wellcome Trust: The Wellcome Trust. Wellcome Collection. [http://www.wellcome.ac.uk/node6500.html]
- United Nations: Report of the United Nations Conference on Environment and Development: Rio Declaration on Environment and Development. General Assembly. 1992, A/CONF.151/26 (Vol. I).Google Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/6/134/prepub
Pre-publication history
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
