Paucity of qualitative research in general medical and health services and policy research journals: analysis of publication rates

Background

Qualitative research has the potential to inform and improve health care decisions but a study based on one year of publications suggests that it is not published in prominent health care journals. A more detailed, longitudinal analysis of its availability is needed. The purpose of this study was to identify, count and compare the number of qualitative and non-qualitative research studies published in high impact health care journals, and explore trends in these data over the last decade.

Methods

A bibliometric approach was used to identify and quantify qualitative articles published in 20 top general medical and health services and policy research journals from 1999 to 2008. Eligible journals were selected based on performance in four different ranking systems reported in the 2008 ISI Journal Citation Reports. Qualitative and non-qualitative research published in these journals were identified by searching MEDLINE, and validated by hand-searching tables of contents for four journals.

Results

The total number of qualitative research articles published during 1999 to 2008 in ten general medical journals ranged from 0 to 41, and in ten health services and policy research journals from 0 to 39. Over this period the percentage of empirical research articles that were qualitative ranged from 0% to 0.6% for the general medical journals, and 0% to 6.4% for the health services and policy research journals.

Conclusions

This analysis suggests that qualitative research it is rarely published in high impact general medical and health services and policy research journals. The factors that contribute to this persistent marginalization need to be better understood.

Historically, quantitative research has been the most sought after evidence to support health care decision making by clinicians, managers and policy makers, henceforth referred to as users. However, many important questions are not easily answered by quantitative methods, and decisions may be sub-optimal in complex health care environments where quantitative data alone do not address varied information needs [1]. Users also require contextual information about the feasibility and appropriateness of interventions, data that could be supplied by qualitative research [2–6]. Qualitative methods allow complex issues to be studied, and can produce rich data on perceptions, beliefs, experiences and behavior to create a thorough understanding of a problem, and how it could be resolved [7]. Qualitative research approaches have been used to improve health service delivery for a variety of clinical conditions and settings [8–14].

Long characterized as anecdotal or subject to biases, qualitative research has had a much lower profile than quantitative research in health care decision making [5, 6]. This may be partly related to the observation that few qualitative studies appear to be published in major health care journals, which remain a primary means of disseminating research. For example, McKibbon's analysis of qualitative studies published in clinical journals during the year 2000 showed that 0.6% of research articles published in 170 general medical, mental health and nursing journals reviewed were qualitative [15]. Their study found that the majority (61.0%) of qualitative studies were published in 17 nursing journals, while few were published in what were considered high impact journals. They also noted that four of the top 20 journals for the year 2000 published 15 qualitative studies, of which 12 were published in the BMJ.

Interpretation of these findings is limited because they were based on studies published during a single year in a convenience sample of journals articles that had been assembled for a specific research project. While some assert that interest in qualitative research is on the rise, it is unclear whether publication rates of qualitative research in prominent sources have similarly risen [16, 17]. A more detailed, longitudinal analysis of the availability of qualitative research in top health care journals is needed. The purpose of our study was to explore whether qualitative research publication rates increased over a ten year period from 1999 to 2008 in health care journals sampled based on their impact in the international scientific community.

Approach

A literature search was conducted to identify and count the number of qualitative research articles compared with the total number of research articles published in top ranked health care journals over the last decade, including general medical, and health services and policy research journals. Ethics approval was not required since analysis was based on publicly available data.

Sampling

The most frequently top ranked ten general medical, and ten health services and policy research journals were identified based on performance across four ranking systems reported in the 2008 ISI Journal Citation Reports (two-year and five-year impact factor, Eigenfactor, ArticleInfluence). We chose the ten most highly ranked journals across the four systems for general medical and health services and policy journals since they each generate variable ranking, and there is no consensus on which approach is most accurate [18]. Eligible journals are presented in Table 1.

Data collection

MEDLINE was searched to identify the total number of qualitative articles published in each of the 20 eligible journals over the period from 1999 to 2008 using the strategy in Table 2. Search strategies for identifying qualitative research in MEDINE have been developed [19]. Others have shown that searching for qualitative research in MEDLINE involves trade-offs between recall and precision [20, 21]. That study found that even the search with the highest recall resulted in poor precision, with 96% of identified items deemed irrelevant. Therefore we opted to use a simple search strategy, assuming that limitations in accurately identifying qualitative research applied equally to each journal.

Two authors (ARG, MJD) independently reviewed titles and abstracts to identify qualitative research studies. Eligible studies included program evaluations, case studies, interviews, focus groups, content analysis of documents or discourse analysis, or field observation focused on any type of policy, management or clinical aspect of health care delivery or organization; which explored, described or compared knowledge, attitudes, beliefs, opinions, views, experiences, behaviour, practices, and contexts or environmental factors influencing any of these issues; and provided methodological details of sampling, recruitment and analysis. Ineligible studies included randomized controlled trials or clinical cohort studies that may have incorporated a qualitative component, interviews involving time trade off choices or close-ended questionnaires, or studies where interviews were conducted to develop questionnaire content but emphasis of the study was on reporting of psychometric testing with little or no qualitative methodologic details, and narrative systematic reviews. Independent selections were tabulated, and full-text articles were retrieved and reviewed to resolve discrepancies.

To validate our findings, additional checks were performed. Tables of contents (or abstract or methods section of article if necessary) were scanned across two different years for two general medical (CMAJ 2001, BMJ 2006) and two health services and policy research journals (Medical Care 2001, Health Affairs 2006) to quantify the number of qualitative and non-qualitative studies, and compare these data with the MEDLINE results. One author independently reviewed selections of qualitative studies made by a research assistant from tables of contents for the four journals.

Data analysis

The number of qualitative and non-qualitative research studies identified in MEDLINE and tables of contents were quantified, and the percentage of total studies that were qualitative calculated per journal per year over the ten year period from 1999 to 2008. This data was scanned to identify changes but trends were not analyzed statistically.

Tables 3 and 4 show the number of qualitative and total studies identified by the literature search, and the percentage of qualitative studies for general medical and health services and policy research journals, respectively. The total number of qualitative studies published in general medical journals from 1999 to 2008 ranged from 0 in the Journal of Internal Medicine and New England Journal of Medicine to 41 in the BMJ. The percentage of qualitative studies in general medical journals over this period ranged from 0.0% to 0.6%. No trends in the number of qualitative studies published yearly were apparent except for a small peak in 2002 for the BMJ.

The total number of qualitative studies published in health services and policy research journals from 1999 to 2008 ranged from 0 in the Journal of Health Economics to 39 in Quality & Safety in Health Care. The percentage of qualitative studies in health services and policy research journals over this period ranged from 0.0% to 6.4%. No trends in the number of qualitative studies published yearly were apparent.

Table 5 outlines the findings of the validity check, comparing the number of qualitative and total research studies identified by literature search and by tables of contents search in a sample of eligible journals. In both health services and policy research journals examined, there were minimal differences in the number of qualitative and non-qualitative studies identified by the literature and tables of contents searches. For one general medical journal, the table of contents search retrieved more qualitative studies. For both general medical journals, the literature search retrieved more non-qualitative studies compared with the table of contents search.

This study found that very few qualitative studies were published in 20 high impact general medical and health services and policy research journals relative to non-qualitative research, and publishing rates of qualitative studies in these journals remained consistently low over the period from 1999 to 2008.

Our findings based on a decade of published research in general medical and health services and policy research journals are similar to those of one study that investigated qualitative research publication rates which reported that 0.6% of studies published in 170 general medical, mental health and nursing journals during the year 2000 were qualitative [15]. Our findings differ from those reported by Weiner et al. in a ten year scan of nine health services and policy research journals from 1998 to 2008, which found that 9% of research articles were qualitative [22]. However, their purpose and methods differed from ours. They focused on the extent to which health services researchers used qualitative methods and for what purpose so they identified qualitative articles and extracted information about the type of qualitative design and how qualitative methods were reported. With respect to methods, it is unclear how they assembled a bibliographic library of all articles published in the nine journals during the specified time period. They sampled the nine journals from those considered important in a survey of health administration faculty in American business schools, whereas we sampled based on several measures of impact. As a result we reviewed the content of three of the nine health services and policy research they examined. In addition we examined general medical journals. Eligibility criteria also differed. Weiner et al. included case studies using both qualitative and quantitative data collection methods (21% of the qualitative studies they identified), and quantitative surveys, whereas we did not. Peripherally relevant research developed optimal search strategies for identifying qualitative research in the nursing [23] and breastfeeding [19] literature, and examined the methods used in studies published in two qualitative research journals [24]. Hence, our findings that consistently few qualitative research studies are published in prominent health care journals are current and unique.

This study is based on the premise that qualitative data are important in health care decision making. While there is no definitive evidence to support this assertion, there are several examples of ways in which qualitative information can be used to improve the quality of health care delivery [8–14]. Still, qualitative research may be considered a "second class citizen" by users unfamiliar with its philosophy and methods [25, 26]. It is notable that many general medical and health services and policy research journals devote considerable space and attention to topical issues in the form of commentaries which, despite often being authored by recognized experts, are largely based on anecdotes or opinion, while rigorously conducted qualitative research is not routinely published.

One study found that health professionals believe qualitative research lacks scientific accuracy [27]. If such views about qualitative research are widespread, this may be contributing to the low publication rates in high impact journals demonstrated by this research study. What would constitute definitive evidence of the impact and need for qualitative research, such that it would convince different users of its validity? What is the relevance of qualitative research to users when delivered in various formats. Interviews with different types of health professionals could explore these issues to generate greater understanding about whether and how they consider and use qualitative research. For example, meta-synthesis is an emerging means by which to integrate qualitative research on a common topic into what might be considered strong evidence for decision making [28, 29]. At the same time, it is acknowledged that qualitative research might be held in higher regard if qualitative studies were of consistently higher quality, and agreement was established, even among trained qualitative researchers, on various approaches and methods [30]. While guidance is available for conducting and appraising qualitative research [25, 31] other forms of education may be required to inform different users about the nature and applications of qualitative research. It is not clear whether journal editors and reviewers use these criteria, or even whether and how the journals referred to in this study accommodate qualitative research by accepting its submission, providing authorship guidelines for qualitative studies, including individuals with qualitative expertise on editorial boards, or training reviewers or providing them with tools to evaluate qualitative submissions [26, 32]. BMJ appeared to have a small spike in the number of qualitative articles published. Interviews with journal editors may provide insight into policies and processes that influence whether and how qualitative research is considered and published. At the same time it would be useful to explore the role that qualitative researchers play in qualitative research publishing trends, including the decisions they make with respect to where they submit manuscripts based on qualitative research, and whether such research is targeted to specialty journals rather then general medical and health services and policy research journals.

The accuracy of these data are limited by the capacity to execute searches in MEDLINE that distinguish empirical research from other publication types. We found that this was mainly true for non-qualitative studies. Validation checking by searching tables of contents showed that the number of non-qualitative studies was inflated in MEDLINE searches, so despite extremely low ratios of qualitative to non-qualitative studies reported here, they may still be somewhat over-estimated. However, the limitation likely applies to each journal, and the overall intent of the study was to explore the degree to which qualitative research is published and whether this has increased over time, rather than generating an accurate statistic. Moreover, the actual number of qualitative articles identified in MEDLINE matched, or was quite similar to the number identified in tables of contents searching, which confirms the paucity of qualitative studies published in high impact general medical and health services and policy research journals. Interpretation of the implications of these data may be limited by the calculation of the journal impact factor metric. While much debated, this has been shown to be an accurate statistic [18]. However, to alleviate any concerns, top-ranked journals were selected from among those most frequently represented across four impact ranking systems. It may be that there are fewer qualitative compared with quantitative researchers. Even if this were so, the number of qualitative articles published in major health care journals appears to be so low that there are likely other contributing factors. It might not be the mandate of general medical or health services and policy research journals to publish qualitative research, but journal publication policies were not examined.

Although qualitative research has the potential to inform and improve health care decisions, this analysis suggests that it is rarely published in high impact general medical and health services and policy research journals. The factors that contribute to this persistent marginalization need to be explored. More insight on a variety of users knowledge of, and views on the utility of qualitative research is needed to better understand how its appropriate use could be expanded.

Author notes

Affiliations

1. Departments of Surgery; and Department of Health Policy, Management and Evaluation, Institute of Medical Science, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada

   • Anna R Gagliardi

2. Cancer Services and Policy Research Unit, Cancer Care Ontario, Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada

   • Mark J Dobrow

Corresponding author

Correspondence to Anna R Gagliardi.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

Both ARG and MJD conceptualized and planned this research study, collected and analyzed data, summarized and interpreted data, prepared the manuscript, and approved this final version.

Anna R Gagliardi and Mark J Dobrow contributed equally to this work.

Reprints and Permissions