Skip to main content
Download PDF

Paucity of qualitative research in general medical and health services and policy research journals: analysis of publication rates

BMC Health Services Research volume 11, Article number: 268 (2011) Cite this article

Abstract

Background

Qualitative research has the potential to inform and improve health care decisions but a study based on one year of publications suggests that it is not published in prominent health care journals. A more detailed, longitudinal analysis of its availability is needed. The purpose of this study was to identify, count and compare the number of qualitative and non-qualitative research studies published in high impact health care journals, and explore trends in these data over the last decade.

Methods

A bibliometric approach was used to identify and quantify qualitative articles published in 20 top general medical and health services and policy research journals from 1999 to 2008. Eligible journals were selected based on performance in four different ranking systems reported in the 2008 ISI Journal Citation Reports. Qualitative and non-qualitative research published in these journals were identified by searching MEDLINE, and validated by hand-searching tables of contents for four journals.

Results

The total number of qualitative research articles published during 1999 to 2008 in ten general medical journals ranged from 0 to 41, and in ten health services and policy research journals from 0 to 39. Over this period the percentage of empirical research articles that were qualitative ranged from 0% to 0.6% for the general medical journals, and 0% to 6.4% for the health services and policy research journals.

Conclusions

This analysis suggests that qualitative research it is rarely published in high impact general medical and health services and policy research journals. The factors that contribute to this persistent marginalization need to be better understood.

Peer Review reports

Background

Historically, quantitative research has been the most sought after evidence to support health care decision making by clinicians, managers and policy makers, henceforth referred to as users. However, many important questions are not easily answered by quantitative methods, and decisions may be sub-optimal in complex health care environments where quantitative data alone do not address varied information needs [1]. Users also require contextual information about the feasibility and appropriateness of interventions, data that could be supplied by qualitative research [2–6]. Qualitative methods allow complex issues to be studied, and can produce rich data on perceptions, beliefs, experiences and behavior to create a thorough understanding of a problem, and how it could be resolved [7]. Qualitative research approaches have been used to improve health service delivery for a variety of clinical conditions and settings [8–14].

Long characterized as anecdotal or subject to biases, qualitative research has had a much lower profile than quantitative research in health care decision making [5, 6]. This may be partly related to the observation that few qualitative studies appear to be published in major health care journals, which remain a primary means of disseminating research. For example, McKibbon's analysis of qualitative studies published in clinical journals during the year 2000 showed that 0.6% of research articles published in 170 general medical, mental health and nursing journals reviewed were qualitative [15]. Their study found that the majority (61.0%) of qualitative studies were published in 17 nursing journals, while few were published in what were considered high impact journals. They also noted that four of the top 20 journals for the year 2000 published 15 qualitative studies, of which 12 were published in the BMJ.

Interpretation of these findings is limited because they were based on studies published during a single year in a convenience sample of journals articles that had been assembled for a specific research project. While some assert that interest in qualitative research is on the rise, it is unclear whether publication rates of qualitative research in prominent sources have similarly risen [16, 17]. A more detailed, longitudinal analysis of the availability of qualitative research in top health care journals is needed. The purpose of our study was to explore whether qualitative research publication rates increased over a ten year period from 1999 to 2008 in health care journals sampled based on their impact in the international scientific community.

Methods

Approach

A literature search was conducted to identify and count the number of qualitative research articles compared with the total number of research articles published in top ranked health care journals over the last decade, including general medical, and health services and policy research journals. Ethics approval was not required since analysis was based on publicly available data.

Sampling

The most frequently top ranked ten general medical, and ten health services and policy research journals were identified based on performance across four ranking systems reported in the 2008 ISI Journal Citation Reports (two-year and five-year impact factor, Eigenfactor, ArticleInfluence). We chose the ten most highly ranked journals across the four systems for general medical and health services and policy journals since they each generate variable ranking, and there is no consensus on which approach is most accurate [18]. Eligible journals are presented in Table 1.

Table 1 Selected general medical, and health services and policy research journals
Full size table

Data collection

MEDLINE was searched to identify the total number of qualitative articles published in each of the 20 eligible journals over the period from 1999 to 2008 using the strategy in Table 2. Search strategies for identifying qualitative research in MEDINE have been developed [19]. Others have shown that searching for qualitative research in MEDLINE involves trade-offs between recall and precision [20, 21]. That study found that even the search with the highest recall resulted in poor precision, with 96% of identified items deemed irrelevant. Therefore we opted to use a simple search strategy, assuming that limitations in accurately identifying qualitative research applied equally to each journal.

Table 2 Search strategy used to identify qualitative articles published in eligible journals during 1999 to 2008
Full size table

Two authors (ARG, MJD) independently reviewed titles and abstracts to identify qualitative research studies. Eligible studies included program evaluations, case studies, interviews, focus groups, content analysis of documents or discourse analysis, or field observation focused on any type of policy, management or clinical aspect of health care delivery or organization; which explored, described or compared knowledge, attitudes, beliefs, opinions, views, experiences, behaviour, practices, and contexts or environmental factors influencing any of these issues; and provided methodological details of sampling, recruitment and analysis. Ineligible studies included randomized controlled trials or clinical cohort studies that may have incorporated a qualitative component, interviews involving time trade off choices or close-ended questionnaires, or studies where interviews were conducted to develop questionnaire content but emphasis of the study was on reporting of psychometric testing with little or no qualitative methodologic details, and narrative systematic reviews. Independent selections were tabulated, and full-text articles were retrieved and reviewed to resolve discrepancies.

To validate our findings, additional checks were performed. Tables of contents (or abstract or methods section of article if necessary) were scanned across two different years for two general medical (CMAJ 2001, BMJ 2006) and two health services and policy research journals (Medical Care 2001, Health Affairs 2006) to quantify the number of qualitative and non-qualitative studies, and compare these data with the MEDLINE results. One author independently reviewed selections of qualitative studies made by a research assistant from tables of contents for the four journals.

Data analysis

The number of qualitative and non-qualitative research studies identified in MEDLINE and tables of contents were quantified, and the percentage of total studies that were qualitative calculated per journal per year over the ten year period from 1999 to 2008. This data was scanned to identify changes but trends were not analyzed statistically.

Results

Tables 3 and 4 show the number of qualitative and total studies identified by the literature search, and the percentage of qualitative studies for general medical and health services and policy research journals, respectively. The total number of qualitative studies published in general medical journals from 1999 to 2008 ranged from 0 in the Journal of Internal Medicine and New England Journal of Medicine to 41 in the BMJ. The percentage of qualitative studies in general medical journals over this period ranged from 0.0% to 0.6%. No trends in the number of qualitative studies published yearly were apparent except for a small peak in 2002 for the BMJ.

Table 3 Number/percentage of qualitative articles published in top general medical journals, 1999 to 2008
Full size table
Table 4 Number/percentage of qualitative articles published in top health services & policy research journals, 1999 to 2008
Full size table

The total number of qualitative studies published in health services and policy research journals from 1999 to 2008 ranged from 0 in the Journal of Health Economics to 39 in Quality & Safety in Health Care. The percentage of qualitative studies in health services and policy research journals over this period ranged from 0.0% to 6.4%. No trends in the number of qualitative studies published yearly were apparent.

Table 5 outlines the findings of the validity check, comparing the number of qualitative and total research studies identified by literature search and by tables of contents search in a sample of eligible journals. In both health services and policy research journals examined, there were minimal differences in the number of qualitative and non-qualitative studies identified by the literature and tables of contents searches. For one general medical journal, the table of contents search retrieved more qualitative studies. For both general medical journals, the literature search retrieved more non-qualitative studies compared with the table of contents search.

Table 5 Comparison of results achieved by literature and tables of contents searches
Full size table

Discussion

This study found that very few qualitative studies were published in 20 high impact general medical and health services and policy research journals relative to non-qualitative research, and publishing rates of qualitative studies in these journals remained consistently low over the period from 1999 to 2008.

Our findings based on a decade of published research in general medical and health services and policy research journals are similar to those of one study that investigated qualitative research publication rates which reported that 0.6% of studies published in 170 general medical, mental health and nursing journals during the year 2000 were qualitative [15]. Our findings differ from those reported by Weiner et al. in a ten year scan of nine health services and policy research journals from 1998 to 2008, which found that 9% of research articles were qualitative [22]. However, their purpose and methods differed from ours. They focused on the extent to which health services researchers used qualitative methods and for what purpose so they identified qualitative articles and extracted information about the type of qualitative design and how qualitative methods were reported. With respect to methods, it is unclear how they assembled a bibliographic library of all articles published in the nine journals during the specified time period. They sampled the nine journals from those considered important in a survey of health administration faculty in American business schools, whereas we sampled based on several measures of impact. As a result we reviewed the content of three of the nine health services and policy research they examined. In addition we examined general medical journals. Eligibility criteria also differed. Weiner et al. included case studies using both qualitative and quantitative data collection methods (21% of the qualitative studies they identified), and quantitative surveys, whereas we did not. Peripherally relevant research developed optimal search strategies for identifying qualitative research in the nursing [23] and breastfeeding [19] literature, and examined the methods used in studies published in two qualitative research journals [24]. Hence, our findings that consistently few qualitative research studies are published in prominent health care journals are current and unique.

This study is based on the premise that qualitative data are important in health care decision making. While there is no definitive evidence to support this assertion, there are several examples of ways in which qualitative information can be used to improve the quality of health care delivery [8–14]. Still, qualitative research may be considered a "second class citizen" by users unfamiliar with its philosophy and methods [25, 26]. It is notable that many general medical and health services and policy research journals devote considerable space and attention to topical issues in the form of commentaries which, despite often being authored by recognized experts, are largely based on anecdotes or opinion, while rigorously conducted qualitative research is not routinely published.

One study found that health professionals believe qualitative research lacks scientific accuracy [27]. If such views about qualitative research are widespread, this may be contributing to the low publication rates in high impact journals demonstrated by this research study. What would constitute definitive evidence of the impact and need for qualitative research, such that it would convince different users of its validity? What is the relevance of qualitative research to users when delivered in various formats. Interviews with different types of health professionals could explore these issues to generate greater understanding about whether and how they consider and use qualitative research. For example, meta-synthesis is an emerging means by which to integrate qualitative research on a common topic into what might be considered strong evidence for decision making [28, 29]. At the same time, it is acknowledged that qualitative research might be held in higher regard if qualitative studies were of consistently higher quality, and agreement was established, even among trained qualitative researchers, on various approaches and methods [30]. While guidance is available for conducting and appraising qualitative research [25, 31] other forms of education may be required to inform different users about the nature and applications of qualitative research. It is not clear whether journal editors and reviewers use these criteria, or even whether and how the journals referred to in this study accommodate qualitative research by accepting its submission, providing authorship guidelines for qualitative studies, including individuals with qualitative expertise on editorial boards, or training reviewers or providing them with tools to evaluate qualitative submissions [26, 32]. BMJ appeared to have a small spike in the number of qualitative articles published. Interviews with journal editors may provide insight into policies and processes that influence whether and how qualitative research is considered and published. At the same time it would be useful to explore the role that qualitative researchers play in qualitative research publishing trends, including the decisions they make with respect to where they submit manuscripts based on qualitative research, and whether such research is targeted to specialty journals rather then general medical and health services and policy research journals.

The accuracy of these data are limited by the capacity to execute searches in MEDLINE that distinguish empirical research from other publication types. We found that this was mainly true for non-qualitative studies. Validation checking by searching tables of contents showed that the number of non-qualitative studies was inflated in MEDLINE searches, so despite extremely low ratios of qualitative to non-qualitative studies reported here, they may still be somewhat over-estimated. However, the limitation likely applies to each journal, and the overall intent of the study was to explore the degree to which qualitative research is published and whether this has increased over time, rather than generating an accurate statistic. Moreover, the actual number of qualitative articles identified in MEDLINE matched, or was quite similar to the number identified in tables of contents searching, which confirms the paucity of qualitative studies published in high impact general medical and health services and policy research journals. Interpretation of the implications of these data may be limited by the calculation of the journal impact factor metric. While much debated, this has been shown to be an accurate statistic [18]. However, to alleviate any concerns, top-ranked journals were selected from among those most frequently represented across four impact ranking systems. It may be that there are fewer qualitative compared with quantitative researchers. Even if this were so, the number of qualitative articles published in major health care journals appears to be so low that there are likely other contributing factors. It might not be the mandate of general medical or health services and policy research journals to publish qualitative research, but journal publication policies were not examined.

Conclusions

Although qualitative research has the potential to inform and improve health care decisions, this analysis suggests that it is rarely published in high impact general medical and health services and policy research journals. The factors that contribute to this persistent marginalization need to be explored. More insight on a variety of users knowledge of, and views on the utility of qualitative research is needed to better understand how its appropriate use could be expanded.

References

  1. 1.

    Atkins D: Creating and synthesizing evidence with decision makers in mind. Med Care. 2007, 45 (Suppl 2): 16-22. 10.1097/MLR.0b013e3180616c3f.

    Article  Google Scholar 

  2. 2.

    Pearson A, Wiechula R, Court A, Lockwood C: A reconsideration of what constitutes evidence in the healthcare professions. Nurs Sci Quarterly. 2007, 20 (1): 85-88. 10.1177/0894318406296306.

    Article  Google Scholar 

  3. 3.

    Mays N, Pope C, Popay J: Systematically reviewing qualitative and quantitative evidence to inform management and policy making in the health field. J Health Serv Res Policy. 2005, 10 (Suppl 1): 6-20.

    Article  PubMed  Google Scholar 

  4. 4.

    Dobrow MJ, Goel V, Lemieux-Charles L, Black N: The impact of context on evidence utilisation: a framework for expert groups developing health policy recommendations. Soc Sci Med. 2006, 63 (7): 1811-24. 10.1016/j.socscimed.2006.04.020.

    Article  PubMed  Google Scholar 

  5. 5.

    Rusinova K, Pochard F, Kentish-Barnes N, et al: Qualitative research: Adding drive and dimension to clinical research. Crit Care Med. 2009, 37 (1): 140-6. 10.1097/CCM.0b013e318192faa3.

    Article  Google Scholar 

  6. 6.

    Dixon-Woods M, Fitzpatrick R, Roberts K: Including qualitative research in systematic reviews: Opportunities and problems. J Eval Clin Pract. 2001, 7 (2): 125-33. 10.1046/j.1365-2753.2001.00257.x.

    CAS  Article  PubMed  Google Scholar 

  7. 7.

    Auerbach CF, Silverstein LB: Qualitative data: an introduction to coding and analysis. 2003, New York: New York University Press

    Google Scholar 

  8. 8.

    Greenhalgh T, Humphrey C, Hughes J, Macfarlane F, Butler C, Pawson R: How do you modernize a health service? A realist evaluation of whole-scale transformation in London. Milbank Q. 2009, 87 (2): 391-416. 10.1111/j.1468-0009.2009.00562.x.

    Article  PubMed  PubMed Central  Google Scholar 

  9. 9.

    Staggers N, Jennings BM: The content and context of change of shift report on medical and surgical units. J Nurs Admin. 2009, 39 (9): 393-8. 10.1097/NNA.0b013e3181b3b63a.

    Article  Google Scholar 

  10. 10.

    Rhodes P, Giles SJ, Cook GA, et al: Assessment of the implementation of a national patient safety alert to reduce wrong site surgery. Qual Saf Health Care. 2008, 17 (6): 409-15. 10.1136/qshc.2007.023010.

    CAS  Article  PubMed  Google Scholar 

  11. 11.

    Bastiens H, Van Royen P, Pavlic DR, Raposo V, Baker R: Older peoples' preferences for involvement in their own care: A qualitative study in primary care in 11 European countries. Patient Educ Couns. 2007, 68 (1): 33-42. 10.1016/j.pec.2007.03.025.

    Article  Google Scholar 

  12. 12.

    Lehti A, Hammarstrom A, Mattsson B: Recognition of depression in people of different cultures: A qualitative study. BMC Fam Pract. 2009, 10: 53-10.1186/1471-2296-10-53.

    Article  PubMed  PubMed Central  Google Scholar 

  13. 13.

    Greenhalgh T: Integrating qualitative research into evidence based practice. Endocrinol Metab Clin North Am. 2002, 31 (3): 583-601. 10.1016/S0889-8529(02)00009-9.

    Article  PubMed  Google Scholar 

  14. 14.

    Sinuff T, Cook DJ, Giacomini M: How qualitative research can contribute to research in the intensive care unit. J Crit Care. 2007, 22 (2): 104-11. 10.1016/j.jcrc.2007.03.001.

    Article  PubMed  Google Scholar 

  15. 15.

    McKibbon KA, Gadd CS: A quantitative analysis of qualitative studies in clinical journals for the 2000 publishing year. BMC Med Inform Dec Mak. 2004, 4: 11-10.1186/1472-6947-4-11.

    Article  Google Scholar 

  16. 16.

    Elliot H, Popay J: How are policy makers using evidence? Models of research utilization and local NHS policy making. J Epidemiol Community Health. 2000, 54: 461-8. 10.1136/jech.54.6.461.

    Article  Google Scholar 

  17. 17.

    Jack SM: Utility of qualitative research findings in evidence-based public health practice. Public Health Nursing. 2006, 23 (3): 277-83. 10.1111/j.1525-1446.2006.230311.x.

    Article  PubMed  Google Scholar 

  18. 18.

    McVeigh ME, Mann SJ: The journal impact factors denominator. Defining citable (counted) items. JAMA. 2009, 302 (10): 1107-9. 10.1001/jama.2009.1301.

    CAS  Article  PubMed  Google Scholar 

  19. 19.

    Shaw RL, Booth A, Sutton AJ, Miller T, Smith JA, Young B, et al: Finding qualitative research: An evaluation of search strategies. BMC Medical Research Methodology. 2004, 4: 5-10.1186/1471-2288-4-5.

    Article  PubMed  PubMed Central  Google Scholar 

  20. 20.

    Mays N, Pope C: Qualitative research in health care: Assessing quality in qualitative research. BMJ. 2000, 320: 50-2. 10.1136/bmj.320.7226.50.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  21. 21.

    Wilczynski NL, Marks S, Haynes RB: Search strategies for identifying qualitative studies in CINAHL. Qualitative Health Research. 2007, 17 (5): 705-10. 10.1177/1049732306294515.

    Article  PubMed  Google Scholar 

  22. 22.

    Weiner BJ, Amick HR, Lund JL, Lee SY, Hoff TJ: Use of qualitative methods in published health services and management research: a 10-year review. Medical Care Research and Reviews. 2011, 68 (1): 3-33. 10.1177/1077558710372810.

    Article  Google Scholar 

  23. 23.

    Flemming K, Briggs M: Electronic searching to locate qualitative research: Evaluation of three strategies. Journal of Advanced Nursing. 2007, 57 (1): 95-100. 10.1111/j.1365-2648.2006.04083.x.

    Article  PubMed  Google Scholar 

  24. 24.

    Rim Shin K, Young Kim M, Eun Chung S: Methods and strategies utilized in published qualitative research. Qualitative Health Research. 2009, 19 (6): 850-8. 10.1177/1049732309335857.

    Article  Google Scholar 

  25. 25.

    Kuper A, Reeves S, Levinson W: An introduction to reading and appraising qualitative research. BMJ. 2008, 337: 404-7.

    Google Scholar 

  26. 26.

    Sale JEM, Hawker GA: Critical appraisal of qualitative research in clinical journals challenged. Arthritis Rheum. 2005, 53 (2): 314-6. 10.1002/art.21088.

    Article  PubMed  Google Scholar 

  27. 27.

    Johansson EE, Risberg G, Hamberg K: Is qualitative research scientific, or merely relevant?. Scand J Prim Health Care. 2003, 21: 10-14. 10.1080/02813430310000492.

    Article  PubMed  Google Scholar 

  28. 28.

    Sandelowski M, Barroso J: Creating metasummaries of qualitative findings. Nursing Research. 2003, 52 (4): 226-33. 10.1097/00006199-200307000-00004.

    Article  PubMed  Google Scholar 

  29. 29.

    Thorne S: The role of qualitative research within an evidence-based context: Can metasynthesis be the answer?. Int J Nursing Studies. 2009, 46: 569-75. 10.1016/j.ijnurstu.2008.05.001.

    Article  Google Scholar 

  30. 30.

    Nelson AM: Addressing the threat of evidence-based practice to qualitative inquiry through increasing attention to quality: A discussion paper. Int J Nurs Studies. 2008, 45: 316-22. 10.1016/j.ijnurstu.2007.01.012.

    Article  Google Scholar 

  31. 31.

    Tong A, Sainsbury P, Craig J: Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007, 19 (6): 349-57. 10.1093/intqhc/mzm042.

    Article  PubMed  Google Scholar 

  32. 32.

    Theobald S, Kemp J, Squire B, Dick J, Macq J: The disciplinary background of the IJTLD editorial board. Int J Tuberc Lung Dis. 2003, 7 (6): 604-5.

    PubMed  Google Scholar 

Pre-publication history

  1. The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/11/268/prepub

Download references

Author information

Author notes

    Affiliations

    1. Departments of Surgery; and Department of Health Policy, Management and Evaluation, Institute of Medical Science, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada

      Anna R Gagliardi

    2. Cancer Services and Policy Research Unit, Cancer Care Ontario, Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada

      Mark J Dobrow

    Authors
    1. Anna R Gagliardi
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Mark J Dobrow
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Anna R Gagliardi.

    Additional information

    Competing interests

    The authors declare that they have no competing interests.

    Authors' contributions

    Both ARG and MJD conceptualized and planned this research study, collected and analyzed data, summarized and interpreted data, prepared the manuscript, and approved this final version.

    Anna R Gagliardi and Mark J Dobrow contributed equally to this work.

    Rights and permissions

    This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

    Reprints and Permissions

    About this article

    Cite this article

    Gagliardi, A.R., Dobrow, M.J. Paucity of qualitative research in general medical and health services and policy research journals: analysis of publication rates. BMC Health Serv Res 11, 268 (2011). https://doi.org/10.1186/1472-6963-11-268

    Download citation

    Keywords

    • Qualitative Research
    • Publication Rate
    • Health Care Decision
    • High Impact Journal
    • General Medical Journal

    BMC Health Services Research

    ISSN: 1472-6963

    Contact us