Quality of outpatient clinical notes: a stakeholder definition derived through qualitative research

  • Janice L Hanson1, 2Email author,

    Affiliated with

    • Mark B Stephens3,

      Affiliated with

      • Louis N Pangaro4 and

        Affiliated with

        • Ronald W Gimbel5

          Affiliated with

          BMC Health Services Research201212:407

          DOI: 10.1186/1472-6963-12-407

          Received: 12 March 2012

          Accepted: 30 October 2012

          Published: 19 November 2012

          Abstract

          Background

          There are no empirically-grounded criteria or tools to define or benchmark the quality of outpatient clinical documentation. Outpatient clinical notes document care, communicate treatment plans and support patient safety, medical education, medico-legal investigations and reimbursement. Accurately describing and assessing quality of clinical documentation is a necessary improvement in an increasingly team-based healthcare delivery system. In this paper we describe the quality of outpatient clinical notes from the perspective of multiple stakeholders.

          Methods

          Using purposeful sampling for maximum diversity, we conducted focus groups and individual interviews with clinicians, nursing and ancillary staff, patients, and healthcare administrators at six federal health care facilities between 2009 and 2011. All sessions were audio-recorded, transcribed and qualitatively analyzed using open, axial and selective coding.

          Results

          The 163 participants included 61 clinicians, 52 nurse/ancillary staff, 31 patients and 19 administrative staff. Three organizing themes emerged: 1) characteristics of quality in clinical notes, 2) desired elements within the clinical notes and 3) system supports to improve the quality of clinical notes. We identified 11 codes to describe characteristics of clinical notes, 20 codes to describe desired elements in quality clinical notes and 11 codes to describe clinical system elements that support quality when writing clinical notes. While there was substantial overlap between the aspects of quality described by the four stakeholder groups, only clinicians and administrators identified ease of translation into billing codes as an important characteristic of a quality note. Only patients rated prioritization of their medical problems as an aspect of quality. Nurses included care and education delivered to the patient, information added by the patient, interdisciplinary information, and infection alerts as important content.

          Conclusions

          Perspectives of these four stakeholder groups provide a comprehensive description of quality in outpatient clinical documentation. The resulting description of characteristics and content necessary for quality notes provides a research-based foundation for assessing the quality of clinical documentation in outpatient health care settings.

          Keywords

          Clinical documentation Outpatient notes Physician notes Quality Patient perspectives Medical records Health care records Electronic health record

          Background

          Quality of outpatient clinical encounter notes relates to patient safety [1, 2], medical education [35], medico-legal issues [6, 7], and justification of reimbursement [69]. The clinical note is the primary tool used to document care, communicate plans [10, 11], and provide guidance for follow-up treatment and care. Gaps in the quality of clinical documentation could, therefore, adversely affect patient care and health care outcomes. Despite the importance of documentation to clinical care, there is no agreement in the literature on a comprehensive definition of the quality of documentation within a clinical note. When identifying elements associated with quality of clinical documentation, authors commonly cite attributes such as legibility, correctness, completeness, readability, appropriateness and accuracy [1218]. Others have focused on the importance of the structure and content of the note. As early as 1969, Dr. Lawrence Weed recommended a structured outpatient clinical note and a problem-oriented medical record [19]. Weed suggested standard elements for organizing clinical documentation, including data from symptoms, physical examination and laboratory studies, a medication list, a problem list, an assessment, and a plan of care. The resulting “subjective, objective, assessment, plan” (SOAP) format has been widely used for clinical documentation for the past 40 years [20]. While it reflected and replaced the previously common but not universal format of history, physical, lab, and plan, neither provided a description of quality beyond the implicit assertion that organization is a primary requirement of quality.

          Presently, many health systems and physician practices are introducing electronic health records (EHR). This may represent a unique opportunity to improve the quality of clinical documentation. One study retrospectively comparing notes of physicians using an EHR with traditional paper records demonstrates that EHRs are superior to traditional paper charts for documenting complete patient problem and medication lists, identifying relevant patient factors when making a decision, and documenting appropriate clinical decisions [17]. A separate cross-sectional review of the content and quality of a sample of records chosen from general practitioners showed that EHR documentation was more understandable (89.2% v. 69.6%), more legible (100% v. 64.3%), more likely to have at least one diagnosis recorded (48.2 % v. 33.2%), more likely to document that anticipatory advice was given (23.75 v. 10.7%), more likely to document specialty referral (77.4% v. 59.5%), and more likely to specify drug dose (86.6% v. 66.2%) compared to documentation on paper records [13]. A cross-sectional qualitative study describing perceived impact of computerized physician documentation on faculty and resident physicians showed that the EHR improved accessibility, legibility, comprehensiveness, and organization of clinical notes [3]. This study also noted, however, that the EHR also increased redundancy of information, contributed to lengthier notes, had a poor display of information, and increased “clutter” within the medical record. In summary, it is not entirely clear whether EHR use increases or decreases overall record quality. Further complicating the discussion is the fact that operational definitions of quality have yet to be consistently validated.

          Stetson et al. have developed and initially validated two versions of an instrument that evaluates the quality of inpatient clinical documentation [21, 22]. In their exploratory factor analysis, they identified four distinct factors relating to quality of documentation. Factor I, notes are well-formed, clear, uncluttered, organized, structured, non-redundant, and synthesized; Factor II, notes are comprehensible, legible, coherent, useful, correct, comprehensible, and consistent; Factor III, notes are up-to-date, complete, accurate, thorough, current, relevant; and Factor IV, notes are brief, concise, succinct, and focused. The items were generated by a small group of clinicians performing inpatient (hospital-based) work and did not explicitly include opinions of other stakeholders who routinely use clinical notes, such as patients, nurses, ancillary staff or administrators. In addition, it is not clear that quality in the context of documentation for inpatient care (where patients are typically sicker and have many more individuals involved in their day-to-day care processes) translates well to the outpatient environment. While analogous, documentation and its purposes in the inpatient and outpatient environments are different and one cannot assume that aspects of quality would be the same in both environments. Furthermore, there are varied and perhaps conflicting uses for clinical documents. Notes that are deemed of high quality for billing, legal, or care coordination purposes may, for example, be deemed of low quality for communication between physicians. This argues for a comprehensive description of quality in clinical notes based on perspectives from varied stakeholder groups.

          Currently, there are no standardized criteria or tools to define or benchmark quality of documentation of outpatient clinical care. Similarly, there are no research-based definitions or descriptions of quality in outpatient clinical notes. No prior studies systematically incorporate into a definition of quality the perspectives of multiple individuals who write and use outpatient clinical notes. Additionally, many outpatient practices are migrating to a delivery system based on the patient-centered medical home. This team-based approach to outpatient care requires that more individuals will be accessing patient records and documenting care accordingly. With this background in mind, we sought to develop a comprehensive definition of quality in outpatient clinical notes, incorporating perspectives from a diverse group of stakeholders—the authors of notes (clinicians), direct users of notes (nursing and support staff, administrators) and patients themselves. Each of these important stakeholder groups has specific expectations for the clinical encounter note and contributes to the description of quality. The purpose of this study is to describe in detail important elements of quality in outpatient clinical notes, incorporating the perspectives of clinicians, nurses, ancillary staff, administrators, and patients.

          Methods

          We used qualitative research methods to engage participants from six health care facilities and multiple stakeholder groups.

          Research setting

          We conducted our study within the Military Health System (MHS). The MHS is a large capitated health system, comparable to several of the large managed care health systems in operation within the United States (e.g., Kaiser Permanente®, Partners™ Healthcare). The MHS cares for more than 9 million beneficiaries, provides approximately 34 million visits annually [23] and is representative of American society with regard to patient demographics, common diagnoses, and medical procedures [24]. The health system includes both direct care delivery and a defined benefit (i.e., TRICARE). The billing and reimbursement process associated with MHS health care delivery is consistent with civilian-managed health systems. Like civilian systems, the MHS follows Medicare regulation and reimbursement procedures (e.g., benefit eligibility, clinical documentation, clinician sign-off on notes). All outpatient clinical visits are documented in a clinical note and coded for evaluation and management codes (E&M), healthcare common procedure coding system (HCPCS) / current procedural terminology codes (CPT) to include modifiers, and International Classification of Disease codes (ICD-9-CM). Formal follow-up billing occurs when a patient has other health insurance (e.g., retirees with civilian employment-funded insurance), when the episode of care relates to an accident or occupational injury (e.g., automobile accident), or when the patient is enrolled in Medicare. Billing for Medicare patients is applied against an established Medicare-Eligible Retiree Health Care Fund which in fiscal year 2011 was approximately $11.1 billion [25]. The MHS also leverages performance-based incentives (similar to civilian pay-for-performance programs) that provide financial incentives for achieving workload output and quality of care targets. The MHS uses a common EHR and employs clinicians and support staff who are government or contract employees.

          Following Institutional Review Board and secondary-level regulatory approval for conducting the study at the six facilities, focus groups and interviews were held at two tertiary medical centers, two community hospitals and two ambulatory clinics in the eastern United States. All research participants were volunteers, at least 21 years of age, and provided written informed consent prior to participating.

          Participants, inclusion criteria and sampling strategy

          We used purposeful sampling to achieve maximum diversity [26]. We continued sampling until we achieved saturation in the codes and themes identified during data analysis. We recruited research participants from four groups, which represented clinicians, nursing/ancillary staff, administrators and patients. See Table 1 for a list of the occupational titles of participants. Other sampling dimensions included gender and race/ethnicity. Each of the four groups was intentionally selected to provide different viewpoints.
          Table 1

          Descriptive characteristics of research participants

          Characteristics

          Clinician group

          Nurse/Ancillary group

          Patient group

          Administrator group

          Number = 163

          61

          52

          31

          19

          Disciplines/fields represented

          Physicians

          Cert. nurse specialists

          Patients

          Quality assurance specialists

          Nurse practitioners

          Nurses

          Caregivers

          Special needs coordinators

          Physician

          Medical technologists

           

          Patient advocates

          Assistants

          Physical therapists

           

          Patient safety managers

          Dentists

          Social workers

           

          Medical coders

          Psychologists

          Pharmacists

           

          Group practice managers

          Optometrists

          Medical assistants

           

          Medical records technicians

           

          Case managers

           

          Patient administrators

           

          Dental assistants

            
           

          Hospital Corpsman

            

          Gender

              

          Female

          29 (47.5%)

          39 (75.0%)

          20 (64.5%)

          13 (68.4%)

          Male

          30 (49.2%)

          11 (21.2%)

          11 (35.5%)

          6 (31.6%)

          Not reported

          2 (3.5%)

          2 (3.8%)

          0

          0

          Age (in years)

              

          Mean

          41.6

          42.1

          51

          41.8

          Maximum

          72

          62

          83

          61

          Minimum

          26

          21

          25

          26

          # participants not reporting

          1

          3

          9

          0

          Race/ethnicity

              

          African American

          2 (3.3%)

          18 (34.6%)

          8 (25.8%)

          10 (52.6%)

          Asian Pacific

          7 (11.5%)

          3 (5.8%)

          0

          0

          Hispanic

          3 (5.0%)

          2 (3.8%)

          3 (9.7%)

          4 (21.1%)

          Caucasian

          48 (78.7%)

          28 (53.8%)

          20 (64.5%)

          5 (26.3%)

          # participants not reporting

          1 (1.6%)

          1 (1.9%)

          0

          0

          Data collected between

          May 2009 –December 2009

          June 2009 –December 2009

          July 2009 –October 2011

          May 2009 –December 2009

          Clinician group

          Participants in the clinician group were licensed physicians or physicians-in-training (intern, resident, fellow), nurse practitioners, physician assistants, dentists, optometrists and psychologists. Each of these individuals was responsible as an individual author of outpatient clinical encounter notes. All were actively credentialed in clinical practice and using AHLTA (Armed Forces Health Longitudinal Technology Application; the MHS EHR) to document clinical care. Most clinicians were from primary care environments, in particular, internal medicine and family medicine.

          Nursing/ ancillary care group

          Participants in this group were nurses, medical technologists, physical therapists, social workers, pharmacists, case managers, medical assistants, dental assistants, or military medical technicians. These individuals commonly read clinical encounter notes as part of their professional duties, using the information in the notes to provide or coordinate patient care. All had completed EHR training and were familiar with the use of AHLTA.

          Administrators

          Participants in this group were actively employed in an administrative capacity at one of the study sites. Administrators routinely reviewed patient clinical encounter notes as part of their official administrative duties. Related occupational positions included billing staff and supervisors, medical records technicians and their supervisors, quality improvement staff, patient safety managers, group practice managers, and patient administration staff members.

          Patients/caregivers

          Participants in the patient and patient caregiver group were actively receiving care at one of the study sites. All had attended at least one outpatient clinical visit during the past 12 months as either a patient or an individual responsible for assisting with patient care in the home.

          Qualitative research methods

          We used the grounded theory approach to advise our study design, data collection and analysis. This approach derives theory from data which have been systematically gathered and repeatedly analyzed [27, 28].

          Data were collected through focus groups and interviews at the six clinical sites. Most focus groups were small (7 to 11 participants) and facilitated by two investigators who de-briefed after each focus group to discuss emerging themes, focus group process and needed adaptations in the focus group guide. Focus groups and interviews followed the semi-structured guide in Appendix 1 and a modified interview guide for patients and caregivers (Appendix 2). All sessions were audiotaped with two recorders and transcribed by contracted transcription professionals. Transcriptions were created in .txt files using Microsoft® Word and linked to HyperResearchTM[29] for organization of data during qualitative analysis.

          Qualitative data analysis

          We applied open, axial and selective coding techniques to the focus groups and interview transcripts for analysis [28]. Two investigators (JH and RG) developed an initial set of codes (open coding), applied them to 6 transcripts, identified and resolved all coding disagreements, and revised the codes. A third investigator (MS) then reviewed the coding of all 6 transcripts and recommended revisions to the names and definitions of the codes. After grouping the codes in three organizing themes (axial coding), all four investigators discussed and refined the coding scheme to explain the entire dataset. All transcripts were coded, with at least 2 investigators reviewing each transcript; all disagreements were discussed and resolved. During all coding and reviewing of coding, all investigators added memos to the transcripts. The theme describing characteristics of quality in a note was identified as the main organizing theme for the dataset (selective coding). We ensured that the data reached saturation (i.e., no new themes were emerging) for each of the four groups of participants (clinicians, nurses/ancillary staff, administrators, and patients) before we stopped collecting data. The transcripts were coded as one complete dataset with one integrated set of codes and themes. After we completed coding, we examined which stakeholder groups’ transcripts contained which codes and themes, in the data as organized in the HyperResearch™ software.

          Results

          A total of 163 research participants representing the four targeted groups (clinicians, nurse/ancillary, patients, and administrators) participated in focus group sessions or interviews between May 2009 and October 2011 (Table 1). These included 61 clinicians in 9 focus groups, 52 nurses and ancillary care providers in 6 focus groups, 19 administrators in 5 focus groups and 31 patients and caregivers in 1 focus group plus interviews with individuals or couples. The distribution across groups with respect to gender was relatively equal, with the exception of the nurse/ancillary and administrator groups, who had more females (75% and 68.4% respectively). The age distribution across groups was balanced (mean age between 41.6 years and 41.8 years), with exception of the patient group, which was slightly older (mean age 51 years, although young adult patients were represented in this group). Participants self-identified as African-American, Asian or Pacific Islander, Hispanic and Caucasian.

          We identified three major organizing themes relating to quality of outpatient clinical documentation; each theme includes specified sub-themes (Table 2). The first organizing theme identifies characteristics associated with quality within a clinical note. With the exception of three characteristics (ease of translation into codes, prioritized and relevance), all four stakeholder groups discussed each of the characteristics in this theme (Table 3). According to participants, a clinical note of high quality is concise, has sufficient information, and explains the clinician’s thought process and the plan of care. A quality note is also clear and understandable to patients, subsequent providers and others who might read the note. The note should contain information that is relevant, current, accurate and prioritized for action. A note of high quality is readable with appropriate font, legible handwriting, correct spelling, few or no abbreviations, and understandable syntax. The note should be organized and tell a continuous story about the patient.
          Table 2

          Themes and codes

          Characteristics of quality in a clinical note [Main organizing theme]

          a

          Conciseness (focused; brief; not redundant)

          b

          Sufficiency of information (enough information for diagnosis, treatment, coding; pertinent details present; complete for its purpose)

          c

          Explanatory (explains clinician thought process; gives reasons for diagnosis and plan)

          d

          Clarity (clear; understandable to patients, to subsequent providers, and to other users)

          e

          Relevance (only relevant information; no extraneous information)

          f

          Prioritized

          g

          Readability (readable font; correct spelling; no abbreviations or only unambiguous abbreviations; readable output from EHR; legible handwriting; understandable syntax)

          h

          Organization (well-organized; logically grouped; chronological; important parts highlighted; can find the information you need easily)

          i

          Continuity of story (tells a story; written in free text with a flow that makes sense; shows continuity from referral to note and from one provider to another; internally and externally consistent; facilitates follow-up with the information provided; synthesizes information; coordinates information from different sources)

          j

          Current and accurate (has current information; up-to-date; correct; from a patient’s perspective, accuracy includes honesty and whether the note includes what the patient said)

          k

          Ease of translation into codes (diagnostic; procedural; other)

          Content elements of the note

          a

          Patient’s complaints

          b

          History of the present illness (“HPI”; “subjective”)

          c

          Problem list

          d

          Past medical history

          e

          Medications list

          f

          Adverse drug reactions and allergies (distinguished from side effects of medications, which is included in prognosis and expectations)

          g

          Social and family history (includes the patient’s reaction to the diagnosis or health condition)

          h

          Review of systems

          i

          Physical findings (pertinent positives and negatives; “objective;” vital signs)

          j

          Assessment (diagnosis; differential)

          k

          Plan of care (with goals and objectives)

          l

          Follow-up information (instructions for the patient; consults; orders; prescriptions; language and other learning barriers for patients)

          m

          Author information (name; title; discipline; date of the encounter)

          n

          Patient identifiers

          o

          Prognosis and expectations (includes side effects of medications)

          p

          Care and education delivered

          q

          Information added by the patient

          r

          Interdisciplinary information

          s

          Infection alerts

          t

          Patient priorities

          System supports for quality documentation

          a

          Reliability and accessibility (works when you need it; you can get into it; notes available when you need them)

          b

          Interoperability (integrated inpatient records, outpatient records, emergency department and pharmacy; information linked between facilities)

          c

          Structures input well (ease of writing; links to templates; time efficient; limits copying and pasting; easy to correct errors)

          d

          Structures output well (for ease of viewing and reading; useable display; links to patient’s history—medical, surgical, medications, allergies, problem list; links information between different notes; you can find needed information about a patient; links from diagnosis to occupational exposure; works well for security and patient privacy)

          e

          Time (time with patient; time to write notes)

          f

          Ancillary staff (available to help in clinic)

          g

          Relationship with patient (good relationship facilitates good note)

          h

          Workstations (place to see patients and write notes is convenient)

          i

          Can correct errors

          g

          Patient computer (for patient to answer questions)

          k

          Education and training (sufficient training on how to write notes in the EHR and use templates or formats)

          Table 3

          Sources of data contributing to “characteristics of quality in a clinical note”

          Characteristics

          Clinicians

          Nurse/ancillary

          Patients

          Administrators

          Conciseness (focused; brief; not redundant)

           

          Sufficiency of information (enough information for diagnosis, treatment, coding; pertinent details present;, complete for its purpose)

          Explanatory (explains clinician thought process; gives reasons for diagnosis, plan)

          Clarity (clear or unclear; understandable to patients, to subsequent providers, to other users)

          Relevance (only relevant information; no extraneous information)

            

          Prioritized

            

           

          Readability (readable font; correct spelling; no abbreviations or only unambiguous abbreviations; readable output from EHR; legible handwriting; understandable syntax)

          Organization (well-organized; logically grouped; chronological; important parts highlighted; can find the information you need easily)

          Continuity of story (tells a story; written in free text with a flow that makes sense; shows continuity from referral to note and from one provider to another; internally and externally consistent; facilitates follow-up with the information provided; synthesis)

          Current and accurate (has current information; up-to-date; correct; from a patient’s perspective, accuracy includes honesty and whether the note includes what the patient said)

          Ease of translation into codes (diagnostic; procedural; other)

            

          The concept of “continuity of story” was noted repeatedly in all four stakeholder groups. Comments coded “continuity of story” encompassed telling a coherent story about the patient from one visit to the next, as well as facilitating the linkage of one encounter note to related information and encounters. A continuous story facilitates follow-up, synthesizes patient information and coordinates information from different sources. Stakeholder emphasis in the characteristics theme varied. For example, clinicians, nurses and administrators discussed “current and accurate” primarily in relation to whether the information in the note was updated for the patient’s current situation and was factually correct. Patients, in contrast, discussed “accuracy” in terms of whether the clinician included “honest” information and incorporated the information that the patient told the clinician. The final code in the organizing theme of characteristics, “ease of translation into codes,” was emphasized by administrators and mentioned by clinicians, but not mentioned by nurses or patients.

          The second organizing theme is content, which details the elements that the four stakeholder groups expect in a quality clinical note. Within this theme, there was more variation in the codes among the four groups. Different groups emphasized aspects of the note that particularly facilitated their individual roles (Table 4). Although terminology sometimes differed, all four groups agreed that a quality note included the following elements: patient complaints; history of the patient’s current illness; a list of the patient’s problems; the past medical history; a medication list; social and family history (patients emphasized this more than the other groups and included their emotional reactions to diagnoses and health conditions); assessment; plan of care; information needed to understand and implement follow-up care. Clinicians and nurses included adverse drug reactions and allergies; clinicians and administrators included a review of systems; clinicians, nurses and administrators included author information and patient identifiers. Nurses stressed that a quality note should include documentation of care and education delivered to the patient, information that patients add to their own notes, interdisciplinary information from the various health care providers involved in the patient’s care, and infection alerts when a patient’s care required attention to infection control. Patients emphasized the importance of including the patient’s priorities and a description of the patient’s prognosis regarding what they could expect in the future, including the effects and side-effects of treatment.
          Table 4

          Sources of data contributing to “content elements of the note”

          Content elements of the note

          Clinicians

          Nurse/ancillary

          Patients

          Admin.

          Patient’s complaints

          History of present illness (“HPI;” “subjective”)

          Problem list

          Past medical history

          Medications list

          Adverse drug reactions and allergies (distinguished from side effects of medications, which is included in prognosis and expectations)

          Social and family history (includes the patient’s reaction to the diagnosis or health condition)

              

          Review of systems

          Physical findings (pertinent positives and negatives; “objective;” vital signs)

          Assessment (diagnosis; differential)

          Plan of care (with goals and objectives)

          Follow-up information (instructions for the patient; consults; orders; prescriptions; language and other learning barriers for patients)

          Author information (name; title; discipline; date of encounter)

          Patient identifiers

          Prognosis and expectations (includes side effects of medications)

              

          Care and education delivered

           

            

          Information added by the patient

           

            

          Interdisciplinary information

           

            

          Infection alerts

           

            

          Patient priorities

            

           
          The third organizing theme discusses healthcare system contributions to the quality of clinical notes. Collectively, the four groups addressed the medical records system, education and training, and the clinical care environment. Clinicians and nurses discussed systems issues in the most detail. According to the stakeholders, the records system supported quality notes through reliability and accessibility, interoperability (between different aspects of the care system), data input structures, opportunity to correct errors and data output structures. Other system issues relating to quality documentation included adequate time to spend with patients and write notes, adequate ancillary support, and efficient workstations for clinicians, staff and/or patients to contribute to their notes. Patients and administrators commented on the appearance of copied notes, patients noted how time and relationship with clinicians affected the note, and administrators noted the importance of education for clinicians, particularly in terms of learning how to efficiently use EHR systems and include information needed for coding (Table 5).
          Table 5

          Sources of data contributing tosystem supports for quality documentation”

          System supports

          Clinicians

          Nurse/ancillary

          Patients

          Admin.

          Reliability and accessibility (works well when you need it; you can get into it; notes available when you need them)

            

          Interoperability (integrated inpatient records, outpatient records, emergency department and pharmacy; information linked between facilities)

            

          Structures input well (ease of writing; links to templates; time efficient; limits copying and pasting; easy to correct errors)

            

          Structures output well (ease of viewing and reading; usable display; links to patient history including medical, surgical, medication, allergy and problem list information; links information between notes; links from diagnosis to occupational exposure; works well for security and patient privacy)

          Time (time with patient; time to write notes)

           

           

          Ancillary staff (available to help in clinic )

             

          Relationship with patient (good relationship facilitates good note)

           

           

          Workstations (place to see patients and write notes that is convenient)

             

          Can correct errors

           

            

          Patient computer (for patient to answer questions)

           

            

          Education and training (sufficient training on how to write notes in the EHR and use templates or formats)

           

          Discussion

          We asked clinicians, nurses/ancillary staff, administrators and patients/caregivers to describe quality in clinical documentation. Three organizing themes emerged: 1) characteristics of quality in clinical note—11 key characteristics were described; 2) content elements of clinical notes—20 key elements were described; 3) system support factors for writing quality clinical notes—11 key factors were described.

          The organizing theme regarding the characteristics of a note provides a collective description of quality on which the four stakeholder groups substantially agreed. The theme delineating desirable content elements within a note provides a list that could be used to measure a note’s comprehensiveness. While the four groups agreed on many important content elements (patient complaints, symptoms, problems, history, medications, assessment and plan of care), they included or emphasized some elements very differently. Nurses and patients emphasized the importance of detailed, clear, practical information about needed follow-up care, as well as information provided by the patient, such as patient entries in the note or other documentation about patient priorities and patients’ explanations of their problems. Nurses also emphasized the importance of interdisciplinary contributions to a note, to assist with continuity of story. Administrators emphasized the importance of easy translation of information in the notes to codes for medical billing purposes. The third organizing theme we identified relates to system factors that facilitate or inhibit quality clinical documentation. Issues such as data entry, clinical workflow, system interoperability, access, reliability and data output all impact the end-product in terms of quality documentation.

          The inclusion of patients in our study calls to mind the work of Delbanco and colleagues in the OpenNotes project [30, 31]. Most of the patients and caregivers in our study had not seen outpatient notes about their care, which is the reason we created a modified interview guide to elicit their perspectives. They were, nevertheless, able to explain what information they wanted to have in a note about them, such as their priorities for their care, and how a note could help them coordinate their own care or explain their healthcare needs to other family members. Perhaps as patients have opportunities to read their notes, as in the OpenNotes project, more conversations will emerge about what quality means to those about whom the notes are written.

          The grounded theory that emerges is that our stakeholder groups agree on most characteristics of quality in a note. A high-quality outpatient clinical note is concise, explanatory, clear, relevant, prioritized, readable, organized, current and accurate. A quality note contains sufficient information for the reader to understand its rationale and tells a coherent and continuous story. Since individuals define quality in terms of content, as it relates to their role when using the note, it follows that quality content as defined by clinicians differs from that as defined by nurses, administrators and patients. Therefore, we plan to devise a quality rating instrument that encompasses multiple perspectives.

          The research participants from the clinician and nurse stakeholder groups also noted that the health care records system impacts quality documentation according to reliability, accessibility, interoperability and the structure of data input and output, and that the clinical system also makes the production of quality notes more or less likely. System issues related to time, staffing, the support of clinician/patient relationships, convenient workstations, patient data entry, and education/training may also influence quality in clinical documentation. When we began this study, we tried to focus the discussion on quality of notes apart from reference to whether the record system was paper-based or electronic. The clinical participants, however, spoke from the context of their work and commented extensively about the influence of the electronic record system on the quality of notes they wrote and read. In accordance with qualitative research methods, we added 2 questions about this to the focus group guide as the study progressed (Appendix 1) and we reported themes about influence of the EHR on quality as they emerged from the analysis. Nevertheless, there are definite limitations to the insight that this study can add to the complex question of how factors related to EHR use may affect the quality of clinical documentation. This question warrants study in future research.

          Our results add additional characteristics and elements of quality in clinical documentation to the existing literature. For example, the traditional structural elements in a SOAP note appear on our list of content elements as well. The clinicians in our study likely acquired this standard terminology during their medical education. Our characteristics also confirm those noted by Stetson et al. that describe quality documentation in the inpatient setting. The themes in the current study, however, describe additional aspects of quality documentation that arise in the outpatient setting, such as integration of insights and care across disciplines and incorporation of patients’ priorities in ongoing treatment plans and care. These may, of course, also be important in the inpatient setting; and they might also be identified by our study design when applied to clinical documentation for inpatients. A key additional finding from our stakeholders is that quality notes are also “explanatory” and provide “continuity of story”. “Continuity of story” relates to “narrative expressivity” as identified by Rosenbloom and colleagues in their work with clinicians [6, 32]. In our study the concept arose across stakeholder groups and included connecting different aspects of the patient’s story and care, which seems to go beyond the “narrative expressivity” and “flexibility” described in Rosenbloom’s work. This continuity of story highlights that a single patient encounter occurs in a clinical context, often as one of a series of encounters. Often different disciplines and multiple healthcare professionals are involved with varying degrees as part of a patients’ longitudinal story. A single clinical note represents one clinical encounter, a ‘chapter’ in an evolving health care story in the life of the patient. The best notes, according to our groups, make clear the connections between different chapters and thereby enrich the patient’s story.

          Our results also provide insight about the information that various stakeholders seek when defining quality in clinical documentation. Administrators, patients and nurses seek different information to fulfill their roles or implement the care recommendations they receive. The clinical note, therefore, contains interdisciplinary practical context that cannot be ignored in a comprehensive definition or evaluation of quality. It may be that the published literature to date has emphasized clinicians’ needs, with less focus on how the clinical note affects other healthcare providers and patients.

          Our data were collected within a single healthcare system, which may be a limitation of the study. While the system is large and the sample diverse, and we believe that the literature shows the system to be comparable to other large managed care organizations [23, 24], future research should replicate our study in other systems. Another limitation applies particularly to the insights put forth about how systems issues may affect the quality of clinical documentation. Since the focus of the study is quality of notes, not the systems within which notes are written, the scope of our findings for this theme is likely limited. A more thorough treatment of the impact of systems on note quality would require a focused study on this question in settings that use a variety of documentation systems.

          Our research-based definition of quality in clinical documentation describes quality in a clinical note from the perspectives of those who most commonly use the note for clinical, nursing or administrative purposes and the patients who are the subject of the note. The inclusion of the voice of those who use these notes, and the patients about whom these notes are written, represents a novel contribution to the understanding of “quality” in this context. We suggest that comprehensive study of quality of clinical documentation should incorporate the perspectives of these various stakeholder groups, and that achieving quality outpatient clinical documentation requires addressing the needs described by those who use clinical notes to plan, implement and pursue care, as well as by those who write the notes and use them to document what happens in the clinical encounter.

          Conclusions

          Perspectives of four stakeholder groups were necessary to provide a comprehensive description of quality in outpatient clinical documentation. The resulting description of characteristics and content necessary for quality documentation provides a research-based foundation for assessing and improving clinical documentation in outpatient health care settings.

          Appendix 1 Focus Group and Interview Topic Guide

          Study on Quality in Notes

          Materials available for review
          • Preliminary themes from other focus groups and interviews in this study, if available

          • Draft list of dimensions of quality of a medical note, after preliminary analysis of initial focus groups and interviews

          Topics for discussion (identified from the literature and discussion among investigators)
          • Comprehensiveness of notes; essential items to include

          • Accuracy/concordance with care/relevance (diagnosis, treatment, follow-up)

          • Intelligibility/comprehensibility/understandability/clarity

          • Effect on patient care

          • Effect on patient/clinician encounter

          • Quality of narratives

          • Speed and ease of use/timeliness

          • Support for medical decision-making

          • Organization

          General, open-ended questions

          (Tailor wording of the questions for each focus group or interview participant.)
          • What makes a quality note? OR What constitutes “goodness” or “badness” in a clinical note?

          • What diminishes the quality of a note?

          • What do you consider essential content for a clinical note?

          • How does format (written system, electronic system, templates, pick-lists) help improve the quality of notes?

          • How does format (written system, electronic system, templates, pick-lists) hurt the quality of notes?

          • What training about writing notes might improve quality?

          • What tips have you discovered that help make the quality of notes better?

          Additional questions…

          …will be developed based on experience in early focus groups and interviews of the study.

          …will be asked during focus groups to clarify and explore comments offered by study participants. These are the “probing questions” of focus group methodology.

          Appendix 2 Patient/Family Member Interview Topic Guide

          Study on Quality in Notes

          Materials available for review
          • Preliminary themes from other focus groups and interviews in this study

          • Draft list of dimensions of quality of a medical note, after preliminary analysis of initial focus groups and interviews

          Topics for discussion (identified from the literature and discussion among investigators)
          • Comprehensiveness of notes; essential items to include

          • Accuracy/concordance with care/relevance (diagnosis, treatment, follow-up)

          • Intelligibility/comprehensibility/understandability/clarity

          • Effect on patient care

          • Effect on patient/clinician encounter

          • Quality of narratives

          • Speed and ease of use/timeliness

          • Support for medical decision-making

          • Organization

          Note: Additional questions, “probing questions,” may be asked to clarify what the interview participant means and to explore new topics that the participant raises during the interview.

          Open-ended interview questions:
          1. 1.

            Have you seen notes that your doctor or another health care provider has written about you?

             
          2. 2.
            If yes:
            1. a.

              What do you find helpful about these notes?

               
            2. b.

              What makes these notes good?

               
            3. c.

              What makes these notes helpful for you?

               
            4. d.

              What makes these notes useful for other health care providers?

               
            5. e.

              What makes these notes less helpful or less useful to you or to anyone else?

               
             
          3. 3.
            If no:
            1. a.

              What do you think your doctors and your other health care providers write about you and your health?

               
            2. b.

              What do you think would make these notes helpful or useful?

               
            3. c.

              What do you think would make these notes unhelpful or not as useful as they could be?

               
             
          4. 4.

            Can you think of anything that would make the notes that your health care providers write about you more useful to you?

             

          Comments on tentative themes from other groups:

          Have a copy of the tentative themes for interviews and focus groups, folded in half.
          1. 1.
            Show the participant the “content” themes and say, “The health care providers that we have talked to have told us that they look for these things in a note about a patient.”
            1. a.

              What do you think about this list of things that might be written in your medical record?

               
            2. b.

              Which of these things do you think might be most important to have in a health care provider’s note about you?

               
            3. c.

              Is there anything on this list that you wish would NOT be written about you?

               
            4. d.

              Is there anything else that you want your health care providers to write about?

               
             
          2. 2.
            Show the participant the “characteristics” themes and say, “The health care providers that we have talked to have told us that they want a note about a patient to have these characteristics.”
            1. a.

              Which of these things do you think would be most important for a note that a health care provider writes about you?

               
            2. b.

              Are there any other characteristics that you can think of that would make the notes that your health care providers write about you excellent notes?

               
             

          Authors’ information

          Dr. Hanson brings expertise in qualitative research design and methods, medical education and communication. Dr. Stephens is an experienced family physician and medical educator with additional expertise in use of electronic medical records. Dr. Pangaro is an experienced internist, geriatrician and endocrinologist with additional expertise in medical education and evaluation. Dr. Gimbel brings expertise in quality improvement and biomedical informatics.

          Abbreviations

          EHR: 

          Electronic health record

          MHS: 

          Military health system

          AHLTA: 

          Armed forces health longitudinal technology application.

          Declarations

          Acknowledgments

          The authors thank Galen Barbour, MD, Erin Bohen, MD, Lanna Forrest, PhD, Patrice Waters-Worley, MBA and Jeffrey Yarvis, PhD for their assistance in conceptualizing and planning elements of the study. The authors thank Paul Bornemann, MD, Molly Grasso, Joshua Hodge, MD, Lori Krayer, MSN, CFNP, Patrick Monahan, MD, Laura Sessums, MD, JD, and Bradford Smith, MD for assistance in coordinating site logistics and recruitment of participants.

          Funding

          This research was supported by the United States Army Medical Research & Materiel Command, Fort Detrick, Maryland (W81XWH-08-2-0056).

          Authors’ Affiliations

          (1)
          Departments of Medicine, Pediatrics & Family Medicine, Uniformed Services University of the Health Sciences
          (2)
          Department of Pediatrics, University of Colorado School of Medicine
          (3)
          Department of Family Medicine, Uniformed Services University of the Health Sciences
          (4)
          Department of Medicine, Uniformed Services University of the Health Sciences
          (5)
          Department of Biomedical Informatics, Uniformed Services University of the Health Sciences

          References

          1. El-Kareh R, Gandhi TK, Poon EG, Newmark LP, Ungar J, Lipsitz S, et al.: Trends in primary care clinician perceptions of a new electronic health record. J Gen Intern Med 2009,24(4):464–468. Epub 2009/01/22PubMedView Article
          2. Johnson KB, Ravich WJ, Cowan JA Jr: Brainstorming about next-generation computer-based documentation: an AMIA clinical working group survey. Int J Med Inform 2004,73(9–10):665–674. Epub 2004/08/25PubMedView Article
          3. Embi PJ, Yackel TR, Logan JR, Bowen JL, Cooney TG, Gorman PN: Impacts of computerized physician documentation in a teaching hospital: perceptions of faculty and resident physicians. J Am Med Inform Assoc 2004,11(4):300–309. Epub 2004/04/06PubMedView Article
          4. Rouf E, Chumley HS, Dobbie AE: Electronic health records in outpatient clinics: perspectives of third year medical students. BMC Med Educ 2008, 8:13. Epub 2008/04/01PubMedView Article
          5. Shaw N: Medical education & health informatics: time to join the 21st century? Stud Health Technol Inform 2010,160(Pt 1):567–571. Epub 2010/09/16PubMed
          6. Rosenbloom ST, Denny JC, Xu H, Lorenzi N, Stead WW, Johnson KB: Data from clinical notes: a perspective on the tension between structure and flexible documentation. J Am Med Inform Assoc 2011,18(2):181–186. Epub 2011/01/15PubMedView Article
          7. Rosenbloom ST, Stead WW, Denny JC, Giuse D, Lorenzi NM, Brown SH, et al.: Generating clinical notes for electronic health record systems. Applied clinical informatics 2010,1(3):232–243. Epub 2010/10/30PubMedView Article
          8. Benge J, Beach T, Gladding C, Maestas G: Use of electronic health record structured text and its payoffs. The approach and barriers to using structured text in EHR to document care encounters. J Healthc Inf Manag 2008,22(1):14–19. Epub 2008/01/01PubMed
          9. Silfen E: Documentation and coding of ED patient encounters: an evaluation of the accuracy of an electronic medical record. Am J Emerg Med 2006,24(6):664–678. Epub 2006/09/21PubMedView Article
          10. Soto CM, Kleinman KP, Simon SR: Quality and correlates of medical record documentation in the ambulatory care setting. BMC Health Serv Res 2002,2(1):22. Epub 2002/12/11PubMedView Article
          11. Wood DL: Documentation guidelines: evolution, future direction, and compliance. Am J Med 2001,110(4):332–334.PubMedView Article
          12. Dhariwal DK, Gibbons AJ: The CRABEL score--setting standards in maxillofacial medical note-keeping. Br J Oral Maxillofac Surg 2004,42(3):200–202. Epub 2004/05/04PubMedView Article
          13. Hippisley-Cox J, Pringle M, Cater R, Wynn A, Hammersley V, Coupland C, et al.: The electronic patient record in primary care--regression or progression? A cross sectional study. BMJ 2003,326(7404):1439–1443. Epub 2003/06/28PubMedView Article
          14. Hogan WR, Wagner MM: Accuracy of data in computer-based patient records. J Am Med Inform Assoc 1997,4(5):342–355. Epub 1997/09/18PubMedView Article
          15. Pringle M, Ward P, Chilvers C: Assessment of the completeness and accuracy of computer medical records in four practices committed to recording data on computer. Br J Gen Pract 1995,45(399):537–541. Epub 1995/10/01PubMed
          16. Stengel D, Bauwens K, Walter M, Kopfer T, Ekkernkamp A: Comparison of handheld computer-assisted and conventional paper chart documentation of medical records. A randomized, controlled trial. J Bone Joint Surg Am 2004,86-A(3):553–560. Epub 2004/03/05PubMed
          17. Tang PC, LaRosa MP, Gorden SM: Use of computer-based records, completeness of documentation, and appropriateness of documented clinical decisions. J Am Med Inform Assoc 1999,6(3):245–251. Epub 1999/05/20PubMedView Article
          18. Tange HJ, Schouten HC, Kester AD, Hasman A: The granularity of medical narratives and its effect on the speed and completeness of information retrieval. J Am Med Inform Assoc 1998,5(6):571–582. Epub 1998/11/24PubMedView Article
          19. Weed LL: Medical records, medical education, and patient care: the problem-oriented record as a basic tool. Chicago: Year Book Medical Publishers, Inc; 1969.
          20. Zierler-Brown S, Brown TR, Chen D, Blackburn RW: Clinical documentation for patient care: models, concepts, and liability considerations for pharmacists. Am J Health Syst Pharm 2007,64(17):1851–1858. Epub 2007/08/29PubMedView Article
          21. Stetson PD, Morrison FP, Bakken S, Johnson SB: Preliminary development of the physician documentation quality instrument. J Am Med Inform Assoc 2008,15(4):534–541. Epub 2008/04/26PubMedView Article
          22. Stetson PD, Bakken S, Wrenn JO, Siegler EL: Assessing electronic note quality using the physician documentation quality instrument (PDQI-9). Applied clinical informatics 2012,3(2):164–174. Epub 2012/05/12PubMedView Article
          23. Gimbel RW, Pangaro L, Barbour G: America’s “undiscovered” laboratory for health services research. Med Care 2010,48(8):751–756. Epub 2010/07/09PubMedView Article
          24. Jackson JL, Strong J, Cheng EY, Meyer G: Patients, diagnoses, and procedures in a military internal medicine clinic: comparison with civilian practices. Mil Med 1999,164(3):194–197. Epub 1999/03/26PubMed
          25. United States Department of Defense: Fiscal year 2011 report to Congress: evaluation of thje TRICARE program. Washington; 2011.
          26. Hanson JL, Balmer DF, Giardino AP: Qualitative research methods for medical educators. Acad Pediatr 2011,11(5):375–386. Epub 2011/07/26PubMedView Article
          27. Glaser BG, Strauss AL: The discovery of grounded theory: strategies for qualitative research. Hawthorne, NY: Aldine Publishing Company; 1967.
          28. Strauss A, Corbin J: Basics of qualitative research: techniques and procedures for developing grounded theory. 2nd edition. Thousand Oaks: Sage; 1998.
          29. ResearchWare I: HyperResearch, version 2.8. Thousand Oaks: Sage Publications Software; 1997–2007.
          30. Delbanco T, Walker J, Darer JD, Elmore JG, Feldman HJ, Leveille SG, et al.: Open notes: doctors and patients signing on. Ann Intern Med 2010,153(2):121–125. Epub 2010/07/21PubMed
          31. Walker J, Leveille SG, Ngo L, Vodicka E, Darer JD, Dhanireddy S, et al.: Inviting patients to read their doctors’ notes: patients and doctors look ahead: patient and physician surveys. Ann Intern Med 2011,155(12):811–819. Epub 2011/12/21PubMed
          32. Rosenbloom ST, Crow AN, Blackford JU, Johnson KB: Cognitive factors influencing perceptions of clinical documentation tools. J Biomed Inform 2007,40(2):106–113. Epub 2006/08/15PubMedView Article
          33. Pre-publication history

            1. The pre-publication history for this paper can be accessed here:http://​www.​biomedcentral.​com/​1472-6963/​12/​407/​prepub

          Copyright

          © Hanson et al.; licensee BioMed Central Ltd. 2012

          This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://​creativecommons.​org/​licenses/​by/​2.​0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.