Ethics Approvals
This study conforms to the ethical principles in the Helsinki Declaration, and received ethics approval from GLA (PCC 2009-010018) and the University of California at Los Angeles (G08-06-103-02) Institutional Review Boards. Because of the minimal risk nature of this project, the Institutional Review Boards waived the requirement for documentation of informed consent.
Setting
The United States Department of Veterans Affairs (VA) is an integrated healthcare delivery system for people who are discharged from active military service [15]. Since the 1990 s the VA has had a strong tradition of quality of care measurement and improvement [16]. Quality improvement initiatives are often tied to External Peer Review Program (EPRP) data, which involves medical record review by an external contractor and is available at the level of individual healthcare facilities, allowing for peer comparisons and national and regional benchmarks [17]. VA facilities are motivated to improve their care because of financial incentives to their senior managers for meeting certain quality goals and a desire to perform well compared to other facilities. Although EPRP quality measures for falls exist, they are not in the top tier of quality measures for which results are tied to financial and non-financial incentives (e.g., prestige) to senior managers. Examples of top tier indicators applicable to older patients include use of ACE inhibitors or angiotensin receptor blockers for patients with systolic heart failure, achieving a target blood pressure of less than 140/90 mmHg in patients with hypertension, and immunization against influenza. Thus the fall prevention program we developed, although viewed in a positive light, was not a prime target for additional investment of resources.
GLA is the largest healthcare system in the VA, and is composed of ten community based outpatient clinics (CBOCs), two larger ambulatory care centers that provide both primary care and some specialty services, and a medical center that provides ambulatory, inpatient and some post-acute care. GLA also has community living centers that provide nursing home care at two of its sites. Finally, GLA either directly provides, or contracts with other organizations to provide, home care services and adult day health care. Spanning five counties in Southern California, GLA's service area includes 1.4 million veterans [18]. GLA's patients are disproportionately older than is typical in non-VA primary care settings, making falls a salient issue for our system. The system is geographically dispersed; the farthest CBOC is nearly 200 miles from the main medical center, where the investigators for this project are based. Because the CBOCs offer mostly primary care, veterans served by these CBOCs may need to travel to the medical center or an ambulatory care center for certain specialty services.
Prior to the inception of our quality improvement program, GLA had an array of fall prevention services located at its core facilities, including a fall prevention clinic, multidisciplinary geriatrics clinic, physical and occupational therapy (including a Tai Chi class), exercise programs, and home safety evaluations (through home care services). However, there was evidence that these programs were not translating into adequate quality of care for fall prevention at the population level, given GLA's suboptimal performance on the population-based quality indicator of asking outpatients age 75 and older about the presence or absence of falls in the past year [11]. Although improving case-finding for patients at risk for falls was one of GLA's priorities based on quality indicator data, developing a system to manage patients found to be at high risk for future falls was important to establish first, so that a system would be in place to handle patients identified by future case-finding efforts.
GLA is one of five healthcare systems in the VA Desert Pacific Healthcare Network, which serves Southern California and Southern Nevada. The Desert Pacific Healthcare Network both finances and directly provides some operations in each of the five systems. The Network's operations include Telecare, a nurse advice line whose primary function is to receive and triage incoming patient calls. Patients can call Telecare via a toll-free number with any concerns, including symptoms. Telecare also has a specific service, Telecare Tuck-In, which at the time of the falls project's inception was operated only for GLA. Telecare Tuck-In enables healthcare professionals to refer patients for non-urgent follow-up or check-ins with a nurse via telephone. In principle, the Tuck-In service could help mitigate some of the geographic barriers patients face in accessing services at GLA by substituting phone care for some in-person care. It is through Telecare Tuck-In that our project (described below) operated.
Description of the project being evaluated
Here, we briefly describe how the quality improvement program began (for more details, see [11]), but subsequently focus on the formative evaluation of a specific project implemented within the umbrella of our program: the Telecare fall prevention pilot project. Although the quality improvement program and the Telecare fall prevention project were initiated simultaneously, the program ultimately became an umbrella for various fall prevention activities throughout GLA. For clarity, we use the word "program" to focus on the umbrella quality improvement initiative, and "project" to indicate the Telecare fall prevention project.
Program development
Program development consisted of an initial leadership meeting with local GLA leaders (February 2008) to see whether fall prevention was a priority, monthly falls workgroup meetings (starting in April 2008) to decide on a specific project, and a second leadership meeting (October 2008) to review and refine the workgroup's care model for the project. Telecare Tuck-In was selected as a platform for the project during a workgroup meeting.
Implementation
Patients participated in the project through one of two routes: 1) GLA healthcare professionals referred them to Telecare Tuck-In for fall prevention, or 2) the Tuck-In nurse was notified by colleagues of patients who had called in to Telecare with a fall or fall-related injury. The clinical champion for the project, a Tuck-In nurse, then called the patient and read scripted questions to the patient (or caregiver) to assess fall risk factors (see Additional File 1: Telecare Tuck-In nurse script for falls prevention project). The patient's answers to the script were placed into the patient's electronic health record, along with free-text notes. Based on the patient's answers to the script, the nurse determined which, if any, referrals to make for further care, using a predetermined algorithm [11]. Options included the falls clinic at the main facility, geriatrics clinic at the main facility or a separate ambulatory care center, or referral to home care services for a home safety evaluation.
Formative evaluation
In order to obtain feedback on how to improve the project, we planned a formative evaluation, defined as an ongoing evaluation process integrated into the entire lifespan of a project, including development and implementation, rather than a classic evaluation at the end of the project's lifespan [19]. The evaluation's goals included 1) documenting how the project was implemented, including whether it was implemented and sustained according to plan, 2) identifying barriers and facilitators to implementation, and 3) studying how patients' quality of care was affected by the project. To address these aims we used three data sources: semi-structured interviews with patient and employee stakeholders, a review of workgroup meeting minutes, and electronic health record review for participating patients.
Semi-Structured Interviews
Semi-structured interviews were iteratively developed and refined by the research team with support from an internal GLA expert on interview design, resulting in standard interview guides (See Additional File 2: stakeholder interview script and response form and Additional File 3: patient interview script and response form). In keeping with the formative nature of the evaluation, research team members discussed interview results multiple times during the interview process, and pertinent results were shared at a monthly falls workgroup meeting on February 19, 2010; we did not, however, analyze interview data using a formal process. From our interview notes we counted up the number of times that various themes were mentioned by interviewees. Themes were not prespecified, but we made note of emergent themes during the interview process.
We identified a convenience sample of employee stakeholders for employee interviews; stakeholders were defined as those who had significant interaction with the project, including individuals who participated in the leadership meetings, were part of the workgroup, referred patients to the project, or received referrals from the project. Stakeholders were invited to participate by e-mail, and if they agreed, were interviewed by phone. The interviews, taking place between October 2009 and February 2010, covered employees' connection to the project, what they viewed the purpose of the project to be, how effective they found the project, what recommendations they might have, as well as any strengths or weaknesses they noted while working on the project. Interviews ranged from roughly two to thirty minutes in length, depending on the participants' responses and extent of involvement with the project.
Patients were interview candidates if the Tuck-In nurse had read them the script asking about falls risk factors between October 2008 and June 2009. The clinical champion maintained a log of all patients who were considered for the project and those who were actually read the script. We recruited, both by mail and by telephone, all eligible patients who had been read the script. We interviewed patients by phone in October 2009, seeking to capture their impressions of the project, with questions asking whether they remembered receiving the phone call about fall prevention, how they felt the project had helped them, and what suggestions or changes they would recommend. Patients were then asked permission for the second component of the evaluation, the electronic health record review. Patient interviews were of similar length to the employee interviews.
Electronic Health Record Review
For the structured electronic health record review, we adapted forms from a similar review in a previous project [20]. The review forms collect data that allow the Assessing Care of Vulnerable Elders-3 (ACOVE-3) quality indicators for falls [21] to be measured (see Additional File 4: Chart abstraction indicators). The ACOVE-3 indicators assess the extent to which the Telecare falls project complemented the quality of routine medical care for falls by specifically noting those recommended care processes that were provided by the Telecare nurse but not provided in the course of patients' usual care.
Cost Estimation
An additional component of the evaluation involved estimating basic costs associated with project development. Using the meeting minutes as our data source, we estimated a minimum cost of employee time for initial project development, since this activity occurred outside of employees' usual duties. The time spent by the primary investigators to evaluate this project was funded through grants. However, most of the time costs associated with project development were implicitly borne by GLA by allowing employees to participate in meetings. In order to calculate a minimum estimate of the personnel costs associated with project development, we tracked personnel attendance at the two leadership meetings and the seven workgroup meetings that occurred in the months in between these two meetings. We used actual federal employee salary data from the U.S. Office of Personnel Management [22], and thus calculated the total hourly contributions of participating workgroup and leadership meeting members. Salaries were from Fiscal Year 2008, and adjusted to include 30 percent fringe benefits standard in VA budgeting. These estimates constitute a lower bound on the overall resources used for project development, since activities of research personnel (including managing monthly workgroup meetings, developing the Telecare Tuck-In nurse script, and organizing leadership meetings) were not included. Also not included are the costs associated with actual program implementation, since we did not track times for phone calls and script administration.