Skip to main content
  • Research article
  • Open access
  • Published:

Measuring patient and family perceptions of team processes and outcomes in healthcare teams: questionnaire development and psychometric evaluation



There is a lack of validated instruments examining dimensions of team functioning from the perspective of patients and families consistent with a conceptual framework. The study aimed to develop and assess the psychometric properties of the Patient-Perceptions of Team Effectiveness (PTE) questionnaire.


A cross-sectional survey was undertaken in three studies. Data were collected from May–October 2016 for Study 1, April 2018-ongoing for Study 2, and October 2016 to June 2017 for Study 3. Online and paper versions of the self-administered questionnaire were available in English and in French. The initial questionnaire included 41 items. Study 1 included 320 respondents. Reliability was assessed using Cronbach alpha. Face validity (n = 250) was assessed using a structured questionnaire. Content validity was examined using subject matter experts and Spearman’s item-total correlations. Construct validity was examined using known group comparisons (i.e., clinical specialty, education, length of follow-up, reason of consultation). Content analysis was used for open-ended questions.


The questionnaire took 10 to 15 min to complete. Positive assessments were noted for instructions, formatting, font size and logical ordering of questions. In Study 1, reliability indices for the PTE-Overall, Team Processes and Outcomes subscales ranged from 0.72 to 0.84. Item-total correlations ranged from 0.551 to 0.794 (p <  0.001). Differences were noted between clinical specialties, education, length of follow-up, reason of consultation, low and high functioning teams. No differences were noted between English and French language respondents. Psychometric properties were re-assessed in Study 2 and 3 after unclear questions were reworked. Reliability indices for the subscales ranged from 0.76 to 0.94 and differences remained significant between low and high functioning teams.


The final 43-item instrument is easy to administer to patients and families. The studies provide evidence of validity to support the propositions in the conceptual framework. The patient-level measures can be aggregated to the team, organizational or system level. The information can be used to assess healthcare team functioning in acute and primary care and determine the role patients and families are playing in teams. Further testing is needed with patients and families who are hospitalized or receiving care from teams in rural areas.

Peer Review reports


Internationally, there is increased interest to implement patient-reported experience measures (PREMs) to report on what happens during a healthcare visit and improve the effectiveness of care processes [1]. Patient-reported outcome measures (PROMs) examine the impact of health conditions from the patient’s perspective [2]. PREMs and PROMs provide complementary information and are being used to examine healthcare system performance and care quality [3,4,5,6]. Researchers with the Belgian Health Care Knowledge Centre [7] completed a review of systematic reviews (n = 11) of PREMs and PROMs published after 2010, and found that much of the literature was focused in oncology and on PROM measures.

Wiering et al. [8] identified 193 PROMs in specialized and primary care following a scoping review of the PROM literature. Studies using generic PROMs (e.g., EQ-5D) have been of limited use because of their lack of responsiveness and ability to measure change [9]. There has been limited consensus on appropriate PROMs to use in primary care to capture the overall outcome of a visit, as opposed to disease-specific outcomes, especially for patients with several chronic conditions [10, 11]. Similarly, Desomer et al. [7] concluded in their review of reviews that research has focused largely on the impact of PROMs on the patient clinician relationship but all other domains such as primary care and chronic conditions are understudied. They identify a lack of standardized PREM measures, and measures that are at the individual and hospital level but not at the level of the healthcare team as important limitations. Patient judgements about how their healthcare team functions are team-level PREM measures [12].

Casimiro and colleagues [13] highlighted that care providers within the team need to remain flexible and shift teamwork patterns from uni- to interprofessional to facilitate patient-engaged care. This underscores the need for patients and families to understand how different providers contribute to their care. Team processes including communication, involvement in decision-making, cohesion, care coordination, and problem-solving help to explain how something is done or how events unfold over time within a specific context [14, 15]. These processes are dynamic and adaptive actions or steps undertaken by patients or providers to achieve a specific goal [16]. However, we have a very limited understanding of processes in interprofessional teams because it is “like looking into a black box” ([17] p.20, [18], p. S11).

Teams are defined as two or more individuals who work interdependently toward a common and valued goal [19]. Delivery models that promote interprofessional care are believed to be the most effective way to provide healthcare services [20]. Studies have demonstrated that improvements in teamwork can improve patient safety and efficiency in different settings including the operating theatre [21], palliative care [22, 23], and cardiopulmonary arrest simulations [24, 25]. Interprofessional healthcare teams were introduced in hospitals over a century ago [26]. However, interprofessional teams are relatively new in primary care [27], and their evaluation is a global imperative [28].

There is some empirical evidence to support the link between processes and outcomes in healthcare teams [29, 30]. In acute care, Tremblay et al. [31] found that high levels of inter-professional teamwork influenced PREMs in cancer care and Sidani and Doran [32] found some evidence to link processes to outcomes for nurse practitioners (NPs) in acute care. In primary care, Beaulieu et al. [33] examined organizational characteristics associated with high quality primary care and found that provider perceptions of work group innovations were moderately associated with outcomes. Poulton and West [34] demonstrated that processes influenced 23% of change in patient outcomes after controlling for different primary care structures. More recently, Strasser et al. [35] examined five measures of team functioning in rehabilitation teams and determined that only three out of 15 process measures of team functioning were sufficiently sensitive to identify changes in patient outcomes. These authors [35] concluded that team functioning measures were in their infancy but they showed promise and warranted further investigation. El Ansari et al. [36] developed a team effectiveness scale to measure team functioning in adult mental health teams in the community. The methodology used allowed them to generate context-specific measures of effectiveness for teams in the United Kingdom.

The impact of poorly functioning teams is staggering. In the United States (US), a review of over 23,000 malpractice claims across a range of practices over four years found that miscommunication was a factor in more than half the claims, costing $1.7 billion US and leading to the loss of 2000 lives [37]. There is a growing interest in understanding how processes in teams influence patient outcomes and patient experience of care [31, 38, 39]. Castro et al. [40] completed a concept analysis to distinguish between patient empowerment, patient participation and patient-centeredness in hospital care. These authors argued that patient centeredness occurs at the micro level of patient and provider interactions. Evidence is emerging to document the relationship between team structures, processes, perceived care quality and patient experience in cancer care [39].

To date, researchers [41] have focussed primarily on patient views of individual-level interactions with providers (e.g., communication, care coordination) to better understand patient experience and patient engagement [42]. Research is now beginning to shift to the team-level and focus on the patient as a partner in the healthcare team [43]. We know little about micro-level processes in healthcare teams or how they support the development of the role of patients and families as team members [40, 44]. Jeffcott and Mackenzie ([45], p., 190) argued that team performance is difficult to measure but there is a recognition of the “need to advance team measurement.” Valentine, Nembhart, and Edmonson [46] completed a systematic review of instruments to measure teamwork. These authors highlighted that existing questionnaires did not measure key dimensions (e.g., communication, coordination) of team functioning that were consistent with a conceptual framework.

There is a large body of research examining teamwork, interprofessional education, patient safety and communication in different healthcare settings (e.g., AHRQ, 2017; Brandt & Schmidtz, 2017; IOM, 2015) [47,48,49]. High quality evidence including systematic reviews have determined that care delivered by interprofessional teams (e.g., nurses, physicians, physiotherapists, nutritionists, etc.) improved patient outcomes (e.g., Andreatta & Marzano, 2012; Epstein, 2014: Rutherford, 2017) [50,51,52]. The quality of patient experiences of care has been identified as a “critical indicator of healthcare performance” ([53], p.17). However, Beaird et al. [54] and Pomey, Ghadiri, Karazivan, Fernandez, and Clavel [43] argued that more work was needed to include patients in the team.

Patients and families are able to identify and assess provider behaviours related to teamwork [55]. As an exemplar, patients and families under the care of interprofessional teams that included NPs believed their team was more effective after the implementation of an NP regardless if they received services in acute or primary care [44]. NPs have additional training and competencies compared with nurses allowing them to prescribe medications and treatments, order and interpret tests, and perform clinical procedures [56]. Patients and families described how teams with NPs used specific processes (e.g., communication, care coordination) to effectively provide care. Although patients and families identified similar processes across healthcare settings, the relative importance of each process changed if patients were cared for by teams in acute or primary care [44].

There is limited research examining how interprofessional teams provide care [14, 35, 57]. Research is needed to determine the influence of different structures, micro-level processes in primary and acute care teams on outcomes using validated measures [30, 58].To help fill this gap in knowledge, valid and reliable instruments are needed to assess team processes from the perspective of patients and families in acute and primary care.

Conceptual framework

The study was supported by the conceptual framework developed specifically to explain how roles are enacted and interprofessional teams function [59]. Three central processes are at the heart of the framework including role enactment, boundary work and perceptions of team effectiveness. Role enactment in teams focuses on what providers do in their role on a day-to-day basis and on how care is provided. Role enactment is influenced by role clarity. Boundary work is a process of shifting or negotiating new professional boundary lines between members of the team when a new role is introduced or roles change in the healthcare team. Boundary work activities evolve over time. The development of trust among team members is a key factor that influences boundary work activities and changes in professional roles [60]. Perceptions of team effectiveness include the beliefs that the team can perform across a range of activities and team members want to continue working together. In separate studies that examined patient, family and provider perceptions of team effectiveness, participants identified six processes in effective teams [41, 61]. They included improved communication, involvement in decision-making, cohesion, care coordination, problem-solving, and a focus on patient and family needs. Structures from the patient to the healthcare system level are represented in the framework using concentric circles that surround the central processes. Structures and processes influence outcomes such as quality, safety, costs and team outcomes. Relationships are symbolized by bi-directional arrows and dotted lines [59]. Within the framework and in the present study, the term interprofessional was used to examine interactions that involved members of more than one professional group (e.g., the nursing group and another professional group other than nursing). The conceptual framework were used to structure the questionnaire, and link structures, team processes and outcomes.



The study aimed to 1) develop and cross-validate English and French language versions of the Patient-Perceptions of Team Effectiveness (Patient-PTE) Questionnaire to assess team processes and perceived outcomes in healthcare teams from the perspectives of patients and families; and 2) assess the psychometric properties (i.e., reliability, face validity, content validity, and construct validity) of the Patient-PTE. We used the approach proposed by Dillman et al. [62] and Peter et al. [63] to develop and administer the questionnaires. This approach includes, among others, patient interviews, item generation, expert interviews, pilot-testing of the draft to identify unclear items and rephrasing, translation, and the pilot study to examine internal consistency and item correlations. The paper reports on the preliminary validation study (Study 1) and subsequent studies (Study 2 and Study 3 described below) that assessed the psychometric properties of the revised questionnaire.

Instrument development

To address Objective 1, the questionnaire was developed using empirical evidence from studies examing team processes that included a descriptive qualitative study of patients and families (n = 49) under the care of acute and primary care teams with NPs [44], multiple-case studies of processes in interprofessional teams [60, 64, 65] and a literature review of processes in healthcare teams [14]. In the qualitative descriptive study [44], patients and families used examples of how care was provided by healthcare teams with and without NPs to identify or explain the similarities and differences with how care was provided by these teams. Patients and families had similar views of what constituted good or poor team functioning. The interviews allowed us to determine that one questionnaire was needed to examine team functioning from the patient and family perspective.

Item generation

Items were generated using data from semi-structured interviews with patients and families that were transcribed verbatim and analyzed using content analysis [44]. We searched for descriptions of micro-level processes and actions identified by patients and families including how team members worked together to solve problems and if patients and families were involved in decisions about their care. We mapped the statements to processes and outcomes in teams to ensure that key processes in the framework were represented [66]. Draft items were developed from the statements and reviewed by members of the research team including a patient (NF) to ensure item clarity and legibility. In the qualitative descriptive study [44], we found that patients and families could name the professionals they included in their healthcare team to estimate the size of the team but several patients and families had difficulties distinguishing between professional groups, particularly between different groups of nurses (e.g., licensed practical nurses, registered nurses and nurse practitioners) in their healthcare team. Thus, one item was included to identify team size and another item ascertained if teams included a specific provider (e.g., nurse practitioners in the validation study). No item was developed to identify the professional groups in the team because of patient and family difficulties to distinguish between provider groups.


Patient-Perceptions of Team Effectiveness Questionnaire

The preliminary Patient-PTE questionnaire included 41 items. Section I describing the characteristics of the healthcare team included 6 items (e.g., type of healthcare setting, location, team size). The PTE-Overall subscale included 14 items divided into Team Processes (e.g., communication, decision-making) and the factors (e.g., trust, role clarity) known to influence these processes. The Outcomes subscale included five items (e.g., timely care, patient knowledge about medication). Responses in PTE-Overall and Outcomes used a 1–7 Likert scale. Responses ranged from strongly disagree, disagree, disagree somewhat, neutral/no opinion, agree somewhat, agree, and strongly agree. The items were summed for each dimension (e.g., communication, coordination) and a mean value was estimated to obtain a score for the dimension (See Table 1). The scores ranged from 1 to 7. Higher scores in PTE-Overall, Team Processes and Outcomes indicated improved perceptions of team processes and outcomes respectively. Section IV describing demographic characteristics of patients and families included 15 items (e.g., perceived health, age, gender, education, marital status). Almost all response options in Section IV were categorical with the exception of length of follow-up which was a numeric value. One open-ended question was included to gather any additional comments from respondents. Single-item measures were used to assess boundary work (i.e., trust), belief about team effectiveness (BE), care coordination, and problem-solving [67]. Two-item measures were used to examine decision-making, communication, cohesion, patient and family focus, and role clarity. To reduce response burden, single-item measures may be more appropriate in vulnerable populations [67, 68].

Table 1 Preliminary Items Related to Healthcare Team Processes and Outcomes in Study 1


Strategies to enhance the cross-cultural adaptation of the questionnaire included translation and back-translation by a professional translator and a review of the questionnaire by bilingual researchers [69]. We adopted the method proposed by Gillemin et al. [69]. Interview participants responded in English and French. We harmonized the statements into English. The initial questionnaire was translated from English to French and back-translated from French to English by back-translators who were unaware of the intent of the survey. The final translation was verified by the research team where four members are bilingual and have experience translating surveys. This allowed us to identify any discrepancies in translation [70].

Following the review of draft items by the research team, English and French language versions of the questionnaire were pre-tested with patients and families (n = 6) to identify unclear wording. As proposed by Kilpatrick et al. [71], the same participants completed the questionnaire after two weeks to determine if their understanding of the questions changed over time. One revision to improve clarity was made to one item following pre-testing (i.e., “Team members work in silos” was changed to “Team members work in isolation (in silos)”). No differences in participants’ responses and understanding of the questions were noted when participants completed the questionnaire after two weeks.


A cross-sectional survey were undertaken (Study 1). A self-administered questionnaire was identified as a cost-effective method to gather information from large groups of participants [72].


In Study 1, a convenience sample was used. We aimed to recruit 300 patients and families to assess instrument performance and identify factors that influenced perceptions of team effectiveness. This sample size was sufficiently large to detect medium (0.30) correlations and a 95% confidence interval half-width of 0.1, a statistical power of 80% and a 5% alpha level [73, 74].

As proposed by Knafl, Leeman, Havill, Crandell, and Sandelowski [75], a broad definition of family was adopted to include persons who provide instrumental and relationship support deemed to be significant by the patient. Patients and families followed by teams in acute or primary care were eligible to participate in the study. Interviews with patients and families in primary care, nephrology, cardiology, and the neonatal intensive care unit indicated that they valued the same processes and perceived outcomes regardless of their health concern [44]. Nine to 10 participants in each group allowed us to attain theoretical and data saturation [44]. Acute care was defined as in-hospital or specialized ambulatory care to address specific health conditions [76]. Primary care was defined as the point of entry to the healthcare system and included comprehensive healthcare services for common health concerns [77]. Participants were recruited face-to-face in waiting rooms [78]. For this reason, it was not possible to estimate a response rate. It is worth noting that almost no participants declined to participate in the study. Those who declined or did not complete the questionnaire 1) were called in for their health appointment and did not have enough time to complete the questionnaire; or 2) they felt unwell or had received pre-medication prior to receiving their chemotherapy treatment. Questionnaires were administered to patients and families attending ambulatory acute care or outpatient care in a large integrated health system that included three hospitals, eight local community care centres, long-term care institutions, and day centres.

Data collection

Following Research Ethics Committee approval (Study 1: 2016–712, 15,065; Study 2: MP-12-2017-841, CIUSSS-EMTL-203, Study 3: 2017–337), data were collected from May to October 2016 for Study 1. Online and paper copies of all study documents were available in English and in French to enhance response rates [79]. For Study 1, recruitment occurred in oncology, maternal and perinatal care, pediatrics, and primary care. Posters were placed in view in the waiting rooms of these clinics to advise potential participants that data collection was underway. An information sheet and consent form, a feedback form and questionnaire were included in each handout. Participants were advised that they were under no obligation to participate in the study and could withdraw at any time. To address Objective 2, cognitive interviewing, a feedback form, and statistical analyses were used to assess the psychometric properties of the questionnaire in Study 1.

Cognitive interviewing

Cognitive interviewing with scripted probes were used to explore the meaning of participant responses [80]. Participants who completed the pre-test and the main survey were asked to describe their understanding of survey items (e.g., What did it mean to you when you responded…). Research assistants noted their responses to determine if their understanding matched the intent of the question [62]. As proposed by Kelley, Clark, Brown, and Sitzia [81], participants were asked to review instructions, questionnaire layout, procedures to complete French and English language questionnaires, and item and response options for clarity.

Feedback form

As proposed by Pomey et al. [43], patients and families were viewed as subject matter experts. All participants in Study 1 were asked to complete a feedback form immediately following the completion of the questionnaire. Participants were asked how long it took to complete the questionnaire to identify a risk of response burden. The feedback form was used by Bryant-Lukosius et al. [82] and Kilpatrick et al. [71] to assess provider questionnaires. Participants were asked if 1) the time to complete the questionnaire was appropriate; 2) the instructions were clear; 3) relevant questions NOT included in the Patient-PTE questionnaire needed to be added; 4) there were any irrelevant, unimportant or redundant questions that could be removed without jeopardizing the completeness of the study results; 5 and 6) format and font size were easy to read; and 7) questions were ordered logically. The eighth item of the feedback form was an open-ended question and asked if patients and families had suggestions to improve the Patient-PTE questionnaire. Responses to the eighth question of the feedback form informed the assessment of content validity.

Data analysis

Data were analyzed using IBM Statistical Package for the Social Sciences, version 25 [83]. Items were positively and negatively worded. The negatively-worded items (n = 7) were reverse-coded prior to the analyses. Descriptive statistics were generated (e.g., mean, standard deviation (SD), range). An average score was estimated to determine an overall Patient-PTE score (PTE-Overall). Scores by subscale and dimension were calculated. Criteria to assess the intrument’s performance included Cronbach alpha (α), item-total correlation of each item, and less than 5% missing data. The Shapiro-Wilk test was performed to test for normality [84]. Responses did not follow a normal distribution (p values < 0.001). Thus, Kruskal-Wallis one-way analysis of variance was used for non-parametric testing of variables with more than two categories and Mann-Whitney U was used to assess variables with two categories. The Mann-Whitney U post hoc test with a Bonferroni correction was applied where α was divided by the number of groups under consideration (e.g., α/3 = 0.017 for variables with three response categories) to correct for multiple comparisons and determine where the differences were between the groups. Spearman’s item-total correlations [rho coefficient] were used to assess the relationships for continuous data [85]. Content analysis was used for the open-ended questions and responses to cognitive interviews.

The numeric responses for length of follow-up were recategorized into less than 12 months, 12–59 months, and 60 months or more to identify differences in team processes over time. Education was recategorized as having completed or not completed high school education to ascertain if completion rates for the questionnaire were lower for respondents with less education. The responses to the item examining beliefs about team effectiveness (BE) from 1 to 5 were recategorized as low functioning teams and responses 6 and 7 for this item were recategorized as high-functionong teams. As proposed by Norman et al. (2003) [86] differences of half a standard deviation of the belief about team effectiveness score were considered to be a minimally significant difference in team functioning. After taking into account that data were not normally distributed and the small sample size in the low functioning group in the Study 2, we compared the median scores for Boundary Work, Team Processes, Role Clarity, and Outcomes between the groups using a Mann-Whitney U test.

Instrument performance

Reliability. Internal consistency of the PTE-Overall, Team Processes and Outcomes subscales was assessed using Cronbach alpha (). Values ranging from 0.7 to 0.9 were considered acceptable to excellent [87]. For Study 1, the within-group variance was compared to the between-group variance to determine the consistency of responses among respondents within the same group [88]. As proposed by Verran, Gerber, and Milton [89], variability within the known groups should be less than the variability across the groups to aggregate data from the individual to the team or organizational level.

Face validity was assessed using the feedback form that examined the Patient-PTE questionnaire, formatting and instructions. Frequencies and percentages were generated.

Content validity was assessed in two steps. In the first step, prior to the questionnaire roll-out, different expert groups including patients/families (n = 6), researchers (n = 7), NP students (n = 25) and graduate students (n = 4) in other professional roles (e.g., physiotherapy, management) examined the questionnaire. Two questions were added to ask if respondents 1) had any one of 16 health conditions; and 2) the extent to which specific chronic conditions (i.e., diabetes, heart and respiratory problems) influenced their day-to-day activities. The second step occurred in Study 1 where all respondents were viewed as subject matter experts and asked to complete the open-ended question in the feedback form.

Construct validity was assessed using the known-group technique where we compared the scores between specific groups [90, 91]. Based on the views of patients and families who participated in the qualitative descriptive study of teams with NPs [44], we hypothesized a priori that no differences in scores would be found between 1) men and women; 2) patients and families; 3) English and French language respondents; 4) respondents in urban and rural locations; 5) marital status; and 6) employment status. Differences were expected according to 1) clinical speciality; 2) team size (small (less than 5 members), medium (5–10 members), large (more than 10 members); 3) respondents’ education; 4) length of follow-up; 5) reason of consultation (i.e., routine/annual examination, pregnancy follow-up, follow-up for a chronic health problem, new health problem, urgent health problem, several chronic health problems); and 6) teams with and without NPs. Factor analysis was not used to determine construct validity because the measures had fewer than three items per scale [92]. Responsiveness was defined as the instrument’s ability to detect a meaningful change [93]. Although there is no agreed-upon method to assess responsiveness, we hypothesized that the questionnaire could distinguish between low and high-functioning teams.

In the following section, results from Study 1 including means, correlations, Cronbach α and p values statistically significant at the 5% level or α/ngroups for post-hoc analysis are reported by measure and known group, followed by the results of Study 2 and 3.


Overall, 355 participants responded in Study 1. The questionnaire took 10 to 15 min to complete. We excluded 20 questionnaires that included only socio-demographic data and 15 questionnaires with more than 20% missing data for PTE-Overall and Outcomes. Thus, 320 questionnaires were included in the analyses. Almost all questionnaires were completed on paper; online questionnaires were completed by 5% of respondents (n = 17).

Socio-demographic characteristics of participants are presented in Table 2. Sixty seven percent (n = 209) of respondents were female, and their average age was 50.7 years ±15.9[mean ± SD]. Seventy nine percent (n = 248) of respondents were patients, and 39% (n = 121) of respondents perceived their health was good to very good (25%, n = 78). Most were born in Canada (79%, n = 242) and living in an urban area (97%, n = 309). Seventy three percent (n = 214) of respondents spoke only French and 82% (n = 262) of respondents completed the questionnaire in French. Seventy one percent (n = 222) of respondents were married or living with a partner. Eighty five percent (n = 266) had completed high school education or above. Thirty five percent (n = 109) of respondents worked full time.

Table 2 Characteristics of Respondents and Their Healthcare Team in Study 1

Respondents were followed by healthcare teams in oncology (64%, n = 205), maternal-perinatal care (17%, n = 55), primary care (9%, n = 28), pediatrics (5%, n = 17); and other specialties (5%, n = 14) including medicine, surgery, emergency-critical care, cardiovascular, rehabilitation, hospice/palliative care, community health, home care, psychiatry, mental health, and long-term care. The number of respondents in the primary care-pediatrics, maternal-perinatal care, and oncology groups were sufficiently large to allow for known-group comparisons.

Respondents consulted the healthcare team for a variety of reasons including new health problem, chronic illness care, and pregnancy follow-up. About half of respondents had been followed by their healthcare team for less than a year (51%, n = 150, 30 ± 53 months; range 1 to 420 months). The length of follow up by specialty was provided in Table 3. Most (90%, n = 286) were followed by healthcare teams that did not include an NP and about one third of participants viewed their team as a small, medium or large size team.

Table 3 Length of Follow-Up (Months) by Specialty in Study 1

Reliability was assessed by estimating Cronbach α values for PTE-Overall, Team Processes and Outcomes. Cronbach α values for PTE-Overall and Team Processes equalled 0.84, and Outcomes equalled 0.72 (See Table 4). Across the groups, the Mean Square (MS) values for the within groups variability were much smaller than the MS values between groups for all the measures indicating that individual level data could be aggregated at the team or organizational level (See Table 5).

Table 4 Scores for Boundary Work, Perceptions of Team Effectiveness, Role Clarity, and Outcomes in Study 1
Table 5 Between and Within-Group Variability by Specialty in Study 1

Face validity was assessed using the responses (n = 250) in the feedback form (See Table 6). All the responses for the feedback form were completed using paper copies. Most (85%) believed that key questions to understand team functioning were included in the questionnaire and no question was unimportant. Positive assessments were noted for the time needed to complete the questionnaire, instructions, formatting, font size and logical ordering of questions.

Table 6 Responses Related to Face Validity in Study 1

Content validity was assessed using cognitive interviewing and the open-ended question in the feedback form (n = 51). Four questions needed to be further clarified. The first question asking if the healthcare team included a nurse practitioner was confusing because some participants were not sure what a nurse practitioner role was. The second and third questions asked about overlapping roles in the healthcare team and the role of patients in the healthcare team. Respondents believed these questions needed to be expanded. The fourth question asked about annual household income. Respondents believed that rather than including a range for family income, it would be easier to respond to the question if it centered on how patients and families perceived their financial situation.

Overall, patients and families rated Team Processes 5.4/7 ± 0.9 (range: 1.6–7) with Boundary Work rated highest (mean: 5.6/7 ± 1.5) followed by PTE-Overall (mean: 5.4/7 ± 0.9), and Role Clarity (mean: 5/7 ± 1.3). Outcomes were rated on average 5.4/7 ± 1.1. The ratings for the dimensions included in the Team Processes ranged from 4.7/7 ± 1.9 for care coordination to 6.1/7 ± 1.2 for the BE score. The rho coefficients (rs) for the dimensions included in Team Processes ranged from 0.55 to 0.79 (p value< 0.001) (See Table 4). The mean values of Team Processes were generally lower for the primary care-pediatrics group with the exception of decision-making and care coordination which were lower in the maternal-perinatal care group (data not shown).

Construct validity was assessed using known groups (See Table 7). As anticipated, no differences were noted between men and women; patients and families; English and French language respondents; according to marital status; and employment status. As hypothesized, differences were identified in groups according to education, length of follow-up, and reason of consultation. We had anticipated differences according to team size and in teams with and without NPs but no significant differences were identified in these groups. Completion rates were similar for respondents who had and had not completed high school education. We were unable to assess differences for urban and non-urban respondents due to the small sample size.

Table 7 Differences in Team Processes and Outcomes by Known Group Comparisonsa,b in Study 1

Responsiveness. As anticipated, differences (p <  0.001) were noted for Boundary Work, Team Processes, Outcomes, and Role Clarity between low and high functioning teams (See Table 8).

Table 8 Differences in Processes and Outcomes in Low and High Functioning Teams in Study 1

Changes made following study 1

Study 1 allowed us to refine the items included in the questionnaire and identify unclear or unnecessary questions. Using cognitive interviewing and respondent suggestions, changes were made to simplify and improve the flow of the questionnaire. The No opinion/Not applicable response option was not identified as an issue by respondents during the cognitive interviews, and the response was used infrequently by respondents who completed the questionnaire. We opted to remove the option to simplify the questionnaire, thus reducing the number of response options to six instead of seven. Other modifications included adding items to measure care coordination and an item pertaining to the team’s in-depth knowledge and skill, and separating the question describing patient and family’s role in the team into two questions to take into account that patients and families believed they had a role in the team and that they could play an active role as a team member. Respondents believed there were too many negatively-worded questions. We revised and left only three negatively worded items. We changed the monetary values for family income to asking about the perceived adequacy of family income, and added a definition for the NP role to ensure that participants understood which provider role was under consideration. The revised items are included in Table 9. The complete instrument included 43 items and is provided as an additional file.

Table 9 Revised Items Related to Healthcare Team Processes and Outcomes in Study 2

Additional evidence of validity

Results from study 2

We used the first 242 completed questionnaires from a larger study of patients and families followed by primary care teams with nurse practitioners. These respondent were randomly selected from lists of patients. Data collection for Study 2 began in April 2018 and is ongoing. Dillman’s tailored approach was used to mail-out questionnaires to respondents and included a pre-notification, complete survey package with a $5 token incentive and a pre-paid return envelope, one reminder post-card, and a final complete package with a postage-paid retun envelope [62]. Respondents included patients (n = 161) and families (n = 74). Seventy percent of respondents were women. Overall, Boundary Work was rated highest 5.3/6 ± 0.92. Team Processes averaged 4.9/6 ± 0.66 with Patient/Family Focus scoring lowest (4.3/6 ± 1.13) and beliefs about team effectiveness (BE) scoring highest (5.4/6 ± 0.86). Outcomes of team functioning scored on average 5.1/6 ± 0.74. The rho coefficients (rs) for the dimensions included in Team Processes ranged from 0.61 (problem-solving) to 0.83 (care coordination) (p value< 0.001). Cronbach α values increased for PTE-Overall (0.88) and Outcomes (0.78), and were unchanged for Team Processes (0.84). The differences between low and high functioning teams for Boundary Work, Team Processes, Outcomes, and Role Clarity noted in Study 1 remained significant (p <  0.001) (Table 10).

Table 10 Differences in Processes and Outcomes in Low and High Functioning Teams in Study 2

Results from study 3

Participants (n = 50) were recruited as part of a prospective observational study examining the impact of follow-up by physiotherapists in primary care on patients’ pain levels for different musculo-skeletal conditions. Data were collected between October 1st 2016 and June 10th 2017. Respondents were primarily women (n = 37, 71%). A majority of participants were married (n = 31, 60%) and 35% (n = 18) were of low socio-economic status. Patients received an average of three therapy sessions with the physiotherapists. Similar results were found in this group of patients where Boundary Work was rated highest (5.3/6 ± 0.80), followed by PTE-Overall (5.1/6 ± 0.67), and Outcomes (5.0/6 ± 0.68). Team Processes averaged 5.1/6 ± 0.67 with Patient/Family Focus scoring lowest (4.7/6 ± 0.81) and beliefs about team effectiveness (BE) scoring highest (5.4/6 ± 0.80). The rho coefficients (rs) for the dimensions included in Team Processes ranged from 0.64 (patient/family focus) to 0.86 (communication) (p value< 0.001). Cronbach α values in this group were higher for PTE-Overall (0.94) and Team Processes (0.92), and slightly lower for Outcomes (0.76).


The purpose of the study was to develop and assess the psychometric properties of English and French language versions of the Patient-PTE questionnaire. The questionnaire was designed to measure patient and family perceptions of how healthcare teams provide care and the teams function in acute and primary care. We identified strong positive correlations between processes in teams and care outcomes that were consistent with a conceptual framework. Reliability indices for the revised questionnaire were good to excellent. The psychometric evaluation of the questionnaires provided evidence of validity.

The changes made to the questionnaire improve our ability to assess team functioning and distinguish between low and high functioning teams. Researchers can use the questionnaire to assess patient and family perceptions of processes in teams for different provider groups (e.g., physiotherapists, respiratory therapist, social workers) by changing the definition of the provider group being considered. This will allow researchers, decision-makers, patients and their families to understand how these providers contribute specifically to team processes and effective team functioning. We added a “Prefer not to respond” option to the question examining gender. Although no respondent asked for the addition, we believe that this will show respect for participants who do not wish to identify their gender or who do not identify with a specific gender. This will allow us to further refine our analyses and better understand the impact of gender on perceptions of team functioning [94]. In addition, we bolded key terms in the negatively worded items (e.g., not) in the Team Processes subscale to ensure that respondents clearly identify the negative statements.

The questionnaire helps to fill an important gap in the literature related PREM and PROM measures as well as the assessment of team functioning by placing patients and families at the center of the healthcare team conceptually and in the measurement tool [46, 95]. The questionnaire provides individual patient- and family-level measures of team processes and outcomes, and can help identify specific areas of team functioning that can be improved. The questionnaire can provide managers and decision makers with before and after data following the implementation of an intervention to improve team functioning or measure patient participation in increasingly complex healthcare teams [96].

The development of high functioning teams is a priority worldwide [27, 97, 98]. An important consideration to assess processes at the team-level is the ability to aggregate individual patient data to the team or organizational level and reduce the risk of measurement bias in aggregate data [88, 99]. To do this, it is necessary to determine the level of agreement of the assessments among the individuals within the same group [100]. Given the tool’s performance, it is possible to aggregate the individual level data to the team-, organizational- or system- level. The aggregated team or organizational assessments of team functioning can inform healthcare decisions (e.g., communication). This information about team processes has the potential to fill a gap in knowledge where the link between structures and outcomes is not always clear.

The questionnaire is short and easy to complete which is an important consideration for patients who are very ill, and families with limited time or energy to complete these instruments. It provides evidence that is consistent with the propositions in the conceptual framework about structures, team processes and outcomes. The questionnaire has a broad application because it measures patient and family views of team functioning in acute and primary care teams and as patients transition through the healthcare system. Patients and families can use the information to assess how their healthcare team is performing and determine the role they are playing in the team. Providers can use the information to tailor care, and support patient involvement as team members. Decision-makers can use aggregated data to monitor team performance. This appears to be particularly important in the current global trend of organizational restructuring and increased health system integration where processes in teams may be affected as teams work across settings, and healthcare systems around the world are becoming more goal-oriented and performance-driven [101, 102].

Some limitations must be kept in mind. Few responded to the questionnaire on-line. Thus, we were unable to examine if their perceptions differed from those who completed paper copies. This may be due in part to the electronic link to the questionnaire in Study 1 that was long to memorize, and the application for smart phones was not available. Future studies will need to recruit patients and families with different characteristics (e.g., hospitalized patients, patients in rural areas) as well as examine PTE from the provider perspective. These studies will allow us to gather additional evidence of validity for the revised questionnaire and test relationships between different variables in the framework and in low and high functioning teams. To test the relationship between perceptions of outcomes and actual outcomes, further research is needed to examine patient and family perceptions of outcomes and clinical or condition-specific indicators for health conditions including glycated hemoglobin A1C for patients with diabetes, systolic and diastolic blood pressure measures for patients with hypertension, adherence to medication regimens for patients with different health conditions.


The study aimed to develop and assess the psychometric properties of the Patient-PTE questionnaire. The study produced evidence to support propositions in the conceptual framework of role enactment and perceptions of team effectiveness. The questionnaire can be used to assess structures, processes and perceived outcomes in low and high functioning teams in acute and primary care from the perspective of patients and their families. The instrument was validated for English and French language respondents. Individual patient-level data can be aggregated to the team-, organization- or system levels. Further research is needed to examine the perspectives of patients and families who are hospitalized or living in rural or remote areas and when other providers are included in the healthcare team.





Agency for Healthcare Research and Quality


Institute of Medicine


Mean square


Nurse practitioners


Perceptions of team effectiveness


Standard deviation


United States


  1. Kingsley C. Patel S patient-reported outcome measures and patient reported experience measures. BJA Education. 2017;17(4):137–44.

    Article  Google Scholar 

  2. Canadian Institute for Health Information. PROMs and PREMs at CIHI Measuring patient-centered care Calgary, Alberta. November 27, 2014. Accessed 17 Oct 2018.

  3. Canadian Institute for Health Information. Patient-centred measurement peer learning day: Translating patient-centred measurement into action. 2017. Accessed 17 Oct 2018.

  4. Deyo P, Swartwout E, Drenkard K. Nurse manager competencies supporting patient engagement. J Nurs Adm. 2016;46(3 Suppl):S19–26.

    Article  PubMed  Google Scholar 

  5. Weenink JW, Braspenning J, Wensing M. Patient reported outcome measures (PROMs) in primary care: An observational pilot study of seven generic instruments. BMC Fam Pract. 2014;15:88.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Mackie TI, Sheldrick RC, de Ferranti SD, Saunders T, Rojas EG, Leslie LK. Stakeholders' perspectives on stakeholder-engaged research (SER): strategies to operationalize patient-centered outcomes research principles for SER. Med Care. 2017;55(1):19–30.

    Article  PubMed  Google Scholar 

  7. Desomer A, Van den Heede K, Triemstra M, Paget J, De Boer D, Kohn L, et al. Use of patient-reported outcome and experience measures in patient care and policy. In: Health Services Research (HSR). Brussels: Belgian Health Care Knowledge Centre (KCE). KCE Reports 303; 2018. Accessed 17 Oct 2018.

    Google Scholar 

  8. Wiering B, de Boer D, Delnoij D. Patient involvement in the development of patient-reported outcome measures: a scoping review. Health Expect. 2017;20(1):11–23.

    Article  PubMed  Google Scholar 

  9. Bryan S, Davis J, Broesch J, Doyle-Waters M, Lewis S, McGrail K. Choosing your partner for the PROM: a review of evidence on patient-reported outcome measures for use in primary and community care. Res Pap. 2014;10(20):39–51.

    Google Scholar 

  10. Canadian Institute for Health Information. PROMs Background document. 2015. Accessed 17 Oct 2018.

  11. Murphy M, Salisbury C, Hollinghurst S. Can the outcome of primary care be measured by a patient reported outcome measure? Br J Gen Pract. 2014;64(629):647–8.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Male L, Noble A, Atkinson J, Marson T. Measuring patient experience: a systematic review to evaluate psychometric properties of patient reported experience measures (PREMs) for emergency care service provision. Int J Qual Health Care. 2017;29(3):314–26.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Casimiro LM, Hall P, Kuziemsky C, O'Connor M, Varpio L. Enhancing patient-engaged teamwork in healthcare: an observational case study. J Interprof Care. 2015;29(1):55–61.

    Article  PubMed  Google Scholar 

  14. Kilpatrick K, Lavoie-Tremblay M, Ritchie JA, Lamothe L. Advanced practice nursing, health care teams, and perceptions of team effectiveness. J Trauma Nurs. 2014;21(6):291–9.

    Article  PubMed  Google Scholar 

  15. Pettigrew AM. What is a processual analysis? SJM. 1997;13(4):337–48.

    Article  Google Scholar 

  16. Van de Ven AH. Suggestions for studying strategy process: a research note. SMJ. 1992;13:169–88.

    Google Scholar 

  17. Thomson A, Perry J. Collaboration processes: Inside the black box. Publ Admin Rev. 2006;66(1 Special Issue):20–32.

    Article  Google Scholar 

  18. Coleman K, Phillips KE, Van Borkulo N, Daniel DM, Johnson KE, Wagner EH, et al. Unlocking the black box: Supporting practices to become patient-centered medical homes. Med Care. 2014;52(11 Suppl 4):S11–7.

    Article  PubMed  Google Scholar 

  19. Salas E, Burke CS, Cannon-Bowers JA. Teamwork: emerging principles. Int J Manag Rev. 2000;2(4):339–56.

    Article  Google Scholar 

  20. Nelson S, Turnbull J, Bainbridge L, Caufield T, Hudon G, Kendel D, et al. Optimizing scopes of practice: new models of care for a new health care system. Ottawa: Canadian Academy of Health Sciences; 2014. Accessed 17 Oct 2018.

    Google Scholar 

  21. Weld LR, Stringer MT, Ebertowski JS, Baumgartner TS, Kasprenski MC, Kelley JC, et al. TeamSTEPPS improves operating room efficiency and patient safety. Am J Med Qual. 2016;31(5):408–14.

    Article  PubMed  Google Scholar 

  22. Brennan CW, Kelly B, Skarf LM, Tellem R, Dunn KM, Poswolsky S. Improving palliative care team meetings: structure, inclusion, and "team care". Am J Hosp Palliat Care. 2016;33(6):585–93.

    Article  PubMed  Google Scholar 

  23. Goebel J, Guo W, Wood K. Teamwork and perceptions of palliative care quality. JHPN. 2016;18(3):242–8.

    Article  Google Scholar 

  24. Mahramus T, Frewin S, Penoyer DA, Sole ML. Perceptions of teamwork among code team members. Clin Nurse Spec. 2013;27(6):291–7.

    Article  PubMed  Google Scholar 

  25. Mahramus T, Penoyer D, Waterval E, Sole M, Bowe E. Two hours of teamwork training improves teamwork in simulated cardiopulmonary arrest events. Clin Nurse Spec. 2016;30(5):284–91.

    Article  PubMed  Google Scholar 

  26. Linker B. Teamwork: metaphors and myths of equality in the health-care setting. Nurs Hist Rev. 2016;24:69–75.

    Article  PubMed  Google Scholar 

  27. Sullivan EE, Ibrahim Z, Ellner AL, Giesen LJ. Management lessons for high-functioning primary care teams. J Health Manag. 2016;61(6):449–66.

    Google Scholar 

  28. World Health Organization. Global strategy on human resources for health: Workforce 2030. 2016. Accessed 17 Oct 2018.

  29. Cavanaugh KL. Patient experience assessment is a requisite for quality evaluation: a discussion of the in-center hemodialysis consumer assessment of health care providers and systems (ICH CAHPS) survey. Semin Dial. 2016;29(2):135–43.

    Article  PubMed  Google Scholar 

  30. Price AR, Elliott MN, Zaslavsky AM, Hays RD, Lehrman WG, Rybowski L, et al. Examining the role of patient experience surveys in measuring health care quality. Med Care Res Rev. 2014;71(5):522–54.

    Article  Google Scholar 

  31. Tremblay D, Roberge D, Touati N, Maunsell E, Berbiche D. Effects of interdisciplinary teamwork on patient-reported experience of cancer care. BMC Health Serv Res. 2017;17(1):218–29.

    Article  Google Scholar 

  32. Sidani S, Doran D. Relationships between processes and outcomes of nurse practitioners in acute care: an exploration. J Nurs Care Qual. 2010;25(1):31–8.

    Article  PubMed  Google Scholar 

  33. Beaulieu MD, Haggerty J, Tousignant P, Barnsley J, Hogg W, Geneau R, et al. Characteristics of primary care practices associated with high quality of care. Can Med Assoc J. 2013;185(12):E590–6.

    Article  Google Scholar 

  34. Poulton B, West M. Effective multidisciplinary teamwork in primary health care. J Adv Nurs. 1993;18:918–25.

    Article  CAS  Google Scholar 

  35. Strasser DC, Burridge AB, Falconer JA, Uomoto JM, Herrin J. Toward spanning the quality chasm: An examination of team functioning measures. Arch Phys Med Rehabil. 2014;95(11):2220–3.

  36. El Ansari W, Lyubovnikova J, Middleton H, Dawson JF, Naylor PB, West MA. Development and psychometric evaluation of a new team effectiveness scale for all types of community adult mental health teams: a mixed-methods approach. Health Soc Care Community. 2016;24(3):309–20.

    Article  PubMed  Google Scholar 

  37. CRICO Strategies- The Risk Management Foundation of the Harvard Medical Institutions Incorporated. Malpractice Risks in Communications Failures- 2015 annual benchmarking report. 2015. Accessed 17 Oct 2018.

  38. Song H, Chien AT, Fisher J, Martin J, Peters AS, Hacker K, et al. Development and validation of the primary care team dynamics survey. Health Serv Res. 2015;50(3):897–921.

    Article  PubMed  Google Scholar 

  39. Song H, Ryan M, Tendulkar S, Fisher J, Martin J, Peters AS, et al. Team dynamics, clinical work satisfaction, and patient care coordination between primary care providers: a mixed methods study. Health Care Manag Rev. 2017;42(1):28–41.

    Article  Google Scholar 

  40. Castro EM, Van Regenmortel T, Vanhaecht K, Sermeus W, Van Hecke A. Patient empowerment, patient participation and patient-centeredness in hospital care: a concept analysis based on a literature review. Patient Educ Couns. 2016;99(12):1923–39.

    Article  PubMed  Google Scholar 

  41. Barello S, Graffigna G, Vegni E. Patient engagement as an emerging challenge for healthcare services: mapping the literature. Nurs Res Pract. 2012;2012:1–7.

    Article  Google Scholar 

  42. Carman KL, Dardess P, Maurer M, Sofaer S, Adams K, Bechtel C, et al. Patient and family engagement: a framework for understanding the elements and developing interventions and policies. Health Aff. 2013;32(2):223–31.

    Article  Google Scholar 

  43. Pomey MP, Ghadiri DP, Karazivan P, Fernandez N, Clavel N. Patients as partners: a qualitative study of patients' engagement in their health care. PLoS One. 2015;10(4):e0122499.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  44. Kilpatrick K, Jabbour M, Fortin C. Processes in healthcare teams that include nurse practitioners: what do patients and families perceive to be effective? J Clin Nurs. 2016;25(5–6):619–30.

    Article  PubMed  Google Scholar 

  45. Jeffcott SA, Mackenzie CF. Measuring team performance in healthcare: review of research and implications for patient safety. J Crit Care. 2008;23(2):188–96.

    Article  PubMed  Google Scholar 

  46. Valentine MA, Nembhard IM, Edmondson AC. Measuring teamwork in health care settings: a review of survey instruments. Med Care. 2015;53(4):e16–30.

    Article  PubMed  Google Scholar 

  47. Agency for Healthcare Research and Quality, U.S. Department of Health & Human Services. Team Strategies & Tools to Enhance Performance & Patient Safety. TeamSTEPPS. Agency for Healthcare Research and Quality. 2017. Accessed 17 Oct 2018.

  48. Brandt BF, Schmitz CC. The US National Center for interprofessional practice and education measurement and assessment collection. J Interprof Care. 2017;31(3):277–81.

    Article  PubMed  Google Scholar 

  49. Institute of Medicine of the National Academies. Measuring the impact of interprofessional education on collaborative practice and patient outcomes. 2015. Accessed 17 Oct 2018.

  50. Andreatta P, Marzano D. Healthcare management strategies: interdisciplinary team factors. Curr Opin Obstet Gynecol. 2012;24(6):445–52.

    Article  PubMed  Google Scholar 

  51. Epstein NE. Multidisciplinary in-hospital teams improve patient outcomes: A review. Surg Neurol Int. 2014;5(Suppl 7):S295–303.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Rutherford JS. Monitoring teamwork: a narrative review. Anaesthesia. 2017;72(Suppl 1):84–94.

    Article  PubMed  Google Scholar 

  53. Guler PH. Patient experience: a critical indicator of healthcare performance. Front Health Serv Manag. 2017;33(3):17–29.

    Article  Google Scholar 

  54. Beaird G, Dent JM, Keim-Malpass J, Muller AG, Nelson N, Brashers V. Perceptions of teamwork in the interprofessional bedside rounding process. J Healthc Qual. 2017;39(2):95–106.

    Article  PubMed  Google Scholar 

  55. Henry BW, McCarthy DM, Nannicelli AP, Seivert NP, Vozenilek JA. Patients' views of teamwork in the emergency department offer insights about team performance. Health Expect. 2013;19(3):702–15.

    Article  PubMed  PubMed Central  Google Scholar 

  56. Donald F, Kilpatrick K, Reid K, Carter N, Bryant-Lukosius D, Martin-Misener R, et al. Hospital to community transitional care by nurse practitioners: a systematic review of cost-effectiveness. Int J Nurs Stud. 2015;52(1):436–51.

    Article  PubMed  Google Scholar 

  57. Strasser DC, Burridge AB, Falconer JA, Herrin J, Uomoto J. Measuring team process for quality improvement. Top Stroke Rehabil. 2010;17(4):282-93.

  58. Frankel A, Haraden C, Federico F, Lenoci-Edwards J. A framework for safe, reliable, and effective care. Institute for Healthcare Improvement and Safe & Reliable Healthcare. 2017. Accessed 17 Oct 2018.

  59. Kilpatrick K, Lavoie-Tremblay M, Lamothe L, Ritchie JA, Doran D. Conceptual framework of acute care nurse practitioner role enactment, boundary work, and perceptions of team effectiveness. J Adv Nurs. 2013;69(1):205–17.

    Article  PubMed  Google Scholar 

  60. Kilpatrick K, Lavoie-Tremblay M, Ritchie JA, Doran D, Lamothe L. Boundary work and the introduction of acute care nurse practitioners in healthcare teams. J Adv Nursing. 2012;68(7):1504–15.

    Article  Google Scholar 

  61. Kilpatrick K. How do nurse practitioners in acute care affect perceptions of team effectiveness? J Clin Nurs. 2013;22:2632–47.

    Article  Google Scholar 

  62. Dillman DA, Smyth JD, Christian LM. Internet, mail, and mixed-mode surveys: The tailored design method. 4th ed. New Jersey, Wiley: Hoboken; 2014. Accessed 17 Oct 2018

  63. Peter E, Baiju RM, Varghese NO, Sivaraman R, Streiner DL. How to develop and validate a questionnaire for orthodontic research. Eur J Dent. 2017;11(3):411–6.

    Article  PubMed  PubMed Central  Google Scholar 

  64. Kilpatrick K. Understanding nurse practitioner communication and decision making activities in health care teams. J Clin Nurs. 2012;22:168–79.

    Article  PubMed  Google Scholar 

  65. Kilpatrick K, Lavoie-Tremblay M, Ritchie JA, Lamothe L, Doran D, Rochefort C. How are acute care nurse practitioners enacting their roles in healthcare teams?: a descriptive multiple-case study. Int J Nurs Stud. 2012;49(7):850–62.

    Article  PubMed  Google Scholar 

  66. Burns KE, Duffett M, Kho ME, Meade MO, Adhikari NK, Sinuff T, et al. A guide for the design and conduct of self-administered surveys of clinicians. CMAJ. 2008;179(3):245–25.

    Article  PubMed  PubMed Central  Google Scholar 

  67. Fisher J, Steggall MJ, Cox CL. Developing the a&E nurse practitioner role. Emerg Nurse. 2016;13:26–31.

    Article  Google Scholar 

  68. Rosenzveig A, Kuspinar A, Daskalopoulou SS, Mayo NE. Toward patient-centered care: a systematic review of how to ask questions that matter to patients. Medicine. 2014;93(22):e120–30.

    Article  PubMed  PubMed Central  Google Scholar 

  69. Guillemin F, Bombardier C, Beaton D. Cross-cultural adaptation of health-related quality of life measures: literature review and proposed guidelines. J Clin Epidemiol. 1993;46(12):1417–32.

    Article  CAS  Google Scholar 

  70. Vallerand RJ. Vers une médologie de validation trans-culturelle de questionnaires psychologiques: Implications pour la recherche en langue française. Can Psychol. 1989;30(4):662–80.

    Article  Google Scholar 

  71. Kilpatrick K, DiCenso A, Bryant-Lukosius D, Ritchie JA, Martin-Misener R, Carter N. Practice patterns and perceived impact of clinical nurse specialist roles in Canada: results of a national survey. Int J Nurs Stud. 2013;50(11):1524–36.

    Article  PubMed  Google Scholar 

  72. Babbie E. The practice of social research. 12th ed. Wadsworth: Belmont; 2010.

    Google Scholar 

  73. Moinester M, Gottfried R. Sample size estimation for correlations with pre-specified confidence interval. TQMP. 2014;10(2):124–30.

    Article  Google Scholar 

  74. Prajapati B, Dunne M, Armstrong R. Sample size estimation and statistical power analyses. Oct 2010, 2010(July).

  75. Knafl K, Leeman J, Havill N, Crandell J, Sandelowski M. Delimiting family in syntheses of research on childhood chronic conditions and family life. Fam Process. 2015;54(1):173–84.

    Article  PubMed  Google Scholar 

  76. Kilpatrick K, Harbman P, Carter N, Martin-Misener R, Bryant-Lukosius D, Donald F, et al. The acute care nurse practitioner role in Canada. Can J Nurs Leadersh. 2010;23(Special Issue):114–39.

    Article  Google Scholar 

  77. Valaitis R, Martin-Misener R, Wong ST, MacDonald M, Meagher-Stewart D, Austin P. Methods, strategies and technologies used to conduct a scoping literature review of collaboration between primary care and public health. Prim Health Care Res Dev. 2012;13(3):219–36.

    Article  PubMed  Google Scholar 

  78. Hogg W, Johnston S, Russell G, Dahrouge S, Gyorfi-Dyke E, Kristjanssonn E. Conducting waiting room surveys in practice-based primary care research: a user’s guide. Can Fam Physician. 2010;56:1375–6.

    PubMed  PubMed Central  Google Scholar 

  79. Kroth PJ, McPherson L, Leverence R, Pace W, Daniels E, Rhyne RL, et al. Combining web-based and mail surveys improves response rates: a PBRN study from PRIME net. Ann Fam Med. 2009;7(3):245–8.

    Article  PubMed  PubMed Central  Google Scholar 

  80. Desai AA, Bolus R, Nissenson A, Bolus S, Solomon MD, Khawar O, et al. Identifying best practices in dialysis care: results of cognitive interviews and a national survey of dialysis providers. Clin J Am Soc Nephrol. 2008;3(4):1066–76.

    Article  PubMed  PubMed Central  Google Scholar 

  81. Kelley K, Clark B, Brown V, Sitzia J. Good practice in the conduct and reporting of survey research. International journal for quality in health care. Int J Qual Health Care. 2003;15(3):261–6.

    Article  Google Scholar 

  82. Bryant-Lukosius D, Green E, Fitch M, Macartney G, Robb-Blenderman L, McFarlane S, et al. A survey of oncology advanced practice nurses in Ontario: profile and predictors of job satisfaction. Nurs Leadersh. 2007;20(2):50–68.

    Article  Google Scholar 

  83. IBM. SPSS Statistics version 25. New York. 2018. Accessed 17 Oct 2018.

  84. Kutner MH, Nachtsheim CJ, Neter J, Li W. Applied Linear Statistical Models. 5th ed. McGraw-Hill Irwin; 2005.

  85. Kellar SP, Kelvin EA. Munro's statistical methods for healthcare research. Philadelphia: Lippincott, Williams & Wilkins; 2013.

    Google Scholar 

  86. Norman GR, Sloan JA, Wyrwich KW. Interpretation of changes in health-related quality of life: the remarkable universality of half a standard deviation. Med Care. 2003;41(5):582–92.

    Article  PubMed  Google Scholar 

  87. Tabachnick BG, Fidell LS. Using multivariate statistics. 5th ed. Boston: Pearson, Allen and Bacon; 2007.

    Google Scholar 

  88. Wolak ME, Fairbairn DJ, Paulsen YR. Guidelines for estimating repeatability. Methods Ecol Evol. 2012;3:129–37.

    Article  Google Scholar 

  89. Verran JA, Gerber RM, Milton DA. Data aggregation: criteria for psychometric evaluation. Res Nurs Health. 1995;18(1):77–80.

    Article  CAS  Google Scholar 

  90. Davidson M. Known-groups validity. Encyclopedia of Quality of Life and Well-Being Research. 2014. 3481–82.

  91. Polit DF, Beck CT. Nursing research: generating and assessing evidence for nursing practice. 8th ed. Philadelphia: Lippincott, Williams, & Wilkins; 2008.

    Google Scholar 

  92. Costello AN, Osborne JW. Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. PARE. 2005;10(7):1–8.

    Google Scholar 

  93. Streiner DL, Norman GR, Cairney J. Health measurement scales: a practical guide to their development and use, 5th ed. Oxford University Press. 2015.

  94. Johnson JL, Greaves L, Repta R. Better science with sex and gender: facilitating the use of a sex and gender-based analysis in health research. Int J Equity Health. 2009;8:14.

    Article  PubMed  PubMed Central  Google Scholar 

  95. Hysong SJ, Knox MK, Haidet P. Examining clinical performance feedback in patient-aligned care teams. J Gen Intern Med. 2014;29(Suppl 2):S667–74.

    Article  PubMed  Google Scholar 

  96. Meyer RM, O'Brien-Pallas L, Doran D, Streiner D, Ferguson-Pare M, Duffield C. Boundary spanning by nurse managers: effects of managers' characteristics and scope of responsibility on teamwork. Nurs Leadersh. 2014;27(2):42–55.

    Article  Google Scholar 

  97. Bunnell CA, Gross AH, Weingart SN, Kalfin MJ, Partridge A, Lane S, et al. High performance teamwork training and systems redesign in outpatient oncology. BMJ Qual Saf. 2013;22(5):405–13.

    Article  PubMed  Google Scholar 

  98. Eggenberger T, Sherman RO, Keller K. Creating high-performance interprofessional teams. Am Nurse Today. 2014;9(11).

  99. Gullickson W, Harper MP. Possible measurement bias in aggregate productivity growth. Mon Labor Rev. 1999;122(2):47–67.

  100. Ginsburg L, Gilin OD. Patient safety climate strength: A concept that requires more attention. BMJ Qual Saf. 2016;25(9):680–7.

    Article  PubMed  Google Scholar 

  101. Agency for Healthcare Research and Quality. 2015 National Healthcare Quality and Disparities Report and 5th Anniversary Update on the National Quality Strategy. 2016. Accessed 17 Oct 2018.

  102. Canadian Institute for Health Information. Benchmarking Canada's Health System: International Comparisons. 2013. Accessed 17 Oct 2018.

Download references


The authors would like to thank the patients and families as well as the staff and managers at the CIUSSS-EMTL site for their support.


Partnership funding for the Study 1 was received from the Réseau de recherche en interventions en sciences infirmières du Québec (RRISIQ)/Quebec Network on Nursing Intervention Research, the Ministère de la santé et des services sociaux du Québec, and Fonds de recherche du Québec-Santé (FRQS), and Study 2 and Study 3 were funded by the Canadian Institutes of Health Research. Kelley Kilpatrick received a Junior 2 researcher award from the FRQS. The funding bodies played no role in study development, data collection, data analysis, interpretation or in the writing of the manuscript.

Availability of data and materials

The dataset analysed during the current study is available from the corresponding author on reasonable request.

Author information

Authors and Affiliations



KK, ET, LP, CG, MJ, VL, and NF made substantial contributions to study conception and design, acquisition of data, analysis and interpretation of data for Study 1. KK, ET, LP, MJ, and VL made substantial contributions to study conception and design, acquisition of data, analysis and interpretation of data for Study 2. FD made substantial contributions to study conception and design, acquisition of data, analysis and interpretation of data for Study 3. All authors were involved in drafting and making revisions for critical intellectual content in the manuscript. All authors gave final approval of the version to be published.

Corresponding author

Correspondence to Kelley Kilpatrick.

Ethics declarations

Ethics approval and consent to participate

Ethics approval was received from the Centre intégré universitaire de santé et des services sociaux de l’Est-de-l’Île-de-Montréal (Study 1: 2016–712, 15,065; Study 2: MP-12-2017-841, CIUSSS-EMTL-203, Study 3: 2017–337). Participants were provided with an information sheet and consent form, and advised that they were under no obligation to participate in the study and could withdraw at any time.

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kilpatrick, K., Tchouaket, É., Paquette, L. et al. Measuring patient and family perceptions of team processes and outcomes in healthcare teams: questionnaire development and psychometric evaluation. BMC Health Serv Res 19, 9 (2019).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: