The representation of patient experience and satisfaction in physician rating sites. A criteria-based analysis of English- and German-language sites

Background Information on patient experience and satisfaction with individual physicians could play an important role for performance measures, improved health care and health literacy. Physician rating sites (PRSs) bear the potential to be a widely available source for this kind of information. However, patient experience and satisfaction are complex constructs operationalized by multiple dimensions. The way in which PRSs allow users to express and rate patient experience and satisfaction could likely influence the image of doctors in society and the self-understanding of both doctors and patients. This study examines the extent to which PRSs currently represent the constructs of patient experience and satisfaction. Methods First, a systematic review of research instruments for measuring patient experience and satisfaction was conducted. The content of these instruments was analyzed qualitatively to create a comprehensive set of dimensions for patient experience and patient satisfaction. Second, PRSs were searched for systematically in English-language and German-language search engines of Google and Yahoo. Finally, we classified every structured question asked by the different PRS using the set of dimensions of patient experience and satisfaction. Results The qualitative content analysis of the measurement instruments produced 13 dimensions of patient experience and satisfaction. We identified a total of 21 PRSs. No PRSs represented all 13 dimensions of patient satisfaction and experience with its structured questions. The 3 most trafficked English-language PRS represent between 5 and 6 dimensions and the 3 most trafficked German language PRSs between 8 and 11 dimensions The dimensions for patient experience and satisfaction most frequently represented in PRSs included diversely operationalized ones such as professional competence and doctor-patient relationship/support. However, other less complex but nevertheless important dimensions such as communication skills and information/advice were rarely represented, especially in English-language PRSs. Conclusions Concerning the potential impact of PRSs on health systems, further research is needed to show which of the current operationalizations of patient experience and satisfaction presented in our study are establishing themselves in PRSs. Independently of this factual development, the question also arises whether and to what extent health policy can and should influence the operationalization of patient experience and satisfaction in PRSs. Here, the challenge would be to produce a set of dimensions capable of consensus from among the wide range of operationalizations found by this study.


Background
For questions about health and sickness in general, a large share of the information relevant to decision-making is publicly available through digital media (particularly the internet). Anyone can learn about the characteristics of certain illnesses (cause, symptoms, diagnostic criteria) and of the available medical measures (effectiveness, dosage, side effects) (e.g. [1,2]). This information is more practically relevant if it facilitates the primary goals and ethical principles of medicine [3,4]: (i) the welfare of patients, (ii) the respect for and promotion of patient autonomy and health literacy, and (iii) social justice.
Alongside information on illnesses and medical measures, it could also be relevant to the primary goals of medicine to acknowledge patients' satisfaction with physicians and provide information on this [5][6][7]. However, health services research has recently criticised the concept of patient satisfaction because of its inherent sources of bias [8]. Patients, for example, can describe high levels of satisfaction at the same time as describing experiences that are suboptimal. Thus in light of the limitations of patient satisfaction research, a recent trend in health services research has favored questions about patients' experiences [8]. Despite the great amount of active research and scientific publications in the field of patients' experience and satisfaction [7][8][9] there is only very little publicly available information on patients' experience and satisfaction concerning individual doctors or hospitals [10].
Physician rating sites (PRSs) are a new web-based source for peer-to-peer information on individual physicians. PRSs provide information about a physician's address, opening hours and certifications [10]. Next to this factual information, a major objective of PRSs is to collect and present information about patients' experience and satisfaction with individual physicians and their practices. PRSs could, therefore, improve informed provider choice [11,12] and are similar to other peer-topeer consumer information websites that allow the rating and discussion of the experience and satisfaction with hotels, restaurants or technological devices. In contrast to the rather expert-driven approaches to information on patients' experience and satisfaction, such as the CAHPS Clinician and Group Survey [7], PRSs are a consumer-driven alternative.
So far, very little research has been done on PRSs [10,13]. In contrast to this lack of research, there is controversial discussion about the impact of PRSs in several health systems. The British National Health Service (NHS) and the leading sickness fund within the German statutory health insurance have encouraged patients to review their physicians and hospitals [14,15]. American and German physician organizations, including the American Medical Association (AMA) and the German Medical Association (BÄK), have been rather opposed to the development of PRSs, arguing, for example, that the identity of patients cannot be confirmed, reviews will be excessively negative, and physicians' responses will be hampered by confidentiality issues [16]. To our knowledge, patient organizations were rather silent about their views on positive and negative consequences of PRSs.
Besides the technical and judicial questions surrounding PRSs, future discussion and decision making on PRSs also have to consider the content and rating options of PRSs. The practical relevance of information on patient experience and satisfaction -with respect to the achievement of the primary goals of medicinedepends significantly on how the complex construct of patient experience and satisfaction is represented in PRSs.
Our study examines, qualitatively and quantitatively, the extent to which PRSs currently represent patients' experiences and the construct of patient satisfaction as measured by research instruments.

Method
The framework of dimensions representing patients' experience and satisfaction An examination of the extent to which PRSs represent patients' experience and the construct of patient satisfaction requires a comprehensive catalogue of dimensions. Given the variety of instruments for measuring patients' experiences and satisfaction, which to some extent vary significantly in their exclusion and inclusion of individual dimensions, we decided to compile one single framework of dimensions on the basis of a systematic review of reviews of relevant measurement instruments. We conducted a systematic search in MEDLINE using the following search algorithm: ("Health Surveys"[Mesh] OR "Questionnaires"[Mesh]) AND ("Patient Satisfaction"[Mesh] OR "Patient Experience") AND "Review "[Publication Type]. We included all reviews that reviewed instruments for measuring patients' experience and satisfaction with resident or clinical doctors in general. We obtained the original versions of the potentially relevant measurement instruments mentioned in the reviews. We included all measurement instruments that (i) provided at least one dimension to measure patients' experience or satisfaction in general and (ii) reported information on reliability (re-test reliability, internal consistency) and validity. Measurement instruments on illness-or setting-specific patient satisfaction were excluded, for example [17,18]. All bibliographies of the reviews and measurement instruments we included were examined for further relevant measurement instruments. We extracted dimensions for patients' experience and satisfaction from the instruments we included through a qualitative content analysis [19]. After an initial reading, both authors individually and jointly organized all mentioned dimensions into conceptual relationships and higher order dimensions. After discussion both authors were able to reach a consensus about all discrepancies in reviewing and analyzing the instruments. The resulting set of 13 dimensions arranged into three subgroups produced the framework for the analysis of the PRSs.

Systematic search for physician rating sites
In November 2009 we did a systematic search for PRSs in the English-language search engines of Google and Yahoo with the following search terms: "doctor rating sites", "physician rating sites", "rate a physician", "rate a doctor", "search a doctor", "find a doctor". In the German-language search engines of Google and Yahoo we used the terms: "Arztbewertung", "Ärzte finden", "Ärzte bewerten", "Arztsuche". We checked the first 100 hits of every search. All websites were labeled PRSs and included that allowed the specific qualitative (narrative) or quantitative expression of patients' experiences or satisfaction with physicians in a structured format. We restricted our search to the first 100 hits of every search because it is to be assumed that the PRSs identified in this way are the sites that a potential user would run into in his or her search, and thus represent the most frequently used sites. Website traffic data was estimated using Alexa (see table 1) http://www.alexa.com/siteinfo.

Structured content analysis of physician rating sites
We used the 13 framework dimensions to classify each rating option or structured question provided by the different PRSs. Multiple classifications for one rating option were possible.

Results
The MEDLINE search yielded 401 references from which 55 were reviews of instruments for assessing patients' experience or satisfaction. From these we identified 20 English-language or German-language instruments (see table 2) with sufficient reliability and validity (range of test reliability: 0.62 [20] -0.94 [21]); range of test validity: 0.54 [22] -0.98 [23]) [24]. The qualitative content analysis of the 20 measurement instruments produced 13 dimensions of patients' experience and satisfaction (see tables 3, 4, and 5).
A total of 12 English-language and 9 Germanlanguage PRSs were identified (see tables 3, 4, and 5) [last checked in April 2010]. No PRS represented all 13 dimensions of patients' experience and satisfaction with its structured questions. The number of dimensions for patients' experience and satisfaction represented in PRSs range from 3 (checkMD.com, doctortree.org) to 11 (Jameda.de, Docinsider.de). For data about the traffic of PRSs see table 1. While the 3 most trafficked Englishlanguage sites (vitals.com, RateMD.com, vimo.com) only represent between 5 and 6 dimensions (see tables 3, 4, and 5) the 3 most trafficked German-language sites (Imedo.de, Jameda.de, Docinsider.de) represent between 8 and 11 dimensions.
Among the dimensions most frequently represented in PRSs are the less complex dimensions such as office organization/waiting time (in 21 PRSs) and office staff (in 17 of 21 PRSs). On the other hand quite complex and thus diversely operationalized dimensions of patient experience and satisfaction such as professional competence (in 18 of 21 PRSs) and doctor-patient relationship (in 16 of 21 PRSs) are also often found on these sites. However, other dimensions that allow users to rate important aspects of the encounter between doctor and patient were rarely represented. For example, only 4 of 12 English and 1 of 9 German PRSs provided structured questions about physicians' communication skills. Also only 3 of 12 English PRSs but 8 of 9 German PRSs provided structured questions concerning how well physicians offer information and advice.
The following sections show how the dimensions of patients' experience and satisfaction are represented both by the research instruments and in the PRSs. We will begin by qualitatively describing the operationalization of each dimension by the research instruments. This operationalization formed the basis for the classification of rating options in the PRSs that we will present subsequently. To facilitate reading we separated the 13 dimensions of patients' experience and satisfaction into 3 subgroups (see tables 3, 4, and 5) that aim to group together (i) characteristics of the encounter between doctor and patient (e.g. doctor-patient relationship, communication) in table 3 (ii) organizational aspects of the medical practice (e.g. equipment, personnel, organization) in table 4 and (iii) overarching assessment categories (e.g. general satisfaction) in table 5. Numbers in parentheses indicate the corresponding PRSs specified in tables 3, 4 and 5.
Characteristics of the encounter between doctor and patient Doctor-patient relationship Patients' experience and satisfaction research instruments include the personal skills of the doctor such as patience, taking the patient seriously, being friendly, caring, trustworthy, diligent, empathetic and humane. Additional characteristics, such as seeing the patient as an equal partner or as a person and not as a number, are often classified under patient-centeredness. Additionally the quality of the doctor-patient relationship has been operationalized by means of doctor behavior such as showing interest in the patient, maintaining a respectful manner, and allowing the patient to ask necessary questions as well as supporting the patient through his or her mental, social and bureaucratic difficulties.
PRS: 16 PRSs (76%) allow for an explicit assessment of certain aspects of the doctor-patient relation as described above (1-10, 12, [14][15][16][17]19). Three Englishlanguage PRSs use the term "bedside manner" as a catchall term for a particular behavior on the part of the doctor, but explain it differently -one more descriptively as the "attitude and conduct of a physician in the presence of a patient" (9) and another in clear normative terms as "do they (physicians) make you feel comfortable" (7). Other criteria that we correlated with the dimension of doctor-patient relation were helpfulness (2, 10, 12), approachability and friendliness (2-5, [14][15][16]19), spending enough time with the patient (1, 17), shows caring or concern (8, 12), and shows compassion (8).

Communication skills
Patients' experience and satisfaction research instruments include the ability to listen, to communicate  Even if there is some concrete overlap between the dimensions of communication and doctor-patient relation, we find it possible and productive to maintain a distinction. The Association of American Medical Colleges (AAMC) also emphasizes this distinction between communication skills and interpersonal skills [25], whereby the communicative skills include above all effective listening, appropriate questioning and the provision of information. Interpersonal qualities are seen more in the establishing of a trusting relation [26]. PRS: Five sites (24%) asked explicitly about this dimension as the "ability to communicate" (8), "understanding" (12), or "listening to you" (6). The criteria of "takes time to answer questions", "feel rushed while talking" (3) and the question of sufficient conversational time (14) also represent the doctor's communicative skills.

Trust
Patients' experience and satisfaction research instruments include questions about whether the patient feels trust in the particular doctor, whether an atmosphere of privacy obtains or whether the patient can open up to entrust the doctor with all of the necessary information.
PRS: Six sites (29%) ask about this dimension. The explanations hardly differ at all; they usually concern the preservation of the sense of privacy (13,17,19) and then ask quite unspecifically about whether the patient trusts the particular doctor, for example with the question of whether one feels oneself in good hands (9,14) or "Do you trust this care professional?" (15).

Professional competence
Patients' experience and satisfaction research instruments include the doctors' knowledge of his or her own (professional) limitations (e.g. prompt referral, collaboration with other doctors, admitting his or her own errors) and the aspect of diligence (e.g. investigating all possible causes, correct diagnosis, thorough examination, responsibility). This dimension also includes knowledge of all aspects of the medical therapy and the appropriateness of the treatment (e.g. medications without or with only minor side effects, no excess treatment, individualized medical treatment, no doubled examinations, awareness of price).
PRS: 18 sites (86%) ask for an assessment of this dimension (1-10, 12-17, 19, 20). The questions range from unspecified aspects such as "knowledgeable" or "Knowledge and Professionalism" (2, 5-7, 9, 10, 12, 17, 19) to more comprehensive descriptions of professional competence such as "Was the doctor's diagnosis later confirmed? Did he follow up with the appropriate treatment?" (14) or "Does your doctor have up-to-date medical skills? Can you get a recommendation to a specialist easily?" (3).  Table 3 Characteristics of encounter between doctor and patient

Information and advice
Patients' experience and satisfaction research instruments include the content and scope of what is communicated. Hence this dimension particularly concerns how well and how transparently the patient is informed during the medical treatment. This includes the information given (e.g. understandable information about the causes and process of the illness and the side effects of diagnosis and treatment) and advice (e.g. information on self-help groups or nutritional advice).
PRS: Eleven sites (52%) covered this dimension in their assessment (4,5,8,(13)(14)(15)(16)(17)(19)(20)(21). Two of the sites only named the dimension as "informative" (5) and "advice" (21), whereas the other assessment templates provided the user with more thorough explanations, for example "Information on preventative examinations, preventative options, diagnosis, therapy, written information, education" (13) or "All of my questions were answered by the doctor and his or her team to my complete satisfaction. I felt that I was advised very well." (20). Organizational aspects of the medical practice

Medical and technical facilities
Patients' experience and satisfaction research instruments include the technical facilities of the physician's practice (or the hospital). PRS: Four of the sites (19%) include this dimension in the assessment (4,11,13,16). This dimension is represented by questions such as "How did you find the technical equipment of the medical practice (technically up-to-date)?" (16) or "Ability to get all of the care for your health problem or illness at this clinic" (4).

Accessibility/availability
Patients' experience and satisfaction research instruments include the doctor's availability by telephone Table 5 Overarching assessment categories outside of visiting hours, the accessibility of the office or clinic (e.g. wheel-chair accessible, directions by public transportation are available, parking nearby, etc.) as well as the possibility of house visits. An acceptable waiting period for an appointment and an arrangement of appointments suitable to the patient are also included. PRS: More than half of the sites (12; 57%) asked about aspects of accessibility/availability (1, 3, 4, 8, 11, 13-17, 19, 20). Often only one representative criterion was named such as "availability" or "arrangement of appointments" (1, 8,16,20). More concrete questions concerned the possibility of house visits, the waiting time for an appointment, the possibility of reaching the doctor by telephone outside of visiting hours or per email, the allocation of appointments, and available parking.

Office characteristics
Patients' experience and satisfaction research instruments include the atmosphere in the office or clinic (team spirit, attractive waiting room, cleanliness), separate changing rooms, or play-areas for children.

Office organization
Patients' experience and satisfaction research instruments include time management (brief waiting periods within the office, good office organization), customerorientation (emphasis on the patient, no interrupting phone calls, equal service for all patients, shorter waiting times for emergencies) and service (priority for mothers with children, referral given on the same day, copies of all test results).
PRS: This dimension is included in all of the rating sites. There are hardly any differences in the weight given individual aspects of office organization: waiting times and promptness are the primary concerns. One site goes into this point in greater detail: "Getting your test results back in a timely manner./After arriving at the office, how many minutes did you wait before seeing the doctor?/About how many minutes did the doctor spend with you in your most recent visit?" (4). On English-language websites this category also sometimes includes an estimate of the costs of the treatment (8)(9)(10)(11).

Office staff
Patients' experience and satisfaction research instruments include the competence and friendliness of the staff as well as their interaction within the team.
PRS: 17 of the sites (81%) cover this aspect. Several sites ask about this without any further explanation, particularly concerning the friendliness, helpfulness and professionalness of the staff (1- 6,8,[11][12][13][14][15][17][18][19][20]. Four sites provide examples of how the friendliness of the staff could be demonstrated, e.g.: "How friendly and well-organized were the office personnel in the office and in making appointments" (13), "Base your opinion on your experience making an appointment, how you are greeted for an office visit, and how well the office staff meets your needs as a patient" (9).
Overarching assessment categories Success of outcome Patients' experience and satisfaction research instruments include assessments of effectiveness (e.g. help by the doctor, effective medications), the suitability of the therapy relative to the diagnosis, the disappearance of symptoms, the amelioration of the complaint, and the increase in functional ability and quality of life.
PRS: Nine of the sites (43%) ask explicitly about the success of the treatment, for example with questions such as: "Were you better after the treatment?" (14), "How would you subjectively assess the success of the treatment?" (18), or "How did his treatments work for you?" (2). All 9 PRSs (2, 4,10,12,14,15,17,18,20) assess this dimension by asking about the subjective treatment success; there are no detailed descriptions of the reduction of specific symptoms, ability to go back to work, increase in quality of life, etc.

General satisfaction
Patients' experience and satisfaction research instruments include the aggregate assessment as a general summary of all previous assessments.
PRS: 13 of the sites asked about general satisfaction (62%), for example using terms such as "overall satisfaction" (1, 6,8,9,14,15) or by asking for a total rating/ assessment (4,10,11,16,18,19,21). No site said anything in greater detail about this total assessment. The visual presentation of the assessment of a doctor (ranging from check-marks to stars, plus points, and other symbols) or the option of writing free commentary might implicitly represent general satisfaction. On this broader definition the dimension general satisfaction would be represented on all sites.

Willingness to recommend the doctor
Patients' experience and satisfaction research instruments include patients' willingness to recommend a doctor and generally describe this dimension as: would the patient be able to recommend this doctor (or this clinic) to friends and relatives?
PRS: Only six sites (29%) explicitly posed this question to their evaluating users (8,9,15,16,19,20). This dimension is similar to the representation of general satisfaction: one would expect a high level of satisfaction to correspond with a high willingness to recommend the doctor, especially as this is precisely the point of the sites: to allow people to share their experiences with those looking for a suitable doctor and to offer their own experiences to others who might find it useful and helpful.

Possibility of free commentary
86% of the sites (18) allow for users to write free commentary (except 5,13,17). One site only offers assessment by commentary without indicating or asking about any other dimensions, and was thus excluded from the criteria-based analysis. "The operators of this site assume that "rating" doctors doesn't give you the kind of information you need. [...] You can't make a "best fit" healthcare choice when all you have is a number between 0 and four" ("Our Philosophy" [27]).

Discussion
The 21 PRSs examined here show a clearly heterogeneous representation of the different dimensions of patient experience and satisfaction as viewed from a quantitative and qualitative perspective. Our quantitative findings show that the most trafficked English-language PRSs currently only represent between 5 and 6 of the 13 dimensions that can play a role in assessing patient experience and satisfaction. However, the most trafficked German-language PRSs represent between 8 and 11 of the 13 dimensions.
Findings from the qualitative analysis demonstrate that the most frequently represented dimensions in PRSs include diversely operationalized ones such as professional competence and doctor-patient relationship. To assess professional competence, for example, PRSs provide heterogeneous questions such as: "Does your doctor have up-to-date medical skills", "Was the doctor's diagnosis later confirmed? Did he follow up with the appropriate treatment?", "Thoroughness of examination" and others..
The PRSs identified using our search dimensions are the sites that a potential user would run into in his or her search. Nevertheless, we cannot exclude the possibility of other PRSs not discovered by our search algorithm. However, we do not aim at completeness; instead we interpret the sites examined here as a faithful representation of existing sites and assume that other sites not examined will deviate minimally.
We only examined the structured questions asked by the PRSs and not the actual patient reviews. Therefore we did not include the content of open and narrative feedback sections which may represent additional specifications of the 13 dimensions for patients' experience and satisfaction.
At the current state of research, we cannot give evidence-based recommendations about which and how many dimensions of patients' experience and satisfaction need to be represented in PRSs to best improve the primary goals of medicine [3,4]. There may be plenty of arguments against the representation of all dimensions in PRSs. However, PRSs should not be confused with other well-investigated expert initiatives that measure physician performance and quality of care [6,7,28]. For example, a rather broad representation of dimensions of patients' experience and satisfaction could have a positive impact on the interactive functions of PRSs and thus on the three levels of patients' health literacy [29,30]: Aside from the direct recommendations of individual physicians PRSs bear the potential to improve users' critical reflection about what aspects could and should be considered prior to choosing a doctor. It seems possible, therefore, that PRSs increase the users' health literacy at the interactive level, since it is precisely the exchange of information that is considered the essential point of these sites. Concerning the critical level of health literacy, one could also expect (in the best case) a similarly positive development. In particular the possibility of evaluating a specific dimension as more or less helpful and the possibility of free narrative commentary could train users to individually filter out what is important -possibly leading the patients to pay more attention to the doctors' competence in shared decision-making and less attention to practices' atmosphere. However, the major precondition for PRSs to achieve these objectives would be to provide possibilities for open narrative commentaries next to structured rating options and to facilitate peer-to-peer communication. Needless to say, PRSs also bear the potential to negatively influence the three levels of health literacy.
Our findings provide the starting point for dealing with another potential impact of PRSs. It also seems plausible to assume that in the future complex constructs such as 'quality of care', 'physician performance', or 'patient satisfaction/experience' will still not be operationalized through consensus on a single gold standard, nor that such a gold standard will be used in all PRSs. Thus the consumer-driven, peer-to-peer operationalization of dimensions such as professional competence and doctor-patient relationship in PRSs could significantly shape the social image of the doctor and the self-understanding on the part of both doctors and patients. An important question in the research accompanying PRSs, therefore, would be: which of the current operationalizations of patients' experiences and satisfaction in general and of professional competence and doctor-patient relation in particular are establishing themselves in PRSs?
Independently of the factual development examined here, the question also arises whether and to what extent health policy can and should influence the operationalization and assessment of patients' experience and satisfaction in consumer-driven PRSs (true Web 2.0 application). One could imagine certifications from professional societies and public institutions entailing not just formal and legal standards but also specifications for a suitable representation and operationalization of patients' experience and satisfaction. Here the concrete difficulty would be to work out a set of dimensions capable of consensus from among the wide range of operationalizations shown by this study. From an ethical viewpoint such certifications would be more legitimate if they arose through a transparent process allowing the participation by various stakeholders (including consumer representatives). This process would have to include discussions of which characteristics of doctors and medical practices are crucial in order to appropriately inform consumers in accordance with the primary goals of medicine (see above).

Conclusion
The practical relevance of all issues we discussed in the previous paragraphs only increases with a more widespread use of PRSs. At present, the use of PRSs is rather limited. Recent work on the state of PRSs found that reviews were scarce, and when present, most were positive concerning the overall satisfaction with the specific physician [10]. For 300 physicians, only 66 written patient narratives across 33 sites could be identified [10]. However, there are many reasons that argue in favour of a more widespread use of PRSs in the near future. For example, the so-called Facebook or MySpace generation has been especially socialized with the internet. When this generation reaches the age in which they are increasingly interested in health questions and thus doctors, it is likely that the internet will also play a significant role in their decisions (see among other things the rapidly growing number of participants in patient communities [31]).
In the last decade several national health systems have adopted public reporting instruments allowing consumers to make explicit comparisons between the performances of health care providers or health plans in order to make an informed choice [32]. While the aims of public reporting approaches -increasing public accountability, supporting consumer choice and finally improving quality of care -are comparable with the aims of PRSs, they currently lack patient experience and satisfaction information with respect to individual physicians. PRSs might be interesting esspecially for countries that do not follow gatekeeper models and therefore have free physician choice (e.g. Germany).
In light of the mentioned trend towards an increasing practical relevance of PRSs, these new websites bear the potential to influence the public health and health literacy. Future ethical and policy analyses should take these various influences into account and explicitly weigh them before drawing any conclusions on the form and content of PRSs [33].