Skip to content

Advertisement

BMC Health Services Research

What do you think about BMC? Take part in

Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

Perceptions and utilization of generic medicines in Guatemala: a mixed-methods study with physicians and pharmacy staff

Contributed equally
BMC Health Services ResearchBMC series – open, inclusive and trusted201717:27

https://doi.org/10.1186/s12913-017-1991-z

Received: 22 September 2015

Accepted: 6 January 2017

Published: 13 January 2017

Abstract

Background

Access to low-cost essential generic medicines is a critical health policy goal in low-and-middle income countries (LMICs). Guatemala is an LMIC where there is both limited availability and affordability of these medications. However, attitudes of physicians and pharmacy staff regarding low-cost generics, especially generics for non-communicable diseases (NCDs), have not been fully explored in Guatemala.

Methods

Semi-structured interviews with 30 pharmacy staff and 12 physicians in several highland towns in Guatemala were conducted. Interview questions related to perceptions of low-cost generic medicines, prescription and dispensing practices of generics in the treatment of two NCDs, diabetes and hypertension, and opinions about the roles of pharmacy staff and physicians in selecting medicines for patients. Pharmacy staff were recruited from a random sample of pharmacies and physicians were recruited from a convenience sample. Interview data were analyzed using a thematic approach for qualitative data as well as basic quantitative statistics.

Results

Pharmacy staff and physicians expressed doubt as to the safety and efficacy of low-cost generic medicines in Guatemala. The low cost of generic medicines was often perceived as proof of their inferior quality. In the case of diabetes and hypertension, the decision to utilize a generic medicine was based on multiple factors including the patient’s financial situation, consumer preference, and, to a large extent, physician recommendations.

Conclusions

Interventions to improve generic medication utilization in Guatemala must address the negative perceptions of physicians and pharmacy staff toward low-cost generics. Strengthening state capacity and transparency in the regulation and monitoring of the drug supply is a key goal of access-to-medicines advocacy in Guatemala.

Keywords

Generic medicinesMedicine perceptionsGuatemalaPharmacist rolesAccess to medicinesEssential medicinesNon-communicable diseases

Background

Access to essential medicines has been on the public health agenda since the 1970s when the World Health Organization (WHO) published the first list of essential drugs and the Alma Ata Declaration proclaimed access to medicines as a pillar of primary health care [1]. Policies to promote drug access initially tended to focus on medicines for acute conditions and infectious diseases. However, in recent years, the rapidly rising burden of chronic non-communicable diseases (NCDs) in low-and-middle income countries (LMICs) has led to calls for improved access to medications for NCDs as well [2, 3]. Surveys conducted in diverse LMICs have shown that essential medicines for NCDs tend to be unaffordable or unavailable in both public and private sectors [47].

Access to low-cost essential generic medicines, and, to an increasing degree, generics for NCDs, is thus a critical health policy goal in LMICs. However, barriers to generic access are complex. Demand-side constraints may include health seekers’ negative perceptions of generics or uncertainties regarding medicine quality [8]. On the supply side, numerous factors influence health providers who prescribe and dispense low-cost essential medicines, including efficacy and safety concerns, lack of trust between physicians and pharmacies, misaligned incentives between patient and provider, and pharmaceutical promotional activity [9].

The pharmaceutical industry and generic medicines in Guatemala

Industry analysts estimate total pharmaceutical sales in Central America in 2014 at US$ 3.7 billion [10]. Guatemala, the most populous country in Central America, is the region’s biggest pharmaceutical market [10, 11]. Although approximately 70% of the US$ 800 million Guatemalan drug market by value is comprised of imported medicines [10], with Mexico serving as the leading source of imports by value as of 2010 [12], there are also 70 domestic and two multinational companies authorized to manufacture medicines within the country [13]. Official documents report that 64% of all drug purchases are made in the private sector, and the vast majority of private health spending is out-of-pocket [14, 15]. Prior research on access to medicines in Guatemala, primarily focused on acute conditions, has suggested that they tend to be unavailable or unaffordable [1619]. In general, accurate, up-to-date, and publically available information on the pharmaceutical and pharmacy industries in Guatemala is limited.

WHO defines a generic medicine as “a pharmaceutical product, usually intended to be interchangeable with an innovator product, that is manufactured without a license from the innovator company… [and] marketed under a non-proprietary or approved name rather than a proprietary or brand name [20].” In contrast, “originator” or “innovator” medicines are usually defined as the first drugs to be authorized or patented [21]. Generics can be branded or unbranded; a “branded generic” refers to a generic drug sold under a trade name that is different from the originator name. In addition, a few pharmaceutical chains do exist in Guatemala which market “similares” (non-bioequivalent copy drugs). These include branches of the Mexican chain Farmacias Similares, which has transformed the private-sector pharmaceutical market in Mexico [22]. However, in Guatemala, the market share for “similares” has remained low compared to the generics market.

In Guatemala, multiple factors impact access to and perceptions of generics. First, Guatemalan intellectual property law offers relatively expansive protection for originator pharmaceuticals relative to peer countries, which delays the introduction of new medicines into the generic market [2325]. For example, in the case of the long-acting insulin Lantus® (insulin glargine), a generic version would still be restricted in Guatemala even after the drug lost its patent in the United States [26]. Second, as in many other LMICs [27], most generics sold in Guatemala are branded generics [28]. Branded generics in Guatemala are aggressively marketed to capture consumer loyalty, thereby allowing a premium to be charged as compared to unbranded generics [25].

An important factor impacting the perceptions of low-cost generics in Guatemala is concern regarding generic drug quality. Although legal frameworks mandate drug and safety monitoring [13], domestic generic manufacturers operate under laws that are outdated with respect to best manufacturing practices [13, 29]. In addition, the limited data available suggest that there are quality issues in the Guatemalan drug market. For example, in the international Medicines Quality Database, whose results have been published in aggregate form [30], eight of the 37 medicine samples submitted from Guatemala failed to pass a quality test [31]. A series of dissolution studies carried out by students and faculty at San Carlos University in Guatemala showed that, in multiple cases, generic versions of drugs were not bioequivalent to originator products [29]. A 2012 Ministry of Health report found that 5.2% of 960 pharmaceutical quality tests did not comply with quality standards [13].

Robust and competitive generic pharmaceutical markets also require that consumers trust the public institutions tasked with monitoring and protecting the drug supply. In Guatemala, however, there is deep mistrust of the political system [32], and there have been numerous scandals in recent years involving collusion between government officials, business elites, and the pharmaceutical and pharmacy industries. For example, in 2014 alone, four large pharmacy chains were fined for advertising false discounts on medicines [33], 12 international pharmaceutical companies operating in Guatemala were found guilty of colluding to raise the prices of some 500 products [34], and the Guatemalan Social Security Institute (IGSS) entered into a fraudulent pharmaceutical procurement contract that ultimately led to the arrest of high-profile government figures including the director of IGSS and the president of the Central Bank [35]. Taken together, such high-profile scandals reinforce public skepticism of the integrity and safety of the drug supply in Guatemala [25].

Finally, a multitude of ground-level pressures and incentives tend to favor originator medicines over generics. First, although physicians in Guatemala are legally required to include generic names in prescriptions, such laws are not enforced in practice. In some cases, physicians may steer patients to their own privately-owned pharmacies, creating a financial conflict of interest towards preferentially prescribing and selling higher-cost products like originator medicines [36]. In addition, retail pharmacies commonly advertise discounted or free in-house clinical consultations as a business strategy to increase sales. Finally, while direct-to-consumer advertising in Guatemala is not legally permitted for any prescription medicine [13], the scope of what is considered a prescription drug is very narrow, and, therefore, the vast majority of drugs—including antibiotics—can be purchased over the counter [37]. In practice, consumers are frequently targeted in advertisements for pharmaceutical products [38]. A newspaper advertisement (Fig. 1) published by Fedefarma, a trade group representing more than a dozen well-known international pharmaceutical companies in Central America, illustrates this dynamic well. Fedefarma promotes originator brands to prospective consumers [39] and clinicians [40] by emphasizing themes of quality, safety, and bioequivalence.
Fig. 1

Advertisement in Guatemala promoting originator medicines over generic medicines. In English the text reads: I have diabetes. If my medicine fails, I could suffer a diabetic coma. I don’t play around; I only use the real thing. Trusted brands [“originator medicines”] have studies that back up their quality, efficacy, and safety. Respect the medical prescription. Ask your doctor or your pharmacist for originator medicines. Image used with permission of Health Action International

Against this background, it remains unclear how the attitudes toward generics of pharmaceutical gatekeepers—prescribing physicians and retail pharmacy staff—impact the access and use of these medicines. One recent study utilized an online survey to demonstrate that many physicians perceived generics to be of lower quality than originator brands, while, at the same time, their prescribing practices were influenced by the affordability of such medicines for patients [41]. However, this study did not examine how other factors such as how interaction with pharmaceutical representatives may also impact perceptions. Additionally, no studies have examined the perceptions of pharmacy staff towards generic medications or how pharmaceutical gatekeepers view generic NCD drugs.

These are important gaps in knowledge since medicines in Guatemala are largely purchased as an out-of-pocket household expense at pharmacies [14, 15]. Even in the public sector, medicines are often out of stock, and, thus, public providers may write prescriptions for patients to purchase medicines privately. Many health seekers eschew clinical providers altogether and directly seek care at pharmacies [37, 4244]. In this setting, pharmacy staff may theoretically steer patients away from generics or “upsell” originator products.

To examine these issues, we undertook a mixed-methods study (1) to explore physician and retail pharmacy staff’s perceptions regarding generic and medicines and (2) to investigate how these perceptions influence on-the-ground prescribing and dispensing decisions of generic medicines including drugs for NCDs.

Methods

Institutional context and ethics

This study was facilitated by Wuqu’ Kawoq | Maya Health Alliance, a non-governmental organization (NGO) working to provide health services, including services for NCDs, in rural indigenous Maya communities in Guatemala. This study was reviewed and approved by the Institutional Review Boards of Wuqu’ Kawoq | Maya Health Alliance and Partners Healthcare, Boston. All participants provided verbal informed consent.

Overview of methods

The study consisted of semi-structured interviews conducted with 42 physicians and pharmacy staff in three locations in central Guatemala in the province of Chimaltenango: San Juan Comalapa (population 42,200), Tecpán (population 81,100), and the provincial capital Chimaltenango (114,400). These towns were selected as study sites because they represented large concentrations of patients with NCDs for the sponsoring NGO.

Sample recruitment

Thirty retail pharmacy staff were recruited to the study through random sampling of pharmacies in San Juan Comalapa, Tecpán, and Chimaltenango. “Pharmacy staff” was defined broadly to include any pharmacy worker who dispensed medications to clients, regardless of level of training or professional certification. People in San Juan Comalapa and Tecpán commonly purchase medicines both locally and in the city of Chimaltenango, due to its central location along the major highway leading to both towns. In each of these three towns, the research team created a comprehensive map of all pharmacies and classified each pharmacy in one of five categories: (1) national pharmacy franchise, (2) discount local pharmacy, (3) non-discount local pharmacy, (4) hybrid clinic-pharmacy; or (5) government-subsidized pharmacy (through the national Program for Medication Accessibility [PROAM]). In each town, three pharmacies were randomly selected within each category. In San Juan Comalapa and Tecpán, fewer than three pharmacies were present for some categories; in these cases, all pharmacies within a town’s category were selected. In total, across the three locations, interviewees were recruited from 30 distinct pharmacies whose classifications are reported in Table 1A.
Table 1

Characteristics of pharmacies and physicians surveyed

A. Pharmacy characteristics (n = 30)

Number (percent)

 Town of pharmacy surveyed

  San Juan Comalapa

8 (27%)

  Tecpán

8 (27%)

  Chimaltenango

14 (47%)

 Category of pharmacy (n = 30)

  National pharmacy franchise

6 (20%)

  Discount local pharmacy

7 (23%)

  Non-discount local pharmacy

8 (27%)

  Hybrid clinic-pharmacy

5 (17%)

  Government-subsidized pharmacy

4 (13%)

B. Physician characteristics (n = 12)

 Town of physician clinic surveyed

  San Juan Comalapa

5 (42%)

  Tecpán

5 (42%)

  Chimaltenango

2 (17%)

 Category of practice environment

  Public

3 (25%)

  Private

6 (50%)

  Private and public

2 (17%)

  NGO

1 (8%)

Twelve physicians were recruited to the study from San Juan Comalapa, Tecpán, and Chimaltenango. Convenience sampling was used. The research team identified and contacted physicians in all three locations through referrals from health professionals working in San Juan Comalapa and Tecpán. In Chimaltenango, the research team performed site visits to clinics where the sponsoring NGO’s patients commonly reported accessing care. Across the three locations, physician participants were recruited from 12 distinct practice environments as reported in Table 1B.

Data collection

Semi-structured interviews lasting 20–45 min were conducted with pharmacy staff and physicians in their place of work. All interviews were conducted in Spanish. Interviews with pharmacy staff focused on training, perceptions of generic medicines, prescription practices regarding medicines for diabetes and hypertension, advice given to clients with diabetes and hypertension, and opinions about the roles of pharmacy staff and physicians in care delivery. Interviews with physicians focused on perceptions of generic medicines including generics for diabetes and hypertension, prescribing practices, and opinions about the roles of pharmacy staff and physicians in care delivery. We opted not to define “originator medicines,” “branded generics,” or “unbranded generics” to research participants given that such definitions can be technically complex and that these terms are not used regularly in clinics and pharmacies in Guatemala. As we were most interested in the perceptions and utilization of low-cost generics, which in Guatemala generally refer to unbranded generics, we used the term “generic” in research questions in contrast to “commercial brands,” as these are the terms most commonly used and understood by participants in this setting. Detailed notes were taken during and after each interview, and these serve as the basis for analysis.

Data analysis

Qualitative data from interviews was analyzed using an inductive approach, allowing dominant themes to emerge from interview notes. Lists of responses to each open-ended question were compiled in matrices organized by participant type. Responses were then coded into thematic categories by one researcher [IM]. Other team members collectively reviewed and arrived at consensus about the application of codes. Finally, as it was hypothesized that pharmacy staff and physicians’ perspectives might differ across interview topics, coded responses in the matrices were compared across participant type. Quantitative data derived from interviews were first entered into a spreadsheet and then imported into Stata (StataCorp; College Station, TX) for analysis. In addition to descriptive statistics, Fisher’s exact test was used to compare categorical responses between physicians and pharmacy staff.

Results

Demographics and education

Thirty pharmacy staff and 12 physicians were interviewed. Demographic characteristics of participants are summarized here and in Table 2. Of the 12 physicians sampled, eight were general practitioners who had completed medical school but no further training, three had completed post-graduate residency programs, and one had received a master’s degree in epidemiology. Ten participants attended medical school in Guatemala and two in Cuba.
Table 2

Demographic characteristics of participants

 

Pharmacy staff

Physicians

Participants (n)

30

12

Formal Education level, n (%)

Elementary school: 2 (7)

Middle school: 9 (30)

High school: 17 (57)

Post-high school: 2 (7)

Medical school: 8 (67)

Residency: 3 (25)

Master’s degree: 1 (8)

Pharmacy Specific Training

 No training, n (%)

15 (50%)

N/A

 Mean duration of training, months ± SD

5.4 ± 8.5

N/A

Pharmacy certification, n (%)

7 (23%)

N/A

Mean Work experience, years ± SD

7.3 ± 5.7

13.5 ± 10.4

Among pharmacy staff, the level of formal education prior to entering the pharmacy business varied widely but was typically low, with only two of 30 respondents having received any formal post-high school education. When pharmacy staff were queried as to why they had entered the retail pharmacy field, they expressed a diverse range of motivations, including professional interest in the field, desire to help people, previous work in health care (such as nursing or social work), or economic necessity. Participants often cited social connections as drawing them into the profession, such as having a neighbor, family member, or spouse who worked in a pharmacy practice.

Level of pharmacy-specific training or certification also varied widely. Half of participants had received no pharmacy-specific training of any kind. Among those who did report receiving such training, the quality and nature of the training was heterogeneous. Some reported training that consisted of “reading pharmacy books” while others had engaged in on-the-job apprenticeships within pharmacies or clinics. Some participants referred to prior experience with medications, gained from working in a different health care environment, such as a nursing home or clinic. Among those who self-reported any kind of pharmacy-specific training, the duration of that training was 5.4 ± 8.5 months. Only seven (23%) self-reported earning a license to practice in a pharmacy setting.

General perceptions of generic cost, safety, and efficacy

Pharmacy staff and physicians’ perceptions of the safety and efficacy of generic medicines are summarized in Table 3. In general, physicians were more likely than pharmacy staff to perceive generic medications to be less safe or less effective. However, these differences in perceptions were not statistically significant, likely due to our small sample size.
Table 3

Perceptions of safety and efficacy by pharmacy staff and physicians

 

Pharmacy staff (n = 30)

Physicians (n = 12)

P-value

Generics are not as safe as branded drugs

12 (41%)a

6 (55%)a

0.50b

Generics are not as effective as branded drugs

10 (33%)

6 (55%)a

0.29b

aA few respondents answered “do not know” and their answers are excluded from this calculation

bFisher’s exact test

Among pharmacy staff, 41% believed that generics were not as safe as drugs with commercial brand names, whereas 33% believed that they were not as effective. When asked to elaborate on their perceptions of generic drugs, nine pharmacy staff cited “inferior manufacturing ingredients” or “low quality.” They also cited specific examples of clients who had experienced different therapeutic responses, such as changes in blood pressure or glycemic control, when switching between branded and generic medications as evidence of generics’ inferiority. Eight participants reasoned that generic medicines’ low cost was in itself proof of low quality, explaining that because they are “too economical” or “cheap,” generic medicines could not possibly have equal effectiveness. Five participants were concerned by reports they had read in the popular press or trade journals about the heterogeneous quality of various companies’ manufacturing processes. Nine pharmacy staff reported that the distinction between generics and “branded medications” was the branding process itself; “the only difference is the brand because the quality is the same,” as one respondent remarked. One participant distinguished between generic medicines made by trusted manufacturers and “generic generics,” which the participant deemed to be less reputable.

Pharmacy staff who worked at a national pharmacy franchise or a for-profit hybrid clinic-pharmacy were more likely to view generics as less safe than those who worked at community or public pharmacies (26% vs. 70%, p = 0.046). A similar trend effect was observed for the perception of efficacy but was not statistically significant (21% vs. 55%, p = 0.11).

Among physicians, 55% believed that low-cost generics were not as safe as “branded drugs,” and 55% also stated that they were not as effective. Nine participants expressed at least some reservations about the use of generic medications, with typical comments including that “they are not adequately manufactured,” “they don’t have good quality control,” and “they don’t offer the same results to the patient.” Physicians also acknowledged the heterogeneity of generic drug quality, which was perceived to be dependent on the reliability of the manufacturer. Like pharmacy staff, some physicians reasoned that the low cost of generic drugs was itself indicative of inadequate quality.

When asked about the differences between generic and non-generic drugs, physicians cited prestige in addition to safety and efficacy concerns. One physician explained that patients feel “proud” to be able to pay for expensive drugs. Three physicians described prescribing low-cost generic medicines and subsequently switching to a more expensive drug with a brand name when the generic version was perceived to be ineffective. One participant explained that patients with previous exposure to high-cost drugs with brand names would not respond as well when “downgraded” to a low-cost generic version.

Interactions with pharmaceutical company representatives

Pharmacy staff and physicians were asked about their interactions with pharmaceutical representatives. Twenty-six of 30 pharmacy staff reported visits from pharmaceutical representatives. Seven said that they had received incentives from representatives in the form of drug samples, price discounts, pens, notebooks, calendars, prizes, and informal presentations. Nineteen pharmacy staff cited pharmaceutical representatives among the resources they used to learn about the latest pharmaceutical products.

Responses were similar among physician participants. All but two of the 12 physicians interviewed said that pharmaceutical sales representatives regularly visited them. Of these 10, four reported receiving incentives from pharmaceutical representatives, including drug samples, electronic office equipment, payment of registration for conferences, calendars, and prescription pads. When asked how they stayed up-to-date on current medical literature, eight physicians cited information provided by pharmaceutical companies and their representatives among other resources. No statistically significant relationship was observed between interactions with pharmaceutical representatives or the receipt of gifts from representatives with perceptions of safety or efficacy of generics for either pharmacy staff or physicians.

Generic drug utilization for diabetes and hypertension

In the case of diabetes and hypertension, prescribing practices of physicians and dispensing practices of pharmacy staff varied widely and were influenced in part by perceptions regarding the efficacy and safety of low-cost generics. When pharmacy staff were asked how they decide whether to dispense low-cost generic medicine for a patient with a chronic disease, they were in general agreement that it depended on patients’ financial resources and the severity of the condition, with wealthier and sicker patients less likely to be dispensed generics. Many pharmacy staff also emphasized that the decision to use a generic was ultimately up to the patient. As one participant commented, regarding chronic disease medicines, “I tell them, do you want the expensive one or cheap one? Some people die because they don’t want to buy a generic medicine.” One pharmacy worker added that, in the case of chronic diseases, it was the role of pharmacies “to educate people, so they don’t get tricked by brand-name marketing.”

Three physicians cited the patient’s clinical history and prior drug regimens as important factors in deciding what to prescribe. For instance, one participant explained that a more severe clinical presentation or a “very chronic disease” would call for a non-generic medicine, although starting with a generic version and switching if it were ineffective might be acceptable practice. Several participants expressed that concerns regarding the effectiveness and safety of generic medicines influenced their clinical practice. One physician commenting on chronic disease medicines stated, “To prescribe generics is to play with the life of the patient.”

Roles of physicians and pharmacy staff in providing chronic disease care

The majority of pharmacy staff (87%) stated that a prescribing physician should determine the medicines for diabetes or hypertension. Similarly, most pharmacy staff reported that it is more common for patients with diabetes or hypertension to arrive at the pharmacy with a physician prescription in hand than to ask for a treatment recommendation at the pharmacy. Only 27% of pharmacy staff reported dispensing a medication other than the prescribed medication to a patient with diabetes or hypertension.

In addition to dispensing medications, most pharmacy staff described providing additional medical counseling to patients with diabetes and hypertension. Eighty percent of participants reported offering advice to patients with diabetes and 63% reported giving advice to patients with hypertension. Pharmacy worker advice for managing diabetes included exercise and decreasing the intake of sugar, sugary foods, and fats. In addition to pharmaceuticals, pharmacy staff participants also recommended vitamins, herbs, and electrolyte solutions specifically marketed for patients with diabetes. When providing advice for hypertension, pharmacy staff similarly emphasized increasing physical activity, making dietary changes, and improving psychological well-being. Participants also suggested natural remedies including chamomile tea and garlic with honey.

When physicians were asked about the overall role of pharmacy staff in patient care, several key themes emerged. In particular, interviewed physicians expressed concern about the lack of formal training among pharmacy staff as well as their competing desire to broker the sale of medicines. Physicians frequently stated that the role of pharmacists should be to dispense the medicines as prescribed by the physician without substitutions and to provide education about medicines.

Discussion

Through semi-structured interviews with physicians and pharmacy staff in Guatemala, this study explored perceptions of low-cost generic drugs in clinical practice and how these perceptions of pharmaceutical gatekeepers impact prescribing and dispensing decisions. Our results offer several insights regarding use and access to low-cost generic medicines, including generics for NCDs, that merit further discussion.

First, both physicians and pharmacy staff expressed doubt as to the safety and efficacy of low-cost generic medicines in Guatemala, in part due to concerns about low manufacturing standards and quality control, which influenced their daily prescribing and dispensing practices. Our results echo studies from other LMICs where physicians and pharmacy staff have expressed mixed or negative perceptions regarding the safety and efficacy of generics [4548]. One review examining physician perspectives of generic drugs found that doctors in LMICs tend to have less positive views of generics compared to doctors in high-income countries, who had more positive perceptions [49]. Another review of both physician and pharmacist perceptions of generics similarly distinguished between the high trust of generics in mature health systems and the low trust in generic quality, manufacturing, and bioequivalence in less mature health systems [50].

Second, we found that both physicians and pharmacy staff in Guatemala utilize cost as a heuristic for the quality of the medicines they prescribe and dispense. The use of cost as proxy for the quality of pharmaceutical products has been previously reported in Guatemala and in other LMICs. Indeed, there is some evidence that price does at times serve as an indicator for quality; a recent study of essential medicines from private pharmacies in 17 LMICs found that failing drugs were on average priced 13.6–18.7% lower than non-failing drugs [51]. Some commentators writing about Guatemala from the field of anthropology have argued that high-priced health care products serve not only to emphasize perceptions of quality but also to appeal to a desire for social status and upward mobility [52, 53]. In the present study, this viewpoint is buttressed by the response of some participants, such as the physician who remarked that some patients “feel proud” to pay for expensive drugs. Furthermore, others have shown that the commercial aspects of medications, such as pricing, may influence the placebo effect [54], and it is therefore reasonable to speculate that cost may indeed affect clinical outcomes independent of a difference in intrinsic efficacy between products.

Third, our results show that physicians, pharmacy staff, and consumers all play a role in determining whether a low-cost generic is ultimately utilized in practice for NCDs. The incentives of each of these actors and the forces that influence them are complex, yet policy implications emerge. We report that pharmacy staff generally, though not always, defer to physician prescriptions, which points to a need to enforce laws requiring generic name in prescriptions and otherwise improve communication between prescribers and dispensers. We also found that pharmaceutical representatives frequently visit physicians and pharmacy staff in Guatemala and provide education about new pharmaceuticals. The ubiquity of these visits and the use of gifts and prizes call attention to a need for a well-articulated government policy regarding interactions between pharmaceutical representatives and providers. Although our study did not show a statistically significant relationship between perceptions of generics and interactions with pharmaceutical representatives, the small sample size and very low numbers of recruited participants who did not interact with pharmaceutical representatives means that our study was significantly underpowered to detect a difference.

Fourth, our results show that pharmacy staff generally have low levels of education and training, yet they play an important role in educating patients about medicines and chronic diseases like diabetes and hypertension. Although this study was not designed to evaluate pharmacy worker decision-making, there is a significant literature showing that pharmacy staff in many LMICs [55, 56], including Guatemala [57, 58], do not deliver treatment advice in accordance with commonly-accepted medical standards. However, in this study, pharmacy staff reported that physicians should be the gatekeepers who determine diabetes and hypertension medicines for patients, that it was more common for patients with diabetes and hypertension to present with physician prescriptions than to ask for advice at pharmacies, and that they did not frequently switch out physician prescriptions. These finding are at odds with the existing literature in the region, which, focusing largely on medications for acute illnesses, finds that most dispensing occurs without the involvement of a prescribing physician [37, 43, 44, 57, 59].

Although these results are subject to methodological bias, they are surprising and unexpected, and they require confirmation and exploration in a larger study. One possibility is that the culture of pharmaceutical dispensing is slowly changing in Guatemala, with a move towards more formal reliance on physician prescriptions. Along these lines, it is worth noting that some key studies of pharmaceutical dispensing in the region are now more than 10 years old [57, 58], and so it is likely time to revisit the issue, including reanalyzing dispensing practices for antibiotics and other acute illness medications. Alternatively, it may be that informally-trained pharmacy workers are simply less comfortably dispensing medications for chronic diseases. This could be because of relative limitations in their knowledge about these medications, exacerbated by the rapid diversification of and growth in products in the chronic illnesses market, which are rapidly outpacing the relatively more stable market for antimicrobials, analgesics, and the like. We also hypothesize that the higher cost of medications for chronic illness is a contributory factor. Given the higher financial stakes, it may be that clients or dispensing practitioners increasingly prefer to formalize transactions with the aid of a written physician prescription.

From a broader perspective, previous literature outlines four main categories of barriers to the implementation of pro-generic policies in LMICs: (1) legal barriers; (2) management and institutional barriers; (3) behavior, perception, and knowledge barriers; and (4) financial barriers [9]. This study principally addresses the third category—in particular, the perceptions of physicians and pharmacy staff and how these perceptions influence clinical behavior. Our results indicate that improving attitudes toward low-cost generics is an important goal. At the same time, as we have detailed in the introduction to this article, there is compelling evidence in Guatemala to support the belief that the safety and efficacy of the drug supply cannot be guaranteed. Indeed, the issue of substandard and counterfeit medicines is a concern in many LMICs [30, 6063]. In the Guatemalan setting, where regulatory information is limited and drug quality is uncertain, it is thus not surprising that the high price of trusted pharmaceutical brands is therefore used as a surrogate marker for quality.

Taken in context, our results also underline the fact that interventions to improve the perception of low-cost generics and increase their utilization by physicians and pharmacy staff would be insufficient without commensurate efforts to strengthen the regulation and transparency of the drug supply. Unfortunately, in Guatemala’s current political climate, it is difficult to imagine any increased state capacity for regulating or ensuring generic drug supplies in the near future. However, we do see two potential promising avenues. First, in recent years, the Guatemalan judicial system, reinforced by backing from the United Nations’ International Commission against Impunity in Guatemala (CICIG), has made considerable inroads in fighting high-level government corruption, including in the health care sector [64]. We hope that CICIG’s activities will reduce high-profile drug procurement scandals, such as those we mentioned in this paper’s Introduction, and may help to increase public confidence in generic medications. Second, we can see a role for voluntary self-organization and self-regulation by generics manufacturers. Commitments to increased transparency and accountability, for example through reporting on compliance with international manufacturing standards and bioequivalence testing, could go a long way both in improving the quality of the generics drug supply and confronting the allegations of inferiority in both the popular press and the promotional materials of originator brands. Such commitments would no doubt come at increased production costs, but these could potentially be offset as public opinion improved and sales increased.

Study limitations and future directions

This study has several methodological limitations that limit the generalizability of our findings. First, the sample size of physicians and pharmacy staff was small with limited power to detect some important differences, especially regarding interaction with pharmaceutical representatives. Physicians were recruited based on a convenience sampling strategy that could have introduced bias. Second, geographically, participants were concentrated in one particular region of Guatemala, and their opinions may not reflect the views of physicians and pharmacy staff throughout Guatemala. In particularly, the study did not sample participants in Guatemala City, which is the country’s dominant urban area from both a population and economic perspective, nor did we consider the self-reported ethnicity of providers or pharmacy staff themselves in creating our sample. Additionally, as we did not define “originator medicines,” “branded generics,” and “unbranded generics” to research participants, we are unable to make firm distinctions among the perceptions of drugs in each of these categories. The “branded generic conundrum” has emerged as a primary challenge of the generic medicine policy agenda [63]. Further inquiry into the perception, price, and utilization of branded generics in Guatemala in relation to originator products and unbranded generics is an important future research goal for our team. We also intend to explore in greater detail the unexpected findings about reluctance to dispense medications for chronic illness without a prescription, and to investigate more closely the relationship between prescribing practices and interactions with pharmaceutical representatives. In this regard, the frequency of interaction with pharmaceutical representatives we measure in this study will permit sample size calculations for an adequately powered subsequent study.

Conclusion

This study demonstrates that physician and pharmacy staff’s perceptions of low-cost generics influence the utilization of these medicines in clinical practice, especially in relation to the treatment of NCDs like diabetes and hypertension. Interventions to improve the perception of generics and increase their utilization by physicians and pharmacy staff are critical. At the same time, additional research on changing dispensing practices of medications for both acute and chronic illnesses is needed, as is further investigation of the impact of interactions with pharmaceutical representatives on prescribing and dispensing practices. Although short-term prospects for increasing state capacity for regulation and monitoring of the drug supply are limited, we anticipate that the emergent anti-impunity movement in Guatemala will increase public trust in generic medications by reducing high-profile scandals over drug procurement schemes. Furthermore, we suggest that the generics industry, through self-regulation and self-organizing, could improve the quality of the generics market while simultaneously increasing their market share.

Abbreviations

CICIG: 

Comisión Internacional contra la Impunidad en Guatemala, International Commission against Impunity in Guatemala

IGSS: 

Instituto Guatemalteco de Seguro Social, Guatemalan Social Security Institute

LMIC: 

Low- and middle-income country

NCD: 

non-communicable disease

NGO: 

non-governmental organization

PROAM: 

Programa de Acesibilidad a Medicamentos, Program for Medication Accessibility

WHO: 

World Health Organization

Declarations

Acknowledgements

The authors wish to thank Brooke Ramay, Alejandro Cerón, and Luis Mendez for their useful comments and suggestions during the drafting process.

Funding

The authors received no specific funding for this work.

Availability of data and materials

The data spreadsheet and interview guides associated with this research are available from the Harvard Dataverse at http://dx.doi.org/10.7910/DVN/Y4WQ10.

Authors’ contributions

DF and IM contributed equally to this study. DF performed the literature review, designed the study, created pharmacy maps, supervised data collection, and drafted the manuscript. IM designed the study, collected and analyzed qualitative data, and drafted the manuscript. AC designed the study, created pharmacy maps, and drafted the manuscript. PG designed the study and edited the manuscript. PR designed the study, analyzed data, and drafted the manuscript. All authors read and approved the final manuscript.

Authors’ information

At the time of this study, all authors were volunteers or staff members with Wuqu’ Kawoq | Maya Health Alliance, a non-governmental organization engaged in developing non-communicable disease programming for indigenous patients in rural Guatemala.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Not applicable.

Ethics approval and consent to participate

This study was approved by the Institutional Review Boards of Wuqu’ Kawoq | Maya Health Alliance (WK-2014-005) and Partners Healthcare, Boston (2014P002427/BWH). All participants provided verbal informed consent based on IRB-approved consent scripts. A waiver for written consent was obtained as the research (1) was deemed to not involve more than minimal risk, (2) was not subject to FDA regulations, and (3) the only record linking the subject and the research was the consent document.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Wuqu’ Kawoq | Maya Health Alliance

References

  1. Laing R, Waning B, Gray A, Ford N, ’t Hoen E. 25 years of the WHO essential medicines lists: progress and challenges. Lancet. 2003;361:1723–9.View ArticlePubMedGoogle Scholar
  2. Hogerzeil HV, Liberman J, Wirtz VJ, Kishore SP, Selvaraj S, Kiddell-Monroe R, et al. Promotion of access to essential medicines for non-communicable diseases: practical implications of the UN political declaration. Lancet. 2013;381:680–9.View ArticlePubMedGoogle Scholar
  3. Kishore SP, Kolappa K, Jarvis JD, Park PH, Belt R, Balasubramaniam T, et al. Overcoming obstacles to enable access to medicines for noncommunicable diseases in poor countries. Health Aff (Millwood). 2015;34:1569–77.View ArticleGoogle Scholar
  4. van Mourik MS, Cameron A, Ewen M, Laing RO. Availability, price and affordability of cardiovascular medicines: a comparison across 36 countries using WHO/HAI data. BMC Cardiovasc Disord. 2010;10:25.View ArticlePubMedPubMed CentralGoogle Scholar
  5. Vialle-Valentin CE, Serumaga B, Wagner AK, Ross-Degnan D. Evidence on access to medicines for chronic diseases from household surveys in five low- and middle-income countries. Health Policy Plan. 2014;30:1044–52.View ArticlePubMedPubMed CentralGoogle Scholar
  6. Cameron A, Roubos I, Ewen M, Mantel-Teeuwisse AK, Leufkens HG, Laing RO. Differences in the availability of medicines for chronic and acute conditions in the public and private sectors of developing countries. Bull World Health Organ. 2011;89:412–21.View ArticlePubMedPubMed CentralGoogle Scholar
  7. Cameron A, Ewen M, Ross-Degnan D, Ball D, Laing R. Medicine prices, availability, and affordability in 36 developing and middle-income countries: a secondary analysis. Lancet. 2009;373:240–9.View ArticlePubMedGoogle Scholar
  8. Bigdeli M, Jacobs B, Tomson G, Laing R, Ghaffar A, Dujardin B, et al. Access to medicines from a health system perspective. Health Policy Plan. 2013;28:692–704.View ArticlePubMedGoogle Scholar
  9. Kaplan WA, Ritz LS, Vitello M, Wirtz VJ. Policies to promote use of generic medicines in low and middle income countries: a review of published literature, 2000–2010. Health Policy. 2012;106:211–24.View ArticlePubMedGoogle Scholar
  10. BMI Research. Central America pharmaceuticals and healthcare report Q3 2015. 2015. https://pdf.marketpublishers.com/businessmonitor/central_america_pharmaceuticals_n_healthcare_report_q3_2011.pdf. Accessed 20 Sept 2015
  11. Petrecolla D. Condiciones de competencia en el sector de medicamentos de centroamérica. 2011. http://www.acodeco.gob.pa:8080/RECAC/InformeSectorMedicamentos_Enero2011.pdf. Accessed 10 Jan 2017.
  12. Oficina Comercial de ProChile en Guatemala. Estudio de mercado medicamentos en guatemala. 2011. http://www.prochile.gob.cl/wp-content/files_mf/documento_06_20_11172536.pdf. Accessed 10 Jan 2017.
  13. World Health Organization. Guatemala: Perfil farmacéutico nacional. 2012. http://apps.who.int/medicinedocs/documents/s19750es/s19750es.pdf. Accessed 20 Sept 2015.
  14. Ministerio de Salud Pública y Asistencia Social. Cuentas nacionales de salud: Reporte de Guatemala al informe de estadísticas sanitarias mundiales. 2015. http://www.mspas.gob.gt/files/Descargas/UNIDADES/CuentasNacionalesSalud/Publicaciones/MSPAS%20(2015)%20Cuentas%20Nacionales%20de%20Salud%20de%20Guatemala.pdf. Accessed 10 Jan 2017.
  15. Bowser DM, Mahal A. Guatemala: the economic burden of illness and health system implications. Health Policy. 2011;100:159–66.View ArticlePubMedGoogle Scholar
  16. Anson A, Ramay B, de Esparza AR, Bero L. Availability, prices and affordability of the World Health Organization’s essential medicines for children in Guatemala. Global Health. 2012;8:22.View ArticlePubMedPubMed CentralGoogle Scholar
  17. Consejo Centroamericano de Protección al Consumidor. Sondeo centroamericano de precios de medicamentos 2008. 2009. http://www.opinandoenelsalvador.com/wp-content/uploads/2009/06/estudiopreciosmedicamentoscentroamerica.pdf. Accessed 20 Sept 2015.
  18. Emmerick IC, Luiza VL, Camacho LA, Ross-Degnan D. Access to medicines for acute illness in middle income countries in Central America. Rev Saude Publica. 2013;47:1069–79.View ArticlePubMedPubMed CentralGoogle Scholar
  19. Perez Schlosser A. Accesibilidad a los medicamentos en Guatemala: Una evaluacion de costo versus indicatores socioeconomics. Guatemala: Universidad del Valle de Guatemala; 2007.Google Scholar
  20. World Health Organization. Glossary. http://www.who.int/trade/glossary/story034/en/. Accessed 10 Jan 2017.
  21. World Health Organization. Measuring medicine prices, availability, affordability and price components. 2nd ed. Geneva: World Health Organization; 2008.Google Scholar
  22. Hayden C. A generic solution? Pharmaceuticals and the politics of the similar in Mexico. Curr Anthropol. 2007;48:475–95.View ArticleGoogle Scholar
  23. Cerón A, Snodgrass Godoy A. Intellectual property and access to medicines: an analysis of legislation in Central America. Bull World Health Organ. 2009;87:787–93.View ArticlePubMedPubMed CentralGoogle Scholar
  24. Godoy AS, Ceron A. Changing drug markets under new intellectual property regimes: the view from Central America. Am J Public Health. 2011;101:1186–91.View ArticlePubMedPubMed CentralGoogle Scholar
  25. Godoy AS. Of medicines and markets: intellectual property and human rights in the free trade era. Stanford: Stanford University Press; 2013.Google Scholar
  26. Shaffer ER, Brenner JE. A trade agreement’s impact on access to generic drugs. Health Aff (Millwood). 2009;28:w957–68.View ArticleGoogle Scholar
  27. Kaplan WA, Wirtz VJ, Stephens P. The market dynamics of generic medicines in the private sector of 19 low and middle income countries between 2001 and 2011: a descriptive time series analysis. PLoS One. 2013;8:e74399.View ArticlePubMedPubMed CentralGoogle Scholar
  28. Martinez GE, Bonilla AC. Propiedad intelectual y acceso a medicamentos esenciales de calidad en Centroamérica. San Salvador: Fundación Nacional para el Desarrollo; 2008.Google Scholar
  29. León SD. ¿Medicamentos genéricos o innovadores? El que sea, pero que tengan el mismo efecto. El Periódico. 26 October 2014. https://web.archive.org/web/20141201102644/http://www.elperiodico.com.gt/es/20141026/domingo/3891/%C2%BFMedicamentos-gen%C3%A9ricos-o-innovadores-El-que-sea-pero-que-tengan-el-mismo-efecto.htm. Accessed 20 Sept 2015.
  30. Hajjou M, Krech L, Lane-Barlow C, Roth L, Pribluda VS, Phanouvong S, et al. Monitoring the quality of medicines: results from Africa, Asia, and South America. Am J Trop Med Hyg. 2015;92:68–74.View ArticlePubMedPubMed CentralGoogle Scholar
  31. The United States Pharmacopeial Convention. Medicines quality database. 2015. http://www.usp.org/global-health-programs/promoting-quality-medicines-pqmusaid/medicines-quality-database-mqdb. Accessed 20 Sept 2015.
  32. Power TJ, Cyr JM. Mapping political legitimacy in Latin America. Int Soc Sci J. 2009;60:253–72.View ArticleGoogle Scholar
  33. Orozco A. Clientes pueden pedir reintegro a farmacias. Prensa Libre. 13 May 2014. http://www.prensalibre.com/noticias/comunitario/descuento-farmacias-citacion-diaco-sobreprecio_de_medicinas-reintegro_0_1137486423.html. Accessed 20 Sept 2015.
  34. Orozco A, Rojas A. Laboratorios habrían acordado aumento en precios de medicinas. Prensa Libre. 28 May 2013. http://www.prensalibre.com/noticias/politica/aumento-en-precios-de-medicamentos-j-i-cohen-informe-pdh-acuerdo-entre-farmaceuticas-0-1145885594. Accessed 20 Sept 2015.
  35. Webber J. Guatemala’s central bank chief arrested. Financial Times. 21 May 2015. http://www.ft.com/cms/s/0/8971ab82-ff3c-11e4-84b2-00144feabdc0.html - axzz3iZypGvAH. Accessed 20 Sept 2015.
  36. Flood D, Rohloff P. “This disease is for those who can afford it”: diabetes in indigenous Maya communities. In: Chary A, Rohloff P, editors. Privatization and the new medical pluralism: shifting healthcare landscapes in Maya Guatemala. Lanham: Lexington Books; 2015. p. 51–68.Google Scholar
  37. Ramay BM, Lambour P, Ceron A. Comparing antibiotic self-medication in two socio-economic groups in Guatemala City: a descriptive cross-sectional study. BMC Pharmacol Toxicol. 2015;16:11.View ArticlePubMedPubMed CentralGoogle Scholar
  38. Consumidores en Acción de Centroamérica. ¿Promoción ética y precio justo en los medicamentos? Una mirada desde las organizaciones de consumidores en Centroamérica. San Salvador: Consumidores en Acción de Centroamérica; 2007.Google Scholar
  39. Laing R. The need for generic policies as part of health reform. Geneva: World Health Organization; 2013.Google Scholar
  40. Fedefarma. Quienes somos. 2012. http://www.fedefarma.org/. Accessed 20 Sept 2015.
  41. Vargas Copland EC, Ramay B, López Garcia ER, Ureta Morales FJ, Mejía Villatoro CR. Percepción de médicos de Guatemala respecto a las especialidades farmacéuticas genéricas. Perspectiva. 2012;13:69–85.Google Scholar
  42. Bruce N, Pope D, Arana B, Shiels C, Romero C, Klein R, et al. Determinants of care seeking for children with pneumonia and diarrhea in Guatemala: implications for intervention strategies. Am J Public Health. 2014;104:647–57.View ArticlePubMedPubMed CentralGoogle Scholar
  43. Heuveline P, Goldman N. A description of child illness and treatment behavior in Guatemala. Soc Sci Med. 2000;50:345–64.View ArticlePubMedGoogle Scholar
  44. Amorim C. Health is a business for everyone and is not a right to anyone: neoliberal health care provision in rural Guatemala (Master’s thesis). Lawrence: University of Kansas; 2012.Google Scholar
  45. Bashaar M, Hassali MA, Saleem F. Community pharmacists’ attitudes toward the quality and price of locally manufactured generic medicines in Kabul, Afghanistan. J Pharm Policy Pract. 2015;8:16.View ArticlePubMedPubMed CentralGoogle Scholar
  46. Chua GN, Hassali MA, Shafie AA, Awaisu A. A survey exploring knowledge and perceptions of general practitioners towards the use of generic medicines in the northern state of Malaysia. Health Policy. 2010;95:229–35.View ArticlePubMedGoogle Scholar
  47. Kumar R, Hassali MA, Saleem F, Alrasheedy AA, Kaur N, Wong ZY, et al. Knowledge and perceptions of physicians from private medical centres towards generic medicines: a nationwide survey from Malaysia. J Pharm Policy Pract. 2015;8:11.View ArticlePubMedPubMed CentralGoogle Scholar
  48. Basak SC, Sathyanarayana D. Exploring knowledge and perceptions of generic medicines among drug retailers and community pharmacists. Indian J Pharm Sci. 2012;74:571–5.View ArticlePubMedPubMed CentralGoogle Scholar
  49. Hassali MA, Wong ZY, Alrasheedy AA, Saleem F, Mohamad Yahaya AH, Aljadhey H. Perspectives of physicians practicing in low and middle income countries towards generic medicines: a narrative review. Health Policy. 2014;117:297–310.View ArticlePubMedGoogle Scholar
  50. Toverud EL, Hartmann K, Hakonsen H. A systematic review of physicians’ and pharmacists’ perspectives on generic drug use: what are the global challenges? Appl Health Econ Health Policy. 2015;13 Suppl 1:35–45.View ArticlePubMed CentralGoogle Scholar
  51. Bate R, Jin GZ, Mathur A. Does price reveal poor-quality drugs? Evidence from 17 countries. J Health Econ. 2011;30:1150–63.View ArticlePubMedGoogle Scholar
  52. Chary A, Rohloff P, editors. Privatization and the new medical pluralism: shifting healthcare landscapes in Maya Guatemala. Lanham: Lexington Press; 2015.Google Scholar
  53. Nelson DM. 100 percent Omnilife: health, economy, and the end/s of war. In: McAllister C, Nelson DM, editors. War by other means: aftermath in post-genocide Guatemala. Durham: Duke University Press; 2013. p. 285–306.View ArticleGoogle Scholar
  54. Waber RM, Shiv B, Carmon Z, Ariely D. Commercial features of placebo and therapeutic efficacy. JAMA. 2008;299:1016–7.View ArticlePubMedGoogle Scholar
  55. Brata C, Gudka S, Schneider CR, Clifford RM. A review of the provision of appropriate advice by pharmacy staff for self-medication in developing countries. Res Social Adm Pharm. 2015;11:136–53.View ArticlePubMedGoogle Scholar
  56. Smith F. The quality of private pharmacy services in low and middle-income countries: a systematic review. Pharm World Sci. 2009;31:351–61.View ArticlePubMedGoogle Scholar
  57. Kroeger A, Ochoa H, Arana B, Diaz A, Rizzo N, Flores W. Inadequate drug advice in the pharmacies of Guatemala and Mexico: the scale of the problem and explanatory factors. Ann Trop Med Parasitol. 2001;95:605–16.View ArticlePubMedGoogle Scholar
  58. Blanco R. Conceptos de empleados de farmacia acerca del manejo de la diarrea en niños. Bol Med Hosp Infant Mex. 1989;46:463–9.PubMedGoogle Scholar
  59. Hall-Clifford R. Oral rehydration therapy in highland Guatemala: long-term impacts of public health intervention on the self (Doctoral dissertation). Boston: Boston University; 2009.Google Scholar
  60. Almuzaini T, Choonara I, Sammons H. Substandard and counterfeit medicines: a systematic review of the literature. BMJ Open. 2013;3:e002923.View ArticlePubMedPubMed CentralGoogle Scholar
  61. Mackey TK, Liang BA, York P, Kubic T. Counterfeit drug penetration into global legitimate medicine supply chains: a global assessment. Am J Trop Med Hyg. 2015;92:59–67.View ArticlePubMedPubMed CentralGoogle Scholar
  62. Caudron JM, Ford N, Henkens M, Mace C, Kiddle-Monroe R, Pinel J. Substandard medicines in resource-poor settings: a problem that can no longer be ignored. Trop Med Int Health. 2008;13:1062–72.View ArticlePubMedGoogle Scholar
  63. Kaplan WA, Wirtz VJ. A research agenda to promote affordable and quality assured medicines. J Pharm Policy Pract. 2014;7:2.View ArticlePubMedPubMed CentralGoogle Scholar
  64. Comisión Internacional contra la Impunidad en Guatemala. Por una cultura de legalidad. 2016. http://www.cicig.org/uploads/documents/broshure/broshure_es.pdf. Accessed 9 May 2016.

Copyright

© The Author(s). 2017

Advertisement