Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

Core indicators evaluation of effectiveness of HIV-AIDS preventive-control programmes carried out by nongovernmental organizations. A mixed method study

  • Anna Berenguera1, 5,
  • Enriqueta Pujol-Ribera1Email author,
  • Concepció Violan1,
  • Amparo Romaguera2,
  • Rosa Mansilla3,
  • Albert Giménez3,
  • Carlos Ascaso5 and
  • Jesús Almeda1, 2, 4
BMC Health Services Research201111:176

DOI: 10.1186/1472-6963-11-176

Received: 30 March 2011

Accepted: 28 July 2011

Published: 28 July 2011

Abstract

Background

The number of nongovernmental organizations working on AIDS has grown. There is great diversity in the type of activities and population groups that have been targeted. The purposes of this study are: to describe and analyze the objectives and HIV-AIDS preventive activities that are carried out by the AIDS-NGOs that work with AIDS in Catalonia and that receive subsidies from the Department of Health; and to develop a comprehensive proposal for measurable and agreed upon core quality evaluation indicators to monitor and assess those objectives and activities that can have an impact on the fight against inequalities and stigmatization, and incorporate the perspectives of the service providers and users.

Methods

A mixed method study has been carried out with professionals from the 36 NGOs that work with HIV/AIDS in Catalonia, as well as their users. This study achieved the completeness model using the following phases:

1. A systematic review of AIDS-NGOs annual reports and preparation of a catalogue of activities grouped by objectives, level of prevention and AIDS-NGOs target population; 2. A transversal study through an ad-hoc questionnaire administered to the AIDS-NGOs representatives; 3. A qualitative study with a phenomenological approach through focus groups, individual interviews and observations; 4. Consensus meetings between AIDS-NGOs professionals and the research team using Haddon matrices in order to establish a proposal of evaluation indicators.

Results

The information was classified according to level of prevention and level of intervention. A total of 248 objectives and 258 prevention activities were identified. 1564 evaluation indicators, addressed to 7 target population groups, were produced. Thirty core activities were selected. The evaluation indicators proposed for these activities were: 76 indicators for 15 primary prevention activities, 43 for 5 secondary prevention activities and 68 for 10 tertiary prevention activities.

Conclusions

The results could help to homogeneously assess the preventive-control activities carried out of AIDS-NGOs. The proposed indicators could help the professionals to improve the evaluation of the preventive-control AIDS-NGOs activities. Furthermore, the Haddon matrix enables us to identify deficiencies of activities at intervention levels and strategies to bear in mind in order to enhance the future AIDS prevention programs.

Background

The 2001 Declaration of Commitment on HIV/AIDS and the 2006 Political Declaration on HIV/AIDS adopted by the United Nations General Assembly are the guiding forces of the global response to AIDS. The process of preparing country progress reports should involve all partners in AIDS response and should provide an opportunity for reflection on the national response, its achievements as well as its shortcomings, in order to reach universal goals. Monitoring the response to the AIDS epidemic is essential to ensure that investments in AIDS achieve the expected results in health and society [1].

HIV-related Non Governmental Organizations (AIDS-NGOs) work to promote prevention and to provide care and help to people affected by HIV in most countries of the world. The role of these AIDS-NGOs has been critical in the fight against HIV infection since the onset of the epidemic [2]. The activities of the AIDS-NGOs are important both around the world and in Catalonia (one of 17 Autonomous Communities in the North-East of Spain with 7,364,078 inhabitants in 2008) [3]. These activities are complementary and are often leaders of public and private health systems in various countries. AIDS-NGOs have lead the initiative against HIV and act as a bridge and "communications space" between the most vulnerable sectors of the population (because of socioeconomic inequalities) and health services [4]. AIDS-NGOs are the largest providers of preventive activities against HIV/AIDS, particularly amongst high-risk behavior groups: commercial sex workers, injecting drug users, men who have sex with men, youths in high-risk situations, prisoners and immigrants and persons living with HIV/AIDS [5, 6]. Furthermore, they have wide experience of working at the community level, and their autonomous nature allows them to respond more quickly and employ innovative methods [7].

Since the inception of the AIDS National Program in Catalonia, the promotion and coordination of activities carried out by AIDS-NGOs has been emphasized. To support the AIDS-NGOs, since 1992 the Department of Health has provided annual subsidies that have progressively increased to reach 1.4 million Euros in 2007 [810]. There has been an increase in the prevention, health promotion and care activities of AIDS-NGOs [6, 11].

The number of AIDS-NGOs and their activities has grown to meet the demand and the needs of individuals in preventive materials, determination of serostatus and HIV care. For this reason there is great diversity in the type of activities, as well as in the social levels and population groups that have been targeted.

In Spain, Peiró et al. systematically classified and evaluated the activities of national AIDS programs, including AIDS-NGOs activities using the Haddon matrix. This study concluded that there is much heterogeneity in the structure and indicators of the various programs. Concepts such as goals, objectives and priorities are poorly defined. Moreover, in all countries the implementation of preventive programs shows a predominance of individual interventions over social ones [12].

A more recent report, the Health Plan of the Community of Valencia 2005-2009 [13] includes a description of objectives and lines of action for HIV prevention based on the Haddon matrix axes [1416]. Other autonomous communities in Spain such as the Canary Islands and Asturias have also provided a basis for developing evaluation indicators for all activities [17, 18]. In Spain, the "AIDS National Programme" of the Ministry of Health addresses the development of preventive programs for HIV and other sexually transmitted infections in individuals that work as commercial sex workers. The plan suggests both qualitative and quantitative evaluation indicators for these activities [9].

Moreover, the methodology that has been proposed to compare the different ways to prioritize issues and to forecast a policy's application in terms of results and outcomes is the Haddon Matrix. This matrix is a way to synthesizes the enormous amount of information available in strategy documents, shorten the time needed to produce an assessment, and improve a policy's value by comparing it with external models. In this paper we use this theoretical framework developed with the explicit intention of covering the need of the AIDS National Program of Catalonia to know what preventive and control interventions are done by AIDS-NGOs with the objective of improving the praxis of these activities and establishing new strategies for the AIDS National program [12]. Furthermore, a proposal of measurable indicators to monitor and assess each activity done by these organizations will be developed.

It is important to provide key constituents who are actively involved in a country's response to AIDS with essential information on core indicators that measure the effectiveness of the national response. The indicators will help to ensure the consistency and transparency of the process used by national governments. Increasingly, countries are strongly encouraged to integrate the core indicators into their ongoing monitoring and evaluation activities [19].

The key motivation for undertaking this work is to propose quality indicators which include users' satisfaction, psychosocial aspects that can have an impact on the fight against inequalities and stigmatization, and the perspectives of professional service providers. Until now, in many evaluation proposals, these aspects were not really addressed in the HIV literature, while their consideration would permit an important step toward the incorporation of strategies for evaluation of NGO prevention and more comprehensive control activities.

The purposes of this study are: to describe and analyze the objectives of and the HIV-AIDS preventive activities carried out by the AIDS-NGOs that work with AIDS in Catalonia and that receive subsidies from the Department of Health; and to develop a more comprehensive set of measurable and agreed upon core quality evaluation indicators to monitor and assess those objectives and activities that can have an impact on health and the fight against inequalities and stigmatization, and incorporate the perspectives of service providers and users.

Methods

A mixed method strategy was used in order to achieve the objectives of the study. The way to combine quantitative and qualitative research was by completeness, in order to achieve more complete and comprehensive answers to research questions. The strategy was divided into four phases and there was full data integration of each phase (each new phase includes the findings of previous phases) [2023] (Figure 1).
https://static-content.springer.com/image/art%3A10.1186%2F1472-6963-11-176/MediaObjects/12913_2011_Article_1691_Fig1_HTML.jpg
Figure 1

Methodological Strategy.

Phase 1-Systematic and exhaustive review of the AIDS-NGOs annual reports

A systematic reading and review of the 36 annual reports from the all AIDS-NGOs working in Catalonia and which received subsidies from the Catalan Government in 2006. The annual reports of AIDS-NGO were provided by the AIDS National Program exclusively for this study.

From this systematic reading, we found objectives and activities of prevention were classified according to the following criteria [12, 15, 16]: AIDS-NGOs target population (commercial sex workers, injecting drug users, men who have sex with men, youths in high risk situations and persons living with HIV/AIDS), levels of prevention (primary, secondary and tertiary), and levels of intervention (individuals, the individual's immediate environment, the external environment and the social environment). This classification was the first step in constructing Haddon matrices. The Haddon matrix offers a conceptual framework that facilitates the analysis of health problems, taking into consideration a three dimensional grid: on the x-axis, the time dimension: primary prevention, secondary prevention and tertiary prevention; on the y-axis, the setting of the intervention: individual factors, immediate environment, external environment and a fourth level that includes the social system and the current regulations. The third axis or dimension includes actions designed to decrease gender, age, ethnic, socio-economic and cultural inequalities [12, 15, 16].

From this synthesis, a database relating the objectives and activities described in the AIDS-NGOs annual reports was created.

What new information does this methodological phase of the study provide?

This phase provides new basic information on the objectives of and the activities carried out by the AIDS-NGOs as described in the annual reports these organisations prepare for the Health Department. A Haddon matrix has also been constructed with the activities classified by strategy, level of prevention, and level of intervention.

Phase 2-Transversal study

An ad-hoc questionnaire was constructed in order to identify activities and objectives classified according to AIDS-NGOs target populations that were not explicit in the AIDS-NGOs annual reports.

A pilot study was carried out in 2008, using a cognitive interview which allowed assessment of the understanding and applicability of the questionnaire [24, 25].

The questionnaire was self-administered and it was sent by mail to all the 36 AIDS-NGOs. It consisted of 15 questions, (5 open questions, 7 closed questions and 3 multiple choice questions with non-mutually exclusive answers) and included the followed sections: (1) Profile of the individual responding to the questionnaire, (2) Information regarding the AIDS-NGOs target population, (3) Objectives of the AIDS-NGOs and preventive activities of the program, (4) Preventive activities carried out by the AIDS-NGOs, (5) Agreement to participate in the future consensus meetings between AIDS-NGOs professionals and the research team to establish an evaluation indicators proposal. In each section, there was one open question to express opinions and experiences that were not covered by the questionnaire.

To facilitate the highest percentage of responses, two rounds of confirmatory phone calls were conducted and a descriptive analysis (absolute frequencies and percentages) was performed.

What new information does this methodological phase of the study provide?

The results of the transverse study complement those of the first phase of the elaboration of the Haddon matrix based on the systematic study of the reports. In this methodological phase we have also been able to establish a relationship between objectives and activities.

Phase 3-Qualitative study of phenomenological perspective

A qualitative study was conducted in order to identify the experiences that professionals in the Catalan AIDS-NGOs had in preventive activities, the potential areas of improvement of the activities, the professionals' evaluation of their activities, the experiences and practices of AIDS-NGOs with regard to HIV infection and prevention, and elements that enhanced their relationship with the AIDS-NGOs.

A theoretical sampling (professional) and opportunistic sample (participants) was proposed. Professionals and users from 36 AIDS-NGO funded by the Health Department to take part in the study. For the professionals, a theoretical sampling based on the previous definition of the characteristics of the participants has been carried out to obtain the greatest variety and discursive wealth to reach data saturation. Variables used to define the informant profile for the professionals of AIDS-NGO are: AIDS-NGO target population (commercial sex workers, injecting drug users, men who have sex with men, youths in high-risk situations, prisoners and immigrants and persons living with HIV/AIDS), age, sex, professional profile, setting (urban or rural) and years of experience.

Due to the difficulties of the theoretical sampling, an opportunistic sampling was finally chosen for the AIDS-NGO users. However, heterogeneity criteria were taken into account. Variables used to define the users are: AIDS-NGO target population, age, sex, nationality, serostatus, and time in contact with the NGO [26].

Different techniques have been used [27]. For the professionals, focus group [28] and triangular groups [29]. In the focus groups, interaction is the instrument to stimulate the individual speech [30]. For the users, semi-structured interviews (because during the interviews some sensitive issues may arise) [30] and open, focused and non-systematic observation of theatre performances in teenagers and young adults have been carried out [31].

Data collection was between February and June 2008. Focus groups took place in a neutral place (IDIAP Jordi Gol) and they included a moderator and an observer. The semi-structured interviews took place in the working place of the users or at the AIDS-NGO places. The observations were made in two secondary schools.

A thematic interpretive content analysis was conducted by three analysts. More details about the methodology of this phase have been published elsewhere [32].

What new information does this methodological phase of the study provide?

This methodological phase of the study has permitted a deeper knowledge of how AIDS-NGO professionals carry out their activities. It has also allowed exploration of the barriers professionals encounter in carrying out evaluation. Users also took part in this methodological phase of the study, in which their knowledge and perception of HIV-AIDS risks was explored.

Phase 4-Consensus meetings between NGO professionals and the research team

From the synthesis of the information obtained in phases 1, 2 and 3, the Haddon matrices described in phase 1 were completed. Later, consensus meetings were held between five AIDS-NGOs professionals and four members of the research team, in order to review and redevelop the Haddon matrices and to identify new preventive activities when possible.

The AIDS-NGOs professional selection was carried out considering the inclusion of different professional profiles, work with different AIDS-NGOs target groups, years of antiquity and the geographical setting.

From the Haddon matrices, a proposal of qualitative and quantitative evaluation indicators was developed, both for process and outcomes of activities carried out by AIDS-NGOs. The structure and methodology that was followed in order to develop the indicators proposal is shown in tables 1 and 2 [33].
Table 1

Structure of the proposal of indicators

AIDS-NGOs target population

   

Prevention and/or intervention level

   

ACTIVITY

   

Objective

   

PROCESS MEASURE

Indicator description

Source of Data

Data Collection Frequency

Measures the process carried out, directly or indirectly, on the participant. A good process indicator is one that is based on activities closely related with the outcome of the performance. These indicators reflect the standards or criteria of adequacy according to the professionals' consensus and from the literature.

Format of the indicator

Institution, person or document from which data are obtained for elaborating the indicators

It measures the frequency in which this process indicator should be evaluated

OUTCOME MEASURE

Indicator description

Source of Data

Data Collection Frequency

Measures the effect of the process on the participant. The grade in which this outcome is influenced by other independent process

Format of the indicator

Institution, person or document from which data are obtained for elaborating the indicators

It measures the frequency with which this process indicator should be evaluated

Table 2

Criteria used to construct and to evaluate the indicators

Criteria used to construct the indicators

1. The indicator measured performance of an intervention or treatment with potential health benefits for the patient

2. The indicator was supported by scientific evidence or professional consensus

3. The indicator was under the control or influence of the care provider or health plan

4. Evidence to evaluate the indicator could be found in the medical record and its absence from the record could be considered a marker for poor quality

Criteria used to evaluate the indicators

Criteria for validity

1. Adequate scientific evidence of professional consensus exists supporting the indicator

2. Identifiable health benefits to patients who receive care specified by the indicator

3. Health professionals with significantly higher rates of adherence to an indicator would be considered higher quality providers

4. Most factors that determine adherence to an indicator are under the control of the health professional (or are subject to influence by the health professional).

Criteria for feasibility

1. The information necessary to determine adherence is likely to be found in a typical medical record

2. Estimates of adherence to the indicator - based on medical record data - are likely to be reliable and unbiased.

3. Failure to document the indicator is itself a marker for poor quality

Notes: Table adapted from Monitoring the Declaration of Commitment of HIV/AIDS: Guidelines on construction of core indicators: 2010 reporting.

Finally, core indicators were selected according to the following criteria: (i) relevance of the indicator (ii) adequacy of the indicator in the AIDS-NGOs context (iii) ability of AIDS-NGOs to change the indicator outcome (iv) feasibility of obtaining the information needed to construct the indicator and (v) activity common to all AIDS-NGOs target group [34, 35].

What new information does this new methodological phase of the study provide?

This new methodological phase, through consensus meetings between AIDS-NGO professionals and the IDIAP Jordi Gol research team, has allowed the elaboration of a Haddon matrix for each target group as well as evaluation indicators for each activity carried out by the AIDS-NGOs with regard to the objective of this activity.

Ethical aspects

The study has been conducted according to Guidelines of the Helsinki Declaration of Good Clinical Research Practice. The project has been approved by the Ethical and Clinical Research Committee of the Institute of Research in Primary Health Care (IDIAP) Jordi Gol.

Informed Consent: The information for the study has been provided orally as well as in writing. Study subjects had sufficient opportunity to ask questions regarding details of the study. Written informed consent following the guidelines of the Helsinki Declaration was obtained.

Data confidentiality: Confidentiality and anonymity of the data have been ensured according to the Spanish law 15/1999 of data confidentiality, both in the implementation phase of the project and in presentations or publications resulting there from. Individual data was encoded to ensure anonymity. Only researchers and monitors have access to the data.

Results

Phase 1-Systematic and exhaustive review of the annual AIDS-NGOs reports

The objectives and activities identified using the review of the reports were aimed at 5 target populations established by the AIDS Program of Catalonia (commercial sex workers, injecting drug users, men who have sex with men, youth in high risk situations, persons living with HIV/AIDS), and 2 more identified target population established for this study (general population and women). Consequently, objectives and activities were collected into seven Haddon matrices.

248 objectives were detected, 118 for Primary Prevention, 44 for Secondary Prevention and 86 for Tertiary Prevention (Figure 2). A total of 242 activities were identified in relation to the objectives in this phase. Some activities were similar or repeated for different objectives and different levels of prevention (Figure 2).
https://static-content.springer.com/image/art%3A10.1186%2F1472-6963-11-176/MediaObjects/12913_2011_Article_1691_Fig2_HTML.jpg
Figure 2

Flow chart from the total objectives and evaluation indicators to core objectives and evaluation indicators.

What new results were obtained?

In this phase we know the number of objectives and activities carried out by the AIDS-NGOs and the relationship between them. We also observe that many activities are common to different target groups, that different activities share the same objective, and vice versa.

Phase 2-Transversal study

The total response to the questionnaires was 72% (26/36). Women accounted for 57.7% of the respondents.

The AIDS-NGO carried out activities for different target population. It was seen that 2/26 of the AIDS-NGOs target their activities to 7 different target groups, 12/26 to 4, 5 or 6 and 11/26 to 2 or 3 target groups.

Regarding the position of the person answering the questionnaire, "coordinator of HIV prevention activities" was the most frequent answer (15/26), followed by "person with most administrative knowledge of the AIDS-NGOs" (11/26). The category "others" was also frequently chosen (9/26). This last category included administrative personnel, managers and individuals responsible for specific programs within the AIDS-NGOs.

Regarding the question about what professionals work in AIDS-NGOs, the answer was that they have interdisciplinary teams, psychologists (64%), social workers (31%), doctors (27%), educators (23%), nurses (17%), and affected volunteers (20%) being the most frequent professional profiles.

What new results were obtained?

The number of identified activities increased to 258 in total, while seventeen activities (6.6%) that were not covered in the annual reports were detected. Three were especially addressed at the primary prevention level (activities that encourage closer ties with NGO services, activities that promote development leisure and interaction activities, legal advice), 3 at secondary prevention (viral load tests, risk pregnancy tests and subsequent referral, detection of tuberculosis and clinical follow-up), 5 for tertiary prevention (individualized psychological care to prison inmates, delivery of diapers to HIV+ families, websites and chats for seropositive individuals, forums and e-mail consultations and at home health care) and 6 activities present at the different levels of prevention (coordination with Spanish, European and African entities that work with HIV+ infants, adolescents and youth, guidelines to improve the doctor-patient relationship, participation in research projects, training courses and continuous education).

Phase 3. A qualitative study with a phenomenological perspective

The main results achieved through the qualitative study were: AIDS-NGOs offered different activities adapted to the needs of the users (people who use AIDS-NGOs' services); The NGO professionals perceived that users were satisfied. The users were satisfied and felt comfortable with the education and health promotion model; the preventive activities of the AIDS-NGOs were based on a participatory health education model adjusted to the people's needs, based on empowerment and focusing on the ideas of knowledge and skills; AIDS-NGOs professionals worked in interdisciplinary groups using a holistic approach with cultural and professional competence; they were flexible, innovative and motivated; and they traveled to where the users are.

Regarding the evaluation of the AIDS-NGOs objectives and activities, the main conclusions were: evaluating the work of AIDS-NGOs is difficult (a concern voiced in most group meetings); some activities require great effort and dedication and quantitative indicators such as the number of cases are not enough to evaluate them; to improve the assessment of the activities and objectives of the AIDS-NGOs, the professionals suggest looking for quantitative and qualitative indicators that provide a reliable measure of the process and the results. More details about results in this phase have been published elsewhere [34].

What new information do the results of this study provide?

AIDS-NGO professionals carry out their activities on a holistic basis, maintaining confidentiality, with professional competence and cultural awareness and respecting the principle of equality. The relationship between AIDS-NGO professionals and users is based on attention, a holistic focus (both socially and in health terms), and user needs.

Users often apply to NGOs for help not only with their basic needs (food, shelter and hygiene) but also for legal aid and assistance with employment and health. Activities aimed at the prevention of HIV-AIDS infection are more effective if they take these basic needs into account. NGOs carry out specific activities aimed at promoting the accessibility of and user links with these associations. NGO professionals form flexible, motivated and innovative interdisciplinary teams ready to go wherever their users are. They require continued training and coordination between the different fields.

NGO professionals recognise the need for an adequate system for the collection of information in order to facilitate information management. They also emphasise the need to support and improve the planning and execution of the current process of the evaluation of objectives and activities. NGO professionals cite perception of a high level of satisfaction among users with regard to the attention received, as well as a lack of recognition of their activities by other professional and institutional groups.

Phase 4. Consensus meetings between the research team and the NGO professionals

Seven Haddon matrices were developed with the results achieved in the three previous phases. In total, 1564 evaluation indicators, 720/1564 of primary prevention, 255/1564 of secondary prevention and 589/1564 of tertiary prevention were proposed (Figure 2). All the information containing the core objectives, activities and indicators are shown in detail in Additional File 1. All the indicators have been published elsewhere [36].

Finally, from all the activities and all the evaluation indicators proposed, the research team selected the core objectives and indicators based on the criteria specified above [34] as shown in table 3 and in Figure 2. Thirty core activities were undertaken: 15 for primary prevention activities (76 indicators), 5 for secondary prevention (43 indicators) and 10 for tertiary prevention (68 indicators) (Table 4).
Table 3

Core objectives of prevention levels by AIDS-NGO

CORE OBJECTIVES

Primary Prevention

Encourage the use of condoms and promotion of proper use

Increase information and prevention of HIV and other STI.

Conduct training and awareness activities for HIV/AIDS and other STI

Contribute to the improvement of the information on HIV and sexual health through various media (telephone, e-mail, internet)

Promote condom and hydro soluble lubricant use as a prevention method

Elaborate informative materials on HIV prevention

Gather written and audiovisual information on initiative studies and process experiences in HIV

Increase public awareness of the epidemic and disseminate information about AIDS

Secondary Prevention

Encourage the early detection of HIV/AIDS and other STI

Encourage the early detection of HIV/AIDS and other STI in the affected person's contacts

Offer medical testing services for the detection of HIV and opportunistic diseases

Tertiary Prevention

Ensure referral to medical services of individuals with positive diagnosis

Provide information and advice on economic, training, social and legal aid.

Provide information and advice on mental health issues

Promote human rights, equality and liberty using cultural mediation

Promote integral incorporation and autonomy in society

Develop mutual help groups of peers and encourage creation of social networks

Training of health and employment specialists

Encourage and foster support for individuals with HIV regardless of their gender

Promote and foster reintegration for seropositive individuals without economic resources

Organize events with broad social impact and including the media (combat stigmatization and achieve normalization)

Contribute to the change of attitudes on treatment adherence and safe sex

Table 4

Haddon matrix of core prevention-control based AIDS-NGOs activities

Prevention Level

Activities intervening on individual factors: Promoting healthy behaviors and attitudes

Intervening on factors related to the immediate

Activities aimed at intervening on factors related to the external environment: infrastructure and population

Activities aimed at intervening on factors related to the social system

Primary Prevention

Activities that encourage closer ties with AIDS-NGO services

 

Activities that encourage closer ties with NGO services

Participation in commemorative acts.

Health promotion and primary prevention (preventing HIV infection)

Promote and development leisure and interaction activities

 

Promote and development leisure and interaction activities

Participation in networking platforms and work groups related to HIV/AIDS

 

Health education and safe sex activities: HIV information and distribution of informative materials

 

Health education and safe sex activities: HIV information and distribution of informative materials

Diffusion and training

 

Educational activities to increase awareness of the male condom, the female condom, lubricants and their correct use

 

Educational activities to increase awareness of the male condom, the female condom, lubricants and their correct use

Training of trainers

 

Distribution of prophylactic material: condoms and lubricants in NGO offices, saunas, etc.

 

Distribution of prophylactic material: condoms and lubricants in NGO offices, saunas, etc.

 
 

Distribution of informative materials on the street, in saunas, pubs, flats, clubs, schools, high schools

 

Distribution of informative materials on the street, in saunas, pubs, flats, clubs, schools, high schools

 
 

Personalized care and information via telephone and email

 

Participation in networking platforms and work groups related to HIV/AIDS

 
 

Editing and preparation of informative material for the prevention of HIV/AIDS

 

Training staff in peer groups

 
 

Legal advice

 

Legal advice

 
 

Workshops and advice on employment

 

Workshops and advice on employment

 

Secondary Prevention

Counselling and rapid testing of HIV and syphilis

Testing for detection of hepatitis A/B/C

Counselling and rapid testing of HIV and syphilis for the couple

  

Measures focusing on early HIV detection

Consultations for STI detection

Referrals for consultations for STI detection

   
 

Consultations for tuberculosis detection

Referrals to Chest Unit for tuberculosis detection

   
 

Information and referral for post-exposure prophylaxis

   

Tertiary Prevention

Receive health services and referrals to other health services, when necessary

   

Attention focused on measures: reducing the effects and promote the rehabilitation reintegration

Promote the adherence to antiretroviral treatments (HAART)

   
 

Conduct emotional support sessions and advice. Conduct individual psychological therapies

   
 

Accompany individuals to medical consultations in order to improvement visits to hospitals and primary care

   
 

Give social and legal aid

 

Give social and legal aid

 
 

Workshops and advice on employment

   
 

Peer help groups, support groups to address HIV

 

Peer help groups, support groups to address HIV

 
 

Conduct alternative medicine workshops: Reiki, Bach flowers, etc.

   
 

Provide flats for HIV+ individuals regardless of their gender

 

Provide flats for HIV+ individuals regardless of their gender

 
 

Telephone attention, internet consultations, chats for seropositive individuals and for serodiscordant couples, forums and consultations via email

   
An example of the core preventive activities and proposed evaluation indicators are described in tables 5, 6 and 7 for each level of prevention. Primary prevention indicators are mainly related to the use of barrier methods for preventing HIV infection and to the increased knowledge of HIV and Sexually Transmitted Infections (STI) (Table 5). Secondary prevention indicators are related to early detection of HIV, STI and opportunistic infections (Table 6). Tertiary prevention indicators are related with comprehensive counselling to achieve effective reintegration of individuals infected with HIV/AIDS (Table 7).
Table 5

Example of proposed indicators for core preventive activities at the primary prevention level

Activities to PROMOTE THE LINK TO NGO SERVICES.

Activities to PROMOTE ACCESSIBILITY THROUGH ON-SITE CARE DURING EXTENDED HOURS

PROCESS MEASURE

Indicator description

Method of Data collection

Data Collection Frequency

Monitoring of days public is served

Days of the year open to the public

NGO

12 months

Monitoring of schedules and days of the week that public is served

Number of hours per day and days of the week open to the public

NGO

12 months

Monitoring of hours open outside of normal schedule

Hours of the day and days of the week open outside of normal schedule

NGO

12 months

Monitoring of number of activity hours for each professional

Hours of activity for each professional (separating volunteers and part time and full time professionals)

NGO

12 months

OUTCOME MEASURE

Indicator description

Method of Data collection

Data Collection Frequency

Monitoring of the individuals attended to in one year

Number of individuals attended in one year

NGO

12 months

Monitoring of the use of the center by the same user

Number of times that the same user uses the centre per year

NGO

12 months

Table 6

Example of a proposed indicators for core preventive activities at the secondary prevention level

COUNSELLING AND RAPID TESTING FOR HIV AND SYPHILIS

PROCESS MEASURE

Indicator description

Method of Data collection

Data Collection Frequency

Monitoring of various media outlets that announce the availability of the test

Number of media outlets announcing the rapid testing/Total number of media outlets used by the association

NGO

12 months

Monitoring of the number of advertisements broadcasted

Number of rapid testing advertisements per month

NGO

12 months

Monitoring number of hours of testing per week

Number of hours of testing per week/Total number of hours of activity per week

NGO

12 months

Availability of equipment and appropriate conditions to conduct rapid testing

Availability of equipment and appropriate conditions to conduct rapid testing

Direct observation

6 months

Availability of personnel trained in giving HIV/AIDS prevention advice personal

Number of professionals trained to give advice regarding rapid testing/Number of professionals that give advice

NGO

12 months

Monitoring of the number of educational materials distributed

Number of educational materials about rapid testing distributed each month

NGO

12 months

Monitoring of the number of meetings and continuing education of personnel that conduct rapid testing

Number of coordination meetings and continuing education between professionals that conduct rapid testing in order to share experiences (analysis of cases, difficulties...)

NGO

12 months

OUTCOME MEASURE

Indicator description

Method of Data collection

Data Collection Frequency

Monitoring of costs to NGOs in promoting rapid testing and the number of applications received

Total costs in the promotion of rapid testing and the number of applications received

NGO

12 months

Monitoring of individuals that solicit appointments for HIV or syphilis rapid testing

Number of individuals that solicit appointments for HIV or syphilis rapid testing, according to sex, age, marital status, and place of origin

NGO

12 months

Monitoring of the number of users that have appointments and come for rapid HIV testing

Number of users that have an appointment and finally come to get tested

NGO

12 months

Monitoring of individuals attended in pre-test, according to sex, age, marital status, sexual preference and place of origin

Number of individuals that are tested, according to sex, age, marital status and place of origin

NGO

12 months

Monitoring of tests conducted per month

Total number of rapid tests conducted per month

NGO

12 months

Monitoring of condoms distributed in CSW.

Number of condoms distributed monthly to people that receive rapid testing

NGO

12 months

Monitoring of the number of rapid testing referrals

Number of people referred to other associations for rapid testing

NGO

12 months

Monitoring of individuals that return to collect syphilis test results

Number of individuals that return to collect syphilis test results/Total number of individuals that are tested for syphilis

NGO

12 months

Monitoring of individuals that return to collect HIV test results

Number of individuals that return to collect an HIV test results/Total number of individuals that are tested for HIV

NGO

12 months

Monitoring of positive syphilis test results

Number of individuals with a positive syphilis test result/Number of individuals that return to collect syphilis test results

NGO

12 months

Monitoring of positive HIV test results

Total number of users with a positive HIV test result/Number of users that return to collect an HIV test results

NGO

12 months

Level of satisfaction with the services

Number of rapid test service users that have responded to the satisfaction questionnaire and have high, moderate or low satisfaction/Total number of rapid test service users

Users service questionnaire

12 months

Table 7

Example of proposed indicators for core preventive activities at the tertiary prevention level

Guarantee the follow-up and referral to medical services of individuals with a positive diagnosis

PROCESS MEASURE

Indicator description

Method of Data collection

Data Collection Frequency

Monitoring of the number of individuals infected with HIV that have received medical care

Number of individuals infected with HIV that have received medical care/Number of individuals infected with HIV attended and that have received health care by the NGO in the evaluation year

NGO

12 months

Monitoring of the number of individuals infected with HIV that are referred to other health services

Number of individuals infected with HIV that are referred to other health services/Number of individuals infected with HIV attended and that have received health care by the NGO in the evaluation year

NGO

12 months

Monitoring of the number of individuals referred to each service

Numbers of individuals referred to each service

NGO

12 months

Monitoring of the number of people working in a service

Number of people working in a service

NGO

12 months

Monitoring of the hours dedicated to each activity

Hours dedicated to each activity

NGO

12 months

OUTCOME MEASURE

Indicator description

Method of Data collection

Data Collection Frequency

Level of satisfaction with the service

Number of individuals infected with HIV that have received care, have responded to the satisfaction questionnaire, and have a high level of satisfaction/Number of infected individuals that have received care and have responded to the questionnaire

NGO

Answers to the satisfaction questionnaire

12 months

Monitoring of the questionnaire of quality of life answers of individuals attended by the NGO

Descriptive analysis and monitoring of the global score and score by dimensions of the questionnaire of quality of life answers of individuals infected with HIV and attended by the NGO

NGO

Answers to the QQV

12 months

All the core evaluation indicators proposed can be seen in Additional Files 1.

Activities were carried out to achieve all of the proposed objectives by the various AIDS-NGO, although there were some objectives covered by more than one activity.

Discussion

The present study followed a mixed method strategy that contributes to improving knowledge of activities undertaken by AIDS-NGOs and serves as a basis from which to develop a final proposal of indicators to assess these prevention-control based activities. As shown in the results, this methodological strategy implies that the different phases complement one another and progressively richer information is obtained [20, 37].

The results allowed us to know all the preventive-control based activities done by AIDS-NGOs in Catalonia and how they could be assessed. The first two phases showed us a relationship between objectives and activities, and what other activities were not registered in the AIDS-NGOs annual reports. The qualitative phase showed how the preventive-control activities are done and what the main barriers and limitations to evaluate it are. Finally, the indicators consensus meetings enabled us to talk with AIDS-NGOs professionals and establish a consensus on the indicators proposed to evaluate the activities in the daily work flow.

The preventive activities of the AIDS-NGO are based on a participatory health education model adjusted to the people's needs, based on empowerment and focusing on the ideas of knowledge and skill. This education strategy follows the principles put forward by the WHO [38].

Results emphasise the importance of offering ancillary services [39] to people in need of HIV-AIDS prevention or treatment. Another study showed that the holistic person-based approach is essential to achieve a change in behaviour [40]. On the other hand, the WHO Report 2008 [38] underlines the impact of person-based care in health improvement, quality of life, user trust and treatment adherence.

In this relationship model, the professional takes into account the values and perspective of the user, and therefore incorporates them in the decision-making process [41]. The applicability of specific programmes requires a community approach to adjust them to match the needs of the target population [39, 42].

The results confirm that the AIDS-NGOs perform most of their activities in relation to target groups at risk of social exclusion or because they are socially vulnerable. Moreover, it reduces social inequalities due to socioeconomic status, gender and social orientation. It also confirms the role of NGO-AIDS as a bridge acting as a "communications space" between health services and the population) and also with other services (legal, social, employment, etc.).

Evaluation is a continuous process that facilitates the identification of areas for improvement. It should also contribute to the recognition of tasks that have been satisfactorily carried out by professionals. The professionals belonging to the participating AIDS-NGOs of this study consider evaluation to be an activity which is relevant to them and share the evaluation needs. Therefore, there is concordance between the necessities detected by the HIV-AIDS program (Public Health Department) and those identified through a revision of international publications [1, 4345].

In order to evaluate the objectives and activities of AIDS-NGOs, it is essential to use good indicators bringing together characteristics such as: acceptability, feasibility, reliability, sensitivity to change, validity and meaningful and possible communicability [19, 46].

The latest update of the "Compendium of Evidence-Based HIV Prevention Interventions", elaborated by the CDC, as well as the "Community guide" website, both show a broad range of evidence-based preventive activities. However, these activities are included in programs aimed at very specific groups (female condom skills training; many men, many voices; personalized cognitive risk-reduction counseling), while our study presents a broader evaluation and suggests multiple indicators to evaluate concrete activities, which can be applicable to diverse intervention programs and to different groups. Thus, this proposal of indicators can help provide new evidence about the effectiveness of specific activities according to the level of prevention and intervention [47].

Through the development of the first phase of this study, two more target groups of population were identified: inmates and immigrants. These two target populations had specific characteristics deserving of special consideration and their own preventive programs on the part of the AIDS-NGOs, and they were added as target populations for purposes of this study. Specific objectives and activities addressed to these groups were identified, and specific evaluation indicators proposed through the seven final Haddon matrices, available via the following link (http://www.gencat.cat/salut/depsalut/html/ca/dir2068/informefinal_actongsida2009.pdf) [36].

Our proposal agrees with some of the quality measures of HIV care proposed by the CDC and UNAIDS [1, 45, 47]. What is new and constitutes a strong point in our study is that we have added other quality indicators that measure user satisfaction (an essential aspect of the evaluation of service quality) and more psychosocial factors (legal advice) that can have an impact on the fight against inequalities and stigmatization [6, 48].

Another important aspect to note is that, by using the Haddon matrix, many preventive interventions and evaluation indicators have been identified and defined on the individual, external and social levels, while less have been directed towards the individual's immediate environment. These findings are consistent with those observed by Peiró et al [12]. A possible interpretation of this phenomenon is the concern of AIDS-NGOs professionals about data confidentiality; in order to act in the immediate environment of the affected person, he or she must agree to share the problem with individuals in his or her intimate environment.

The consensus phase among AIDS-NGOs professionals and the research team to develop the indicators has meant the cooperation and participation of the professionals in this study and constitutes a key element in facilitating the acceptance and implementation of the improvements proposed in the evaluation. It is also worth mentioning that professionals belonging to the AIDS-NGOs that participated in the consensus phase make up a multidisciplinary team much like the AIDS-NGOs. This gives the proposed indicators wide variability and richness, encompassing all of the activities undertaken by the AIDS-NGOs [34, 35, 45].

The main limitation of this proposal of indicators is that it has not been applied. It would be important to share and disseminate it among professionals of the various programs and organizations that work with HIV in order to test its applicability, usefulness, validity and practical relevance.

A possible selection bias could also be argued, given that these indicators have been designed with the participation of professionals from AIDS-NGOs that work in Catalonia and are financed by this autonomous community's Department of Health. Nonetheless, as mentioned above, the enormous variability in the number of activities and proposed evaluation indicators minimizes this bias and raises their potential usefulness for AIDS-NGOs located in other geographical areas. Furthermore, one limitation of the study was not to have studied the views of other health providers that work with HIV-AIDS prevention or control-programmes (Primary Care, Public Health, Reproductive and Sexual Health, Faith-Based Organizations and Community Based Organizations). Although their contributions would have been of great interest, for reasons of practicality and insufficient resources this aspect could not be included.

We also want to mention that indicators for global results such as HIV mortality, rate of hospitalizations and approach to other health problems (diabetes, hypertension...) have not been proposed. However, the improvements in these epidemiological indicators can be attributed to multiple factors (improvements in security and effectiveness of drug treatments, healthcare, etc.) and not only to the AIDS-NGOs' activities. Nonetheless, studies that directly involve classic systems of epidemiological surveillance for HIV-AIDS would be of interest.

For the selection of indicators in future evaluations of preventive activities in AIDS-NGOs, the elements that make up the third dimension of the Haddon matrix should be taken into consideration: effectiveness, cost, freedom, equity, stigmatization, preferences of the affected community or individual and viability. These factors can be considered from different points of view and can have greater or lesser weight when selecting indicators, based on the general objectives of future health policies and on HIV-AIDS [12, 15, 16].

It is also important that the strategic plan of future HIV-AIDS policies emphasize activities that address the population that makes up the affected or at-risk person's immediate environment [12]. Although activities directed towards the social environment can have an indirect impact and their execution can be more difficult in some cases, it is necessary to emphasize the key role of the AIDS-NGOs in the development of these activities and the important social impact they can have.

Conclusions

This study showed the importance of methodological combination for a better understanding of results and the mechanisms of changes in the evaluation of AIDS-NGO activities. Although the study had some limitations, the combination of quantitative and qualitative data gave the opportunity to construct core evaluation activities and the corresponding indicators. The results could help to homogeneously assess the preventive-control activities carried out by several AIDS-NGOs. Moreover, the indicators could help the professionals to understand and therefore to improve the evaluation of the preventive-control AIDS-NGO activities. The Haddon matrix enables us to detect the gaps in intervention levels and, as a consequence, to keep in mind strategies that could enhance AIDS programs in the future. A further important message of the research was that qualitative approaches are essential in order to identify underlying processes and to detect activity procedures in order to better define the indicators.

Declarations

Acknowledgements

This study has been funded by the AIDS Programme, Public Health Department of the Ministry of Health, Government of Catalonia. We would like to thank all the AIDS-NGO professionals, and in particularly Kati Zaragoza, Anna Rafael, Ricardo Fuertes, Alba Mestre and Anna Lara, who took part in the consensus meetings. We specially acknowledge all users who have taken part in the qualitative study. We thank the IDIAP Jordi Gol for the funding of the study and translation of the manuscript into English.

Authors’ Affiliations

(1)
Research Department, Primary Health Care Research Institute, (IDIAP-Jordi Gol), Catalan Health Institute (ICS), (Gran Via de les Corts Catalanes 587 àtic)
(2)
Primary Health Department Costa de Ponent, Catalan Health Institute (ICS), IDIAP-Jordi Gol, Gerència Territorial Metropolitana Sud, Hospital Duran i Reynals
(3)
AIDS Programme. Public Health Department, Ministry of Health. Government of Catalonia
(4)
CIBER, Epidemiology and Public Health (CIBERESP), Instituto de Salud Carlos III
(5)
Public Health Department, Medicine Faculty, Barcelona University

References

  1. UNAIDS: Monitoring the Declaration of Commitment of HIV/AIDS: Guidelines on construction of core indicators: 2010 reporting. 2010, Geneva: UNAIDS
  2. UNAIDS: 2008. Report on the global AIDS epidemic. 2009
  3. Anuari estadístic de Catalunya. 2010 Població. 1900-2010. Províncies. 2011, Sèrie temporal. Institut d'Estadística de Catalunya Ed. Jule 2010 Barcelona, ISSN: 1130-166X. Available in http://www.idescat.cat/pub/?id=aec&n=245
  4. de SR: Creating "communicative spaces": a case of NGO community organizing for HIV/AIDS prevention. Health Commun. 2009, 24: 692-702. 10.1080/10410230903264006.View Article
  5. Crane SF, Carswell JW: A review and assessment of non-governmental organization-based STD/AIDS education and prevention projects for marginalized groups. Health Educ Res. 1992, 7: 175-194. 10.1093/her/7.2.175.View ArticlePubMed
  6. Kelly JA, Somlai AM, Benotsch EG, Amirkhanian YA, Fernandez MI, Stevenson LY, Sitzler CA, McAuliffe TL, Brown KD, Opgenorth KM: Programmes, resources, and needs of HIV-prevention nongovernmental organizations (NGOs) in Africa, Central/Eastern Europe and Central Asia, Latin America and the Caribbean. AIDS Care. 2006, 18: 12-21. 10.1080/09540120500101757.PubMed CentralView ArticlePubMed
  7. Sehgal PN: Prevention and control of AIDS: the role of NGOs. Health Millions. 1991, 17: 31-33.PubMed
  8. Generalitat de Catalunya: Pla de salut de Catalunya a l'horitzó 2010. Els 5 eixos estratègics que articulen les polítiques de salut. Servei del Pla de Salut. 2010, Departament de salut. 2007, Accessible a: http://www.gencat.cat
  9. Plan Nacional sobre el Sida, Grupo de trabajo sobre prevención del VIH/Sida y otras ITS en personas que ejercen la prostitución: Prevención del VIH/Sida y otras infecciones de transmisión sexual en personas que ejercen la prostitución: Elementos clave para el desarrollo de programas. 2003, Ministerio de Sanidad y Consumo, 2003. Madrid
  10. RESOLUCIÓ SLT/1948/2007, de 21 de juny per la qual s'adjudiquen les subvencions per finançar activitats de prevenció de la infecció pel VIH/Sida a entitats de caràcter no lucratiu corresponents a l'any 2007. 2010, (DOGC núm. 4915 de 29/6/2007, pàg. 21947)
  11. Sullivan PS, Hamouda O, Delpech V, Geduld JE, Prejean J, Semaille C, Kaldor J, Folch C, Op de Coul E, Marcus U, Hughes G, Archibald CP, Cazein F, McDonald A, Casabona J, van Sighem A, Fenton KA: Reemergence of the HIV epidemic among men who have sex with men in North America, Western Europe, and Australia, 1996-2005. Ann Epidemiol. 2009, 19: 423-431. 10.1016/j.annepidem.2009.03.004.View ArticlePubMed
  12. Peiró R, Alvarez-Dardet C, Plasencia A, Borrell C, Colomer C, Moya C, Pasarín MI, Zafra E: Rapid appraisal methodology for 'health for all' policy formulation analysis. Health Policy. 2002, 62: 309-328. 10.1016/S0168-8510(02)00053-2.View ArticlePubMed
  13. Generalitat Valenciana: Plan de Salud de la Comunidad Valenciana 2005-2009. 2010, Conselleria de Salut. 2006, ISBN: 84-482-4365-X
  14. Barnett DJ, Balicer RD, Blodgett D, Fews AL, Parker CL, Links JM: The application of the Haddon matrix to public health readiness and response planning. Environ Health Perspect. 2005, 113: 561-566.PubMed CentralPubMed
  15. Runyan CW: Using the Haddon matrix: introducing the third dimension. Inj Prev. 1998, 4: 302-307. 10.1136/ip.4.4.302.PubMed CentralView ArticlePubMed
  16. Runyan CW: Introduction: back to the future--revisiting Haddon's conceptualization of injury epidemiology and prevention. Epidemiol Rev. 2003, 25: 60-64. 10.1093/epirev/mxg005.View ArticlePubMed
  17. Dirección General de Salud Pública y Participación: Evaluación del Programa de Prevención y Atención a las Personas Afectadas por el VIH-sida en Asturias 2003-2008. 2010, Consejería de Salud y Servicios Sanitarios. Asturias, 2009
  18. García Ruiz M, Suárez Álvarez O: Programa de Educación Sexual y Prevención del VIH en Centros de Menores. Experiencias en Asturias y País Vasco. 2000, Consejería de Salud y Servicios Sanitarios del principado de Astúrias. edn
  19. NHS Institute for Innovation and improvement: The good indicators guide: Understanding how to use and choose indicators. 2011, Coventry: 2008
  20. Bryman A: Mixed methods research: combining quantitative and qualitative research. In Social Research Methods. 2011, Oxford: Oxford University Press; 2008
  21. Lingard L, Albert M, Levinson W: Grounded theory, mixed methods, and action research. BMJ. 2008, 337: a567-10.1136/bmj.39602.690162.47.View ArticlePubMed
  22. O'Cathain A, Murphy E, Nicholl J: Three techniques for integrating data in mixed methods studies. BMJ. 2010, 341: c4587-10.1136/bmj.c4587.View ArticlePubMed
  23. Sandelowski M: Combining qualitative and quantitative sampling, data collection, and analysis techniques in mixed-method studies. Res Nurs Health. 2000, 23: 246-255. 10.1002/1098-240X(200006)23:3<246::AID-NUR9>3.0.CO;2-H.View ArticlePubMed
  24. Herdman M, Fox-Rushby J, Badia X: A model of equivalence in the cultural adaptation of HRQoL instruments: the universalist approach. Qual Life Res. 1998, 7: 323-335. 10.1023/A:1008846618880.View ArticlePubMed
  25. Valderas JM, Ferrer M, Alonso J: [Health-related quality of life instrumentsand other patient-reported outcomes]. Med Clin (Barc). 2005, 56-60.
  26. Tuckett AG: Qualitative research sampling: the very real complexities. Nurse Res. 2004, 12: 47-61.View ArticlePubMed
  27. Tong A, Sainsbury P, Craig J: Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007, 19: 349-357. 10.1093/intqhc/mzm042.View ArticlePubMed
  28. Krueger R, Casey MA: Focus Group: A Practical Guide for Applied Research. 2011, London: Sage Publications; 2000
  29. Conde F: Los grupos triangulares como espacios transicionales para la producción discursiva. La vivienda en Huelva. Culturas e identidades urbanas. 1996, Sevilla: Junta de Andalucia-Fundación El Monte, 275-307. 2011
  30. Bryman A: Social Research Methods. 2008, Oxford: Oxford University Press; 2008, 3
  31. Gilbert N: Researching Social life. 2011, London: Sage Publications; 2001, 2
  32. Berenguera A, Pujol-Ribera E, Violan C, Romaguera A, Mansilla R, Giménez A, et al: Experiences about HIV-AIDS preventive-control activities. Discourses from non-governmental organizations professionals and users: qualitative study. Gac Sanit. 2011, 25 (3): 184-190. 10.1016/j.gaceta.2010.10.015.View ArticlePubMed
  33. Asch SM, McGlynn EA, Hogan MM, Hayward RA, Shekelle P, Rubenstein L, et al: Comparison of quality of care for patients in the Veterans Health Administration and patients in a national sample. Ann Intern Med. 2004, 141: 938-945.View ArticlePubMed
  34. Plaza TA, Guarga RA, Farres QJ, Zara YC: [Consensus on a process of benchmarking in primary care in Barcelona]. Aten Primaria. 2005, 35: 130-139. 10.1157/13071938.View Article
  35. Plaza TA, Zara YC, Guarga RA, Farres QJ: [Results of a benchmarking exercise for primary care teams in Barcelona, Spain]. Aten Primaria. 2005, 35: 122-127. 10.1157/13071936.View Article
  36. Almeda J, Berenguera A, Pujol-Ribera E, Romaguera A, Violan C: Proposta d'indicadors d'avaluació de les activitats de les ONG-SIDA. 2008
  37. Popay J: Qualitative research and the epidemiological imagination: a vital relationship. Gac Sanit. 2003, 17 (Suppl 3): 58-63.View ArticlePubMed
  38. World Health Organization: The World Health Report 2008 - primary Health Care (Now More Than Ever). 2008
  39. Conviser R: Catalyzing system changes to make HIV care more accessible. J Health Care Poor Underserved. 2007, 18: 224-243.View ArticlePubMed
  40. Estrada JH: Prevention models in the fight against HIV/AIDS. Acta Bioethica. 2006, 12: 91-100.View Article
  41. Ruiz MR, Rodriguez JJ, Epstein R: [What style of consultation with my patients should I adopt? Practical reflections on the doctor-patient relationship]. Aten Primaria. 2003, 32: 594-602.View Article
  42. Rhodes SD, Hergenrather KC, Wilkin A, Alegria-Ortega J, Montano J: Preventing HIV infection among young immigrant Latino men: results from focus groups using community-based participatory research. J Natl Med Assoc. 2006, 98: 564-573.PubMed CentralPubMed
  43. de Souza EM: Evaluation methods in health promotion programmes: the description of a triangulation in Brazil. Cien Saude Colet. 2010, 15: 2521-2532. 10.1590/S1413-81232010000500026.View ArticlePubMed
  44. Duber HC, Coates TJ, Szekeras G, Kaji AH, Lewis RJ: Is there an association between PEPFAR funding and improvement in national health indicators in Africa? A retrospective study. J Int AIDS Soc. 2010, 13: 21-10.1186/1758-2652-13-21.PubMed CentralView ArticlePubMed
  45. Wilson IB, Landon BE, Hirschhorn LR, McInnes K, Ding L, Marsden PV, et al: Quality of HIV care provided by nurse practitioners, physician assistants, and physicians. Ann Intern Med. 2005, 143: 729-736.View ArticlePubMed
  46. Campbell SM, Braspenning J, Hutchinson A, Marshall MN: Research methods used in developing and applying quality indicators in primary care. BMJ. 2003, 326: 816-819. 10.1136/bmj.326.7393.816.PubMed CentralView ArticlePubMed
  47. CDC 2009 Compendium of Evidence-Based Interventions. 2009, [http://www.cdc.gov/hiv/topics/research/prs/]
  48. Kelly JA, Somlai AM, Benotsch EG, McAuliffe TL, Amirkhanian YA, Brown KD, et al: Distance communication transfer of HIV prevention interventions to service providers. Science. 2004, 305: 1953-1955. 10.1126/science.1100733.View ArticlePubMed
  49. Pre-publication history

    1. The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/11/176/prepub

Copyright

© Berenguera et al; licensee BioMed Central Ltd. 2011

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.