Skip to main content

Development and validation of the adolescent behavioural change Counselling Assessment Tool in Indonesia



Primary care provides an important context to engage adolescents and their families in healthy lifestyles with the goal of reducing future behaviour-related health problems. Developing a valid tool to assess health professionals’ skills in behavioural change counselling is integral to improving the quality of clinical care provided to adolescents in Indonesia.


This work was nested within a project to develop a training program to enhance the behaviour-change counselling of adolescents and their parents by Indonesian primary care professionals. Initial item development was based on the content of the training module and the domain structure of the Behavioral Change Counselling Index (BECCI), a commonly used tool to assess counselling quality in healthcare settings. Expert panels were used to test content validity, while face validity was assessed by a group of trained psychologists. Inter-rater agreement was calculated prior to tests of construct validity and reliability, which involved psychologists rating 125 audio-taped counselling sessions between the health professional and adolescent patients, together with a parent.


An initial 13-item tool was developed using a 1–5 Likert scale. Validity and reliability testing resulted in the decision to use a 14-item tool with a 0–3 Likert scale. The scale was found to have a Cronbach’s α coefficient of 0.839 (internal consistency), and there was strong inter-rater agreement (0.931).


The assessment tool known as the Adolescent Behavioural Change Counselling Assessment Tool, is a valid and reliable instrument to measure Indonesian health professionals’ behavioural-change counselling skills with adolescent patients. The tool provides an evaluation framework for future interventions that aim to improve health professionals’ skills in addressing adolescent behaviour-related health problems.

Peer Review reports


Adolescence is a time of great developmental opportunity, shaped by socioeconomic resources, education, cultural norms, family values, and peer contexts. These same factors influence behavioural patterns which contribute to much adolescent morbidity and mortality. This includes behaviours that contribute to unintentional injury, exposure to violence, sexual health outcomes including sexually transmitted infections and unplanned pregnancy, smoking and substance use, unhealthy eating habits and sedentary lifestyles, each of which carry risks to adolescents’ health and well-being [1, 2]. These behaviours also contribute to adult health. For example, each year 15 million people die prematurely between the ages of 30-69-years-old from non-communicable diseases (NCDs), with over 85% occurring in low- and middle-income countries (LMICs) [3]. In Indonesia, NCDs contribute a major proportion of the country’s adult health burden [4].

Risky behaviours can be identified, managed and monitored [1]. Behaviour counselling interventions have been found to be associated with reduction in risky behaviours without evidence of unintended harmful effects [5, 6]. Despite recognition of the significant short- and long-term impacts of behaviour-related health problems, and the availability of effective interventions, only about a third of adolescents with a diagnosable behaviour-related health disorder receive appropriate care [7], consistent with evidence that counselling interventions to address behaviour-related health problems are underutilized in healthcare settings [7, 8].

Many health professionals are poorly prepared to provide counselling to address behaviour-related health problems, especially in LMICs [8, 9]. In 2018, a national review in Indonesia revealed that poor counselling quality was a major weakness within health services for adolescents [10]. One approach to improve the identification and treatment of behaviour-related health problems within the health care system is to integrate behavioural health services into medical settings [7]. In Indonesia behaviour-change counselling techniques have not yet been included in the majority of health professional training curricula [11, 12] and there are no available tools to assess counselling quality. We recently developed a training program to promote the capacity of primary care providers to provide behavioural-change counselling to adolescent patients. In this paper, we describe the development and validation of an assessment tool, the Adolescent Behavioural Change Counselling Assessment Tool (ABC-CAT) to evaluate the effectiveness of the clinical training.

Materials and methods

Setting and context

An internet-based training to improve health professionals’ counselling skills was developed as an initiative to provide more accessible training on adolescent health within primary health services in Indonesia. The clinical training module was designed in three stages. Firstly, we undertook a literature review using a series of keywords such as weight management counselling, obesity prevention, behaviour change counselling, motivational interviewing (MI) and parenting in adolescents. Secondly, we undertook a qualitative study of adolescents and parents from different socio-demographic backgrounds in three provinces of Indonesia to explore barriers to healthy eating in the home [13]. These two steps informed the development of the initial training material. In the third step, we engaged with relevant professional organisations in Indonesia, the Ministry of Health (Republic of Indonesia) and individual adolescent health practitioners to review and refine the training material.

The final training module comprised sections on adolescent growth and development, healthy lifestyles, parenting, how to start a counselling session with adolescent patients, psychosocial screening, behaviour change principles and stages, motivational interviewing, and how to work with parents to promote their ability to foster their children make healthy choices. The development of the training material used constructive alignment theory, an integrative design for teaching in which the alignment between intended learning outcomes, teaching and learning activities, and assessment tasks is emphasized [14].

The assessment tool was developed to evaluate the counselling skills that were taught as part of this training program. Although the training was explicitly aimed at obesity prevention, the principles of the behaviour-change counselling skills were intended to be applicable to any behaviour-related health problem. The training and the assessment tool were developed in the Indonesian language (Bahasa Indonesia).

Item development

The first step in the development of the assessment tool was to identify the key aspects of behaviour-change counselling skills that were taught within the training material [14]. The training was based on motivational interviewing (MI), a specific behaviour-change counselling approach that is widely used within health services, including for adolescent patients in primary care [15,16,17]. A number of assessment tools have been developed to assess the quality of counselling following MI training. Among the most widely used is the Behavioral Change Counselling Index (BECCI) which has been shown to have a Cronbach α-coefficient of 0.71 [18,19,20].. We initially created a 13-item measure that used the 11-item BECCI for items related to MI counselling techniques, to which we added two items that corresponded to the more adolescent-specific aspects of the training (psychosocial screening and parental involvement). All questions used a Likert scale response rating of 1–5. The conceptual framework for the development of the assessment tool and its constructs is shown in Fig. 1.

Fig. 1
figure 1

Conceptual framework for construct development of the assessment tool


A questionnaire requires tests of validity and reliablity to evaluate its accuracy and consistency. Validity expresses the degree to which the measurement of something (in this case, a survey or questionnaire) measures what it purports to. We assessed two major types of validity: content validity and construct validity. Content validity is the extent to which a questionnaire includes the most relevant and important aspects of a concept in the context of a given measurement application [21]. Face validity, an aspect of content validity, is the ability of an instrument to be understandable and relevant to the targeted user [21, 22]. Construct validity is the degree to which an instrument measures the theoretical construct that it is intended to measure. Reliability concerns the extent to which the measurement of a phenomenon provides stable or consistent results [21].

Content validity

Content validity was undertaken by an expert panel of 16 people that consisted of: representatives of the Indonesian Pediatric Society, the Indonesian Family Doctors Association, and the Indonesian Clinical Psychologist Association; six trained adolescent health care providers (physicians and nurses); two clinical psychologists (one child and adolescent psychologist and one MI trained psychologist); two paediatricians (one with particular expertise in child growth and development and one with specific expertise in nutrition and metabolic diseases); and one sports medicine specialist. Each member of the expert panel scored (0–4) the appropriateness of each item in the context of the training module, which provided the content validity index (CVI). Focus group discussions were then conducted with the expert panel, in which each item was discussed with the goal of improving the overall assessment tool.

Face validity

Face validity was assessed by the ability of the ABC-CAT to be understandable and relevant for clinical psychologists, who we used to evaluate the tool. In Indonesia, clinical psychologists are the only health professionals trained on MI. Using the Indonesian young clinical psychologist network, we recruited eight clinical psychologists who had been trained in MI. The clinical psychologists used the assessment tool to rate audiotaped consultations of counselling sessions of primary care professionals and adolescent patients and their parents. The intention of the validation was to confirm that there was clear understanding of each item and each assessment description by this group of professionals. Previously, the psychologist team had been given access to the online training modules and all training materials on the training website. After discussing the module content and the pedagogical intention of the training, the psychologist team conducted a reading test of the ABC-CAT. Following this, they individually used the assessment tool to evaluate the same two audio-taped counselling sessions. These then formed the basis of focus group discussions to refine the assessment tool [22].

Construct validity and internal reliability

We completed an inter-rater reliability test to ensure that there was acceptable agreement between all raters (the eight trained clinical psychologists) for each of the assessment items. For this purpose, an inter-class correlation coefficient (ICC) test was conducted using a sample size of 15 audio-taped counselling sessions, based on the simplified Winer and Walter formula which is sufficient to detect a strong level of agreement (above 0.7) [23]. Construct validity was established by corrected item-total correlation analysis of a sample size of 125, which was obtained from audio-taped counselling sessions of adolescent health practitioners from 17 provinces across Indonesia [24]. Items with a correlation coefficient less than 0.3 were omitted [25]. The internal consistency (reliability) was then examined using the Cronbach α coefficient, for which a coefficient of 0.7 or higher is considered reliable [25].


The Health Research Ethics Committee - Faculty of Medicine Universitas Indonesia and Cipto Mangunkusumo Hospital approved this study (approval number 829a/UNZ.F1/ETIK/PPM.00.02/2021). All participants gave written informed consent prior to their participation.


Content validation

All 16 experts gave a score of 3 or 4 (appropriate or very appropriate) for the 13 assessment items which resulted in a total CVI score of 1. Discussion of each item within the subsequent focus groups resulted in two major changes. Firstly, a new item was added, namely “provided adolescents with the opportunity to be seen alone and made a confidentiality statement”. Secondly, the original Likert 1–5 scale was replaced with a simpler 0–3 scale. Finally, minor changes to the explanation for each score on each assessment item were made to more clearly differentiate a continuum of performance levels.

This process resulted in the assessment tool (ABC-CAT version 2.0) having 14 items across four sub-themes. The four sub-themes are the opening of a session, psychosocial screening, the specific behaviour-change counselling approach (MI) and parental involvement (see Table 1).

Table 1 The domain structure of the BECCI* and ABC-CAT**

Following the face validation process with the group of eight clinical psychologists, revisions were made to four items. Further revisions were also made to each of the assessment descriptions for each score for all 14 items. The reading level assessment also led to some simplification of language. This final assessment tool (ABC-CAT version 3.0) (see Table 2) then underwent tests of construct validity and internal reliability.

Table 2 Final scoring rubric for the Adolescent Behaviour Change– Counselling Assessment Tool (ABC– CAT) and inter-item correlation score for each item

Construct validity and internal reliability

The mean value of the ICC measurement of the eight psychologists who rated 15 audio-taped counselling sessions was 0.931 (CI 0.868–0.973), indicating strong agreement. Using the 125 counselling audiotapes, the construct validation test of this assessment tool obtained a Cronbach value of 0.839. The correlation coefficient values were > 0.3 for all but two assessment items (items 1 and 14). To preserve the content validity of the final assessment tool, both items were retained after some minor revision to wording.

The overall process for developing the assessment tool is summarised in Fig. 2.

Fig. 2
figure 2

The processes and measurements tested with each version of the ABC-CAT


Within the tool, the assessment of four themes (the opening, psychosocial screening, specific behaviour-change counselling approach, and parental involvement) is consistent with our earlier identification of these themes as relevant for training in behavioural counselling. A focus on psychosocial screening is consistent with recommendations for counselling adolescents with behaviour-related health problems [16, 26, 27] and the foundational communication requirements with adolescent patients [28, 29]. Parental involvement is another distinctive element of this counselling training as parents critically influence adolescent growth and development, including adolescent behaviours [30, 31].

The careful development and validation process we undertook resulted in a series of changes to the assessment tool at each stage of its development. In particular, the face validity process resulted in changes to the scoring explanation of each item in response to evidence that our earlier explanations for scoring were at times unclear or ambiguous. This included changes to the Likert response scale, consistent with evidence that detailed, specific explanations of scoring for each item improves a tool’s performance [32, 33]. Each rater provided their reason for scoring each item, which informed the simplification of the scale from a 5-point to 3-point scale) and its response options, which ended up having more clear and concise definitions of each score for each item [33]. This is reflected in substantially better inter-rater reliability and internal consistency in the latter versions of the tool. Moreover, the performance level descriptions featured criteria that were more appropriate with the assessment’s purpose [34]. The final 14-item assessment tool revealed acceptable reliability (internal consistency) using Cronbach α scores. While cut-off values for questionnaire reliability are recognised to vary according to the field of application, a Cronbach α value ranging between 0.65 and 0.8 is typically considered adequate [35]. Comparison with other assessment tool is challenging due to differences with previously validated tools around the issues covered within the assessment tool, the number of items and type of questions, and the target population [36].

This assessment tool was used to assess behavioral change counselling for weight management in adolescents. While MI was initially developed for adults with substance use disorders, the application of MI has widely expanded to include other areas of behavior-related health and disease management such as adherence with medication for those with chronic diseases requiring long-term treatment adherence [16]. The general approach to MI has also expanded beyond clinical care to include preventive health measures such us developing healthy lifestyles and immunization programs [6, 16]. While MI was initially developed in adult populations, there is a large body of evidence about its relevance with adolescents [6, 37]. Other MI based assessment tools have not included specific items on the more generic aspects of counselling with adolescents. In particular, behaviour change for adolescents typically requires parent involvement due to their critical role in creating enabling home environments for healthy growth and development [13, 26, 27]. Notwithstanding its use in evaluating this training program that had a particular focus on weight, there is every expectation that this assessment tool could also be utilized to appraise the quality of counselling for other behavior-related health problems with adolescent patients.

A strength of this validation is that it was based on the assessments of clinical psychologists who understood basic communication skills with adolescent patients, the MI approach and the importance of involving parents in behavioral-change counselling with adolescents. The robust validity we have demonstrated no doubt reflects these skill sets. It is unknown to what extent less experienced professionals would achieve the same results, although it is anticipated that the development of an accompanying assessment guidance that includes real-life examples to guide evaluation may enhance consistent performance of the tool. Regardless, a current limitation is that the tool can only be used by trained raters who understand MI and the adolescent counselling principles covered in the assessment items.


In sum, this study provides the first brief, feasible and validated assessment tool to evaluate health professionals’ skills in behaviour-change counselling for adolescent patients in Indonesia. Prior to expanding its use, the development of written guidance would be an important strategy to ensure the validity of future assessment. Future research is needed to assess the value of this assessment tool in research that aims to assess health professionals’ skills in working with adolescents on behavioral-change counselling beyond those relevant to overweight.

Data availability

Data and materials used for this study are available from the corresponding author upon request.



Non-communicable diseases


Content validity index


Inter-class correlation coefficient


Motivational Interviewing


  1. Bozzini AB, Bauer A, Maruyama J, Simões R, Matijasevich A. Factors associated with risk behaviors in adolescence: a systematic review. Brazilian J Psychiatry. 2020;43:210–21.

    Article  Google Scholar 

  2. Mokdad AH, Forouzanfar MH, Daoud F, Mokdad AA, El Bcheraoui C, Moradi-Lakeh M, et al. Global burden of diseases, injuries, and risk factors for young people’s health during 1990–2013: a systematic analysis for the global burden of Disease Study 2013. The Lancet. 2016;387(10036):2383–401.

    Article  Google Scholar 

  3. Organization WH. Non-communicable disease prevention and control: a guidance note for investment cases. World Health Organization; 2019.

  4. Marthias T, Anindya K, Ng N, McPake B, Atun R, Arfyanto H, et al. Impact of non-communicable disease multimorbidity on health service use, catastrophic health expenditure and productivity loss in Indonesia: a population-based panel data analysis study. BMJ open. 2021;11(2):e041870.

    Article  PubMed  PubMed Central  Google Scholar 

  5. O’Connor EA, Perdue LA, Senger CA, Rushkin M, Patnode CD, Bean SI, et al. Screening and behavioral counseling interventions to reduce unhealthy alcohol use in adolescents and adults: updated evidence report and systematic review for the US Preventive Services Task Force. JAMA. 2018;320(18):1910–28.

    Article  PubMed  Google Scholar 

  6. Gayes LA, Steele RG. A meta-analysis of motivational interviewing interventions for pediatric health behavior change. J Consult Clin Psychol. 2014;82(3):521.

    Article  PubMed  Google Scholar 

  7. Richardson LP, McCarty CA, Radovic A, Suleiman AB. Research in the integration of behavioral health for adolescents and young adults in primary care settings: a systematic review. J Adolesc Health. 2017;60(3):261–9.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Dietz WH, Baur LA, Hall K, Puhl RM, Taveras EM, Uauy R, et al. Management of obesity: improvement of health-care training and systems for prevention and care. Lancet. 2015;385(9986):2521–33.

    Article  PubMed  Google Scholar 

  9. Delisle H, Shrimpton R, Blaney S, Du Plessis L, Atwood S, Sanders D, et al. Capacity-building for a strong public health nutrition workforce in low-resource countries. Bull World Health Organ. 2017;95(5):385.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Nicholas J, Goodwin PPD, Susanto P, Rahmah A. Tulodo, Rutgers WPF Indonesia. What’s Ailing You, Kid? Review of Youth Friendly Health Services in Indonesia. Jakarta, Indonesia: Rutgers WPF; 2018.

    Google Scholar 

  11. Indonesia KK. Standar pendidikan profesi dokter Indonesia. Jakarta: Konsil Kedokteran Indonesia. 2012;5.

  12. Indonesia PPN. Standar kompetensi perawat Indonesia. Bidang Organisasi PP-PPNI: Jakarta. 2005.

  13. Agung FH, Sekartini R, Sudarsono NC, Hendarto A, Dhamayanti M, Werdhani RA, et al. The barriers of home environments for obesity prevention in Indonesian adolescents. BMC Public Health. 2022;22(1):1–10.

    Article  Google Scholar 

  14. Biggs J, Tang C. Teaching for quality learning at university. McGraw-hill education (UK); 2011.

  15. Resnicow K, McMaster F, Bocian A, Harris D, Zhou Y, Snetselaar L, et al. Motivational interviewing and dietary counseling for obesity in primary care: an RCT. Pediatrics. 2015;135(4):649–57.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Rollnick S, Miller WR, Butler C. Motivational interviewing in health care: helping patients change behavior. Guilford press; 2008.

  17. DiClemente CC, Corno CM, Graydon MM, Wiprovnick AE, Knoblach DJ. Motivational interviewing, enhancement, and brief interventions over the last decade: a review of reviews of efficacy and effectiveness. Psychol Addict Behav. 2017;31(8):862.

    Article  PubMed  Google Scholar 

  18. Lane C, Huws-Thomas M, Hood K, Rollnick S, Edwards K, Robling M. Measuring adaptations of motivational interviewing: the development and validation of the behavior change counseling index (BECCI). Patient Educ Couns. 2005;56(2):166–73.

    Article  PubMed  Google Scholar 

  19. Lundahl B, Droubay BA, Burke B, Butters RP, Nelford K, Hardy C, et al. Motivational interviewing adherence tools: a scoping review investigating content validity. Patient Educ Couns. 2019;102(12):2145–55.

    Article  PubMed  Google Scholar 

  20. Hurlocker MC, Madson MB, Schumacher JA. Motivational interviewing quality assurance: a systematic review of assessment tools across research contexts. Clin Psychol Rev. 2020;82:101909.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Tsang S, Royse CF, Terkawi AS. Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine. Saudi J Anaesth. 2017;11(5):80.

    Article  Google Scholar 

  22. Trakman GL, Forsyth A, Hoye R, Belski R. Developing and validating a nutrition knowledge questionnaire: key methods and considerations. Public Health Nutr. 2017;20(15):2670–9.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Bujang MA, Baharum N. A simplified guide to determination of sample size requirements for estimating the value of intraclass correlation coefficient: a review. Archives of Orofacial Science. 2017;12(1).

  24. Bujang MA, Baharum N. Sample size guideline for correlation analysis. World. 2016;3(1):37–46.

    Google Scholar 

  25. Zijlmans EA, Tijmstra J, Van der Ark LA, Sijtsma K. Item-score reliability as a selection tool in test construction. Front Psychol. 2019:2298.

  26. Finan SJ, Yap MB. Engaging parents in preventive programs for adolescent mental health: A socio-ecological framework. J Family Theory Rev. 2021;13(4):515–27.

    Article  Google Scholar 

  27. Bean MK, Caccavale LJ, Adams EL, Burnette CB, LaRose JG, Raynor HA et al. Parent involvement in adolescent obesity treatment: a systematic review. Pediatrics. 2020;146(3).

  28. Organization WH. Making health services adolescent friendly: developing national quality standards for adolescent friendly health services. 2012.

  29. Chown P, Kang M, Sanci L, Newnham V, Bennett DL. Adolescent Health. Enhancing the skills of General practitioners in caring for young people from culturally diverse backgrounds. NSW Centre for the Advancement of Adolescent Health and Transcultural Mental Health Centre; 2008.

  30. Patton GC, Sawyer SM, Santelli JS, Ross DA, Afifi R, Allen NB, et al. Our future: a Lancet commission on adolescent health and wellbeing. The Lancet. 2016;387(10036):2423–78.

    Article  Google Scholar 

  31. Hancock Hoskins D. Consequences of parenting on adolescent outcomes. Societies. 2014;4(3):506–31.

    Article  Google Scholar 

  32. Batterton KA, Hale KN. The likert scale what it is and how to use it. Phalanx. 2017;50(2):32–9.

    Google Scholar 

  33. Nemoto T, Beglar D, editors. Likert-scale questionnaires. JALT 2013 conference proceedings; 2014.

  34. Brookhart SM, editor. Editor appropriate criteria: key to effective rubrics. Frontiers in Education. Frontiers Media SA; 2018.

  35. Vaske JJ, Beaman J, Sponarski CC. Rethinking internal consistency in Cronbach’s alpha. Leisure Sci. 2017;39(2):163–73.

    Article  Google Scholar 

  36. Rosi A, Ferraris C, Guglielmetti M, Meroni E, Charron M, Menta R, et al. Validation of a general and sports nutrition knowledge questionnaire in Italian early adolescents. Nutrients. 2020;12(10):3121.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Naar S, Suarez M. Motivational interviewing with adolescents and young adults. Guilford Publications; 2021.

Download references


The authors thank the Ministry of Health of the Republic of Indonesia, especially the director and the staff of The Directorate of Nutrition, Child and Maternal Health, the provincial health offices from 17 provinces and the clinical psychology team whose expertise greatly contributed to this assessment tool (Dian Oriza, Annisa Rahmalia, Utari Krisnamurthi, Sri Wulandari, Fina Dwi Putri, Widya Gunawan, Diajeng Tri, Fadhilah Erynanda, Risky Adinda).


This research was supported by Universitas Indonesia (Grant number NKB–434/UN2.RST/ HKP.05. 00/2022).

Author information

Authors and Affiliations



FHA, RS, NCS, AH, RAW, SRP and SMS contributed to the conception of the work. FHA, LH and AN carried out the data collection. FHA did the data analysis and wrote the initial draft, which was refined with SMS. All authors reviewed and approved the final version of the article to be published. FHA and SMS did the final review before submission.

Corresponding author

Correspondence to Fransisca Handy Agung.

Ethics declarations

Ethics approval and consent to participate

The study was performed in accordance with the relevant Declaration of Helsinki and was approved by the ethic committee of The Faculty of Medicine, University of Indonesia, with approval number 829a/UNZ.F1/ETIK/PPM.00.02/2021. A thorough explanation of the study and written informed consent was obtained from all study participant.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Agung, F.H., Sekartini, R., Sudarsono, N.C. et al. Development and validation of the adolescent behavioural change Counselling Assessment Tool in Indonesia. BMC Health Serv Res 24, 257 (2024).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: