Skip to main content

Rewiring care delivery through Digital Therapeutics (DTx): a machine learning-enhanced assessment and development (M-LEAD) framework



Digital transformation has sparked profound change in the healthcare sector through the development of innovative digital technologies. Digital Therapeutics offer an innovative approach to disease management and treatment. Care delivery is increasingly patient-centered, data-driven, and based on real-time information. These technological innovations can lead to better patient outcomes and support for healthcare professionals, also considering resource scarcity. As these digital technologies continue to evolve, the healthcare field must be ready to integrate them into processes to take advantage of their benefits. This study aims to develop a framework for the development and assessment of Digital Therapeutics.


The study was conducted relying on a mixed methodology. 338 studies about Digital Therapeutics resulting from a systematic literature review were analyzed using descriptive statistics through RStudio. Machine learning algorithms were applied to analyze variables and find patterns in the data. The results of these analytical analyses were summarized in a framework qualitatively tested and validated through expert opinion elicitation.


The research provides M-LEAD, a Machine Learning-Enhanced Assessment and Development framework that recommends best practices for developing and assessing Digital Therapeutics. The framework takes as input Digital Therapeutics characteristics, regulatory aspects, study purpose, and assessment domains. The framework produces as outputs recommendations to design the Digital Therapeutics study characteristics.


The framework constitutes the first step toward standardized guidelines for the development and assessment of Digital Therapeutics. The results may support manufacturers and inform decision-makers of the relevant results of the Digital Therapeutics assessment.

Peer Review reports


Chronic diseases and comorbidities affect a significant portion of the population, and older adults are particularly vulnerable [1,2,3]. In Italy, around 50% of people aged between 65 and 75 suffer from at least one chronic disease [4]. Italy has the highest median age among all European countries [5]. This highlights the urgent need to develop, adopt, and implement strategies that can help manage, treat, and care for older adults and their chronic health conditions [6]. This is particularly crucial considering the limited healthcare resources available and the fact that over 50% of the country’s surface area is made up of rural regions that have limited access to medical care [7, 8].

Healthcare digitalization involves using digital technologies to revolutionize the delivery, management, and accessibility of health services and information [9, 10]. Digitalization has enabled various innovations, including electronic health records, telemedicine, mobile health applications, wearable devices, artificial intelligence, and Digital Therapeutics (DTx) [9, 11,12,13,14,15]. These advancements have the potential to improve healthcare quality, efficiency, and equity, and achieve better outcomes, bringing benefits to patients and healthcare professionals when used effectively [16]. However, healthcare digitalization is complex and dynamic, involving various stakeholders, challenges, and opportunities [17].

This paper focuses on DTx as a specific aspect of digitalization in healthcare [18, 19]. DTx are software-based interventions that deliver evidence-based therapeutic interventions to patients using high-quality software programs to prevent, manage, or treat a medical disorder or disease. DTx have the potential to mitigate the challenges posed by chronic conditions, such as improving patient adherence, self-management, and quality of life [20, 21]. DTx for chronic diseases such as diabetes, for instance, provide patients with tailored digital coaching and insights to optimize treatment plans (e.g., BlueStar) [22, 23]. EndeaverRx, an example of DTx for the treatment of ADHD in children, is designed with a game-based approach for attention management, resulting in more engagement than other standard treatments [24]. Therefore, DTx can potentially be effective and evidence-based tools to support patients’ health [25], and it is crucial to involve patients in the development and use of these solutions to take the expected benefits [26]. The concept of value in healthcare is becoming increasingly important, with value being defined as the health outcomes that matter to patients relative to the costs of delivering such results [27,28,29]. This notion has driven most efforts to assess the value of health technology, including Health Technology Assessment (HTA), a process designed to promote an equitable, efficient, and high-quality health system [30]. As digital technologies continue to evolve, the healthcare sector must be ready to integrate them into processes to take advantage of their benefits, ensuring their safety and equal access to care services [31, 32].

Digital Therapeutics are digital health technologies that deliver medical interventions through software programs to prevent, manage, or treat medical disorders or diseases [20]. These products can be used alone or in combination with other therapies to improve patient care and health outcomes [21]. Although there is still a debate about the cost-effectiveness of digital technologies for health purposes, an increasing amount of ongoing evidence indicates that they have the potential to offer a cost-effective solution for managing chronic diseases. However, ongoing studies are still investigating the efficacy of these emerging technologies and no definitive evidence has been established yet [33,34,35]. In this sense, the development, adoption, and implementation of DTx also entail significant ethical, legal, and social challenges including data privacy and security, the digital divide and health equity, the reliability and validity of machine learning algorithms, and the need for rigorous clinical evaluation and regulation of DTx [31, 36,37,38]. On the dark side, it is also important to include the unintended social and behavioral impacts of DTx, such as the exacerbation of health disparities and the loss of human touch and empathy [39]. The overreliance and addiction to digital devices, as well as the ethical dilemmas and moral conflicts that may arise from the use of DTx, are additional concerns that need to be critically examined and addressed. Therefore, it is essential to ensure that the benefits of DTx outweigh the harms and that the implementation of DTx is guided by evidence-based practices and ethical principles [20, 21].

In many countries, including Italy, DTx products are considered medical devices and must be reviewed, certified (e.g., CE marking s required), and authorized by regulatory bodies to ensure their safety and effectiveness [20]. At the European level, the relevant legislation in the field of medical devices, thus including DTx, is the Medical Devices Regulation (MDR) 2017/745 [40]. According to the Regulation, a Digital Therapeutic can be considered Software as a Medical Device (SaMD), referring to the central role of the algorithm underlying their functioning [41]. Despite the growing interest in DTx, Italy has not yet produced an official definition or development pathway for these products [42]. This lack of clarity has hindered the diffusion of DTx technologies in Italy [43]. To address this issue, this study aims to provide original insights and develop a framework for developing DTx in Italy.

The paper is organized as follows: the first subsection reviews the literature on the assessment of health technologies to lay the foundation for the research framework and hypotheses. Section 2 outlines the methods of data collection and analysis. Section 3 presents the findings of the empirical investigation and discusses the significant achievements of the paper. Finally, Sect. 4 discusses the main results and their implications for researchers, managers, and policymakers.

Assessment of health technologies

According to Tunis and Ommaya (2002), clinical research is an essential component of medical and health research to produce knowledge valuable for understanding human disease, preventing and treating illness, and promoting health [44]. Two main types of clinical studies exist: interventional (or experimental) and observational [45]. While local regulations define the characteristics of clinical research for new products (e.g., drugs, medical devices), little is known when dealing with new digital technologies such as Digital Therapeutics. With some exceptions (e.g., Germany), there is no standard or shared framework for the study design of a DTx. In the literature, it is possible to find examples of frameworks that summarize the existing study types and can help select the most suitable alternative. For instance, Grimes and Schulz (2002) developed an algorithm for classifying the kinds of clinical research that can be successfully applied to traditional medical products and technologies [46]. The framework, however, cannot be fully employed when dealing with Digital Therapeutics. Indeed, the development and assessment of disruptive innovation such as DTx introduces additional hurdles (e.g., regulatory frameworks, ethics considerations, organizational challenges) calling for the design of updated and potentially new methodologies for research and evaluation [47].

A relevant stream of literature that can be considered to frame a new health technology is the Health Technology Assessment. The main objective of HTA is to determine the actual and potential effects of given health technologies - as well as the consequences on the healthcare system, national economy, and society resulting from their adoption - both a priori and along their whole lifecycle [48]. As medical devices, pharmaceuticals, medical procedures, and health programs are the most common technologies assessed in HTA practices, Digital Therapeutics might become frequently assessed in the future [49]. Since most HTA agencies are governmental (hence, publicly funded), adopting formal HTA pathways is relevant to ensure a balance between equal and sustainable care and the adoption of innovative health technologies. Further, besides informing decision-makers, HTA has another vital role: orientating innovation in the healthcare industry and informing relevant stakeholders (e.g., clinicians, patients) of the drivers behind selecting the best health technologies. Each of the several existing frameworks (e.g., EUnetHTA Core Model, EVIDEM, and MAST) points out that to promote equitable, efficient, high-quality health, any HTA framework requires generating multidisciplinary, high-quality evidence. As Busse et al. (2002) pointed out, any HTA process should include the systematic retrieval of reliable evidence covering all the relevant domains of the assessment [50]. However, as far as digital health innovations (e.g., DTx) are concerned, traditional assessment approaches have partly failed since introducing digital health technologies disrupts the conventional paradigms of care (e.g., the patient-clinician relationship) [51, 52]. Additionally, the generalizability of results related to digital health technologies is more challenging than traditional health technologies since it is inherently context- and actors-specific [53, 54].

The present research aims to develop a framework to help fill the DTx validation gap.


The study has been conducted relying on mixed methods, combining analytical and qualitative methodologies to increase the reliability of results [55] and to strengthen their evaluations [56]. Figure 1 shows the sequence of phases of the study.

Fig. 1
figure 1

Flow chart of the methodology adopted


A systematic literature review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines was performed (Fig. 2) [57].

Fig. 2
figure 2

PRISMA diagram

The literature search was conducted in Scopus, PubMed, and databases. Search query included keywords concerning DTx (e.g., “digital therap*”, “DTx”), and only English-language records were considered. The search was constrained by documents published from 2015 onwards, considering the approval of the first American DTx (i.e., reSET) by the FDA in 2017. Records meeting the following two criteria were incorporated into the review: (1) the study aligns with the Digital Therapeutics Alliances’ [21] definition of DTx, and (2) the study explicitly reports results and implications related to DTx. Conversely, records were excluded if they met any of the following criteria: (1) lack of relevance to human medicine, (2) lack of relevance to digital medicine, (3) provision of a vague definition for DTx with a general perspective, and (4) absence of studies assessing specific DTx in their content. Two researchers (AC and SM) performed title and abstract screening and full-text reading. Disagreements have been solved through discussion between the two reviewers and by consulting a third researcher (EL).

Aggregated data from the systematic literature review were analyzed using descriptive statistics via RStudio. Then, a subset of 33 variables related to the 338 included studies was selected and divided into predictors [23] and targets [10]. Three machine learning (ML) algorithms (namely, K-NN, decision trees, and random forests) were iteratively applied to build models to predict the target variables starting from the predictors and find patterns in the data. According to the specific characteristics of the dataset, a metric for comparison of model performances, namely Model Comparison Metric (MCM), was defined based on the weighted average of accuracy (0.5), specificity (0.25), precision (0.125), and recall (0.125) [58]. As decision trees reported better MCM values than the other algorithms, their outputs, combined and interpreted considering the recommendations from the literature, were used to develop a framework for assessing Digital Therapeutics. The framework, organized in five sections, guides DTx study design and assessment.

Starting from the multiple-step procedure developed by Knol et al. (2010), the following sequence of steps was followed: (i) translation of the framework’s pillars into hypotheses in the form of statements and visual representations, (ii) definition of the scope and format of the elicitation, (iii) selection of experts, (iv) design of the elicitation protocol, (v) preparation of the elicitation session, (vi) elicitation of expert judgments, (vii) text analysis, aggregation, and reporting, (viii) feedback mechanisms to correct the framework [59]. Therefore, the paper is based on the integration of quantitative and qualitative data. The results of the literature review were analyzed through ML algorithms, allowing interpretation and attribution of meaning in the form of a framework [60]. In addition, the qualitative approach brought further insights through interviews and validated the hypothesized framework [61].

The framework was decomposed into 18 statements expressed in the form of hypotheses (Table 1).

Table 1 Hypotheses development and framework main components


A purposeful sampling approach has been adopted to select critical stakeholders to be involved in the expert opinion elicitation [62]. To maximize the breadth of the perspectives, it was decided to include all three types of professional experts mentioned by Knol et al. (2010) (namely generalists, subject-matter experts, and normative experts), resulting in a total of eight people with a heterogeneous background [59]. Their expertise ranged from business and product development to healthcare management, health regulatory policies, clinical research, and clinical practice. Table 2 summarizes the profiles and roles of the people involved.

Table 2 Professionals involved during the expert opinion elicitation

In addition, contacted stakeholders have been asked to disseminate this study invitation to other profiles of interest and refer to eligible contacts (i.e., snowballing). Eligible participants needed to be experts in the field of (i) Digital Therapeutics, (ii) clinical development of new medical technologies, and (iii) involved in new digital health product development project(s).

Data collection

Focus group discussions and semi-structured interviews have been conducted via Microsoft Teams between September and December 2022. In addition, focus group discussions and interviews were recorded and performed in Italian and were about 1–2 h long. The day before the interviews, after collecting informed consent from the participants and confirmation of their willingness to participate, an outline of the questions that would be asked during the interview was shared with them. Most were planned; however, others emerged during the sessions. Participants were allowed to answer each question and appeared engaged during the interview, sharing their experiences. The experts involved were asked to comment on the framework by highlighting its strengths and limitations and suggesting possible improvements based on their expertise. At the beginning of each interaction, the moderator (AC or SM) introduced himself and explained the purpose and procedures. Next to the moderator, the observer (SM or AC) was present in discussions and responsible for time management and taking notes. A PowerPoint presentation guided participants through the focus group discussion and interviews. Several recommendations were followed to minimize inherent biases in subjective judgment and errors related to the elicited outcomes (e.g., to avoid misperception of notions about likelihood and probability) [63].

Data analysis

Upon completing the interview process, two researchers (AC and SM) engaged in a collaborative debriefing session to share their observations and facilitate the recording of salient details, including non-verbal behaviors. The data obtained were analyzed deductively (i.e., based on the research objectives) and inductively (i.e., based upon the recognition of patterns), relying on a process of data familiarization followed by the coding phase using Microsoft Excel, following a thematic analysis approach [64]. The verbatim transcription of each interview was accompanied by meticulous notetaking, enabling the identification of key sentences to generate codes and emergent themes. These themes were subsequently grouped, organized, and abstracted, with the attainment of data saturation and replication in themes serving as a guiding principle. Additionally, the senior researcher (EL) conducted an independent review of the transcripts to generate codes and themes. A working analytical framework (namely, a coding tree) has been created and applied to the other transcripts. Finally, the data were processed into a matrix and interpreted. The final themes, sub-themes, and relative categories were then subjected to a thorough discussion by a team of three researchers with the aim of refining and clarifying their significance by following a consensus-driven approach [65, 66].


The analytical analysis of aggregated data deriving from the systematic literature review of 338 studies about DTx has been considered for the development of M-LEAD, a Machine Learning-Enhanced Assessment and Development framework. The framework comprises five components (namely, study type and randomization, patients and study duration, comparators and study arms, outcomes and scales, and sources of evidence). The resulting framework, therefore, combines background literature, machine learning outputs, and expert opinion elicitation. An overview of the insights derived from the expert opinion elicitation is provided in Table 3.

Table 3 Summary of the findings from the expert opinion elicitation

The following sections will focus on three of the components of the framework (namely, comparators and study arms, outcomes and scales, and sources of evidence), whose implications are the most relevant to discuss.

Comparators and study arms: the role of digital placebos

A significant portion of the expert elicitation was dedicated to study comparators and arms. This framework section is consistent with what experts have directly experienced with the study design of products under development. Both the CRO representatives and the researcher stated that comparators depend on product features and are often related to the target disease, as the standard of care is disease-specific. Additionally, according to the CRO managing director and the medical evidence advisor, now DTx typically represent the digital edge of an existing non-digital pathway– the traditional standard of care.

“I think that an arm comparing DTx with the standard of care should exist to prove that digitalization has improved both efficacy and efficiency of the patient’s care pathway.” (1, CRO representative).

Considering the most used comparators in the literature (namely standard of care or no intervention), several participants pointed out that the standard of care and no intervention are significantly different options. No intervention might indeed be unethical and not even recommended by the regulations, as the objective of a clinical study should be to demonstrate that the product is better than - or at least equal to - the standard of care.

“Comparing a Digital Therapeutics product candidate with a no intervention arm would likely generate a positive assessment of the intervention, which may be worse than existing standards of care.” (6, psychologist).

Finally, according to the researcher, the three hospital representatives, and the CRO managing director, any DTx can have a digital placebo. Further, for the physician, digital placebos represent a potentially valuable tool for the study of DTx.

“I believe that it should not be recommended to compare completely different intervention modalities, like the DTx vs a non-digital intervention, to minimize the risk of confounding.” (5, physician).

Hence, a digital placebo would be correct from a methodological standpoint, producing more reliable and generalizable results.

Outcomes and scales: spotlight on the organizational aspects

Another relevant part of the discussion concerned HTA domains and outcomes. When discussing this section, there was substantial agreement on the results proposed. Among the main HTA domains (namely clinical, human factor, societal, organizational, ethical, and economic), the MedTech manager presented his opinion about the organizational implications of a DTx.

“According to my experience, DTx do not determine organizational changes in terms of processes and structure, but mainly in terms of market access and regulations.” (7, MedTech company representative).

Further, cultural change, considered part of the organizational domain, is also relevant. DTx strongly affect the patient and his willingness to pay and be empowered for his care. This influence should be considered when developing business models and reimbursement policies in countries (e.g., Italy) where the propensity toward out-of-pocket healthcare expenses is limited. Hence, before economic barriers to DTx, organizational aspects, in terms of cultural change, should be carefully assessed.

Sources of evidence: the opportunities of real-world data

As argued by both the health policy researcher and the CRO managers, study designs must support the answer to a clinical question. Hence, even in case of a lack of sources of evidence to demonstrate an outcome, it is necessary to strategically plan study design and data collection so that the evidence needed can be generated.

“It is a good practice to collect more than strictly necessary data also in early stages as such data, combined with administrative and cost information, can provide additional evidence during validation stages.” (2, CRO representative).

From this perspective, according to the CRO managers and all the hospital representatives, it is true that study design and purposes guide the selection of sources of evidence and not vice versa. Several experts stressed the importance of developing data collection, in particular of real-world data (RWD) and statistical analysis plans to correctly infer clinical, human factor, societal, organizational, economic, and ethical evidence from insightful extra data collecting during trials, which might be valuable for many purposes (included HTA) when analyzed ex-post in real-world evidence (RWE) settings.


The field of Digital Therapeutics (DTx) is rapidly evolving, with various stakeholders involved in the development, regulation, procurement, and use of these digital health solutions. DTx have gained significant attention in recent years for their potential to address a wide range of health conditions, from chronic illnesses to mental health disorders, and to improve the overall quality of care. However, the lack of standardized methods for assessing their multidimensional impact is a significant barrier to their adoption and integration into clinical practice and reimbursement schemes. There are various HTA frameworks available, including the EUnetHTA Core Model, EVIDEM, and MAST that is designed specifically for telemedicine. However, traditional frameworks have limitations when applied to digital health and may not fully cater to the requirements of innovative digital medicine applications, such as DTx. This gap affects not only the mainstream parameters of benefit, such as effectiveness and cost-effectiveness, but also broader parameters, like therapy adherence, user satisfaction, engagement, and organizational adaptation to real-life environments [54, 67,68,69].

To address this gap, our study aimed to develop and validate a comprehensive, evidence-based framework that systematically guides the development and assessment of DTx in healthcare. While we build on the existing frameworks and guidelines for Health Technology Assessment of digital health interventions, such as the one recently proposed by Tarricone et al. (2022) [68], the M-LEAD framework offers unique insights, as it is explicitly designed to address the distinct elements and criteria of DTx. The following sections provide critical insights with the final goal of establishing a common and shared framework for fostering the adoption of DTx in Italy. Each framework component has been validated, relying on interviews and focus groups. Experts’ knowledge and experience of participants improved study reliability and validity [70].

Theoretical contributions and managerial implications

The development and validation of the M-LEAD framework is a significant contribution to the literature on Digital Therapeutics and Health Technology Assessment. The M-LEAD framework provides a comprehensive and systematic approach to guide the development and assessment of DTx, addressing the gap in the existing frameworks and guidelines for digital health interventions.

The M-LEAD framework incorporates specific elements and criteria relevant to DTx, such as selecting a suitable comparator, measuring user adherence and engagement, and assessing the ethical and organizational implications of DTx. By integrating the perspectives and preferences of the end-users and the healthcare system, the framework ensures that the DTx solutions are aligned with the needs and expectations of their users and stakeholders. What sets this framework apart is its emphasis on involving patients and other stakeholders in the co-design and co-assessment of DTx. This approach ensures that these digital interventions meet the end-user’s and healthcare system’s needs and preferences. The framework also addresses the challenges and opportunities of integrating DTx into existing care pathways and workflows. It recognizes the potential impact of DTx on the roles and responsibilities of healthcare professionals and patients.

Moreover, the framework addresses the ethical and legal issues arising from using DTx, such as data privacy and security, informed consent, liability, and accountability. Another distinctive characteristic of this framework is its focus on the potential of DTx to reduce health inequalities and improve access to care, especially for underserved and vulnerable populations. Finally, the framework underscores the need for adaptive and iterative assessment methods for DTx. This approach accounts for the dynamic and evolving nature of these interventions and their contexts of use.

The M-LEAD framework has important implications for the stakeholders involved in the DTx sector, including developers, researchers, regulators, payers, clinicians, and patients. The framework can help developers and researchers plan, design studies, and develop evidence-based products demonstrating their value and impact. It can also assist regulators and payers in developing standardized guidelines and criteria for assessing DTx, essential for governing a rapidly growing and evolving sector. The framework can also support clinicians and patients in making informed decisions about adopting and using DTx and assessing their outcomes and experiences. By promoting evidence-based and cost-effective DTx, the M-LEAD framework can potentially enhance the quality and efficiency of healthcare delivery and outcomes, benefiting patients, clinicians, payers, and society.

The M-LEAD framework also offers valuable knowledge for decision-making and policy-making processes, particularly regarding the adoption and reimbursement of DTx. It highlights the need for a coordinated and standardized approach to assessing the value and impact of DTx, considering the multidimensional and dynamic nature of these interventions and their contexts of use.

The framework also emphasizes the importance of involving and training healthcare professionals and patients to assess clinical practices associated with DTx, as they are central to the paradigm shift in the patient-clinician relationship introduced with DTx. It stresses the need to determine the users’ awareness and the ethical and organizational implications of this paradigm change.

The M-LEAD framework can help establish a common language and principles for developing and assessing DTx, enabling stakeholders to collaborate and communicate effectively. It can also foster innovation and collaboration among the stakeholders involved in the DTx sector, creating platforms and networks for sharing best practices, data, and experiences and facilitating the co-creation and co-assessment of DTx solutions. The framework can also help address the challenges and opportunities of applying the framework to different types and stages of DTx, such as prevention, diagnosis, treatment, and monitoring, and how the framework can be adapted to different contexts and settings. It can also help ensure the sustainability and scalability of the DTx solutions by addressing the gaps and uncertainties in the evidence base, incorporating the perspectives and preferences of the end-users, and ensuring the safety and quality of the DTx solutions.

Strengths and limitations of the study and further research

This study has several strengths and limitations that need to be acknowledged. Firstly, the study has a strength in developing and validating a comprehensive and systematic framework for developing and assessing DTx, addressing the gap in the existing frameworks and guidelines for digital health interventions. The M-LEAD framework incorporates specific elements and criteria relevant to DTx, such as selecting a suitable comparator, measuring user adherence and engagement, and assessing the ethical and organizational implications of DTx. The M-LEAD framework also integrates the perspectives and preferences of the end-users and the healthcare system, ensuring that the DTx solutions are aligned with the needs and expectations of the stakeholders.

Secondly, the study has a strength in conducting the validation process in collaboration with the DTx developers, who are the primary users of the framework. The framework was refined iteratively, considering the various development phases of the DTx. Throughout the development and validation process of the framework, the technology readiness level (TRL) increased from the preliminary stages, indicating the progress and maturity of the DTx solutions. The study also involved key players in Italy, providing a comprehensive representation of the actors involved in developing DTx in the country. However, depending solely on expert opinion can lead to subjective biases that affect the consistency of the results. Nevertheless, this methodology of generating evidence is crucial given the early development stage (i.e., lower than TRL 7) of these emerging technologies [59, 71]. Future research conducted at higher TRL levels can adopt more advanced evidence-generation approaches.

Additionally, the study also has other limitations that need to be addressed. Firstly, the experts consulted to conduct this research were only from Italy, so the findings may not apply to other regions and countries. The DTx sector is rapidly evolving and diverse, with different regulatory and reimbursement frameworks, market dynamics, and stakeholder preferences. Even though the expert selection process was systematic in mapping all the DTx under development close to the Italian market, there is a potential limitation in fully capturing the diversity of opinions and experiences in the target field, with the risk of incurring representation bias. Therefore, further research is needed to update and validate this framework in different contexts and settings and to test its applicability and feasibility in real-world scenarios.

Secondly, the developed framework was based on the current state of the art of DTx and HTA, which may change over time as new technologies and methodologies emerge. The M-LEAD framework is intended to be a flexible and adaptable tool that can accommodate the dynamic and evolving nature of DTx and their contexts of use. However, the framework may need to be revised and updated periodically to reflect the latest evidence and best practices in the field. Therefore, further research is required to monitor and evaluate the framework’s performance and impact and identify the areas for improvement and innovation.


The development and assessment of Digital Therapeutics (DTx) is an emerging field that has gained significant attention in recent years. DTx products are software-based interventions that aim to prevent, manage, or treat various health conditions and diseases. However, despite the growing interest in DTx, there is a significant gap in the literature regarding the development and assessment of these products. To address this gap, this study has created a new framework that seizes an opportunity and contributes to filling the relevant literature gap about DTx product development and assessment. The proposed framework, called M-LEAD, has been improved and validated using expert elicitation and represents the first step towards creating standardized guidelines for developing Health Technology Assessment of Digital Therapeutics.

The M-LEAD framework consists of principles and guidelines that can be used to assess the effectiveness, safety, and usability of DTx products. These guidelines are intended to be flexible enough to accommodate the unique features of different DTx products while still providing a standardized framework for their assessment. The M-LEAD framework has been validated through expert elicitation, which involved a group of experts in the field of DTx product development and assessment. The experts reviewed the proposed framework and provided feedback on its clarity, completeness, and usefulness. The feedback was used to refine the framework and ensure that it accurately reflects the current state of the field.

In conclusion, the proposed M-LEAD framework for DTx product development and assessment is an essential contribution to the field. It provides a standardized approach to assessing the multidimensional impact of DTx products in real-world environments, which will help to ensure that these products are developed and assessed consistently and transparently. The M-LEAD framework also represents the first step towards creating standardized guidelines for developing health technology assessments of Digital Therapeutics. Some of the original insights that the M-LEAD framework offers are: (i) incorporation of the perspectives and preferences of the end-users and the healthcare system, ensuring that the DTx solutions are aligned with the needs and expectations of the stakeholders; (ii) emphasis on the measure of user adherence and engagement, as they are critical factors for the success and sustainability of DTx products; (iii) inclusion of the ethical and organizational implications of DTx, such as data privacy and security, informed consent, liability, and accountability, and guidance on how to address them; (iv) co-design and co-assessment of DTx products, involving patients and other stakeholders in the development and assessment process, to ensure that the DTx products are user-centered and evidence-based.

Data availability

The datasets generated and analyzed during the current study are not publicly available. Although identifying information has been removed, authors cannot risk identification by making the data available for public inspection, as anonymity to respondents was guaranteed. Datasets could be available from the corresponding author on reasonable request.



Contract Research Organization


Digital Therapeutics


Health Technology Assessment


K-Nearest Neighbors


Machine Learning


Model Comparison Metric


Medical Devices Regulation


Real-World Data


Real-World Evidence


Software as a Medical Device


Technology Readiness Level


  1. EpiCentro. Patologie croniche nella popolazione residente in Italia secondo i dati PASSI e PASSI d’Argento. [cited 2023 Nov 1]. Available from:

  2. Population structure and ageing. - Statistics Explained. [cited 2023 Nov 1]. Available from:

  3. Maresova P, Javanmardi E, Barakovic S, Barakovic Husic J, Tomsone S, Krejcar O, et al. Consequences of chronic diseases and other limitations associated with old age– a scoping review. BMC Public Health. 2019;19(1):1431.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Melchiorre M, Socci M, Quattrini S, Lamura G, D’Amen B. Frail older people Ageing in Place in Italy: Use of Health Services and Relationship with General Practitioner. IJERPH. 2022;19(15):9063.

    Article  PubMed  PubMed Central  Google Scholar 

  5. istat. [cited 2023 Nov 2]. An ageing population. Available from:

  6. Car J, Tan WS, Huang Z, Sloot P, Franklin BD. eHealth in the future of medications management: personalisation, monitoring and adherence. BMC Med. 2017;15(1):73. s12916-017-0838–0.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Cepparulo A, Giuriato L. The residential healthcare for the elderly in Italy: some considerations for post-COVID-19 policies. Eur J Health Econ. 2022;23(4):671–85.

    Article  PubMed  Google Scholar 

  8. Cingolani M, Scendoni R, Fedeli P, Cembrani F. Artificial intelligence and digital medicine for integrated home care services in Italy: opportunities and limits. Front Public Health. 2023;10:1095001.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Kraus S, Schiavone F, Pluzhnikova A, Invernizzi AC. Digital transformation in healthcare: analyzing the current state-of-research. J Bus Res. 2021;123:557–67.

    Article  Google Scholar 

  10. Carini E, Villani L, Pezzullo AM, Gentili A, Barbara A, Ricciardi W, et al. The Impact of Digital Patient Portals on Health outcomes, System Efficiency, and patient attitudes: updated systematic literature review. J Med Internet Res. 2021;23(9):e26189.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Verily AA, Adams L, National Academy of Medicine, Barrett M, ResMed et al. The Promise of Digital Health: Then, Now, and the Future. NAM Perspectives. 2022 [cited 2023 Nov 6];6(22). Available from:

  12. Wattanapisit A, Teo CH, Wattanapisit S, Teoh E, Woo WJ, Ng CJ. Can mobile health apps replace GPs? A scoping review of comparisons between mobile apps and GP tasks. BMC Med Inf Decis Mak. 2020;20(1):5.

    Article  Google Scholar 

  13. Gautam N, Ghanta SN, Mueller J, Mansour M, Chen Z, Puente C, et al. Artificial Intelligence, Wearables and Remote Monitoring for Heart failure: current and future applications. Diagnostics. 2022;12(12):2964.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Chakraborty I, Edirippulige S, Vigneswara Ilavarasan P. The role of telehealth startups in healthcare service delivery: a systematic review. Int J Med Informatics. 2023;174:105048.

    Article  Google Scholar 

  15. Mumtaz H, Riaz MH, Wajid H, Saqib M, Zeeshan MH, Khan SE, et al. Current challenges and potential solutions to the use of digital health technologies in evidence generation: a narrative review. Front Digit Health. 2023;5:1203945.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Granström E, Wannheden C, Brommels M, Hvitfeldt H, Nyström ME. Digital tools as promoters for person-centered care practices in chronic care? Healthcare professionals’ experiences from rheumatology care. BMC Health Serv Res. 2020;20(1):1108.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Schiavone F, Mancini D, Leone D, Lavorato D. Digital business models and ridesharing for value co-creation in healthcare: a multi-stakeholder ecosystem analysis. Technol Forecast Soc Chang. 2021;166:120647.

    Article  Google Scholar 

  18. Cronin S, Madden S, Ardill L, O’Reilly K, Brinth A, Fitzgerald C et al. The Past, Present, and Future of Digital Therapeutics. In: Digital Therapeutics [Internet]. 1st ed. Boca Raton: Chapman and Hall/CRC; 2022 [cited 2023 Nov 6]. p. 405–30. Available from:

  19. Nomura A. Digital health, digital medicine, and digital therapeutics in cardiology: current evidence and future perspective in Japan. Hypertens Res. 2023;46(9):2126–34.

    Article  PubMed  PubMed Central  Google Scholar 

  20. ISO/TR 11147.:2023(en), Health informatics — Personalized digital health — Digital therapeutics health software systems. [cited 2023 Oct 19]. Available from:

  21. DTA_DTx-Definition-and -Core-Principles.pdf. [cited 2023 Nov 1]. Available from:

  22. Agarwal P, Mukerji G, Desveaux L, Ivers NM, Bhattacharyya O, Hensel JM, et al. Mobile App for Improved Self-Management of type 2 diabetes: Multicenter Pragmatic Randomized Controlled Trial. JMIR Mhealth Uhealth. 2019;7(1):e10321.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Phan P, Mitragotri S, Zhao Z. Digital therapeutics in the clinic. Bioeng Transla Med. 2023;8(4):e10536.

    Article  Google Scholar 

  24. Kollins SH, Childress A, Heusser AC, Lutz J. Effectiveness of a digital therapeutic as adjunct to treatment with medication in pediatric ADHD. Npj Digit Med. 2021;4(1):58.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Wang C, Lee C, Shin H. Digital therapeutics from bench to bedside. Npj Digit Med. 2023;6(1):38.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  26. Leonardsen ACL, Hardeland C, Helgesen AK, Grøndahl VA. Patient experiences with technology enabled care across healthcare settings- a systematic review. BMC Health Serv Res. 2020;20(1):779.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Porter ME, Lee TH. The strategy that will fix Health Care. Harvard Business Rev. 2013;91(10):50–70.

    Google Scholar 

  28. Sacristán JA. Patient-centered medicine and patient-oriented research: improving health outcomes for individual patients. BMC Med Inf Decis Mak. 2013;13(1):6.

    Article  MathSciNet  Google Scholar 

  29. Teisberg E, Wallace S, O’Hara S. Defining and implementing value-based Health Care: a Strategic Framework. Acad Med. 2020;95(5):682–5.

    Article  PubMed  Google Scholar 

  30. Drummond MF, Schwartz JS, Jönsson B, Luce BR, Neumann PJ, Siebert U, et al. Key principles for the improved conduct of health technology assessments for resource allocation decisions. Int J Technol Assess Health Care. 2008;24(03):244–58.

    Article  PubMed  Google Scholar 

  31. Petersson L, Larsson I, Nygren JM, Nilsen P, Neher M, Reed JE, et al. Challenges to implementing artificial intelligence in healthcare: a qualitative interview study with healthcare leaders in Sweden. BMC Health Serv Res. 2022;22(1):850.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Raja M, Kymre IG, Bjerkan J, Galvin KT, Uhrenfeldt L. National digital strategies and innovative eHealth policies concerning older adults’ dignity: a document analysis in three scandinavian countries. BMC Health Serv Res. 2023;23(1):848.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Bashi N, Fatehi F, Mosadeghi-Nik M, Askari MS, Karunanithi M. Digital health interventions for chronic diseases: a scoping review of evaluation frameworks. BMJ Health Care Inf. 2020;27(1):e100066.

    Article  Google Scholar 

  34. Eze ND, Mateus C, T Cravo Oliveira Hashiguchi. Carter HE editor. 2020 Telemedicine in the OECD: an umbrella review of clinical and cost-effectiveness, patient experience and implementation. PLoS ONE 15 8 e0237585.

  35. Von Huben A, Howell M, Howard K, Carrello J, Norris S. Health technology assessment for digital technologies that manage chronic disease: a systematic review. Int J Technol Assess Health Care. 2021;37(1):e66.

    Article  Google Scholar 

  36. Asan O, Bayrak AE, Choudhury A. Artificial Intelligence and Human Trust in Healthcare: Focus on clinicians. J Med Internet Res. 2020;22(6):e15154.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Gooding P, Kariotis T. Ethics and Law in Research on Algorithmic and Data-Driven Technology in Mental Health Care: scoping review. JMIR Ment Health. 2021;8(6):e24668.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Manetti S, Cumetti M, De Benedictis A, Lettieri E. Adoption of novel biomarker test parameters with machine learning-based algorithms for the early detection of sepsis in hospital practice. J Nurs Manage. 2022;30(8):3754–64.

    Article  Google Scholar 

  39. Digital health divide.: only 1 in 2 countries in Europe and central Asia have policies to improve digital health literacy, leaving millions behind. [cited 2024 Feb 5]. Available from:

  40. EUR-Lex– 02017R0745-20200424 - EN. - EUR-Lex. [cited 2024 Feb 6]. Available from:

  41. md_mdcg_2019_11_guidance_qualification_classification_software_en_0.pdf. [cited 2024 Feb 6]. Available from:

  42. Improving Access To Digital Therapeutics In. Europe, efpia, June 2023. [cited 2023 Oct 23]. Available from:

  43. Recchia G, Maria Capuano D, Mistri N, Verna R. Digital therapeutics-what they are, what they will be. Act Scie Medic. 2020;4(3):01–9.

    Article  Google Scholar 

  44. Institute of Medicine (US) Clinical Research Roundtable, Tunis S, Ommaya A. The Role of Purchasers and Payers in the Clinical Research Enterprise: Workshop Summary. Washington, D.C.: National Academies Press; 2002 [cited 2023 Nov 1]. Available from:

  45. Learn About Studies.| [cited 2023 Nov 1]. Available from:

  46. Grimes DA, Schulz KF. An overview of clinical research: the lay of the land. Lancet. 2002;359(9300):57–61.

    Article  PubMed  Google Scholar 

  47. Michie S, Yardley L, West R, Patrick K, Greaves F. Developing and Evaluating Digital Interventions to Promote Behavior Change in Health and Health Care: Recommendations Resulting From an International Workshop. J Med Internet Res. 2017;19(6):e232.

  48. [cited 2023 Nov 1].| health technology assessment. Available from:

  49. Gianfrate F. Economia E management dei servizi sanitari. Strategie E strumenti per una sanità migliore. Tecniche nuove; 2013.

  50. Busse R, Orvain J, Velasco M, Perleth M, Drummond M, G¨rtner F, Best Practice in Undertaking and Reporting Health Technology Assessments, et al. Working Group 4 Report. Int J Technol Assess Health Care. 2002;18(2):361–422.

    Article  Google Scholar 

  51. Zanaboni P, Lettieri E. Institutionalizing telemedicine applications: the challenge of legitimizing decision-making. J Med Internet Res. 2011;13(3):e72.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Carrera A, Zoccarato F, Mazzeo M, Lettieri E, Toletti G, Bertoli S, et al. What drives patients’ acceptance of Digital Therapeutics? Establishing a new framework to measure the interplay between rational and institutional factors. BMC Health Serv Res. 2023;23(1):145.

    Article  PubMed  PubMed Central  Google Scholar 

  53. Hashiguchi TCO, Bringing Health Care to the Patient:. An Overview of the Use of Telemedicine in Oecd Countries. (116).

  54. Kidholm K, Ekeland AG, Jensen LK, Rasmussen J, Pedersen CD, Bowes A, et al. A Model for Assessment of Telemedicine Applications: Mast. Int J Technol Assess Health Care. 2012;28(1):44–51.

    Article  PubMed  Google Scholar 

  55. Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Adm Policy Ment Health. 2011;38(1):44–53.

    Article  PubMed  Google Scholar 

  56. Dopp AR, Mundey P, Beasley LO, Silovsky JF, Eisenberg D. Mixed-method approaches to strengthen economic evaluations in implementation research. Implement Sci. 2019;14(1):2.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Moher D. Preferred reporting items for systematic reviews and Meta-analyses: the PRISMA Statement. Ann Intern Med. 2009;151(4):264.

    Article  PubMed  Google Scholar 

  58. Ali A, Kalatian A, Choudhury CF. Comparing and contrasting choice model and machine learning techniques in the context of vehicle ownership decisions. Transp Res Part A: Policy Pract. 2023;173:103727.

    Google Scholar 

  59. Knol AB, Slottje P, Van Der Sluijs JP, Lebret E. The use of expert elicitation in environmental health impact assessment: a seven step procedure. Environ Health. 2010;9(1):19.

    Article  PubMed  PubMed Central  Google Scholar 

  60. Kim M, Yang J, Ahn WY, Choi HJ. Machine Learning Analysis To Identify Digital Behavioral Phenotypes for Engagement and Health Outcome Efficacy of an mHealth intervention for obesity: Randomized Controlled Trial. J Med Internet Res. 2021;23(6):e27218.

    Article  PubMed  PubMed Central  Google Scholar 

  61. De Sutter E, Borry P, Huys I, Barbier L. Informing a European guidance framework on electronic informed consent in clinical research: a qualitative study. BMC Health Serv Res. 2023;23(1):181.

    Article  PubMed  PubMed Central  Google Scholar 

  62. Creswell JW. Research design: qualitative, quantitative, and mixed methods approaches. 4th ed. Thousand Oaks: SAGE; 2014. p. 273.

    Google Scholar 

  63. Tversky A, Kahneman D. Judgment under uncertainty: Heuristics and biases: biases in judgments reveal some heuristics of thinking under uncertainty. Science. 1974;185(4157):1124–31.

    Article  CAS  PubMed  ADS  Google Scholar 

  64. Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13(1):117.

    Article  PubMed  PubMed Central  Google Scholar 

  65. Patton MQ. Enhancing the quality and credibility of qualitative analysis. Health Serv Res. 1999;34(5 Pt 2):1189–208.

    CAS  PubMed  PubMed Central  Google Scholar 

  66. Patton MQ. Qualitative Research. In: Everitt BS, Howell DC, editors. Encyclopedia of Statistics in Behavioral Science. 1st ed. Wiley; 2005 [cited 2024 Feb 6]. Available from:

  67. Martini N, Calabria S, Recchia G, Gussoni G, Ciani O. Digital therapeutics, HTA and reimbursement in Italy. Tendenze Nuove. 2021;4:117–26.

    Google Scholar 

  68. Tarricone R, Petracca F, Cucciniello M, Ciani O. Recommendations for developing a lifecycle, multidimensional assessment framework for mobile medical apps. Health Econ. 2022;31(S1):73–97.

    Article  PubMed  PubMed Central  Google Scholar 

  69. Yan K, Balijepalli C, Druyts E. The Impact of Digital Therapeutics on current Health Technology Assessment Frameworks. Front Digit Health. 2021;3:667016.

    Article  PubMed  PubMed Central  Google Scholar 

  70. Yam CH, Wong EL, Cheung AW, Chan FW, Wong FY, Yeoh Ekiong. Framework and components for effective discharge planning system: a delphi methodology. BMC Health Serv Res. 2012;12(1):396.

    Article  PubMed  PubMed Central  Google Scholar 

  71. Mitton C, Dionne F, Damji R, Campbell D, Bryan S. Difficult decisions in times of constraint: Criteria based Resource Allocation in the Vancouver Coastal Health Authority. BMC Health Serv Res. 2011;11(1):169.

    Article  PubMed  PubMed Central  Google Scholar 

Download references


We thank Maria Pinelli and Giovanni Giorgetti for their support during the study conduction. We would also like to thank the stakeholders for participating in this study and for their helpful insights.


Not applicable.

Author information

Authors and Affiliations



All authors meet the journal’s authorship guidelines. A.C. has made substantial contributions to the design of the work, the acquisition, analysis, and interpretation of data, and has drafted the work together with S.M. E.L. has made substantial contributions to the conception and design of the work and has substantively revised it. All authors read and approved the final manuscript. All authors reviewed the manuscript.

Corresponding author

Correspondence to Alessandro Carrera.

Ethics declarations

Ethics approval and consent to participate

This study has been entirely performed following guidelines and regulations. The study did not require approval from an ethics committee because this study did not change in any way patients’ therapy plans, as stated by the Decree of the Minister of Health dated February 1, 2022, and by the Regulations of the “National Ethics Committee for trials of public research institutions (EPRs) and other public institutions of national character,” at the Istituto Superiore di Sanità (ISS). This study included interviews with stakeholders other than patients; no personal data or health-related information was collected. Such laws state that ethics committee approval is required in the following circumstances: evaluation of clinical trials under EU Regulation 536/2014 carried out by ISS, EPRs, and other public bodies of national character; evaluation of protocols of other studies (including clinical, observational, diagnostic, public health, with medical devices, with the use of human biological material, with environmental ethics implications) carried out by ISS and, possibly, by other EPRs and public bodies of national character; Advice and expression of opinions on ethical aspects of activities and interventions of ISS, concerning its functions and duties as defined by the Statute of ISS, the Rules of Organization and Operation of ISS, and in compliance with applicable regulations. In the current paper, experts about Digital Therapeutics were involved in this study just to express their anonymous opinions about these technologies. Participants were informed of the purpose of this study and the policy for data collection, processing, and communication (aggregated and anonymously) coherently to the GDPR indications. Participants expressed their acceptance through an informed consent form. Participants were informed about the treatment of the data according to the European General Data Protection Regulation. Participants were also explicitly asked to accept the conditions regarding the data used for this study. The semi-structured interviews and focus groups used in this study were developed for this research and have never been published elsewhere.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Carrera, A., Manetti, S. & Lettieri, E. Rewiring care delivery through Digital Therapeutics (DTx): a machine learning-enhanced assessment and development (M-LEAD) framework. BMC Health Serv Res 24, 237 (2024).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: