Skip to main content

Developing and validating high-value patient digital follow-up services: a pilot study in cardiac surgery



The existing digital healthcare solutions demand a service development approach that assesses needs, experience, and outcomes, to develop high-value digital healthcare services. The objective of this study was to develop a digital transformation of the patients’ follow-up service after cardiac surgery, based on a remote patient monitoring service that would respond to the real context challenges.


The study followed the Design Science Research methodology framework and incorporated concepts from the Lean startup method to start designing a minimal viable product (MVP) from the available resources. The service was implemented in a pilot study with 29 patients in 4 iterative develop-test-learn cycles, with the engagement of developers, researchers, clinical teams, and patients.


Patients reported outcomes daily for 30 days after surgery through Internet-of-Things (IoT) devices and a mobile app. The service’s evaluation considered experience, feasibility, and effectiveness. It generated high satisfaction and high adherence among users, fewer readmissions, with an average of 7 ± 4.5 clinical actions per patient, primarily due to abnormal systolic blood pressure or wound-related issues.


We propose a 6-step methodology to design and validate a high-value digital health care service based on collaborative learning, real-time development, iterative testing, and value assessment.

Peer Review reports


Emergent digital solutions can impact healthcare positively, but it remains a challenge for service providers and developers to demonstrate the value of their digital innovations in healthcare [1, 2]. Conventional methodologies based on high investment for technology development followed by a robust clinical study for validation fail to cope with the fast pace of digital health innovations [3]. More pragmatic approaches are needed to support evidence gathering, incremental development, and accumulated knowledge base that cope with low initial resources and gradually demonstrate the value in real-world healthcare environments to support scale-up [2,3,4].

Design science research (DSR) develops knowledge from the design, development, and iterative evaluation of artefacts, i.e., incrementally improved solutions to real context problems [5]. DSR methodologies have been applied to innovation in digital services for healthcare and demonstrated to allow both a theoretical and experimental approach to real-world healthcare problems [6, 7].

Patient follow-up is essential in cardiovascular patients’ health pathway [8]. In cardiac surgery, complications during surgery or hospitalization may occur [9]. However, risk also extends to the postoperative period, leading to hospital readmission of 15 to 20% of patients during the first month and 30% in the first year [10,11,12]. Post-discharge telemonitoring can be a valuable tool to maximize surgery outcomes [13]. Despite limited studies on such programs in cardiothoracic surgery, its use is well implemented in chronic heart failure, positively impacting the quality of life and preventing hospital readmission and mortality [14, 15]. It also allows for a reduction of costs, both for the patient (hospital commutes and consultations) and the hospital (patients’ transportation, treatment of complications, and complementary diagnostic exams) [16]. When applied to post-surgery follow-up, there are not enough studies demonstrating value from these digital services, namely reducing readmissions and costs of care [12]. Alongside, studies of postoperative patient-reported outcomes measures (PROMs) are low in volume, and evidence needs to be strengthened, namely with more digital resources [17].

This project started with minimal investment, based on the collaboration of technology partners that made available a set of Digital Health Kits (DHK), composed of smartphones with internet connection and IoT devices developed for cardiac insufficiency [18]. Driven by the need to use the available resources to develop the Remote Patient Monitoring (RPM) system in such a way that could add value to the established follow-up program, we followed a participatory approach to implement a pilot study with the active involvement of all stakeholders [19]. We used the design science research methodology (DSRM) as a baseline framework to develop the digital solution, supporting the research team to iteratively respond to the real context challenges and assess its value [5, 20]. A lean startup approach was needed to help researchers to quickly start with the minimal development needed to initiate patients’ and clinicians’ experience with the new service.

This paper presents the work undertaken to implement a postoperative digital telemonitoring service for patients submitted to cardiac surgery in Hospital de Santa Marta, a central public hospital in Lisbon, Portugal. The service was developed from existing technological equipment supplied by technology partners that supported the project.

The primary objective of this study was to develop a digital transformation of the patients’ follow-up service of a cardiac surgery department, that would collect patient reported outcomes and respond to the real context challenges. The secondary objective was to evaluate the feasibility of such digital follow-up service for delivering high-value care to the patients.



This study followed the framework of DSRM and incorporated concepts from the Lean startup method [21] to start designing a minimal viable product (MVP) from the available equipment (DHK) and resources (provided by the clinical team and the research team). DSRM oriented the researchers to develop a rigorous design-oriented framework centered on the actual context and needs of patients and healthcare professionals by continuously communicating with them. The lean startup approach guided the team to bear uncertainty by quickly starting a pilot study and developing an overall solution in short deployment cycles. The entrepreneurial mindset drove the research team to validate learning during the DSR process and follow a “Build-Measure-Learn” sequence. During the pilot study, the overall RPM service was co-designed and developed by researchers, patients, and the clinical team, relying on short cycle times and rapid iteration with small batches of patients submitted to cardiac surgery [21,22,23]. We started with an MVP based on the available DHK. Furthermore, we developed a digital platform artifact to support the clinical team in managing patients’ data reported from the DHK.

The applied methodology followed a 6-step workflow, based on the DSRM, as depicted in Fig. 1 and described below.

Fig. 1
figure 1

Methodology to design and develop-test-learn cycles of the RPM service for cardiac surgery follow-up

Step 1 – Identify the problem, characterize the context, and engage the stakeholders

According to DSRM, the specification of the problem and motivation should anticipate any design or development to involve the teams and develop sustainable solutions. The project started with a core team of surgeons and nurses responsible for postoperative care in cardiac surgery patients. From four visits to the hospital service and conversations with the clinical team, a description of the existing postoperative follow-up procedure was made to understand the need for an RPM service that effectively collects patients’ outcomes helps to minimize post-surgery complications, and provides reassurance to patients. From each conversation, notes were taken to prepare drafts of the pathway and requirements of follow-up, that were discussed in the next meeting, until a final version was reached. Concerns on its feasibility to engage patients and nurses were discussed; patients’ adoption and nurses’ adherence to the service were set as requirements for the digital innovation. The output of this step was the design of the as-is patient care pathway. It describes the postoperative care program, which was agreed by the clinical team. The used process design was based on a Business Process Model and Notation (BPMN), that is used to support process modeling in healthcare and identify opportunities for quality improvement [24]. Also, we identified stakeholders relevant to the healthcare design and characterized their working needs, values, and expected benefits and contributions. This step allowed us to define a strategy for creativity of the involved stakeholders, and improvement of understanding of their potential contributions to the new digital service design. Not only were patients and clinical teams considered to participate, but technology providers who invested in this proof-of-concept were engaged and motivated towards optimal collaboration, as well.

Step 2 – Map a new patient pathway with the clinical team for an RPM service and prepare a validation study

This step was a creative stage of the project, where, according to the DSRM, the objectives are defined. In two meetings at the hospital, we moderated a research discussion with the clinical team on use cases and scenarios for a digital remote follow-up service after cardiothoracic surgery. First, the technological opportunities (the DHK) were presented to the clinical team as the available resources to implement an RPM service. Then, the requirements for the service (target population, the outcome variables with relevance for follow-up, and the period for telemonitoring) were discussed and agreed upon by the clinical team. By suggestion of the clinical team, a patient whom we called the patient zero voluntarily tested the available technology as-is to assess its feasibility in terms of patient easiness-to-use and technological robustness. Finally, technology providers assessed and discussed the viability (based on the constraints of the low initial investment in technological development) of the proposed requirements. After making necessary adjustments, we discussed these with the clinical team, and a final viable consensus was reached to a new patient pathway that included a RPM service. The output of this step was a new to-be patient pathway and the design of a pilot study to test the new digital follow-up service. The protocol for the pilot study was submitted to the Ethical Committee of the hospital.

Step 3–5 – Develop-test-learn in iterative cycles (DSRM cycles)

After the objectives were set and the new follow-up pathway was defined and validated by the clinical team and the technology providers, an iterative method of consecutive develop-test-learn cycles supported the development of the telemonitoring service. For each cycle, we developed new features, tested these with the participants in the study and collected their experience at the end of their 30-day follow-up period, with interviews taken in presence at the hospital.

We developed the first instantiation of the RPM service with an MVP based on minimal development efforts for a rapid move to a demonstration. The MVP was developed and tested in real context. The first patient was selected with the criteria of being admitted for cardiac surgery and having high potential motivation to collaborate on a digital remote monitoring experience. After the first MVP, we conducted significant improvements. Successive instantiations were developed, tested and evaluated with patients, following an iterative method of consecutive develop-test-learn cycles. After the 30-day telemonitoring period of each patient in each cycle, the patient experience was collected through interviews in person, at the hospital. From the reported experience, and clinicians’ feedback, we identified new requirements and defined a roadmap for further development iterations and the number of patients to test each iteration. During in presence weekly follow-up meetings with the research team, nurses and surgeons were asked for feedback on their experience with the RPM and returned necessary inputs on their needs and suggestions. These were noted and considered the priority requirements to implement in the next iteration cycle.

Software development and User Experience were relevant skills to perform the develop-test-learn cycles. The output of the pilot study was a telemonitoring solution with all the necessary features to make it feasible for cardiothoracic surgery care.

Step 6 – Communicate and prepare for scale-up

According to the literature, following the DSRM, communication “establishes repeatability of the research project and builds the knowledge base for further research extensions” [5]. We developed a communication plan for each stakeholder involved to raise awareness of the project progression, opportunities, and pitfalls.

Support digital and health literacy to patients

It has been demonstrated that patients’ active role significantly impacts the innovation of services, products, or processes in healthcare [25]. Some authors highlight responsiveness and reflectiveness to be part of the process [26, 27]. This step is fundamental to inform and involve the patient, as well [27].

A 2-page written guide illustrating the reporting procedure was delivered to the patient as supportive material. Furthermore, text messages to support health literacy during the recovery period were implemented. The nurse team defined a set of messages organized in a 30-day schedule to be sent to patients’ smartphones when they daily reported data. At the end of each telemonitoring period, when patients returned to the hospital for a clinical appointment, two researchers assessed each patient’s experience by a questionnaire and a structured interview.

Pilot study and evaluation


Participants were both the clinical team responsible for the follow-up service and 30 patients. The sampling size was based on a flat rule of thumb based on theoretical optimal values of pilot trial sample size, which the clinical team considered a feasible dimension for the context of this pilot study [28].

All patients submitted to the cardiac surgery ward were eligible. The clinical team selected the patients during the post-surgery hospitalization, with the following selection criteria: ability to read and write, having a mobile phone and willingness to participate. Patients who could not manage the smartphone of the DHK, either due to functional limitations or very low digital literacy, and did not have the daily support of a caregiver, were excluded. Due to specific pandemic organization of the public health system, the hospital was COVID-19 free, i.e., patients with COVID-19 were transferred to another hospital that was referenced for that purpose.

After being selected by the clinical team, each patient was invited to participate in a 30-minute education session. One nurse and two researchers from the telemonitoring support team were present in each session. Four parts composed the session: 1) the nurse explains the project’s main goals and the telemonitoring data process and assures that the patient understands them; 2) the support team shows all the steps that the patient needs to perform when at home, and checks the patient’s ability to perform each action; 3) the patient independently repeats the routine by following the provided guidelines, and self-assessed his/her performance deciding whether he/she wants to participate, and if so, 4) the patient voluntarily signs the study informed consent, receives a 1-page instruction and the support team provides their phone contact, in case of any equipment failure or misuse. Whenever the patient was willing to participate but had very low digital literacy, the caregiver received the instructions to support the patient.

Instruments and evaluation metrics

The DHK included a smartphone with a SMARTBEAT app to collect data from a smartwatch (to measure steps and continuous heart rate), a sphygmomanometer (to measure blood pressure and heart rate), and a scale (to measure bodyweight) [18]. A chatbot application was also included to exchange messages and a picture of the wound. The selection of patients was independent of their home conditions related to internet connection because each kit included a 4G card to exchange data via a telecom network.

Patient experience

At the end of the telemonitoring program, patients’ experience was collected using a Portuguese validated version of User Experience Questionnaire (UEQ), the Net Promoter Score (NPS), and three open questions related to the recovery period: “What was most important to you during the recovery period?”, “What was most difficult?” and “What would you recommend to improve the follow-up service?” [29, 30]. NPS was used in this study as a metric for patient adoption that is simple to apply to patients with low literacy. This scale is based on a single question: “how likely are you, on a scale from zero to 10, to recommend telemonitoring to a friend or a colleague?”. Responders are grouped according to scores: promoter (9, 10), passive (7, 8) and detractor (< 7). UEQ assesses user experience and contains 26 items organized in 6 scales: attractiveness, perspicuity, efficiency, dependability, stimulation, novelty. Each item is scored from − 3 (horribly bad) to + 3 (extremely good), 0 is a neutral answer. Each scale is based on a set of items and its score is calculated as the mean of its items’ scores. The questionnaires were applied in a paper-pencil form at the hospital.

Feasibility of digital service

Feasibility was measured by: (i) patient’s adoption and adherence, (ii) clinical team’s adherence to the telemonitoring service and engagement (iii) the rate of technical support occurrences during the pilot. Further, we observed how the RPM service was used for clinical support to patients, by analyzing actions that were triggered by patients’ reported data.

Patient adoption was assessed with the NPS. Patient adherence was measured as the ratio of the number of days that each patient-reported outcomes to the total days that the patient had the DHK. Indicators for clinical adherence were: the number of daily accesses to the platform and the number of clinical interventions generated from data in the telemonitoring platform. Indicators for clinical engagement were: the total number of clinical users of the RPM platform.

The clinical effectiveness of the RPM service was analyzed by comparing the critical incidents observed in the group of patients that participated in the pilot study with a control group of patients. Each participant was compared with patients from a hospital surgery follow-up registry of critical incidents (readmissions, surgeries, death) that matched the age, sex and type of cardiac surgery.

Develop-test-learn cycles

During the pilot study, patients were recruited in 4 stages. Each stage corresponded to a develop-test-learn cycle, as described in previously. The defined strategy was to follow a 4-cycle iterative process, with batches of a growing number of patients: 1,5,10 and 14, respectively.

Data analysis

Simple descriptive statistical analysis was used for evaluating the results. We used the Wilcoxon Signed-Rank Test to compare the critical incidents of the participants with the average of the registration of critical incidents for a control group from the same cardiothoracic surgery department [31]. The control group was selected from a database of patients that were operated in the same cardiothoracic surgery department in the last 10 years. Patients in the study were compared on a 1-to-n matching, where the n was the group of patients that matched age/sex/type of surgery (the 3 main factors influencing the risk of surgery) of each participant. A p-value of 0.05 was considered the threshold for statistical significance.

Interviews with the patients were manually transcribed. An inductive thematic analysis was used to identify the most relevant topics raised by patients concerning the clinical support in postoperative rehabilitation.



This overall methodology was implemented for 16 months, from June 2019 to October 2020.


Stakeholders were identified and interviewed to identify their specific requirements for the context of the digital telemonitoring service. Table 1 describes each stakeholder’s expectations and their level (low/medium/high) of importance and influence in the design of the digital health service. The clinical leaders were identified as the ones with more influence in the design of the RPM service. The actual follow-up service and the process were characterized and validated by clinicians.

Table 1 Characterization of the stakeholders in the current project


Researchers met the clinical team in three meetings to design the new pathway and define the outcomes to be monitored (Fig. 2). It was defined that patients had to report, once a day, measures that were relevant to prevent most common complications being blood pressure, heart rate, weight, and steps. Also, a picture of the wound had to be sent. Symptoms related to pain, dyspnea, feet edema, blackouts, palpitations, and wound, were collected via a 6-item questionnaire. The RPM service was set for the first month after hospital discharge, a critical period of higher complication rate. Alarm rules based on the collected outcomes were defined. The nursing team had to access to patients’ data every day, between 2 pm and 4 pm, and patients were instructed to daily report until 12 pm. Researchers worked with DHK suppliers to adjust it to the defined pathway. Moreover, the requirements and resources needed to cover the missing parts, i.e., features not implemented in the available DHK, were identified.

Fig. 2
figure 2

The new telemonitoring process: process underwent in the hospital (yellow) and process implemented by the RPM service (blue)

Steps 3–5

The pilot study started 3 months after the first step. Between October 2019 and October 2020, 30 patients accepted to participate in the study and used the RPM service after being discharged from cardiac surgery ward. From this group, one patient left the telemonitoring program after the first week, due to mental health conditions. The patients’ average (±standard deviation) age was 60.6 ± 13.5 years old, 15 (50%) of the participants were female. In terms of geographical distribution, half of the patients lived more than 100 km away from the hospital, the furthest district being 277 km. Patients’ average education level was 2, according to the European Qualifications Framework (EQF) (

As we were informed, none of the patients was infected with the new coronavirus since the cardiac surgery service was set to accept non-covid patients exclusively. Patients’ recruitment was interrupted from March to May 2020 due to workflow reorganization to cope with the pandemic. Nevertheless, none of the patients already using the telemonitoring system interrupted the follow-up during that period. Besides the interruption of recruitment for 2 months, there were no other specific problems related to the COVID-19 context with the pilot study.

The DHK suffered no modifications; all patients had the same setting. This helped us to identify the need for development of an application to manage the telemonitoring data, engage the clinical team and assess value indicators. The pilot study started without developments from the application to manage patients’ data (secure links to patients’ reports were manually sent to one contact of the clinical team by email). Nurses and surgeons were proactive in suggesting features and modifications to the system or reporting satisfaction.

The iteration process, developments, and feedback used to iterate the next cycle are summarized in Table 2. Most of the developments were for the clinical side of the RPM system. The patient side of the RPM was kept the same: a smartphone to collect weight, steps, blood pressure, 6 questions related to symptoms, and the receipt of the picture of the wound. A minor change was implemented for the patient: to receive literacy text messages from nurses, just after the daily data reporting procedure. This feature was introduced in the fourth develop-test-learn cycle.

Table 2 Description of the four iterations performed during the pilot study to develop the digital platform for managing data from the clinical side, based on clinicians’ and patients’ feedback

Step 6

Further research funding was obtained from dissemination activities to develop intelligent interaction and risk prediction from telemonitoring data. Also, communication of the results to technology partners is moving investment interest to scale up the solution.

Support to patients’ literacy

Education session for digital literacy

All selected patients received the DHK and learned how to use it during the education session. During this session, one withdrawal was registered from a patient that lost interest in participating after trying the measurement procedure. Three other patients with very low digital literacy but willing to participate involved the informal caregivers in the education session to ensure data reporting with the DHK.

Text messages for health literacy

As a feature that was suggested by the clinical team, a list of text messages was created to send to the patients via the chatbot. These messages were based on the contents of the hospital flyer that nurses provide to the patients. Messages were divided into six categories: informative, educational/preventive, motivational, commemorative, technical support, and alert. Withal, the nurse team defined a scheduled plan to deliver recommendations for better recovery during the 30 days, weekly.

Messages were implemented in the last develop-test-learn cycle, applied to 14 patients (Table 2). We divided the scheduled messages by the days of the week so patients would not receive them all on the same day. The trigger mechanism for sending the text messages was the daily receipt of the photograph of the wound from the patient, as depicted in Fig. 3. This was a deliberate strategy to increase patients’ attention to these educational messages, as these were immediately sent in response to patient’s message.

Fig. 3
figure 3

Simulation of a literacy message sent in reply to the daily picture of the surgical wound that is sent by the patients (this image includes a fiction name and is translated to English language)


Patient experience

We interviewed 26 participants after the 30-day telemonitoring period. It was not possible to interview 3 of the participants on the 30-day appointment at the hospital. This is justified by difficulties in coordinating the patients’, clinical, and researchers’ schedules. Nevertheless, these participants engaged in the whole telemonitoring process and reported data that supports this study.

From the responses to the three open questions, it was observed that ten patients (38%) explicitly reported that the remote monitoring allowed them to feel safer, with 3 (12%) referring to the relevance of the collected health data and 9 (35%) highlighting the support and interest demonstrated by the healthcare professionals in their recovery. Four patients (15%) reported difficulties in using the equipment related to taking the wound pictures. It is to note that the questions did not relate directly to the DHK but to what was most relevant in postoperative rehabilitation. Some statements of the patients that relate to the use of DHK are:

  • “I felt like I was at the hospital, that I was being accompanied by them all...”

  • “Yesterday, I was thinking: when I get there to return the devices, I am going to have a surprise, they will give me one of this [referring to the DHK]...”

  • “It is a friend that we do not see, but that is there with us every day [referring to the literacy messages in the chatbot].”

An NPS of 84 was obtained, reflecting an excellent level of patient satisfaction [32, 33]. Furthermore, 88% of patients were promoters (score of 9 or 10), enthusiastically using the technology and referring it to others, 8% were passives (score of 7 or 8), i.e., satisfied but not enthusiasts of the service, and 4% detractors (scores of 0 to 6), i.e., non-supporters of the service.

After applying the full UEQ questionnaire to 4 patients, we observed that this group of patients was not able to answer most of the questions, due to their low literacy level and complexity of the UEQ concepts. For the following participants, we applied only the scale UEQ-Stimulation (items 5,6,7,18 of UEQ), which refers to concepts that we had observed that patients could understand and relate to their experience (Table 3). From the 26 answers collected for the UEQ-Stimulation, the estimated mean value was 2.49 (confidence interval: 2.32–2.6, p = 0.05), as described in Table 3. The technology achieved an excellent evaluation (score higher than 1.55).

Table 3 Results of the Stimulation items of the UEQ


The mean score of NPS of 84 indicates high patient adoption of the digital telemonitoring service. Two patients even mentioned their willingness to use the DHK for some more time. Patients’ adherence was, on average, 91.6 (±15.1), all above 88 except for 4 patients: two had difficulties in using technology and were depended on caregivers’ availability, and two others had been readmitted to the hospital due to clinical complications.

Every day during the pilot study, the clinical team assessed patients’ reported data. Three nurses and two surgeons were engaged to develop the first iterations of the telemonitoring system. A scale-up was observed in iteration 4. A larger group of 25 nurses requested to be assigned to have login access to full use the web application that supported the telemonitoring process to follow patients.

Evaluation related to quantitative service metrics was only possible from cycle 3 since we had not implemented the telemonitoring management platform in the first two iterations. For the total actions registered by the clinical team on the platform, we observed an average of 7 (±4.45) occurrences per patient, most (84.3%) related to clinical interventions (Table 4). The most frequent interventions were related to nursing calls to the patient for clarification about the outcomes (measured or reported), and the least frequent were related to hospital readmission (Table 5). These interventions were triggered by data reported by the patients in the RPM system and accessed by the clinical team through the telemonitoring platform.

Table 4 Results of the analysis of the occurrences registered by the nurse team in the telemonitoring platform during the pilot study
Table 5 Description of the clinical interventions that were triggered by the telemonitoring system

Occurrences related to technical support (15,7%) were due to the synchronization of the smartwatch, or when the patient did not send the daily picture of the wound. Most of these problems were solved by phone calls between the patient and the technical support team.

Clinical interventions were labelled by nurses when they registered them in the telemonitoring application. Most of the clinical interventions were related to abnormal systolic blood pressure (19.5%) or issues related to the wound (12.9%).

The number of entries in the registry of critical incidents that matched each participant varied from 3 to 47 (mean:16,97). We used the Wilcoxon Signed-Rank Test [31] to compare the critical incidents between each participant and the mean of his/her control group. The average of critical incidents of the telemonitored group was 0.07 ± 0.26, and the average of the control group was 0.13 ± 0.13. The result of the test was w = 41 (p-value = 0.01), rejecting the null hypothesis that there is no difference between means.


This paper presents a collaborative and iterative method developed to drive a digital innovation context in a cardiac surgery department of a public hospital in Portugal. The aim was to design, implement and validate a RPM service for post-surgery follow-up.

Due to the COVID-19 outbreak, the pilot study was interrupted for just 2 months due to necessary organizational rearrangement. However, it continued with recognized added value to the patients, when there were clinical guidelines to strictly avoid hospital visits after cardiac surgery. The average age of the participants is slightly below the expected average in cardiac surgery, probably biased by the small sample and a higher willingness of younger patients to participate in the study and use the DHK. Regarding the sex distribution, it is coherent with the population in the hospital department is balanced with similar proportions of males and females [34].

DSRM enabled us to design an artifact in a participatory way, responding to the identified problem and engaging stakeholders. After the artifact was built, the lean-startup approach was used to start experimentation with a minimal valuable instantiation, based on available resources and minimal investment.

Our approach responds to the need to research novel methodologies to support faster technology development-validation cycles while ensuring its value demonstration. The high pace of technological innovation, in contrast to the long process of clinical validation is challenging in healthcare [1, 2]. The traditional product development process dedicates a long effort to analyzing the needs, defining the requirements, developing a prototype, and ultimately, testing and validating before market entry [35]. In healthcare, this process is heavier due to the need to validate and certificate products before reaching the market. This classical approach for product development to be validated is slow, and the delay risks digital technologies: being too late when getting to the market, being unable to prove value after a long development process, and failing to test stakeholders’ adoption in real-world settings [36, 37]. In this work, we developed the RPM service while simultaneously performing validation in different domains during the pilot study. According to the global digital health scorecard proposed by Mathews et al. [38], our methodology included the necessary elements for usability (satisfaction and adoption from patients and clinical team), clinical (reduction of critical incidents), and technical (overall performance of the system in real-world context) validation. Although it is out of the scope of this paper, we already have preliminary results of a cost analysis, which will fill the cost element of the validation scorecard [37].

The results from this study suggest the potential of patient-reported outcomes monitoring to reduce critical clinical incidents. However, a larger study is needed to support the statistical robustness of the comparison between groups of patients. Moreover, we observed that the most frequent complications solved with the RPM service in this study are comparable with a previous study, which used a similar RPM setup in post-cardiac surgery care [12]. For validation of patient experience, we found that UEQ was difficult to apply due to complex concepts that were not in the context of the patients with low literacy. Further research is needed to identify adequate instruments to evaluate the experience with technology that fits the healthcare context, being the short version UEQ-S [39, 40]. Nevertheless, the reduced rate of dropouts (only one patient) and high patient satisfaction in our study suggest that most patients’ needs were addressed.

While the clinical benefits of digital technologies in RPM have been recognized, the adoption is challenging [41]. Participatory methodologies may be a catalyzer for successful implementation and adoption of digital technologies in healthcare [42,43,44,45]. In opposition, disruption in clinical workflows by imposing an external digital tool hamper the adoption and demonstration of its value to the patients and the hospital. Accordingly, the co-design of a new digital service, along with its iterative integration in the clinical workflows, as we used, is an essential part of our contribution to optimize the feasibility and adoption of the technology [46]. Our method considered the stakeholders’ needs and motivations, the process, and the digital tools, as parts of the RPM service design. This shift from focusing on product implementation to service design has been suggested as a needed approach to ensure value co-creation through patients’ and healthcare professionals’ experiences [47, 48].

The continuous involvement of the research team with patients and clinical professionals improved their compliance with the technology, as previously identified in other studies and stressed in the Responsible Research and Innovation guidelines [1, 42, 49, 50].

The role of the researchers was also to challenge the clinical team by proposing new features and assessing acceptance and adoption. A combined strategy of demand-pull and technology-push successfully engaged the clinical team in the innovation process [48, 51]. Our iterative methodology and interdisciplinary research team stimulated innovation and promoted the adoption under this scenario.

A limitation of this study is that, despite the concern of providing a stable instantiation to the patient, the iterative method imposed different experiences to the clinical team and may hinder the reliability of the results. Another limitation is the small number of participants, which did not allow a robust statistical analysis to claim clinical effectiveness of the RPM service. Also, the small number facilitated close support of the technical team and clinicians to patients, probably increasing their adoption and masking potential outliers in terms of negative experiences with the RPM. A clinical study with higher number of patients and a stable version of the RPM service, implemented as a final iteration of our methodology, would increase the robustness of the results. From the results presented in this paper, we plan the scale-up to a randomized clinical trial with 300 patients to test the impact of a more extended digital follow-up period and analyze for which groups of patients the RPM returns higher value. Furthermore, we are working on intelligent prediction models based on patients’ outcomes, that can support a personalized care plan and high-value decisions for this follow-up service [52].

The future application of the method proposed in this work to other use-cases will be essential to validate, generalize, and improve it.


We present the method we undertook to develop, implement, and validate a digital telemonitoring follow-up service in cardiac surgery care. We parted from a need to improve follow-up in cardiac surgery and the opportunity to use resources from our technological partners.

This study contributes to identifying methods that can be applied by digital health innovators, considering the demand for addressing clinicians’ and patients’ needs, available resources, and demonstration of value in short cycles of development and real-context validation.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.



Design science research methodology


Digital Health Kits




Minimal viable product


Net Promoter Score


Patient-reported outcomes measures


Remote Patient Monitoring


User Experience Questionnaire


  1. Peirce SC, Hardisty AR, Preece AD, Elwyn G. Designing and implementing telemonitoring for early detection of deterioration in chronic disease: defining the requirements. Health Informatics J. 2011;17:173–90.

    Article  PubMed  Google Scholar 

  2. Drummond M, Griffin A, Tarricone R. Economic evaluation for devices and drugs—same or different? Value Health. 2009;12:402–4.

    Article  PubMed  Google Scholar 

  3. Guo C, et al. Challenges for the evaluation of digital health solutions—a call for innovative evidence generation approaches. Npj Digit Med. 2020;3:110.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Murray E, et al. Evaluating digital health interventions. Am J Prev Med. 2016;51:843–51.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Hevner M, Park, & Ram. Design science in information systems research. MIS Q. 2004;28:75.

    Article  Google Scholar 

  6. Mulgund P, Sharman R, Rifkin D, Marrazzo S. Design, development, and evaluation of a telemedicine platform for patients with sleep apnea (Ognomy): design science research approach. JMIR Form Res. 2021;5:e26059.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Lapão LV, et al. Implementation of digital monitoring services during the COVID-19 pandemic for patients with chronic diseases: design science approach. J Med Internet Res. 2021;23:e24181.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Driggin E, et al. Cardiovascular considerations for patients, health care workers, and health systems during the COVID-19 pandemic. J Am Coll Cardiol. 2020;75:2352–71.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. Crawford TC, et al. Complications after cardiac operations: all are not created equal. Ann Thorac Surg. 2017;103:32–40.

    Article  PubMed  Google Scholar 

  10. Efthymiou CA, O’Regan DJ. Postdischarge complications: what exactly happens when the patient goes home? Interact Cardiovasc Thorac Surg. 2011;12:130–4.

    Article  PubMed  Google Scholar 

  11. Khoury H, et al. Readmission after surgical aortic valve replacement in the United States. Ann Thorac Surg. 2020;110:849–55.

    Article  PubMed  Google Scholar 

  12. McElroy I, et al. Use of digital health kits to reduce readmission after cardiac surgery. J Surg Res. 2016;204:1–7.

    Article  PubMed  Google Scholar 

  13. Park DK, et al. Telecare system for cardiac surgery patients: implementation and effectiveness. Healthc Inform Res. 2011;17:93.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Pekmezaris R, et al. Home Telemonitoring in heart failure: a systematic review and meta-analysis. Health Aff (Millwood). 2018;37:1983–9.

    Article  Google Scholar 

  15. Yun JE, Park J-E, Park H-Y, Lee H-Y, Park D-A. Comparative effectiveness of telemonitoring versus usual care for heart failure: a systematic review and Meta-analysis. J Card Fail. 2018;24:19–28.

    Article  PubMed  Google Scholar 

  16. Seto E. Cost comparison between Telemonitoring and usual care of heart failure: a systematic review. Telemed E-Health. 2008;14:679–86.

    Article  Google Scholar 

  17. Mori M, et al. Characterizing patient-centered postoperative recovery after adult cardiac surgery: a systematic review. J Am Heart Assoc. 2019;8(21):e013546.

  18. Silva-Cardoso J, et al. SmartBEAT: a smartphone-based heart failure telemonitoring solution. In: Henriques J, Neves N, de Carvalho P, editors. XV Mediterranean conference on medical and biological engineering and computing – MEDICON 2019, vol. 76: Springer International Publishing; 2020. p. 1369–76.

  19. Clemensen J, Rothmann MJ, Smith AC, Caffery LJ, Danbjorg DB. Participatory design methods in telemedicine research. J Telemed Telecare. 2017;23:780–5.

    Article  PubMed  Google Scholar 

  20. Peffers K, Tuunanen T, Rothenberger MA, Chatterjee S. A design science research methodology for information systems research. J Manag Inf Syst. 2007;24:45–77.

    Article  Google Scholar 

  21. Eisenmann T, Ries E, Dillard S. Hypothesis-driven entrepreneurship: the lean startup. Harv Bus Sch Backgr Note. 2011; Case No. 812-095. Available at SSRN:

  22. NEJM Catalyst. What Is Lean Healthcare? (2018).

    Google Scholar 

  23. Bhattacharyya O, et al. Redesigning care: adapting new improvement methods to achieve person-centred care. BMJ Qual Saf. 2019;28:242–8.

    Article  PubMed  Google Scholar 

  24. Pufahl L, Zerbato F, Weber B. Weber I. BPMN in healthcare: Challenges and best practices. Information Systems. 2022;107:102013.

  25. Hibbard JH, Greene J. What the evidence shows about patient activation: better health outcomes and care experiences; fewer data on costs. Health Aff (Millwood). 2013;32:207–14.

    Article  Google Scholar 

  26. Stilgoe J, Owen R, Macnaghten P. Developing a framework for responsible innovation. Res Policy. 2013;42:1568–80.

    Article  Google Scholar 

  27. Iakovleva T. Responsible innovation in digital health. In: Responsible Innovation in Digital Health: Edward Elgar Publishing; 2019. p. 1–8.

  28. Whitehead AL, Julious SA, Cooper CL, Campbell MJ. Estimating the sample size for a pilot randomised trial to minimise the overall trial sample size for the external pilot and main trial for a continuous outcome variable. Stat Methods Med Res. 2016;25:1057–73.

    Article  PubMed  Google Scholar 

  29. Cota MP, Thomaschewski J, Schrepp M, Gonçalves R. Efficient measurement of the user experience. A Portuguese Version. Procedia Comput Sci. 2014;27:491–8.

    Article  Google Scholar 

  30. Hamilton DF, et al. Assessing treatment outcomes using a single question: the net promoter score. Bone Jt J. 2014;96-B:622–8.

    Article  CAS  Google Scholar 

  31. Woolson RF. Wilcoxon signed-rank test. In: D’Agostino RB, Sullivan L, Massaro J, editors. Wiley encyclopedia of clinical trials: John Wiley & Sons, Inc.; 2008. p. eoct979.

  32. Reichheld FF. The one number you need to grow. Harv Bus Rev. 2003;81:46–54, 124.

    PubMed  Google Scholar 

  33. Sauro J, Lewis JR. Did we meet or exceed our goal? In: Quantifying the user experience: Elsevier; 2012. p. 41–62.

  34. Coelho PNMP, Miranda LMRPC, Barros PMP, Fragata JIG. Quality of life after elective cardiac surgery in elderly patients. Interact Cardiovasc Thorac Surg. 2019;28:199–205.

    Article  PubMed  Google Scholar 

  35. Cooper RG. Perspective: the stage-gate ® idea-to-launch process—update, What’s new, and NexGen systems. J Prod Innov Manag. 2008;25:213–32.

    Article  Google Scholar 

  36. Riley WT, Glasgow RE, Etheredge L, Abernethy AP. Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise. Clin Transl Med. 2013; 2(1):10.

  37. Pham Q, Wiljer D, Cafazzo JA. Beyond the randomized controlled trial: a review of alternatives in mHealth clinical trial methods. JMIR MHealth UHealth. 2016;4:e107.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Mathews SC, et al. Digital health: a path to validation. Npj Digit Med. 2019;2:38.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Maramba I, Chatterjee A, Newman C. Methods of usability testing in the development of eHealth applications: a scoping review. Int J Med Inf. 2019;126:95–104.

    Article  Google Scholar 

  40. Schrepp M, Hinderks A, Thomaschewski J. Design and evaluation of a short version of the user experience questionnaire (UEQ-S). Int J Interact Multimed Artif Intell. 2017;4:103.

    Google Scholar 

  41. Vegesna A, Tran M, Angelaccio M, Arcona S. Remote patient monitoring via non-invasive digital technologies: a systematic review. Telemed E-Health. 2017;23:3–17.

    Article  Google Scholar 

  42. Cresswell KM, Bates DW, Ten Sheikh A. Key considerations for the successful implementation and adoption of large-scale health information technology. J Am Med Inform Assoc. 2013;20:e9–e13.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Dugstad J, Eide T, Nilsen ER, Eide H. Towards successful digital transformation through co-creation: a longitudinal study of a four-year implementation of digital monitoring technology in residential care for persons with dementia. BMC Health Serv Res. 2019;19:366.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Morton E, Barnes SJ, Michalak EE. Participatory digital health research: a new paradigm for mHealth tool development. Gen Hosp Psychiatry. 2020;66:67–9.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Hanley J, Pinnock H, Paterson M, McKinstry B. Implementing telemonitoring in primary care: learning from a large qualitative dataset gathered during a series of studies. BMC Fam Pract. 2018;19:118.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Ware P, et al. Evaluating the implementation of a mobile phone–based telemonitoring program: longitudinal study guided by the consolidated framework for implementation research. JMIR MHealth UHealth. 2018;6:e10768.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Lee D. A model for designing healthcare service based on the patient experience. Int J Healthc Manag. 2019;12:180–8.

    Article  Google Scholar 

  48. Shaw J, et al. Beyond “implementation”: digital health innovation and service design. Npj Digit Med. 2018;1:48.

    Article  PubMed  PubMed Central  Google Scholar 

  49. Maeder A, Poultney N, Morgan G, Lippiatt R. Patient compliance in home-based self-care telehealth projects. J Telemed Telecare. 2015;21:439–42.

    Article  PubMed  Google Scholar 

  50. Lettice F, Rogers H, Yaghmaei E, Pawar KS. Responsible research and innovation revisited: aligning product development processes with the corporate responsibility agenda. In: Brem A, Viardot E, editors. Revolution of innovation management: Palgrave Macmillan UK; 2017. p. 247–69.

  51. Di Stefano G, Gambardella A, Verona G. Technology push and demand pull perspectives in innovation studies: current findings and future research directions. Res Policy. 2012;41:1283–95.

    Article  Google Scholar 

  52. Mollaei N, Londral A. Application of machine learning approaches for long of stay in acute intensive care unit in cardiothoracic surgery patients; 2021.

    Google Scholar 

Download references


The pilot study included a great contribution from the team of Fraunhofer AICOS, that actively supported the functional requests of DHK.


This work was supported by Fraunhofer AICOS and Vodafone Portugal and funded by the National Foundation of Science and Technology under the projects DSAIPA/AI/0094/2020 and Lisboa-05-3559-FSE-3.

Author information

Authors and Affiliations



A.L. led the writing of the manuscript. A.L.,S.A.,J.S.,P.C.,J.G.F.,L.L. contributed to developing and refining the concepts. P.D.,C.R. collected and analyzed data from the pilot study. P.D., J.S., F.M., A.G., J.F., H.S., C.V. S.A. collected data and developed the features and concepts in RPM system. R.S., P.D., C.R., A.G., H.S., C.V. contributed to the literacy methodology. P.C., J.G.F. led the clinical research in the pilot study. All authors provided comment, revisions, and approved the final manuscript.

Authors’ information

Not applicable.

Corresponding author

Correspondence to A. Londral.

Ethics declarations

Ethics approval and consent to participate

The clinical study was approved by the Ethical Committee of Centro Hospitalar Lisboa Central and given the study registration number: CA2651. A written informed consent was obtained from all subjects that participated in the study. All methods were carried out in accordance with Declaration of Helsinki and the European General Data Protection Regulation.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Londral, A., Azevedo, S., Dias, P. et al. Developing and validating high-value patient digital follow-up services: a pilot study in cardiac surgery. BMC Health Serv Res 22, 680 (2022).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: