Skip to main content
Download PDF
  • Research article
  • Open access
  • Published:

Measuring the effectiveness of digital nursing technologies: development of a comprehensive digital nursing technology outcome framework based on a scoping review

BMC Health Services Research volume 20, Article number: 243 (2020) Cite this article

Abstract

Background

Digital nursing technologies (DNT) comprise an expanding, highly diverse field of research, explored using a wide variety of methods and tools. Study results are therefore difficult to compare, which raises the question how effectiveness of DNT can be adequately measured. Methods currently used might not be sufficient for certain specific nursing contexts. A comprehensive outcome framework that shows the multitude of possible outcome areas could be useful to generate more comparable results. The aim of the present study is to develop an outcome framework for DNT and to indicate which outcome areas have been most frequently evaluated in previous studies and how this has been done.

Methods

We combined an inductive and deductive approach to develop the framework. The numerical analysis is based on a scoping review focussing on the effectiveness of DNT for persons in need of care, formal or informal caregivers or care institutions. Nine databases were included in the screening: Medline, Scopus, CINAHL, Cochrane Library, ACM Digital Library, IEEE Xplore, the Collection of Computer Science Bibliographies, GeroLit and CareLit. Additional literature searches and expert interviews were included.

Results

The developed framework comprises four outcome target groups and 47 outcome areas. There are considerable differences in the researched outcome areas for the individual outcome target groups. Persons in need of care were by far the most frequently surveyed, particularly with respect to their psychological health. There are much fewer studies on formal and informal caregivers, and it is particularly noticeable that the quality of life of both groups has rarely been investigated. Care process quality was most frequently researched for organisations.

Conclusion

We were able to provide a comprehensive DNT outcome framework, thereby identifying the outcome tools used and the less researched outcome areas. We recommend a detailed investigation of all areas and tools in future research projects with a view to initiating a discussion on the differing importance of existing outcome areas and on a standardisation of outcome tools. We also recommend the development of outcome areas for the macro level of effectiveness assessment.

Peer Review reports

Background

Research on digital nursing technologies is an emerging field. An initial analysis of 715 articles on Digital nursing technologies (DNT) showed the existing variety of research. The field is highly diverse and is explored using a multitude of methods and instruments, which makes the measured effects very difficult to compare [1]. The question how effectiveness of DNT can be adequately measured is becoming increasingly relevant. There are no comprehensive systematisations that can help to structure this measurement, which is why only frameworks that cover partial areas [2] of the field or from other related healthcare contexts such as eHealth [3] or HTA [4] can be used.

Many systematic reviews in the field of technology and nursing conclude that solid evidence with respect to effectiveness is lacking [5,6,7,8,9,10,11,12], mainly due to a weak level of evidence and the incomparability of the study results. For this reason, a comprehensive outcome framework detailing the multitude of possible outcome areas could be useful for generating more comparable results. A systematisation of outcome areas and outcome tools for DNT could promote scientific exchange, improve the comparability of results and facilitate the identification of research gaps. Hence, the aim of this research is to systematically develop a comprehensive outcome framework for DNT and to indicate which outcome areas have been most frequently evaluated in previous studies, and on this basis to provide an overview on research focuses and possible gaps in current research on DNT. The development of the framework and the mapping of current research are based on a scoping review. The categories of the framework were further elaborated on the basis of additional literature and expert knowledge.

Other frameworks, like the evaluation framework for Health Information Systems (HOT-fit) [13], the General Framework for Evaluating Health Information Technology [14], the Canadian Health Information Performance Framework [15], the OECD Framework for Health System Performance Measurement [16], the Infoway Benefits Framework [17] or the MAST Framework [18], have been proposed to structure the process of effectiveness evaluation in health care areas using digital technologies. However, the existing frameworks and categorization systems have different focal points, and none of them are – from our point of view – adequately geared to the specific needs of the complex nursing care context. Outcomes relating to caregivers, and in particular informal caregivers, are often neglected or overlooked. The Infoway Benefits Evaluation Framework [17] that has been proposed for the analysis of health information systems in Canada, and the Model for Assessment of Telemedicine (MAST) [18], pose an exception here, as they are very well elaborated. Both frameworks were developed in a similar context, but for different purposes. The frameworks do not specifically consider informal caregivers, but they nevertheless indicate important sub-areas for the evaluation of DNT (such as, for example, patient safety, care quality, access to care or organisational productivity) that should be integrated into a comprehensive DNT framework. However, they are inadequate and inappropriate for a specific application relating to DNT, because they have been developed for different purposes. In addition to these two frameworks, there is also a framework which especially displays the impact of ICT solutions on nursing care. The adapted version of the Nursing Care Performance Framework [2] focusses on organisational issues, formal caregivers and aspects relating to people in need of care. This makes a helpful contribution in these areas. However, outcomes for technologies on informal caregivers are not represented in this framework. Besides these frameworks, there is also specific work on the measurement of the effectiveness of technology for ageing people in general, including effects on physical and psychological health, mobility, social connectedness, safety, everyday activities and leisure [19].

As we have established, therefore, a number of different instruments for categorizing outcome measures and tools in healthcare contexts have already been developed. To the best of our knowledge, however, there is no extensive analysis of relevant outcome areas and outcome tools pertaining to digital nursing technology published in the English nursing literature. We therefore decided to develop a new comprehensive outcome framework that is applicable to the design of effectiveness evaluation studies in the field of digital nursing technologies.

Definitions

In order to elucidate the conceptual differences between the individual terms in this study, we shall first define our understanding of the most important terms. The main subject of this article is digital nursing technologies (DNT). DNT are required i) to support the immediate action of a caregiver (e.g. decision support systems for guideline compliance [20]); ii) to contribute to the self-reliance of the person in need of care in such a way that direct on-site care assistance can be avoided (e.g. ambient assisted living support at home [21]); iii) to substitute the nursing support by using technology (e.g. robot that measures vital signs prior to consultation) [22]; or iv) to support the training or education of nurses (e.g. high fidelity simulator systems [23]) [1]. Technological support may relate to the person in need of care, formal or informal caregivers, or to an organisational process.

We also distinguish between the terms “outcome measure” and “outcome tool”. An outcome measure is a specific measure used to quantify (quantitative) or gauge (qualitative) an effect, e.g., of an eHealth intervention, and an outcome tool is a specific instrument used to collect quantitative or qualitative data [24, 25]. Outcome tools or outcome measures are indicators that represent effects in a specific outcome area. A distinction is also made between the terms “outcome target group” and “outcome area”. The outcome target group refers to the assignment of the outcome of a technology to a specific group of people (e.g. formal caregivers) or to an organisation (e.g. hospital). The outcome area specifies the content layer on which an effect occurs (e.g. well-being or functional health). There is also a distinction between the terms ‘effectiveness’ and ‘efficacy’. Efficacy studies measure (expected) effects under ideal circumstances, effectiveness studies measure (beneficial) effects under “real world” conditions [26]. Since we have found an incoherent use of the terms in the studies included, we use the term “effectiveness” to cover both concepts. This decision will be further justified and discussed in the discussion section.

Objective and research question

The ultimate objective of this article is to develop an outcome framework for DNT that enables systematic classification into different outcome areas and can be used to support future effectiveness research. It is essential to the development of such a framework that past attempts to evaluate effectiveness in previous studies are understood. This review is thus guided by the following main research questions: (i) Which possible outcome areas for measuring effectiveness of digital nursing technologies can be identified? (ii) Which outcome areas have so far been the focal point of research on effectiveness of digital nursing technologies, and which areas have been researched less frequently or not at all? (iii) How has effectiveness been measured in previous studies?

Methods

Our analysis is based on a previous scoping review (phase one) [1], which we conducted on the basis of Arksey and O’Malley’s scoping review framework [27]. We used processual advice drawn up by Levac, Colquhoun et al. [28] to enhance the scientific process. This was particularly important because the search of the previous scoping review generated a large number of titles, which made the identification, selection and charting of the relevant studies very time consuming and resource intensive. We have tried to counteract this by using the advice to refine the search and selection strategy in an iterative process. We also jointly developed, tested and updated a data charting form that allowed us to review and extract each full text by one researcher, a second author was consulted in case of uncertainties regarding the classification.

The scoping review included 715 studies focussing on acceptance, effectiveness or efficiency DNT. The full search strategy, analysis and results of the scoping review are published in Krick et al. 2019 [1]. The initial analysis of the scoping review yielded strong indications that a more in-depth analysis of tools and research areas could be useful for further research. Therefore, we decided to extend the evaluation to include methodological questions in a second research phase. In the following, the method of the initial scoping review will be briefly presented, followed by an explanation of the method used in the second phase.

Methodical foundation of the initial scoping review

We screened 19.510 titles, based on a search in nine electronic databases, covering studies published between 2011 and March 2018. The initial scoping review (phase one) was conducted with a view to a broader research question insofar as studies were included that related to the acceptance, the effectiveness or the efficiency of a digital nursing technology.

Eligibility criteria of the initial scoping review and the analysis of effectiveness

In order to understand how the preselection of articles for this review took place, the eligibility criteria of Phase one will now be explained briefly. Articles were included if they reported on study results relating to acceptance, effectiveness (on any evidence level) or efficiency (including cost analysis); target settings include residential long-term care, formal and informal care at home, hospital care, palliative inpatient care, intensive care unit care, day-care centre care and cross-sectoral care. Based on this preselection, all studies that reported on the effectiveness of DNT were included for the second phase of analysis. In the next step, as we aimed to focus on effects on persons in need of care, caregivers and care organisations, all studies that aimed primarily at an educational environment and studies conducted in a laboratory environment were excluded. Studies in a laboratory environment were excluded as most of them measure technical effectiveness, which is not the subject of our analysis. The remaining articles were analysed according to the outcome areas, outcome measures and outcome tools to create an empirical basis for the DNT outcome framework. Again, studies were excluded that focussed only on technical effectiveness. Based on these limitations, 123 individual studies were subjected to the analysis presented in this article.

Development of the outcome framework and data assignment

A combined inductive/deductive approach was used to develop the outcome framework. A basic model was developed by drawing on the analysis of the effectiveness articles (n = 123) of the initial scoping review [1]. Two authors screened the full texts to identify all relevant information. The identified outcome measures and outcome tools of the studies were used to inductively derive a preliminary systematic draft of outcome areas. In addition, a narrative literature search was carried out in the databases PubMed and Google Scholar in order to review whether further outcome areas could be found or identified that were not considered in the 123 studies. We combined the search terms “nursing”, “framework”, “outcome”, “digital” and “technology” and decided which articles should be included in the analysis. The search focussed on studies that explicitly referred to specific frameworks, developed frameworks themselves or provided a systematic structure to measure effectiveness. We analysed the texts and reference lists for relevant frameworks or systematisations. This snowballing method is important for such complex search fields. It helped us to provide meaningful additional information as a supplement to the systematic approach [29]. The aim was to identify studies or other frameworks that describe or contain further outcome areas for DNT. All relevant information that could be used to determine further outcome areas was extracted.

Deductive reasoning was used to gather information from general frameworks and inductive reasoning was used for single studies. A definition of each potential outcome area was then drawn up by one author and discussed with the other authors as to their relevance and fit for the framework. The development process included a multi-stage discussion and iteration with multiple revisions to ensure high quality decisions on the outcome areas and their definitions. The outcome framework was then validated by a group of ten German experts in the measurement of effectiveness of DNT. At the time of the survey, the experts in question were all involved in projects evaluating different digital technologies in nursing in the German healthcare system. The experts were requested to report their own project experiences in order to supplement missing relevant outcome areas. This survey took place as part of a regular exchange among experts.

The final outcome areas of the framework described for persons in need of care were developed with strong references to the Nursing Outcomes Classification (NOC) criteria [30] and existing frameworks [17, 18]. The caregiver-related outcome areas were derived from the initial scoping review and supplemented by categories based on the literature review (e.g. [18, 31,32,33,34,35,36,37,38]), while the organisation-related outcome areas emanate from the literature review (e.g. [17, 18, 39,40,41,42,43,44,45]), supplemented by expert opinions. All information is included in the final definitions for each outcome area documented in the Additional files 1, 2, 3, 4.

Data assignment

In the last step of the second phase, the extracted outcome tools from all the studies included were assigned to the outcome areas in the outcome framework using the collectively developed criteria. One researcher re-reviewed all studies (n = 123) to ensure a consistent assignment of outcome measures to outcome areas based on the iteratively developed framework. As a result, the numerical analysis can be presented according to the DNT outcome framework.

Results

Analysis results

A total of 123 studies from the initial scoping review were included in the analysis. These studies refer to the following technology categories: ICT, robotics, monitoring, sensors, assistive devices, ambient assisted living and virtual reality as defined by Krick et al. (2019) [1]. A more detailed systematization of the included technologies together with a list of all included studies is provided in Additional file 5.

The PRISMA Flow Chart is presented in Fig. 1.

Fig. 1
figure 1

Search results and publication selection process

Full size image

Digital nursing technology (DNT) outcome framework

We developed an outcome framework to provide a systematic classification and orientation scheme for outcome measures and outcome tools in the field of DNT (Fig. 2). The classification of the framework differentiates between the four outcome target groups: persons in need of care, formal caregivers, informal caregivers and healthcare organisations. For persons in need of care and formal caregivers we also distinguish between individual effectiveness and individual-related organisational effectiveness. This distinction is used to categorize outcome measures or tools that clearly apply to an individual but are closely related to organisational effectiveness and thus in an intermediate area. Overall, the model comprises 47 different outcome areas (e.g. functional health, well-being, patient satisfaction). The outcome areas included in the framework refer to micro (individual) or meso (organisational) levels of evaluation [46]. The macro level is deliberately not included here because it implies different study perspectives and so far, has only been very rarely analysed in the field of DNT. Detailed definitions and examples for the individual categories are provided in Additional files 1, 2, 3, 4.

Fig. 2
figure 2

Outcome Framework

Full size image

Overall outcome areas

This chapter comprises a detailed analysis of all outcome tools and measures used in the 123 studies included (Table 1). The numerical analysis shows which outcome areas have or have not been extensively analysed with the respective measures or tools. Each study that used an outcome tool or outcome measure in a specific area is only included once in this analysis. Most of the included studies investigated the effectiveness of the technologies in question on persons in need of care (n = 77). Aspects of organisational effectiveness were measured in 45 studies and only 30 studies referred to caregiver outcomes (formal n = 20; informal caregivers n = 10). There are substantial differences in the researched outcome areas for the individual outcome target groups. Psychosocial health (n = 33), intensity of service utilization (n = 19) and organisational patient safety indicators (n = 19) for persons in need of care were measured much more frequently than needs assessment/ fulfilled needs (n = 3), health knowledge & attitudes (n = 3) or intersectoral continuity of care (n = 1). For formal caregivers the outcome areas most frequently covered are relationship to person in need of care (n = 7), guideline compliance (n = 7) and physical/psychological workload (n = 6). Well-being/quality of life, health-related quality of life (QOL), medical condition and caregiver burden were not measured once in our sample. The other outcome areas were only rarely analysed.

Table 1 Framework with overall results
Full size table

For informal caregivers, caregiver burden is the most frequently analysed outcome area (n = 7), while all other outcome areas were rarely analysed or not at all (e.g. knowledge & competences n = 0; compliance n = 0; use of caregiving support n = 0). The most frequently analysed outcome areas for healthcare organisations are care process quality (n = 21), operational efficiency (n = 21) and communication/social interaction (n = 15). Little research has been done on financial performance (n = 4), access to care (n = 1) or working conditions (n = 1), while aspects of recruitment and staffing were not evaluated at all.

Outcome areas of tools

This section provides a numerical analysis of all outcome tools included in the 123 articles. The numbers in Table 2 indicate the outcome tools used in each outcome area. The psychosocial state of persons in need of care was evaluated broadly using different tools (n = 69), as well as well-being/quality of life (n = 14), whereas social condition (n = 0) or health knowledge & attitudes (n = 0) were not analysed using tools. Formal caregivers were seldom examined using tools. Looking at the distribution of the available studies, psychological health (n = 3) was proportionally the most frequently measured outcome area for formal caregivers. Individual-related outcome areas such as medical condition or relationship to the person in need of care, or organisation-related individual outcomes such as employee satisfaction or impact on workload were not measured at all using outcome tools. Outcomes for informal caregivers were also rarely analysed using tools. The most frequently used concept was caregiver burden (n = 10). This is an overarching concept, which includes many of the single outcome areas listed for informal caregivers. A similar picture emerges for organisational factors. Only communication/social interaction (n = 4) and hospital/nursing home quality were evaluated using tools (n = 3). In the next sections, a detailed analysis is carried out to show which tools were used in the individual result areas.

Table 2 Numerical analysis of outcome tools
Full size table

Outcome tools used for specific target groups

Effectiveness relating to persons in need of care

In order to establish how past studies have attempted to measure the effectiveness of care technologies, we have listed all outcome tools that have been used and categorized them according to the outcome framework. Most of the studies included here investigated the effectiveness of a technology on persons in need of care (n = 77). Almost half of these studies (n = 38) used standardised instruments for measuring effectiveness. All instruments are listed in Tables 3 and 4 (for psychological measures) with a reference to the studies that use them. The corresponding outcome area is indicated in the top row of both Tables. A total of 69 different instruments were identified, most of which measure the psychological health of the person in need of care: 40 different instruments were used to measure the psychological health condition. The change effect is most frequently measured in terms of a state of depression, for example by using the Geriatric Depression Scale (n = 8) or the Cornell Scale for Symptoms of Depressions in Dementia (n = 4). The most frequently used single instrument for cognition is the Mini-Mental-State Examination (n = 8). In addition to mental state, the impact of technology on the quality of life was also frequently measured. The Quality of Life in Alzheimer’s Disease scale (QOL-AD) was most frequently used here (n = 7). Tools for the measurement of organisational patient safety indicators (n = 3) or intensity of service utilization (n = 3) were used less often. Most of the tools were only used in single studies.

Table 3 Outcome tools for person in need of care
Full size table
Table 4 Psychological health outcome tools for person in need of care
Full size table

Effectiveness relating to caregiver

Significantly fewer studies from our sample relate to results for caregivers. Only 30 studies referred to caregiver outcomes, and eleven of them used standardised instruments (Table 5). A total of 20 different instruments were found, four of them specifically for formal caregivers and, ten for informal caregivers; five are universal instruments. Though there are fewer studies on informal caregivers (n = 10) than on formal caregivers (n = 20), the informal caregiver burden was the most frequently addressed outcome area for outcome tools. The second most frequently used category of tools addresses psychological changes and the third most frequently used category of instruments evaluates changes in satisfaction. Instruments for QOL, knowledge and workload were the least used. We did not find any instruments for measuring the physical load of caregivers. No tool can be named that has been used particularly frequently for the evaluation of caregivers. Almost all tools were used only once in the sample.

Table 5 Caregiver relevant outcome tools
Full size table

Organisational effectiveness

Aspects of organisational effectiveness were measured in 45 studies. Seven studies used different standardised instruments. We found four instruments to analyse communication/social interaction in the respective organisation and three instruments to analyse hospital quality (Table 6). Each tool was used only once in the sample.

Table 6 Organisation-related outcome tools
Full size table

Discussion

The aims of this scoping review are (i) to show which possible outcome areas for measuring effectiveness of DNT can be identified, (ii) to depict which areas have been the focal point of research on effectiveness of DNT so far, and which areas have been researched less, and (iii) to show how effectiveness has been measured in previous studies. The discussion section is structured around these research questions. Therefore, we divided the discussion section into three main parts: discussion of the framework development (i), critical reflection in relation to the scientific literature (i) and discussion of the quantitative analysis (ii & iii).

Framework development

The comprehensive DNT outcome framework was developed to show which possible outcome areas can be identified for the evaluation of DNT. This framework can be used by researchers to structure their effectiveness evaluation and to check whether essential outcome areas are considered in their evaluation. Thus, its purpose is thus to encourage researchers to focus on specific outcome areas or include additional outcome areas in their work. It is also intended to promote and structure discussion and reflection on desirable or necessary research objectives of DNT and may help to draw inferences on areas in which undesirable negative effects may emerge. The framework was developed using deductive and inductive methods, and therefore comprises elements that have already been researched specifically for the field of technology as well as a derivation from general nursing contexts to the specific context of DNT. Parts of the framework are therefore generic and could also be used for the evaluation of general nursing care interventions. We have also included both effectiveness and efficacy studies in the development of the Framework. Being aware of the differences of these two concepts, we assume that the incoherent use of the words within the analysed studies was caused by the fact that “efficacy and effectiveness exist on a continuum” [91] and the generalizability depends on the viewpoint of the observer and the observed condition [91]. The incorrect or incoherent classification of the two terms has already been described in the scientific literature [92]. In order to allow researchers to choose from a variety of possible outcomes adapted to the particular circumstances and context of the study, and to decide which outcome areas are to be evaluated, it is necessary to develop the most comprehensive framework possible. This led us to include studies referring to themselves as effectiveness studies and studies that refer to themselves as efficacy studies. We leave it to the judgement of the respective researchers to decide which are the optimal outcome areas and corresponding outcomes for their aspired studies.

Critical reflection of the framework in relation to the scientific literature

When comparing our framework with existing frameworks in the field of digital technologies in health care we can state that to the best of our knowledge there is no other such comprehensive framework with a special focus on nursing. As stated in the methods section, we incorporated other frameworks in the development of the DNT framework. To highlight the specific strengths of the newly developed framework, we shall now briefly describe the differences between the DNT framework and some other frameworks in similar contexts. The most comprehensive framework available (MAST) was incorporated in the design of the DNT framework [18]. MAST provides seven domains, including a focus on patients and a focus on the organisation. Five of the seven domains are highly relevant to the nursing context and were therefore included in the development of the DNT outcome framework (safety, clinical effectiveness, patient perspectives, economic aspects, organisational aspects). Formal caregivers are only rarely considered in MAST, and informal caregivers are not considered at all. These target groups are presented and highlighted in much greater detail in the DNT results framework. As a holistic framework, MAST also contains references to socio-cultural, ethical and legal aspects. These are important areas for the evaluation of DNT in general, but they do not fit into the specific context of an effectiveness evaluation for DNT that we wanted to depict in this study. The analysis of ethical and socio-cultural effects requires different research approaches, which are not reflected in our sample. In order to cover these areas, it would be necessary to include a macro perspective underpinned by scientifically sound data. We have deliberately not focused on the macro level, but it would be a possibility to complement this with further research.

For a second comparison, the Infoway Benefits Evaluation Framework [17] is used. This framework divides the evaluation of health information systems into six main dimensions: system, information, service, use, satisfaction and net benefits. The “net benefits” dimension includes many outcome areas such as patient safety, health outcomes, access to care and productivity that were integrated into the DNT outcome framework. User (in this case patient) satisfaction was also integrated into the DNT outcome Framework, but we added areas relating to formal and informal carers, as they were not taken sufficiently into account in the Infoway Benefits Evaluation Framework. Other attributes such as accuracy, performance or functionality refer primarily to the effectiveness of the respective technology, so we therefore excluded them from the DNT Framework.

For another comparison we refer to the adapted version of the Nursing Care Performance Framework [2], which displays the impact of ICT solutions on nursing care. This framework shows important areas especially for formal caregiving, which can also be found in the DNT Outcome Framework. Informal caregivers are not represented, which is certainly due to the focus. The effects on patients are presented in a very specific way. Our DNT outcome frameworks can be helpful to complement some details on outcome areas, such as psychological health or health behaviour. The comparison of the DNT framework with the systematization in a systematic review of effectiveness studies in the field telemedicine [93], indicates that the DNT Framework covers all important outcome areas on the micro and meso level. Ekelanda et al. also include a few aspects on the macro level, e.g. in the area of politics [93]. This level was excluded in the DNT outcome framework, as it implies a different perspective of analysis, and none of the studies in the scoping review related to the macro level. The exploration of outcome areas, outcome tools and outcome measures on the macro level, however, is an interesting field for future research. In sum, it can be said that the developed DNT outcome framework closes an existing gap in nursing and technology research by including all important outcome areas relevant to nursing.

Reflection on the included outcome areas and outcome tools

We evaluated all outcome tools and measures of the included 123 articles with a view to establishing which outcome areas have so far been focused on by research on DNT effectiveness, and which areas have been researched less. There are considerable differences in the researched outcome areas for the individual outcome target groups. It should be pointed out, however, that no valuation of the significance of an outcome area can be made at present. This could be the topic of further research. The study presented here constitutes a first step towards summarizing existing trends.

Persons in need of care were by far the most frequently researched target group. Psychological health, intensity of service utilization, and organisational patient safety indicators were measured much more frequently than needs assessment/ fulfilled needs, health knowledge & attitudes or intersectoral continuity of care. There might be several reasons for this. On the one hand, it might be more difficult to capture fulfilled needs, intersectoral continuity of care, or health knowledge & attitudes with the existing standardised instruments or outcome measures. On the other hand, it might simply not have been of such profound interest during the evaluation because the respective technologies did not target these areas. It is interesting to note that intersectoral care is an area that has so far largely been neglected. Intersectoral care in form of communication or collaboration between different healthcare sectors (e.g. ambulatory care and inpatient care) [94] could, for example, help to prevent hospitalisations [95]. One reason for the neglection of intersectoral care might be that while it is already difficult to implement digital technologies in a single sector, sector boundaries possible constitute a major challenge. The decision-making structures of a healthcare system might be seen to be a barrier to change in this context [96].

Overall in our sample, formal and informal caregivers have not been researched frequently in terms of the effectiveness of DNT. Relationship to person in need of care was most frequently measured among formal caregivers, along with guideline compliance and physical/psychological workload. Well-being/quality of life, health-related quality of life, medical condition and caregiver burden were not measured once in our sample. Several other outcome areas were very rarely analysed. This shows that these aspects were neglected in the past, while quality of life and caregiver burden on professional caregivers are still not being evaluated. The reasons for this should be clarified. It is unclear whether this is the case because these outcome areas are generally considered to be less important than others, or whether there are other reasons for non-evaluation. Assuming that an important goal of digital technologies in nursing care is to relieve nursing staff, it seems inappropriate to only evaluate the direct workload (e.g. physical load or psychological stress through direct work) and not evaluate the effects of digital technologies on the general burden on or the quality of life of formal caregivers. Existing instruments such as the Professional Care Team Burden (PCTB) scale harbour the potential to contribute to the evaluation of DNT in this context [97].

The caregiver burden of informal caregivers was the most frequently analysed outcome area, while all other outcome areas were analysed very seldom or not at all. One reason might be that most instruments for measuring caregiver burden have been developed for informal caregivers [97]. On the other hand, there seems to have been a socio-political interest in reducing the burden on family members through technology in the past. At the same time, the medical health condition, knowledge and competences or the use of caregiving support of informal caregivers have not been analysed in a single study. Medical health may not have been recorded for reasons of personal data protection. Specialised nursing knowledge and skills do not seem to play such a large role in research on technological support for informal carers. We expect enhanced support for informal caregivers to play a more important role in the future, and, hence, those research areas that are seldom examined now to become more important.

The most frequently analysed outcome areas for healthcare organisations were care process quality, operational efficiency and communication/social interaction. Little research has been done on financial performance, access to care and working conditions, while recruitment and staffing areas were not evaluated at all. The frequently researched areas correspond with the potential goal of technologies to improve efficiency while maintaining a high quality of care [98]. Aspects of working conditions or effects on the recruitment or staffing processes from an organisational perspective have apparently never been analysed. If it is assumed that, from an organisational point of view, the main purpose of the technologies is to increase efficiency while maintaining or improving the quality of care, factors such as working conditions for carers play a minor role. This is consistent with the results on formal caregivers. Nevertheless, it is interesting that the impact of existing technologies on recruitment has not been investigated, as technologies are often claimed to be beneficial in terms of enhancing the attractiveness of a healthcare organisation for prospective and current employees [99], which could be expressed in an evaluation of the effects of a technology on recruitment figures. In the light of the current lack of skilled workers, proof of such effects might be an interesting finding.

The third research question was addressed by showing which outcome areas were evaluated with outcome tools and which areas were more likely to be covered by other measures. At the same time, the variety and range of the tools used were presented. The wide range of outcome tools – especially in the field of psychological health - makes it difficult to compare the studies’ results, and a common set of standards for using outcome tools shared by several studies would considerably help improve comparability To this end, further research is needed to assess and evaluate existing outcome tools.

Limitations

Limitations that refer to the underlying scoping review are described in Krick et al. 2019. Especially important for this additional methodological analysis are the following aspects:

Publication bias in particular should be considered here. Studies without positive results are often not published in journals. Outcome areas for which effects are difficult to prove may be underrepresented in this article due to this publication bias, whereby, there is a possibility that studies on certain technologies may have been over-represented, under-represented or not presented at all due to negative results. This may indirectly affect the presented outcome tools and measures.

The included outcome areas and outcome tools presented could also be influenced by the fact that we included both effectiveness and efficacy studies in the development of the framework. It could be criticised that a further breakdown of a framework for efficacy studies and a framework for effectiveness studies is necessary because they may differ in their outcomes and tools. However, we decided to combine these concepts for the reasons of applicability and comprehensiveness, as described in the discussion section. Based on the included publication period of 7 years, the question needs to be considered whether outcome tools or outcome measures are only mapped for the indicated period, and therefore relevant measures of the effectiveness of DNT applied outside that period are missing. There is also the possibility that outcome areas overlap or might have been composed differently by other researchers. Overall, we have tried to ensure the highest possible standard for the outcome framework, by combining literature and expert knowledge. The current version is very comprehensive, but the field of research on DNT is very dynamic, and future adaptions should be included.

Conclusion

This scoping review provides a broad overview on outcome areas and outcome tools used for the evaluation of digital nursing technologies. All outcome tools and measures have been categorised according to our newly developed DNT outcome framework to show which areas have been focused on by research on effectiveness of DNT so far and which areas have rather been neglected. We highly recommend the use of this framework (and the further explanations given in the Additional files) as a basis for future research. Researchers can use the DNT outcome framework as a tool to structure their effectiveness evaluations and to examine whether essential outcome areas have been overlooked in their evaluations. Currently, the DNT outcome framework mainly provides an overview of all outcome areas. The weighting of the importance or significance of the different outcome areas – especially those that have been less explored so far – should be subjected to further research. This would require a more detailed assessment of the individual outcome areas, including a valuation of the areas in the subsequent research. We also consider it important to investigate the heterogeneity of interventions in DNT and to deepen the understanding of different important outcomes linked to these DNTs.

Our systematized overview of the tools in the individual areas can be used as a starting point for further research, in order to share and compare information about appropriate tools and initiate a discussion on the standardisation of tools used for similar questions. An appropriate exchange would also certainly be helpful for outcomes measures used that are not listed here in detail due to the large number.

So far, our findings indicate that intersectoral continuity of care for persons in need of care, quality of life of formal and informal caregivers, caregiving support for informal caregivers and working conditions from an organisational perspective are outcome areas which have only been scantily researched so far and would benefit considerably from future research. At the same time, we recommend the development of outcome areas for the macro level of effectiveness assessment, which is not included in the current version of the DNT outcome framework. Overall, the DNT outcome framework already offers a very good overview of the possible outcome areas and we are confident that future research will benefit from this structured approach.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

ACM:

Association for Computing Machinery

DNT:

Digital Nursing Technologies

MAST:

Model for Assessment of Telemedicine

NOC:

Nursing Outcomes Classification

QOL:

Quality of life

QOL-AD:

Quality of Life - Alzheimer Disease

References

  1. Krick T, Huter K, Domhoff D, Schmidt A, Rothgang H, Wolf-Ostermann K. Digital technology and nursing care: a scoping review on acceptance, effectiveness and efficiency studies of informal and formal care technologies. BMC Health Serv Res. 2019;19:400.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Rouleau G, Gagnon MP, Côté J, Payne-Gagnon J, Hudson E, Dubois CA. Impact of information and communication technologies on nursing care: Results of an overview of systematic reviews. J Med Internet Res. 2017;19:4.

    Article  Google Scholar 

  3. Lau F. Kuziemsky C (eds.): handbook of eHealth evaluation: an evidence-based approach. Victoria: University of Victoria; 2016.

    Google Scholar 

  4. EUnetHTA. EUnetHTA Joint Action 2, Work Package 8. HTA Core Model ® version 3.0. In; 2016. p. 410.

    Google Scholar 

  5. Bemelmans R, Gelderblom GJ, Jonker P, de Witte L. Socially Assistive Robots in Elderly Care: A Systematic Review into Effects and Effectiveness. J Am Med Directors Assoc. 2012;13(2):114–20 e111.

    Article  Google Scholar 

  6. Capurro D, Ganzinger M, Perez-Lu J, Knaup P. Effectiveness of eHealth interventions and information needs in palliative care: a systematic literature review. J Med Internet Res. 2014;16(3) e72.

  7. Fleming R, Sum S. Empirical studies on the effectiveness of assistive technology in the care of people with dementia: a systematic review. J Assist Technol. 2014;8(1):14–34.

    Article  Google Scholar 

  8. Van der Roest HG, Wenborn J, Pastink C, Droes RM, Orrell M. Assistive technology for memory support in dementia. Cochrane Database Syst Rev. 2017;6:Cd009627.

    PubMed  Google Scholar 

  9. Khosravi P, Ghapanchi AH. Investigating the effectiveness of technologies applied to assist seniors: a systematic literature review. Int J Med Inform. 2016;85(1):17–26.

    Article  PubMed  Google Scholar 

  10. Peretz D, Arnaert A, Ponzoni NN. Determining the cost of implementing and operating a remote patient monitoring programme for the elderly with chronic conditions: a systematic review of economic evaluations. J Telemed Telecare. 2018;24(1):13–21.

    Article  PubMed  Google Scholar 

  11. Calvaresi D, Cesarini D, Sernani P, Marinoni M, Dragoni AF, Sturm A. Exploring the ambient assisted living domain: a systematic review. J Ambient Intell Humaniz Comput. 2017;8(2):239–57.

    Article  Google Scholar 

  12. Bemelmans R, Gelderblom GJ, Jonker P, de Witte L. Effectiveness of robot Paro in intramural psychogeriatric care: a multicenter quasi-experimental study. J Am Med Dir Assoc. 2015;16(11):946–50.

    Article  PubMed  Google Scholar 

  13. Yusofa MM, Kuljis J, Papazafeiropouloub A, Stergioulasb LK. An evaluation framework for Health Information Systems: human, organization and technology-fit factors (HOT-fit). Int J Med Informatics. 2008;77:386–98.

    Article  Google Scholar 

  14. Sockolow P, Crawford P, Lehmann HP. Health services research evaluation principles. Methods Inf Med. 2012;51(02):122–30.

    Article  CAS  PubMed  Google Scholar 

  15. Information CIfH. A performance measurement framework for the Canadian health system. In. Ontario: Canadian Institute for Health Information; 2012.

    Google Scholar 

  16. Carinci F, Van Gool K, Mainz J, Veillard J, Pichora EC, Januel JM, Arispe I, Kim SM, Klazinga NS. Group aoBoTOHCQIE et al: towards actionable international comparisons of health system performance: expert revision of the OECD framework and quality indicators. Int J Qual Health Care. 2015;27(2):137–46.

    CAS  PubMed  Google Scholar 

  17. Francis Lau F, Hagens S, Muttitt S. A Proposed Benefits Evaluation Framework for Health Information Systems in Canada. Healthcare Quarterly. 2007;10:1.

    Google Scholar 

  18. Kidhom K, Bowes A, Dyrehauge S, Ekeland AG, Flottorp SA, Jensen LK, Pedersen CD, Rasmussen J. The MAST Manual. MAST - Model for ASsessment of Telemedicine. In: MethoTelemed team; 2010.

    Google Scholar 

  19. Schulz R, Wahl H-W, Matthews JT. Advancing the aging and technology agenda in gerontology. Gerontologist. 2015;55(5):724–34.

    Article  PubMed  Google Scholar 

  20. Lang RLN. Evaluating the effectiveness of nurse-focused computerized clinical decision support on urinary catheter practice guidelines: Gardner-Webb University; 2012.

  21. Hattink BJJ, Meiland FJM, Overmars-Marx T, de Boer M, Ebben PWG, van Blanken M, Verhaeghe S, Stalpers-Croeze I, Jedlitschka A, Flick SE, et al. The electronic, personalizable Rosetta system for dementia care: exploring the user-friendliness, usefulness and impact. Disabil Rehabil. 2016;11(1):61–71.

    CAS  Google Scholar 

  22. Broadbent E, Orejana JR, Ahn HS, Xie J, Rouse P, MacDonald BA. The cost-effectiveness of a robot measuring vital signs in a rural medical practice. In: 2015 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN): Aug. 31 2015-Sept. 4 2015 2015; 2015. p. 577–81.

    Chapter  Google Scholar 

  23. Kunkel C, Kopp W, Hanson M. A matter of life and death: end-of-life simulation to develop confidence in nursing students. Nurs Education Perspectives (Wolters Kluwer Health). 2016;37(5):285–6.

    Article  Google Scholar 

  24. Scott R, Saeed A. Global eHealth - measuring outcomes: why, what, and how a report commissioned by the World Health Organization's global observatory for eHealth. In. Bellagio: World Health Organization’s Global Observatory for eHealth; 2008.

    Google Scholar 

  25. Scott RE, McCarthy FG, Jennett PA, Perverseff T, Lorenzetti D, Saeed A, Rush B, Yeo M. Telehealth outcomes: a synthesis of the literature and recommendations for outcome indicators. J Telemed Telecare. 2007;13(Suppl 2):1–38.

    Article  PubMed  Google Scholar 

  26. Gartlehnera G, Hansenb RA, Nissmanc D, Lohrd KN, Careye TS. A simple and valid tool distinguished efficacy from effectiveness studies. J Clin Epidemiol. 2006;59:1040e1048.

    Google Scholar 

  27. Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

    Article  Google Scholar 

  28. Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5(1):69.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematicreviews of complex evidence: audit of primary sources. BMJ. 2005;331:1064–5.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Maas M, Swanson E, Herrmann M, Moorhead S, Johnson M, Aquilino M. Pflegeergebnisklassifikation (NOC) vol. 2., vollst. überarb. und erw. Aufl. Bern: Huber; 2013.

    Google Scholar 

  31. Nabbout R, Auvin S, Chiron C, Irwin J, Mistry A, Bonner N, Williamson N, Bennett B. Development and content validation of a preliminary core set of patient- and caregiver-relevant outcomes for inclusion in a potential composite endpoint for Dravet syndrome. Epilepsy Behav. 2018;78:232–42.

    Article  PubMed  Google Scholar 

  32. Schwartz S, Laura Darlak L, Whitlatch CJ. Selected caregiver Assessment measures: a resource inventory for practitioners. In., vol. 2nd edition. Cleveland: Margaret Blenkner Research Institute of the Benjamin Rose Institute on Aging; 2012.

    Google Scholar 

  33. Zendjidjian XY, Boyer L. Challenges in measuring outcomes for caregivers of people with mental health problems. Dialogues Clin Neurosci. 2014;16(2):159–69.

    PubMed  PubMed Central  Google Scholar 

  34. Henriksson A, Arestedt K. Exploring factors and caregiver outcomes associated with feelings of preparedness for caregiving in family caregivers in palliative care: a correlational, cross-sectional study. Palliat Med. 2013;27(7):639–46.

    Article  PubMed  Google Scholar 

  35. Deeken JF, Taylor KL, Mangan P, Yabroff KR, Ingham JM. Care for the caregivers: a review of self-report instruments developed to measure the burden, needs, and quality of life of informal caregivers. J Pain Symptom Manag. 2003;26(4):922–53.

    Article  Google Scholar 

  36. Son J, Erno A, Shea DG, Femia EE, Zarit SH, Stephens MA. The caregiver stress process and health outcomes. J Aging Health. 2007;19(6):871–87.

    Article  PubMed  Google Scholar 

  37. Hoefman R, Al-Janabi H, McCaffrey N, Currow D, Ratcliffe J. Measuring caregiver outcomes in palliative care: a construct validation study of two instruments for use in economic evaluations. Qual Life Res. 2015;24(5):1255–73.

    Article  PubMed  Google Scholar 

  38. Carod-Artal FJ, Mesquita HM, Ziomkowski S, Martinez-Martin P. Burden and health-related quality of life among caregivers of Brazilian Parkinson's disease patients. Parkinsonism Relat Disord. 2013;19(11):943–8.

    Article  PubMed  Google Scholar 

  39. Sim J, Crookes P, Walsh K, Halcomb E. Measuring the outcomes of nursing practice: a Delphi study. J Clin Nurs. 2018;27(1–2):e368–78.

    Article  PubMed  Google Scholar 

  40. Epstein MJ, Rejc A. Evaluating performance in information technology. In. Canada: The Society of Management Accountants of Canada; 2005.

    Google Scholar 

  41. Trierweiller AC, Peixe BC, Tezza R, Pereira VL, Pacheco W Jr, Bornia AC, de Andrade DF. Measuring organizational effectiveness in information and communication technology companies using item response theory. Work. 2012;41(Suppl 1):2795–802.

    Article  PubMed  Google Scholar 

  42. Jain R, Shermon G. Creating an optimized organisation key opportunities and challenges. In. India: KPMG; 2012.

    Google Scholar 

  43. Cascio W, Montealegre R. How technology is changing work and organizations. Ann Rev Org Psychol Organizational Behav. 2016;3:349–75.

    Article  Google Scholar 

  44. Damberg CL, Sorbero ME, Lovejoy SL, Martsolf GR, Raaen L, Mandel D. Measuring success in health care value-based purchasing programs: findings from an environmental scan, literature review, and expert panel discussions. Rand Health Quarterly. 2014;4(3):9.

    PubMed  PubMed Central  Google Scholar 

  45. Jalagat R. Amur said Al-Habsi N: evaluating the impacts of IT usage on organizational performance. Eur Acad Res. 2017;5:5111–64.

    Google Scholar 

  46. Organisation WH: Innovative Care for Chronic Conditions: Building Blocks for Action In.: World Health Organisation; 2002.

    Google Scholar 

  47. Lazarou I, Karakostas A, Stavropoulos TG, Tsompanidis T, Meditskos G, Kompatsiaris I, Tsolaki M. A novel and intelligent home monitoring system for care support of elders with cognitive impairment. J Alzheimers Dis. 2016;54(4):1561–91.

    Article  PubMed  Google Scholar 

  48. Zaccarelli C, Cirillo G, Passuti S, Annicchiarico R, Barban F. Computer-based cognitive intervention for dementia Sociable: motivating platform for elderly networking, mental reinforcement and social interaction. In: 2013 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops: 5–8 May 2013 2013; 2013. p. 430–5.

    Google Scholar 

  49. Florczak B, Scheurich A, Croghan J, Sheridan P Jr, Kurtz D, McGill W, McClain B. An observational study to assess an electronic point-of-care wound documentation and reporting system regarding user satisfaction and potential for improved care. Ostomy Wound Manag. 2012;58(3):46–51.

    Google Scholar 

  50. Appari A, Johnson EM, Anthony DL. Information technology and hospital patient safety: a cross-sectional study of US acute care hospitals. Am J Managed Care. 2014;20(17):eSP39–47.

    Google Scholar 

  51. Mitchell S, Yaylacicegi U. EHR prescription for small, medium, and large hospitals: an exploratory study of Texas acute care hospitals. Int J Electron Healthc. 2012;7(2):125–40.

    Article  PubMed  Google Scholar 

  52. Bennett CC, Sabanovic S, Piatt JA, Nagata S, Eldridge L, Randall N. A Robot a Day Keeps the Blues Away. In: 2017 IEEE International Conference on Healthcare Informatics (ICHI): 23–26 Aug. 2017 2017; 2017. p. 536–40.

    Chapter  Google Scholar 

  53. Lexis M. Activity monitoring technology to support homecare delivery to frail and psychogeriatric elderly persons living at home alone. Technol Disabil. 2013;25(3):189–97.

    Article  Google Scholar 

  54. Sahota O, Drummond A, Kendrick D, Grainge MJ, Vass C, Sach T, Gladman J, Avis M. REFINE (REducing falls in in-patieNt elderly) using bed and bedside chair pressure sensors linked to radio-pagers in acute hospital care: a randomised controlled trial. Age Ageing. 2014;43(2):247–53.

    Article  PubMed  Google Scholar 

  55. Mierlo L, Meiland F, Ven P, Hout H, Dröes R. Evaluation of DEM-DISC, customized e-advice on health and social support services for informal carers and case managers of people with dementia; a cluster randomized trial.In. Int Psychogeriatrics. 2015;27:1365–78.

    Article  Google Scholar 

  56. Angst CM, Devaraj S, D'Arcy J. Dual role of IT-assisted communication in patient care: a validated structure-process-outcome framework. J Manag Inf Syst. 2012;29(2):257–92.

    Article  Google Scholar 

  57. Restuccia JD, Cohen AB, Horwitt JN, Shwartz M. Hospital implementation of health information technology and quality of care: Are they related? BMC Med Informatics Decision Making. 2012;12(1).

  58. Marek K, Stetzer F, Ryan P, Bub L, Adams S, Schlidt A, Lancaster R, O'Brien A. Nurse care coordination and technology effects on health status of frail older adults via enhanced self-management of medication: randomized clinical trial to test efficacy In. Nursing Res. 2013;62:269–78.

    Article  Google Scholar 

  59. Broadbent E, Kerse N, Peri K, Robinson H, Jayawardena C, Kuo T, Datta C, Stafford R, Butler H, Jawalkar P, et al. Benefits and problems of health-care robots in aged care settings: a comparison trial. Australasian J Ageing. 2016;35(1):23–9.

    Article  Google Scholar 

  60. Moyle W, Cooke M, Beattie E, Jones C, Klein B, Cook G, Gray C. Exploring the effect of companion robots on emotional expression in older adults with dementia: a pilot randomized controlled trial In. J Gerontological Nurs. 2013;39:46–53.

    Article  Google Scholar 

  61. Nordheim J, Hamm S, Kuhlmey A, Suhr R. Tablet computers and their benefits for nursing home residents with dementia: results of a qualitative pilot study. Zeitschrift fur Gerontologie und Geriatrie. 2015;48(6):543–9.

    Article  PubMed  Google Scholar 

  62. Robinson H, MacDonald B, Kerse N, Broadbent E. The psychosocial effects of a companion robot: a randomized controlled trial. J Am Med Dir Assoc. 2013;14(9):661–7.

    Article  PubMed  Google Scholar 

  63. Lazar A, Demiris G, Thompson HJ. Evaluation of a multifunctional technology system in a memory care unit: opportunities for innovation in dementia care. Informatics Health Soc Care. 2016;41(4):373–86.

    Article  Google Scholar 

  64. Portela FR, Correia RJC, Fonseca JA, Andrade JM. Wiitherapy on seniors - effects on physical and metal domains. In: 2011 IEEE 1st International Conference on Serious Games and Applications for Health (SeGAH): 16–18 Nov. 2011 2011; 2011. p. 1–5.

    Google Scholar 

  65. Rodriguez CS. Enhancing the communication of suddenly speechless critical care patients. Am J Crit Care. 2016;25(3):e40–7.

    Article  PubMed  PubMed Central  Google Scholar 

  66. Miller K, Rodger S, Kipping B, Kimble RM. A novel technology approach to pain management in children with burns: A prospective randomized controlled trial. Burns (03054179). 2011;37(3):395–405.

    Article  CAS  Google Scholar 

  67. Gustafsson C, Svanberg C, Müllersdorf M. Using a robotic cat in dementia care. J Gerontol Nurs. 2015;41(10):46–56.

    Article  PubMed  Google Scholar 

  68. Jøranson N, Pedersen I, Rokstad AMM, Ihlebæk C. Change in quality of life in older people with dementia participating in Paro-activity: a cluster-randomized controlled trial. J Adv Nurs. 2016;72(12):3020–33.

    Article  PubMed  Google Scholar 

  69. Valenti SM, Aguera-Ortiz L, Olazaran RJ, Mendoza RC, Perez MA, Rodriguez PI, Osa RE, Barrios SA, Herrero CV, Carrasco CL, et al. Social robots in advanced dementia. In: Frontiers in aging neuroscience, vol. 7; 2015.

    Google Scholar 

  70. Trukeschitz BSC, Ring-Dimitriou S. Smartes Betreutes Wohnen: Nutzung, Systemakzeptanz und Wirkungen von “meinZentrAAL”. Norderstedt: Deutschland; 2018.

    Google Scholar 

  71. Rantz M, Phillips LJ, Galambos C, Lane K, Alexander GL, Despins L, Koopman RJ, Skubic M, Hicks L, Miller S, et al. Randomized trial of intelligent sensor system for early illness alerts in senior housing. J Am Med Dir Assoc. 2017;18(10):860–70.

    Article  PubMed  PubMed Central  Google Scholar 

  72. Chen ST, Huang YGL, Chiang IT. Using Somatosensory Video Games to Promote Quality of Life for the Elderly with Disabilities. In: 2012 IEEE Fourth International Conference On Digital Game And Intelligent Toy Enhanced Learning: 27–30 March 2012 2012; 2012. p. 258–62.

    Chapter  Google Scholar 

  73. Kipping B, Rodger S, Miller K, Kimble RM. Virtual reality for acute pain reduction in adolescents undergoing burn wound care: a prospective randomized controlled trial. Burns (03054179). 2012;38(5):650–7.

    Article  Google Scholar 

  74. Liang A, Piroth I, Robinson H, MacDonald B, Fisher M, Nater UM, Skoluda N, Broadbent E. A pilot randomized trial of a companion robot for people with dementia living in the community. J Am Med Dir Assoc. 2017;18(10):871–8.

    Article  PubMed  Google Scholar 

  75. Petersen S, Houston S, Qin H, Tague C, Studley J. The utilization of robotic pets in dementia care In. J Alzheimer's Dis. 2017;55:569–74.

    Article  Google Scholar 

  76. Subramaniam P, Woods B. Digital life storybooks for people with dementia living in care homes: an evaluation In. Clinical Interventions Aging. 2016;11:1263–76.

    Article  Google Scholar 

  77. Jøranson N, Pedersen I, Rokstad AMM, Ihlebæk C. Effects on symptoms of agitation and depression in persons with dementia participating in robot-assisted activity: a cluster-randomized controlled trial. J Am Med Dir Assoc. 2015;16(10):867–73.

    Article  PubMed  Google Scholar 

  78. Moyle W, Jones CJ, Murfield JE, Thalib L, Beattie ERA, Shum DKH, O'Dwyer ST, Mervin MC, Draper BM. Use of a robotic seal as a therapeutic tool to improve dementia symptoms: a cluster-randomized controlled trial. J Am Med Dir Assoc. 2017;18(9):766–73.

    Article  PubMed  Google Scholar 

  79. Patterson D, Soltani M, Teeley A, Morse D, Wiechman S, Gibran N. Hypnosis delivered through immersive virtual reality for wound care: a randomized, controlled study. In. J Burn Care Res. 2012;33:S70.

    Article  Google Scholar 

  80. Wagemaker E, Dekkers TJ. Agelink van Rentergem JA, Volkers KM, Huizenga HM: advances in mental health care: five N = 1 studies on the effects of the robot seal Paro in adults with severe intellectual disabilities. J Ment Health Res Intellect Disabil. 2017;10(4):309–20.

    Article  Google Scholar 

  81. van der Heide LA, Willems CG, Spreeuwenberg MD, Rietman J, de Witte LP. Implementation of CareTV in care for the elderly: the effects on feelings of loneliness and safety and future challenges. Technol Disabil. 2012;24(4):283–91.

    Article  Google Scholar 

  82. Zhuang J, Fang R, Feng X, Xu X, Liu L, Bai Q, Tang H, Zhao Z, Chen S. The impact of human-computer interaction-based comprehensive training on the cognitive functions of cognitive impairment elderly individuals in a nursing home In. J Alzheimer's Dis. 2013;36:245–51.

    Article  Google Scholar 

  83. White C, McIlfatrick S, Dunwoody L, Watson M. Supporting and improving community health services-a prospective evaluation of ECHO technology in community palliative care nursing teams. BMJ Support Palliat Care. 2015;0:1–7. Published online first: 01.12.2015.

  84. Shukla J, Barreda-Ángeles M, Oliver J, Puig D. Effectiveness of socially assistive robotics during cognitive stimulation interventions: Impact on caregivers. In: 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN): Aug. 28 2017-Sept. 1 2017 2017; 2017. p. 62–7.

    Chapter  Google Scholar 

  85. Hicken BL, Daniel C, Luptak M, Grant M, Kilian S, Rupper RW. Supporting caregivers of rural veterans electronically (SCORE). J Rural Health. 2017;33(3):305–13.

    Article  PubMed  Google Scholar 

  86. Wakefield BJ, Vaughan-Sarrazin M. Home Telehealth and caregiving appraisal in chronic illness. Telemedicine E-Health. 2017;23(4):282–9.

    Article  PubMed  Google Scholar 

  87. Wieck M, Blake B, Sellick C, Kenron D, DeVries D, Terry S, Krishnaswami S. Utilizing technology to improve intraoperative family communication. In. Am J Surg (no pagination). 2017; vol. Date of Publication: January 10; 2017.

  88. Pot AM, Willemse BM, Horjus S. A pilot study on the use of tracking technology: feasibility, acceptability, and benefits for people in early stages of dementia and their informal caregivers. Aging Ment Health. 2012;16(1):127–34.

    Article  PubMed  Google Scholar 

  89. Ching JM, Williams BL, Idemoto LM, Blackmore CC. Using lean 'Automation with a human Touch' to improve medication safety: a step closer to the 'Perfect Dose'. Joint Commission J Qual Patient Safety. 2014;40(8):341–50.

    Article  Google Scholar 

  90. Bettinelli M, Lei Y, Beane M, Mackey C, Liesching T. Does robotic Telerounding enhance nurse-physician collaboration satisfaction about care decisions? In. Telemedicine J E-Health. 2015;21:637–43.

    Article  Google Scholar 

  91. Gartlehner G, Hansen R, Nissman D, Lohr K, Carey T. Criteria for distinguishing effectiveness from efficacy trials in systematic reviews. Technical review 12. In., vol. no. 06–0046. Rockville: Agency for Healthcare Research and Quality; 2006.

    Google Scholar 

  92. Dal-Ré R, Rosendaal F. Efficacy and effectiveness: the wrong use of different terms. Eur J Internal Med. 2018;54:e17–8.

    Article  Google Scholar 

  93. Ekelanda AG, Bowes A, Flottorp S. Effectiveness of telemedicine: A systematic review of reviews. Int J Med Informatics. 2010;79:736–71.

    Article  Google Scholar 

  94. Ommen O, Ullrich B, Janssen C, Pfaff H. The ambulatory-stationary interface in medical health care: problems, model of explanation, and possible solutions. Medizinische Klinik (Munich, Germany : 1983). 2007;102(11):913–7.

    Article  Google Scholar 

  95. Sundmacher L, Fischbach D, Schuettig W, Naumann C, Augustin U, Faisst C. Which hospitalisations are ambulatory care-sensitive, to what degree, and how could the rates be reduced? Results of a group consensus study in Germany. Health Pol (Amsterdam, Netherlands). 2015;119(11):1415–23.

    Article  Google Scholar 

  96. Busse R, Blümel M. Germany: health system review. Health Syst Transition. 2014;16(2):1–296.

    Google Scholar 

  97. Auer S, Graessel E, Viereckl C, Kienberger U, Span E, Luttenberger K. Professional care team burden (PCTB) scale – reliability, validity and factor analysis. Health Qual Life Outcomes. 2015;13(1):17.

    Article  PubMed  PubMed Central  Google Scholar 

  98. Health EEPoewoii. Assessing the Impact of digital transformation of health service - Expert Panel on effective ways of investing in Health (EXPH). Luxembourg: European Union; 2018.

    Google Scholar 

  99. Wiskow C, Albreht T. Pietro cd: how to create an attractive and supportive working environment for health professionals. In: Policy Brief. Copenhagen: World Health Organization; 2010.

    Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

This study arises from the project “Pflegeinnovationszentrum” (PIZ) which is funded by the Federal Ministry of Education and Research (BMBF) (Grant number: 16SV7821). The BMBF was not actively involved in the design of the study, the data collection, the analysis, the interpretation of the data or in writing the manuscript. The authors are solely responsible for the results.

Author information

Authors and Affiliations

  1. SOCIUM Research Center on Inequality and Social Policy, University of Bremen, Mary-Somerville-Straße 3, 28359, Bremen, Germany

    Tobias Krick & Kai Huter

  2. High-profile Area of Health Sciences, University of Bremen, 28359, Bremen, Germany

    Tobias Krick, Kai Huter, Kathrin Seibert, Dominik Domhoff & Karin Wolf-Ostermann

  3. Institute for Public Health and Nursing Research, University of Bremen, Grazer Straße 4, 28359, Bremen, Germany

    Kathrin Seibert, Dominik Domhoff & Karin Wolf-Ostermann

Authors
  1. Tobias Krick
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kai Huter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kathrin Seibert
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dominik Domhoff
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Karin Wolf-Ostermann
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

TK and KH conceptualized the study, developed the outcome framework, the data abstraction form and screened the studies with respect to outcomes measures and outcome tools. TK interpreted the results and wrote the first draft of the manuscript. KWO obtained funding for the study. TK, KH, DD, KS and KWO read and provided substantial edits on the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Tobias Krick.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

There are no competing interests to declare.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

Definitions and Examples for Outcome Areas for Persons in Need of Care.

Additional file 2.

Definitions and Examples for Outcome Areas for Outcome Areas Formal Caregivers.

Additional file 3.

Definitions and Examples for Outcome Areas for Outcome Areas Informal Caregivers.

Additional file 4.

Definitions and Examples for Outcome Areas for Outcome Areas Organisation.

Additional file 5.

Included Technologies.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Krick, T., Huter, K., Seibert, K. et al. Measuring the effectiveness of digital nursing technologies: development of a comprehensive digital nursing technology outcome framework based on a scoping review. BMC Health Serv Res 20, 243 (2020). https://doi.org/10.1186/s12913-020-05106-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12913-020-05106-8

Keywords

BMC Health Services Research

ISSN: 1472-6963

Contact us