Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

Development and evaluation of the content validity, practicability and feasibility of the Innovative dementia-oriented Assessment system for challenging behaviour in residents with dementia

BMC Health Services ResearchBMC series – open, inclusive and trusted201717:554

https://doi.org/10.1186/s12913-017-2469-8

Received: 23 February 2017

Accepted: 24 July 2017

Published: 14 August 2017

Abstract

Background

One of the most difficult issues for care staff is the manifestation of challenging behaviour among residents with dementia. The first step in managing this type of behaviour is analysing its triggers. A structured assessment instrument can facilitate this process and may improve carers’ management of the situation. This paper describes the development of an instrument designed for this purpose and an evaluation of its content validity and its feasibility and practicability in nursing homes.

Methods

The development process and evaluation of the content validity were based on Lynn’s methodology (1998). A literature review (steps 1 + 2) provided the theoretical framework for the instrument and for item formation. Ten experts (step 3) evaluated the first version of the instrument (the Innovative dementia-oriented Assessment (IdA®)) regarding its relevance, clarity, meaningfulness and completeness; content validity indices at the scale-level (S-CVI) and item-level (I-CVI) were calculated. Health care workers (step 4) evaluated the second version in a workshop. Finally, the instrument was introduced to 17 units in 11 nursing homes in a field study (step 5), and 60 care staff members assessed its practicability and feasibility.

Results

The IdA® used the need-driven dementia-compromised behaviour (NDB) model as a theoretical framework. The literature review and expert-based panel supported the content validity of the IdA®. At the item level, 77% of the ratings had a CVI greater than or equal to 0.78. The majority of the question-ratings (84%, n = 154) and answer-ratings (69%, n = 122) showed valid results, with none below 0.50. The health care workers confirmed the understandability, completeness and plausibility of the IdA®. Steps 3 and 4 led to further item clarification. The carers in the study considered the instrument helpful for reflecting challenging behaviour and beneficial for the care of residents with dementia. Negative ratings referred to the time required and the lack of effect on residents´ behaviour.

Conclusions

There was strong evidence supporting the content validity of the IdA®. Despite the substantial length and time requirement, the instrument was considered helpful for analysing challenging behaviour. Thus, further research on the psychometric qualities, implementation aspects and effectiveness of the IdA® in understanding challenging behaviour is needed.

Keywords

Instrument development Validity Challenging behaviour Dementia Nursing home Content validity index

Background

The process of caring for people with dementia in nursing homes is complicated by their manifestation of behavioural and psychological symptoms, also referred to as challenging behaviours [1]. Challenging behaviours occur throughout all phases of the disease [2] and are highly prevalent in nursing homes (82%) [3]. In contrast with loss of memory, cognitive decline, and functional disability, challenging behaviours are more difficult to manage and are often perceived to be a burden by care staff [4, 5].

Studies that investigate patients’ perspectives on the disease commonly consider challenging behaviours to be a way of coping with dementia rather than a behavioural problem [6]. Research has indicated that challenging behaviours often emerge from an incongruence between a person’s needs and the degree to which the environment fulfils those needs [7]. Thus, challenging behaviours can be explained as the best response a person can provide given the limitations imposed by dementia, his or her abilities and elements of the environment [8]. It is highly important that challenging behaviours are not taken for granted; rather, they should be explored by others to discover the possible triggers and causes of these behaviours [6].

Understanding the underlying causes of challenging behaviours is both demanding and complex. Assessment tools designed to systematically guide nursing staff through this process can facilitate these complex behavioural analyses [9]. These instruments enable an objective description and evaluation of challenging behaviours and can provide insight into their potential triggers and causes [10].

At the time of this study, existing assessment instruments (e.g., Behavioural Pathology in Alzheimer’s Disease (Behave-AD) [11], Neuropsychiatric Inventory-Nursing home version (NPI-NH) [12], Cohen-Mansfield-Agitation-Inventory (CMAI) [13], Challenging Behaviour Scale (CBS) [14]) include topographical information about the behaviour (nature, intensity, and frequency) and a measure of its consequences (stress, safety), but they do not comprise information about its possible triggers. These instruments had objectives other than understanding a person’s challenging behaviour. No instruments for use in nursing homes have been developed that capture all the information needed to systematically guide nursing and care staff through the complex process of understanding residents’ challenging behaviours using a biomedical and psychosocial definition of this behaviour. However, several structured approaches for the analysis of challenging behaviour within nursing homes have now been developed [1520] and these approaches include assessment instruments, charts or tools for the analysis of challenging behaviour [21]. Few approaches also address a behavioural analysis in the community setting [2225]. Due to a lack of detailed information on these documents, a thorough comparison of these instruments is not possible. These instruments appear to be broadly targeted and address different topics, although the determination of medical and physical causes of challenging behaviour is prominent in most assessment instruments [19, 21]. The assessment instruments often require the expertise of professionals other than nurses (e.g., nursing home physicians or psychologists) [1619], or the assessment process is led by a psychologist with a multidisciplinary team [15]. This situation makes it more difficult or even impossible to transfer the assessment instrument to countries, like Germany, where psychologists or nursing home physicians are not employed in nursing homes. In Germany, the nurses are the professionals who are responsible for resident’s care. Additionally, these instruments have objectives other than understanding a person’s challenging behaviour.

To address these shortcomings, an assessment tool called the “Innovative dementia-oriented Assessment system (IdA®)” was developed and validated for use in nursing practice. The objectives of this assessment instrument were as follows: (1) it should comprehensively assess the aspects relevant to evaluating challenging behaviour such as its topography, consequences and need for action; (2) it should collect information that helps identify the triggers and causes and better understand the behaviour; (3) it should include information relevant to providing nursing care for people with dementia; and (4) it should be applicable by care staff.

This paper presents the (1) development process and the (2) content validity of the IdA® as well as the (3) results of its first feasibility and practicability tests.

Methods

For the development and content validity of the new instrument, the approach described by Lynn [26] was used. This approach advocates two stages, in which stage I (development) results in the generation of the instrument’s items and stage II evaluates the performance of the instrument’s items (judgement and quantification). In this study, these two stages were supplemented by a third stage (evaluation), in which the feasibility and practicability of the new instrument were tested within an evaluation study (Fig. 1).
Fig. 1

Stages of the development and validity and feasibility testing of the IdA®

Stage I. Development

The development of the IdA® consisted of two comprehensive literature reviews. The first review aimed to identify a conceptual framework for challenging behaviour and to identify the assessment’s components and items [2628]. The second literature review focused on the specific behaviour of wandering and aimed to confirm the validity of the conceptual framework and of the specific components and items found in the first review. Both reviews were conducted in parallel and used an iterative process.

The first literature review was guided by following research questions:
  1. 1.

    Which conceptual framework is most suitable for assessing challenging behaviour and its causes in people with dementia who are living in nursing homes?

     
  2. 2.

    Which components of the conceptual framework are the most important for establishing the instrument’s dimensions?

     
  3. 3.

    Which items are the most important for representing/assessing the instrument’s dimensions?

     
The following research question guided the second literature search:
  1. 4.

    Are additional dimensions or items needed when considering a specific challenging behaviour?

     

The search process for both reviews has been described elsewhere [29].

Based on the results of both literature reviews, a first draft of the instrument and its items was developed (IdA® version 1.0).

Stage II. Judgement and quantification

The Judgement and Quantification phase included two evaluations. The first assessment consisted of an expert panel that focused on the relevance and clarity of the questions as well as the significance and completeness of responses in the IdA®. The second evaluation comprised a workshop with health care workers that focused on the understandability, completeness, plausibility and management of the IdA®.

1st evaluation: Expert panel

Experts in the expert panel were defined as individuals who worked in practice as a practitioner or in science in the field of dementia care, i.e., the criteria included being a nurse or being very familiar with nursing work and having an understanding of the field of assessment instruments. We focused mainly on nurses since nurses in Germany are responsible for the nursing process and thus for recognition and management of challenging behaviour. Fifteen experts were identified and contacted.

Measurement of content validity

As the IdA® was composed of questions and answers, a content validity index was calculated for both.

Each question was evaluated by rating a) its relevance to the instrument’s aim and b) its understandability/clarity. Each answer was assessed regarding its c) comprehensiveness/completeness and d) meaningfulness/significance for the related question. The four attributes (a-d) of the questions and answers were rated on a 4-point scale (1 = not relevant/not clear/incomplete/meaningless; 4 = highly relevant/clear/complete/meaningful) [26, 30]. In addition, the experts were asked to evaluate whether the items covered all important aspects or if there were missing components. The experts could also comment on every item.

The experts in the first expert panel also completed a questionnaire about their demographics and expertise.

Measurement of the content validity index

A content validity index was calculated both at the item level (I-CVI) and scale level (S-CVI) for all four attributes [26, 31]. The I-CVI was calculated as the number of experts providing a score of 3 or 4 divided by the total number of experts [26]. With more than 5 experts, the I-CVI should not be lower than 0.78 [31].

To calculate the S-CVI, two different indices were calculated: 1) the proportion of the items on one scale that the experts scored as valid (ratings of 3 or 4) (universal agreement by experts = S-CVI/UA) and 2) the average proportion of the items on one scale rated 3 or 4 (average agreement by experts = S-CVI/Ave) [31]. The S-CVI/UA is sensitive to the number of experts: the more experts are included, the greater the possibility of generating a low S-CVI. The S-CVI/Ave is more liberal and is preferred by Polit and Beck [31]; however, they recommend presenting both indices. The acceptable standard for the S-CVI/UA and the S-CVI-Ave is 0.8 [30, 31]; values up to 0.9 indicate an excellent average [21].

Additionally, a modified Kappa index was computed to estimate the I-CVI [32, 33]. The modified Kappa (k*) is an index of agreement among experts that indicates beyond chance that the item is relevant, clear, or another characteristic of interest [32]. We used the formula suggested by Polit et al. [32] (Table 5). The standards recommended by Fleiss [34] and Cicchetti and Sparrow [35] were used to interpret k*.

The results of the expert panel contributed to the second version of the instrument (IdA® 2.0).

2nd evaluation: Workshop with health care workers

According to Lynn [26], the same experts should re-evaluate the modified version of an instrument. Due to the comprehensiveness of the instrument and the limited resources of the first expert panel, a second round of evaluations with the same experts was not possible. Therefore, a second evaluation was organized with other experts, in which key persons from health care practice were invited to participate in a workshop. The nursing homes were free to select the key persons; the only inclusion criteria were that the persons were responsible for implementing the IdA® in this study, e.g., organizing the application of the IdA® and/or using it within the planned study, and that they had experience caring for people with dementia.

During the workshop, which lasted for several hours, the modified version of the IdA® (version 2.0) was introduced, and the objectives were explained. Subsequently, the experts were asked to assess the understandability, plausibility and completeness of the items. The discussion and suggested modifications were recorded, resulting in a further revised version of the IdA® (version 3.0).

Stage III: Evaluation

The third stage aimed to evaluate the feasibility of the IdA®, its usefulness in managing challenging behaviour and its practicability in daily work through an evaluation study. Data from a questionnaire administered in a before-after study were used to evaluate the feasibility and relevance of the IdA® in nursing practice.

Sample

A convenience sample of 11 nursing homes with 17 units participated in the evaluation study. Twelve were traditional care units and 5 were specialized care units for residents with dementia. All units had structures that enabled the use of the IdA® within a team context (e.g., team meetings, case conferences, nursing rounds). All care staff members were invited to the survey.

Application of the IdA®

In this study, the IdA® was expected to be used for 6 weeks with a minimum of 3 residents in one care unit (traditional and/or specialized). The IdA® was introduced to key persons on the units’ care staff team, who received a comprehensive one-day training that included lectures about dementia and challenging behaviour, guidelines about managing challenging behaviour and the IdA® dimensions and items. The remaining care staff team members received an one-hour training on how to use the IdA®.

The IdA® should be used in the study as a team instrument for care staff: the collection of information, the formulation of hypotheses about the causes and triggers of the behaviour and the development of the intervention plan should be conducted within the care staff team (e.g., team meetings, case conferences). Discussions and negotiation are necessary for an appropriate use of the IdA®.

Data collection and analysis

To evaluate the feasibility and relevance of the IdA®, self-developed questionnaires were used. The questionnaires included ratings about its structural aspects (15 items: clarity, meaningfulness, completeness, length, comprehensibility and background information required to use the IdA®); its benefit for daily care (8 items: professionalism and use of the IdA® as a memory aid, communication aid, planning aid, evaluation aid and information tool); carers’ experiences with the IdA® and an overall rating of the IdA®. An open question format was used to ask about missing information, problems with comprehension and other aspects. Finally, the caregivers were asked whether the IdA® supported them in decision making about residents’ challenging behaviour. The data were analysed using descriptive statistics (proportion and mean).

Results

Stage I: Development

The literature review identified a few models or approaches that aimed to explain the development of challenging behaviour in people with dementia (the multidimensional model [36], Progressively Lowered Stress Threshold model [37], Treatment Routes for Exploring Agitation (TREA) [38], Need driven dementia-compromised behaviour model (NDB model) [39], Serial Trial Intervention (STI) [40], Comprehensive model of behaviour [41], Antecedents-Behavior-Consequences (ABC)-approach [31]. In addition to the above mentioned approaches, further dementia-relevant psychosocial models exist [42]. These comprehensive and multidimensional models did not focus solely on behaviour but incorporated assumptions about the development of challenging behaviour in dementia.

The complexity of the behaviour and the influence of multiple determinants call for an explanation model that considers all aspects mentioned in the different approaches [33].

From the nursing perspective, the NDB model fulfilled all of the requirements for this study. It integrated the bio-medical and psychosocial perspective and provided a literature-based, comprehensible and comprehensive framework for assessing the causes of challenging behaviour. The origin of the model was in nursing science [39].

In this model, the causes of challenging behaviour are the unmet needs of people with dementia. They result from the increasing inability of these people to care for themselves, to interpret their needs as such and to communicate them in an understandable manner. The interaction between proximal (physiological and psychological aspects, characteristics of the environment) and background (e.g., neurological factors, cognitive abilities, health status, physical functioning, premorbid personality) factors is hypothesized to lead to the development of challenging behaviour. Background factors are stable characteristics of the person, with low changes over time, and act as risk factors and mediators for the influence of proximal factors on the development of challenging behaviour. Proximal factors are changeable aspects of the person with dementia or the direct environment and can directly produce challenging behaviour [39, 43]. For example, a person has a feeling of fear or experiences pain (proximal factors). The impairments in verbal communication (background factor) and a sensitive and fearful personality (background factor) determine how (through repetitive questioning “Hallo, Help, Hallo, Help”), in which situation, and how rapidly the feeling of anxiety evolves. The interplay between background and proximal factors determines how the person’s needs are communicated, expressed or satisfied. The literature on wandering did not produce new dimensions or aspects that were not covered by the NDB model. The results of the literature review provided substantial information about the priorities of the model and how to define and assess different triggers.

Additionally, other aspects were identified as relevant for documenting the behaviour itself and for deciding whether the behaviour was challenging or problematic and whether it required further consideration. These factors included a description of the type of behaviour, its quantity, context, burden and risk potential. The review showed a large number of scales and measurements, but none of them included all of the issues mentioned above. Thus, an assessment of all attributes of challenging behaviour was developed. The Cohen-Mansfield agitation model [44, 45] was identified as the one most commonly used in research to describe challenging behaviour. The underlying categorization of behaviour types established the theoretical foundation for the development of items in the IdA®. Apathy was added because the Cohens-Mansfield agitation model did not focus on this type of behaviour and apathy is one of the most prevalent challenging behaviours in nursing homes [3].

The first version (1.0) of the instrument consisted of 6 domains within 2 main parts: assessment of behaviour (part 1) and assessment of the causes and triggers of the behaviour (part 2). The behavioural assessment included 18 items, and part 2 was composed of 5 domains (health status & independence in everyday life, communication, personality & life style, moods & emotions, and environmental influences) with 8 dimensions (cognitive status, physical health status, physical discomfort, independence in everyday life, communication, personality and lifestyle before the onset of dementia, mood & emotions, and environmental influences). In total, 93 items were developed (Table 1). The items had different formats: closed-ended and open-ended. Examples are shown in Table 2.
Table 1

Different versions of the IdA®; results from stages I and II

Domains

Version 1.0

(93 items)

Prior to the expert evaluation as a result of Stage I, steps 1 & 2

Version 2.0

(72 items)

after expert evaluation as a result of Stage II, step 3

Version 3.0

(89 items)*

After evaluation by key persons in Stage II, step 4

Dimensions

Number of items

Dimensions

Number of items

Dimensions

Number of items

PART 1

 1. Assessment of the behaviour

Challenging behaviour

18

Challenging behaviour

14

Challenging behaviour

14

PART 2

 2. Health status & independence in everyday life

Cognitive status

7

Cognitive status

11

Cognitive status

11

Physical health status

5

Physical health status and discomfort

11

Physical health status and discomfort

10

Physical discomfort

13

Independence in everyday life

17

Independence in everyday life

1

Independence in everyday life

2

 3. Communication

Communication

6

Communication

6

Communication

9

 4. Personality & lifestyle before dementia

Personality & lifestyle before the onset of dementia

8

Personality & lifestyle before the onset of dementia

8

Personality & lifestyle before the onset of dementia

8

 5. Moods & emotions

Moods & emotions

7

Moods & emotions

10

Moods & emotions

10

 6. Environmental influences

Environmental influences

11

Environmental influences

11

Environmental influences

11

PART 3

 7. Summary

    

Overview and summary

14

Table 2

Examples of items in version 1.0

Stage 2: Judgement and quantification

1st evaluation: Expert panel

Sample

Ten out of 15 experts who were contacted completed the evaluation forms. The only reason for non-completion was a lack of time. The data confirmed that these persons had the required expertise to evaluate the instrument (Table 3). The experts were from a broad spectrum of disciplines; most of them had a double qualification, for example, as a nurse and a gerontologist. Almost all the experts had additional qualification in dementia-relevant aspects. All the experts had longstanding experience in the dementia field. Four of the experts were members of the group of experts who developed the German guidelines for challenging behaviours of residents living in nursing homes [46].
Table 3

Sample characteristics of the expert panel

Characteristics of expert panel, N = 10

Gender

3 men, 7 women

Academic Disciplines (some with more than one degree)

4 nursing/nursing science, 1 psychiatry/psychotherapy, 2 pedagogy/gerontology, 1 psycho-gerontology, 1 philosophy, 1 literature, 1 economics, 1 cultural studies

Vocational training

2 geriatric nurses, 5 nurses

Position

3 employed by the university/development and research institution (scientist, lecturer), 2 in leading positions in nursing homes, 2 working in quality development, 2 consultants in dementia care, 1 senior physician/freelancer

Additional dementia-related training

2 validation, dementia care mapping; 3 geriatric psychiatry/gerontology; 2 cognitive training, communication, organizational development and quality management; 1 palliative medicine

General work experience

On average, 20 years (min 4; max 35)

Work experience in dementia care

On average, 12 years (min 3; max 25)

Practice with people with dementia:

 Care of people with dementia

9 experts

 Assessment instruments/process documentation

9 experts

Theoretical knowledge about:

 Care of people with dementia

10 experts

 Process documentation

10 experts

 Assessment instruments in nursing

9 experts

Research experience:

 In field of dementia

4 experts

 With assessment instruments/process documentation

3 experts

Self-assessment of own expertise in the following:

 

 Dementia care

Median 3a

 Nursing assessment/documentation

Median 2a

aRanked between 0 = no expertise and 3 = comprehensive expertise

Average S-CVI

The S-CVI was calculated for all 9 dimensions, resulting in 36 CVI indices for each calculation method (average and universal agreement).

The S-CVI/Ave ranged from 0.64 to 1.00. Six (16.7%) of the 36 indices were lower than 0.80, and 10 indices (28%) were lower than 0.90. The majority of the dimensions indicated content validity according to both cut-offs (83% and 72% for 0.80 and 0.90, respectively). The questions showed a better performance than the answers. All S-CVI/UA values except one were below the acceptable minimum of 0.80 (Table 4).
Table 4

S-CVI for the IdA® Version 1.0

Dimensions of IdA®, 1.0

Relevance of Questions

S-CVI/Ave

Clarity of Questions

S-CVI/Ave

Significance of Answers

S-CVI/Ave

Completeness of Answers

S-CVI/Ave

1. Assessment of behaviour

0.92

0.87

0.87

0.83

2. Cognitive status

0.95

0.85

0.83

0.74

3. Physical health status

1.00

0.90

0.88

0.79

4. Physical discomfort

0.92

0.81

0.82

0.77

5. Independence in everyday life

0.81

0.85

0.72

0.72

6. Communication

0.93

0.91

0.81

0.83

7. Personality & lifestyle before the onset of dementia

0.95

0.92

0.87

0.90

8. Mood & emotions

0.87

0.64

0.80

0.80

9. Environmental influences

0.85

0.81

0.80

0.76

Questions in the independence in everyday life dimension received the lowest relevance scores for S-CVI/Ave (S-CVI/Ave 0.81), followed by the environmental influences (S-CVI/Ave 0.85) and mood & emotions (S-CVI/Ave 0.87). The best score for the relevance of questions was for the dimension physical health status at 1.0, followed by cognitive status and personality & lifestyle before the onset of dementia (both S-CVI/Ave 0.95) (Table 4).

The average S-CVI for understandability ranged between 0.64 and 0.92. The questions on mood & emotions were the least understood (S-CVI/Ave 0.64). Personality & lifestyle before the onset of dementia (S-CVI/Ave 0.92) as well as communication (S-CVI/Ave 0.91) appeared to be best understood (Table 4).

The scores for meaningfulness of the answers ranged from 0.72 to 0.88. The cognitive status dimension received the fewest negative values (S-CVI/Ave 0.88) followed by the assessment of behaviour (S-CVI/Ave 0.87). The weakest dimension regarding the meaningfulness of answers was independence in everyday life (S-CVI/Ave 0.72) (Table 4).

The answers for 5 of 9 subject areas appeared to be incomplete (S-CVI/Ave below 0.8). S-CVI/Ave varied between 0.72 and 0.9. The dimension personality & lifestyle before the onset of dementia had the highest scores, and the subject areas independence in everyday life, cognitive status and environmental influences received the lowest scores.

Item -CVI

The experts scored 93 items regarding 4 attributes, two for each set of questions and answers, and thus a total of 360 CVI indices and k* were calculated. On the item level, 77% (276) of the ratings had a CVI greater than or equal to 0.78. The majority of the question-ratings (84%, n = 154) and answer-ratings (69%, n = 122) showed valid results. None of the ratings were below 0.50, which would indicate rejection of the item. Fifty-one complete items (question plus answer options) showed an I-CVI value less than 0.78 and k* less than 0.74 for at least one of the 4 rated attributes. Only 6 had I-CVI values ≤0.5 and k* < 0.4. The results for each item are shown in Tables 5, 6, 7, 8, 9, 10, 11, 12 and 13.
Table 5

Content validity of the dimension “Assessment of behaviour”

 

Relevance of the questions

Clarity of the questions

Meaningfulness of the answers

Completeness of the answers

No

Content of the items

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationc

1

Type of behaviour

10

10

1.00

0.001

1.00

****

10

8

0.80

0.044

0.79

****

9

8

0.89

0.018

0.89

****

9

7

0.78

0.070

0.76

****

2

Description of behaviour

10

10

1.00

0.001

1.00

****

10

9

0.90

0.010

0.90

****

9

8

0.89

0.018

0.89

****

8

6

0.75

0.109

0.72

***

3

First occurrence of behaviour

10

9

0.90

0.010

0.90

****

10

10

1.00

0.001

1.00

****

9

9

1.00

0.002

1.00

****

8

8

1.00

0.004

1.00

****

4

Event before the occurrence of the behaviour

10

10

1.00

0.001

1.00

****

10

10

1.00

0.001

1.00

****

9

8

0.89

0.018

0.89

****

8

7

0.88

0.031

0.87

****

5

Time at which the behaviour occurs

10

10

1.00

0.001

1.00

****

10

9

0.90

0.010

0.90

****

9

9

1.00

0.002

1.00

****

9

6

0.67

0.164

0.60

***

6

Duration of the behaviour

10

10

1.00

0.001

1.00

****

10

9

0.90

0.010

0.90

****

9

7

0.78

0.070

0.76

****

9

7

0.78

0.070

0.76

****

7

Frequency of the behaviour

10

10

1.00

0.001

1...00

****

10

10

1.00

0.001

1.00

****

8

7

0.88

0.031

0.87

****

9

9

1.00

0.002

1.00

****

8

Area in which the behaviour occurs

9

8

0.89

0.018

0.89

****

9

9

1.00

0.002

1.00

****

9

8

0.89

0.018

0.89

****

9

7

0.78

0.070

0.76

****

9

Person present when the behaviour occurs

10

8

0.80

0.044

0.79

****

10

9

0.90

0.010

0.90

****

9

7

0.78

0.070

0.76

****

9

7

0.78

0.070

0.76

****

10

Events directly before the behaviour

10

10

1.00

0.001

1.00

****

10

9

0.90

0.010

0.90

****

9

8

0.89

0.018

0.89

****

8

6

0.75

0.109

0.72

***

11

Events directly after the behaviour

10

10

1.00

0.001

1.00

****

10

9

0.90

0.010

0.90

****

9

9

1.00

0.002

1.00

****

8

8

1.00

0.004

1.00

****

12

Helpful interventions

10

8

0.80

0.044

0.79

****

10

5

0.50

0.246

0.34

*

9

7

0.78

0.070

0.76

****

8

7

0.88

0.031

0.87

****

13

Behaviour unpleasant for the resident

10

8

0.80

0.044

0.79

****

10

9

0.90

0.010

0.90

****

9

8

0.89

0.018

0.89

****

9

6

0.67

0.164

0.60

***

14

Danger for the person with dementia

10

10

1.00

0.001

1.00

****

10

9

0.90

0.010

0.90

****

9

9

1.00

0.002

1.00

****

9

7

0.78

0.070

0.76

****

15

Behaviour unpleasant for the resident or for other residents

10

8

0.80

0.044

0.79

****

10

9

0.90

0.010

0.90

****

8

6

0.75

0.109

0.72

***

9

9

1.00

0.002

1.00

****

16

Danger for other residents

10

10

1.00

0.001

1.00

****

11

9

0.82

0.027

0.81

****

9

8

0.89

0.018

0.89

****

9

7

0.78

0.070

0.76

****

17

Impact on carers

10

8

0.80

0.044

0.79

****

10

6

0.60

0.205

0.50

**

9

8

0.89

0.018

0.89

****

9

7

0.78

0.070

0.76

****

18

Danger for carers

10

8

0.80

0.044

0.79

****

8

7

0.88

0.031

0.87

****

9

9

1.00

0.002

1.00

****

10

9

0.90

0.010

0.90

****

  

S-CVI/AVE 0.92

   

S-CVI/AVE 0.87

****

  

S-CVI/AVE 0.87

   

S-CVI/AVE 0.83

   
  

S-CVI/UA 0.56

   

S-CVI/UA 0.22

   

S-CVI/UA 0.28

   

S-CVI/UA 0.17

   

a I-CVI (content validity index) = number of experts providing a rating of 3 or 4/number of experts

b pc (probability of chance occurrence) = [N!/A!(N-A)!] × 0.5N, N = number of experts; A = number of experts agreeing on a rating of 3 or 4

c k* (modified kappa) = (I-CVI-pc)(1-pc)

d Evaluation criteria for the level of content validity: relationship between I-CVI and k*; excellent validity = I-CVI ≥ 0.78 and k* >0.74 (****); good validity I-CVI < 0.78 and ≥ 0.60 and k* ≤ 0.74 (***); fair validity I-CVI < 0.6 and ≥ 0.40 and k* ≤ 0.59 (**); and poor validity I-CVI < 0.4 and k* <0.40 (*)

Table 6

Content validity of the dimension “Cognition status”

 

Relevance of the questions

Clarity of the questions

Meaningfulness of the answers

Completeness of the answers

No

Content of the item

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationad

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

1

Diagnosis of dementia

10

10

1.00

0.001

1.00

****

10

9

0.90

0.010

0.90

****

10

8

0.80

0.044

0.79

****

9

7

0.78

0.070

0.76

****

2

Type of dementia

10

10

1.00

0.001

1.00

****

10

10

1.00

0.001

1.00

***

10

8

0.80

0.044

0.79

****

10

7

0.70

0.117

0.66

***

3

Dementia stage

10

10

1.00

0.001

1.00

****

10

8

0.80

0.044

0.79

****

10

6

0.60

0.205

0.50

**

8

6

0.75

0.109

0.72

***

4

Memory functions

8

6

0.75

0.109

0.72

***

8

7

0.88

0.031

0.87

****

9

9

1.00

0.002

1.00

****

9

6

0.67

0.164

0.60

***

5

Long-term memory

10

9

0.90

0.010

0.90

****

10

7

0.70

0.117

0.66

***

9

9

1 00

0.002

1.00

****

7

6

0.86

0.055

0.85

****

6

Short-time memory

10

10

1.00

0.001

1.00

****

10

8

0.80

0.044

0.79

****

10

8

0.80

0.044

0.79

****

9

6

0.67

0.164

0.60

***

7

Orientation

9

9

1.00

0.002

1.00

****

10

9

0.90

0.010

0.90

****

10

8

0.80

0.044

0.79

****

9

6

0.67

0.164

0.60

***

  

S-CVI/AVE 0.95

   

S-CVI/AVE 0.85

   

S-CVI/AVE 0.83

   

S-CVI/AVE 0.74

   
  

S-CVI/UA 0.71

   

S-CVI/UA 0.14

   

S-CVI/UA 0.33

   

S-CVI/UA 0.00

   

a I-CVI (content validity index) = number of experts providing a rating of 3 or 4/number of experts

b pc (probability of chance occurrence) = [N!/A!(N-A)!] × 0.5N, N = number of experts; A = number of experts agreeing on a rating of 3 or 4

c k* (modified kappa) = (I-CVI-pc)(1-pc)

d Evaluation criteria for level of content validity: relationship between I-CVI and k*; excellent validity = I-CVI ≥ 0.78 and k* >0.74 (****); good validity I-CVI < 0.78 and ≥ 0.60 andk* ≤0.74 (***); fair validity I-CVI < 0.6 and ≥ 0.40 and k* ≤0.59 (**); poor validity I-CVI < 0.4 and k* <0.40 (*)

Table 7

Content validity of the dimension “Physical health status”

 

Relevance of the questions

Clarity of the questions

Meaningfulness of the answers

Completeness of the answers

No

Content of the items

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationad

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

1

Physical impairment

9

9

1.0

0.002

1.0

****

9

8

0.89

0.018

0.89

****

-

-

-

-

-

-

-

-

-

-

-

-

2

Mobility-related impairment

10

10

1.0

0.001

1.0

****

10

7

0.70

0.117

0.66

***

10

9

0.90

0.010

0.90

****

10

7

0.70

0.117

0.66

***

3

Food intake

10

10

1.0

0.001

1.0

****

10

10

1.00

0.001

1.00

****

10

9

0.90

0.010

0.90

****

9

7

0.78

0.070

0.76

****

4

Limitations with bowel or bladder function

10

10

1.0

0.001

1.0

****

10

10

1.00

0.001

1.00

****

10

8

0.80

0.176

0.76

****

9

8

0.89

0.018

0.89

****

5

Other physical limitations

10

10

1.0

0.001

1.0

****

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

10

8

0.80

0.044

0.79

****

  

S-CVI/AVE 1.0

   

S-CVI/AVE 0.90

   

S-CVI/AVE 0.88

   

S-CVI/AVE 0.79

   
  

S-CVI/UA 1.0

   

S-CVI/UA 0.40

   

S-CVI/UA 0.00

   

S-CVI/UA 0.00

   

a I-CVI (content validity index) = number of experts providing a rating of 3 or 4/number of experts

b pc (probability of chance occurrence) = [N!/A!(N-A)!] × 0.5N, N = number of experts; A = number of experts agreeing on a rating of 3 or 4

c k* (modified kappa) = (I-CVI-pc)(1-pc)

d Evaluation criteria for level of content validity: relationship between I-CVI and k*; excellent validity = I-CVI ≥ 0.78 and k* >0.74 (****); good validity I-CVI < 0.78 and ≥ 0.60 and k* ≤ 0.74 (***); fair validity I-CVI < 0.6 and ≥ 0.40 and  k* ≤0.59 (**); poor validity I-CVI < 0.4 and k* <0.40 (*)

Table 8

Content validity of the dimension “Independence in everyday life”

 

Relevance of the questions

Clarity of the questions

Meaningfulness of the answers

Completeness of the answers

No

Content of the items

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

1

Dependence in movement

10

9

0.90

0.010

0.90

****

10

10

1.00

0.001

1.00

****

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

-

2

Resources in movement

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

***

3

Dependence in personal hygiene

10

7

0.70

0.117

0.66

****

10

10

1.00

0.001

1.00

****

10

7

0.70

0.117

0.66

***

10

7

0.70

0.117

0.66

****

4

Resources in personal hygiene

10

8

0.80

0.044

0.79

****

10

10

1.00

0.001

1.00

****

10

7

0.70

0.117

0.66

****

10

7

0.70

0.117

0.66

****

5

Dependence in dressing

10

8

0.80

0.044

0.79

****

10

9

0.90

0.010

0.90

****

10

8

0.80

0.044

0.79

****

10

8

0.80

0.044

0.79

***

6

Resources in dressing

10

8

0.80

0.044

0.79

****

10

8

0.80

0.044

0.79

****

10

7

0.70

0.117

0.66

***

10

7

0.70

0.117

0.66

***

7

Dependence in nutrition

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

8

Resources in nutrition

9

8

0.89

0.018

0.89

****

10

9

0.90

0.010

0.90

****

10

8

0.80

0.044

0.79

****

10

8

0.80

0.044

0.79

****

9

Dependence in elimination

10

10

1.00

0.001

1.00

***

8

7

0.88

0.031

0.87

****

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

10

Resources in elimination

10

10

1.00

0.001

1.00

****

10

10

1.00

0.001

1.00

****

10

8

0.80

0.044

0.79

****

10

8

0.80

0.044

0.79

****

11

Dependence in leisure time activities

10

7

0.70

0.117

0.66

***

10

6

0.60

0.205

0.50

**

10

6

0.60

0.205

0.50

**

10

6

0.60

0.205

0.50

**

12

Resources in leisure time

10

7

0.70

0.117

0.66

***

10

8

0.80

0.044

0.79

***

10

6

0.60

0.205

0.50

**

10

6

0.60

0.205

0.50

**

13

Dependence in social contacts

10

8

0.80

0.044

0.79

****

10

8

0.80

0.044

0.79

***

10

7

0.70

0.117

0.66

***

10

7

0.70

0.117

0.66

***

14

Resources in social contacts

10

8

0.80

0.044

0.79

****

10

8

0.80

0.044

0.79

****

10

6

0.60

0.205

0.50

**

10

6

0.60

0.205

0.50

**

15

Other dependencies

10

6

0.60

0.205

0.50

**

10

7

0.70

0.117

0.66

***

10

5

0.50

0.246

0.34

*

10

5

0.50

0.246

0.34

*

16

Other resources

9

5

0.56

0.246

0.41

**

9

7

0.78

0.070

0.76

****

9

5

0.56

0.246

0.41

**

9

5

0.56

0.246

0.41

**

17

Burden through dependencies

10

9

0.90

0.010

0.90

****

9

7

0.78

0.070

0.76

****

9

6

0.67

0.164

0.60

***

9

6

0.67

0.164

0.60

***

  

S-CVI/AVE 0.81

   

S-CVI/AVE 0.85

****

  

S-CVI/AVE 0.72

   

S-CVI/AVE 0.72

   
  

S-CVI/UA 0.12

   

S-CVI/UA 0.24

   

S-CVI/UA 0.00

   

S-CVI/UA 0.00

   

a I-CVI (content validity index) = number of experts providing a rating of 3 or 4/number of experts

b pc (probability of chance occurrence) = [N!/A!(N-A)!] × 0.5N, N = number of experts; A = number of experts agreeing on a rating of 3 or 4

c k* (modified kappa) = (I-CVI-pc)(1-pc)

d Evaluation criteria for level of content validity: relationship between I-CVI and k*; excellent validity = I-CVI ≥ 0.78 and k* > 0.74 (****); good validity I-CVI < 0.78 and ≥ 0.60 and k* ≤ 0.74 (***); fair validity I-CVI < 0.6 and ≥ 0.40 and k* ≤0.59 (**); poor validity I-CVI < 0.4 and k* <0.40 (*)

Table 9

Content validity of the dimension “Physical discomfort”

 

Relevance of the questions

Clarity of the questions

Meaningfulness of the answers

Completeness of the answers

No

Content of the items

Number of experts

Number of ratings of 3 or 4

I-CVIa

pb

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

1

Health problems generic

9

8

0.89

0.018

0.89

****

9

7

0.78

0.070

0.76

****

-

-

-

-

-

-

-

-

-

-

-

-

2

Pain

10

10

1.00

0.001

1.00

****

10

10

1.00

0.001

1.00

****

9

8

0.89

0.018

0.89

****

9

6

0.67

0.164

0.60

***

3

Depression

10

10

1.00

0.001

1.00

****

8

8

1.00

0.004

1.00

****

8

7

0.88

0.031

0.87

****

9

7

0.78

0.070

0.76

****

4

Hallucination/delusion

10

10

1.00

0.001

1.00

****

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

10

8

0.80

0.044

0.79

****

5

Sleep

10

10

1.00

0.001

1.00

****

10

8

0.80

0.044

0.79

****

10

8

0.80

0.044

0.79

****

10

7

0.70

0.117

0.66

***

6

Bowel elimination

10

10

1.00

0.001

1.00

****

10

9

0.90

0.010

0.90

****

10

8

0.80

0.044

0.79

****

10

8

0.80

0.044

0.79

****

7

Urinary elimination

10

10

1.00

0.001

1.00

****

10

8

0.80

0.044

0.79

****

10

9

0.90

0.010

0.90

****

10

8

0.80

0.044

0.79

****

8

Thirst/hunger

8

8

1.00

0.004

1.00

****

10

8

0.80

0.044

0.79

****

10

8

0.80

0.044

0.79

****

10

8

0.80

0.044

0.79

****

9

Need for movement

10

7

0.70

0.117

0.66

***

10

8

0.80

0.044

0.79

****

10

8

0.80

0.044

0.79

****

10

8

0.80

0.044

0.79

****

10

Breathing/circulatory problems

10

9

0.90

0.010

0.90

****

9

7

0.78

0.070

0.76

****

10

8

0.80

0.044

0.79

****

10

8

0.80

0.044

0.79

****

11

Discomfort

10

6

0.60

0.205

0.50

**

10

5

0.50

0.246

0.34

*

9

6

0.67

0.164

0.60

***

9

6

0.67

0.164

0.60

***

12

Other relevant health problems

9

8

0.89

0.018

0.89

****

8

6

0.75

0.109

0.72

***

8

5

0.63

0.219

0.52

**

8

6

0.75

0.109

0.72

***

13

Side effects of medication

9

9

1.00

0.002

1.00

****

9

6

0.67

0.164

0.60

***

9

9

1.00

0.002

1.00

****

9

8

0.89

0.018

0.89

****

  

S-CVI/AVE 0.92

   

S-CVI/AVE 0.81

****

  

S-CVI/AVE 0.82

   

S-CVI/AVE 0.77

   
  

S-CVI/UA 0.62

   

S-CVI/UA 0.15

   

S-CVI/UA 0.08

   

S-CVI/UA 0.00

   

a I-CVI (content validity index) = number of experts providing a rating of 3 or 4/number of experts

b pc (probability of chance occurrence) = [N!/A!(N-A)!] × 0.5N, N = number of experts; A = number of experts agreeing on a rating of 3 or 4

c k* (modified kappa) = (I-CVI-pc)(1-pc)

d Evaluation criteria for level of content validity: relationship between I-CVI and k*; excellent validity = I-CVI ≥ 0.78 and k* >0.74 (****); good validity I-CVI < 0.78 and ≥ 0.60 and k* ≤0.74 (***); fair validity I-CVI < 0.6 and ≥ 0.40 and  k* ≤ 0.59 (**); poor validity I-CVI < 0.4 and k* <0.40 (*)

Table 10

Content validity of the dimension “Communication”

 

Relevance of the questions

Clarity of the questions

Meaningfulness of the answers

Completeness of the answers

No

Content of the items

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

1

To express oneself verbally

10

10

1.00

0.001

1.00

****

10

10

1.00

0.001

1.00

****

10

8

0.80

0.044

0.79

****

10

8

0.80

0.044

0.79

****

2

To understand verbal communication

10

9

0.90

0.010

0.90

****

10

7

0.70

0.117

0.66

***

10

7

0.70

0.117

0.66

***

10

7

0.70

0.117

0.66

***

3

To understand written language

10

7

0.70

0.117

0.66

***

9

7

0.78

0.070

0.76

****

9

6

0.67

0.164

0.60

***

9

7

0.78

0.070

0.76

****

4

Hearing

10

10

1.00

0.001

1.00

****

10

10

1.00

0.001

1.00

****

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

5

Vision

10

10

1.00

0.001

1.00

****

10

10

1.00

0.001

1.00

****

10

8

0.80

0.044

0.79

****

10

9

0.90

0.010

0.90

****

6

Communication of wishes and needs

10

10

1.00

0.001

1.00

****

10

10

1.00

0.001

1.00

****

10

10

1.00

0.001

1.00

****

9

8

0.89

0.018

0.89

****

  

S-CVI/AVE 0.93

   

S-CVI/AVE 0.91

   

S-CVI/AVE 0.81

   

S-CVI/AVE 0.83

   
  

S-CVI/UA 0.67

   

S-CVI/UA 0.67

   

S-CVI/UA 0.16

   

S-CVI/UA 0.00

   

a I-CVI (content validity index) = number of experts providing a rating of 3 or 4/number of experts

b pc (probability of chance occurrence) = [N!/A!(N-A)!] × 0.5N, N = number of experts; A = number of experts agreeing on a rating of 3 or 4

c k* (modified kappa) = (I-CVI-pc)(1-pc)

d Evaluation criteria for the level of content validity: relationship between I-CVI and k*; excellent validity = I-CVI ≥ 0.78 and k* >0.74 (****); good validity I-CVI < 0.78 and ≥ 0.60 and k* ≤0.74 (***); fair validity I-CVI < 0.6 and ≥ 0.40 and k* ≤0.59 (**); poor validity I-CVI < 0.4 and k* <0.40 (*)

Table 11

Content validity of the dimension “Personality & life style before the onset of dementia”

 

Relevance of the questions

Clarity of the questions

Meaningfulness of the answers

Completeness of the answers

No

Content of the items

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

1

Personality prior to the onset of dementia

10

9

0.90

0.010

0.90

****

9

8

0.89

0.018

0.89

****

10

8

0.80

0.044

0.79

****

10

7

0.70

0.117

0.66

***

2

Stress and frustration tolerance

10

9

0.90

0.010

0.90

****

10

8

0.80

0.044

0.79

****

10

9

0.90

0.010

0.90

****

10

8

0.80

0.044

0.79

****

3

Coping with stressful situations before the onset of dementia

10

10

1.00

0.001

1.00

****

10

8

0.80

0.044

0.79

****

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

4

Past situations or events related to negative emotions

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

10

10

1.00

0.001

1.00

****

5

Past situations or events related to positive emotions

9

9

1.00

0.002

1.00

****

9

9

1.00

0.002

1.00

****

9

7

0.78

0.070

0.76

****

9

9

1.00

0.002

1.00

****

6

Leisure activities preferred by the resident prior to dementia

9

8

0.89

0.018

0.89

****

9

9

1.00

0.002

1.00

****

9

8

0.89

0.018

0.89

****

9

8

0.89

0.018

0.89

****

7

Type of job/housework before dementia

9

9

1.00

0.002

1.00

****

9

9

1.00

0.002

1.00

****

9

7

0.78

0.070

0.76

****

9

8

0.89

0.018

0.89

****

8

Important routine rhythms or established daily rituals

9

9

1.00

0.002

1.00

****

9

9

1.00

0.002

1.00

****

9

9

1.00

0.002

1.00

****

8

8

1.00

0.004

1.00

****

  

S-CVI/AVE 0.95

   

S-CVI/AVE 0.92

   

S-CVI/AVE 0.87

   

S-CVI/AVE 0.90

   
  

S-CVI/UA 0.50

   

S-CVI/UA 0.50

   

S-CVI/UA 0.13

   

S-CVI/UA 0.38

   

a I-CVI (content validity index) = number of experts providing a rating of 3 or 4/number of experts

b pc (probability of chance occurrence) = [N!/A!(N-A)!] × 0.5N, N = number of experts; A = number of experts agreeing on a rating of 3 or 4

c k* (modified kappa) = (I-CVI-pc)(1-pc)

d Evaluation criteria for level of content validity: relationship between I-CVI and k*; excellent validity = I-CVI ≥ 0.78 and k* >0.74 (****); good validity I-CVI < 0.78 and ≥ 0.60 and k* ≤0.74 (***); fair validity I-CVI < 0.5 and ≥ 0.40 and k* ≤0.59 (**); poor validity I-CVI < 0.4 and k* <0.40 (*)

Table 12

Content validity of the dimension “Mood & emotions”

 

Relevance of the questions

Clarity of the questions

Meaningfulness of the answers

Completeness of the answers

No

Content of the items

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationad

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

1

Boredom

10

10

1.00

0.001

1.00

****

10

6

0.60

0.205

0.50

**

10

8

0.80

0.044

0.79

****

10

9

0.90

0.010

0.90

****

2

Anxiety

10

10

1.00

0.001

1.00

****

10

6

0.60

0.205

0.50

**

10

8

0.80

0.044

0.79

****

10

9

0.90

0.010

0.90

****

3

Exhaustion

10

9

0.90

0.010

0.90

****

10

7

0.70

0.117

0.66

***

10

8

0.80

0.044

0.79

****

10

9

0.90

0.010

0.90

****

4

Impression of loneliness or isolation

10

9

0.90

0.010

0.90

****

10

6

0.60

0.205

0.50

**

10

7

0.70

0.117

0.66

***

10

9

0.90

0.010

0.90

****

5

Trusting relationships

10

7

0.70

0.117

0.66

***

10

5

0.50

0.246

0.34

*

10

8

0.80

0.044

0.79

****

10

6

0.60

0.205

0.50

**

6

Amount of occupation

10

8

0.80

0.044

0.79

****

9

7

0.78

0.070

0.76

****

10

8

0.80

0.044

0.79

****

10

5

0.50

0.246

0.34

*

7

Type of occupation

10

8

0.80

0.044

0.79

****

10

7

0.70

0.117

0.66

***

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

  

S-CVI/AVE 0.87

   

S-CVI/AVE 0.64

   

S-CVI/AVE 0.80

   

S-CVI/AVE 0.80

   
  

S-CVI/UA 0.29

   

S-CVI/UA 0.50

   

S-CVI/UA 0.00

   

S-CVI/UA 0.00

   

a I-CVI (content validity index) = number of experts providing a rating of 3 or 4/number of experts

b pc (probability of chance occurrence) = [N!/A!(N-A)!] × 0.5N, N = number of experts; A = number of experts agreeing on a rating of 3 or 4

c k* (modified kappa) = (I-CVI-pc)(1-pc)

d Evaluation criteria for level of content validity: relationship between I-CVI and k*; excellent validity = I-CVI ≥ 0.78 and k* >0.74 (****); good validity I-CVI < 0.78 and ≥ 0.60 and k* ≤0.74 (***); fair validity I-CVI < 0.6 and ≥ 0.40 and k* ≤0.59 (**); poor validity I-CVI < 0.4 and k* <0.40 (*)

Table 13

Content validity of the dimension “Environmental influences”

 

Relevance of the questions

Clarity of the questions

Meaningfulness of the answers

Completeness of the answers

No

Content of the items

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

Number of experts

Number of ratings of 3 or 4

I-CVIa

pc b

k* c

Evaluationd

1

Surroundings

9

8

0.89

0.018

0.89

****

8

5

0.63

0.219

0.52

**

-

-

-

-

-

       

2

Lighting

10

9

0.90

0.010

0.90

****

10

7

0.70

0.117

0.66

***

10

8

0.80

0.044

0.79

****

10

5

0.50

0.246

0.34

*

3

Noises

10

9

0.90

0.010

0.90

****

10

8

0.80

0.044

0.79

****

10

8

0.80

0.044

0.79

****

10

7

0.70

0.117

0.66

***

4

Smells

10

7

0.70

0.117

0.66

***

10

8

0.80

0.044

0.79

****

9

6

0.67

0.164

0.60

***

9

7

0.78

0.070

0.76

****

5

Stimuli

10

8

0.80

0.044

0.79

****

10

7

0.70

0.117

0.66

***

7

5

0.71

0.164

0.66

***

10

8

0.80

0.044

0.79

****

6

Atmosphere

10

6

0.60

0.205

0.50

**

9

6

0.67

0.164

0.60

***

10

7

0.70

0.117

0.66

***

10

8

0.80

0.044

0.79

****

7

Security

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

9

7

0.78

0.070

0.76

****

10

7

0.70

0.117

0.66

***

8

Private space

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

10

7

0.70

0.117

0.66

***

9

Opportunities for contact

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

10

9

0.90

0.010

0.90

****

10

7

0.70

0.117

0.66

***

10

Significant carer/caregivers

9

9

1.00

0.002

1.00

****

9

9

1.00

0.002

1.00

****

9

9

1.00

0.002

1.00

****

9

9

1.00

0.002

1.00

****

11

Continuity of being in contact with significant carer

8

7

0.88

0.031

0.87

****

8

7

0.88

0.031

0.87

 

9

7

0.78

0.070

0.76

****

9

8

0.89

0.018

0.89

****

  

S-CVI/AVE 0.85

   

S-CVI/AVE 0.81

   

S-CVI/AVE 0.80

   

S-CVI/AVE 0.76

   
  

S-CVI/UA 0.09

   

S-CVI/UA 0.09

   

S-CVI/UA 0.10

   

S-CVI/UA 0.10

   

a I-CVI (content validity index) = number of experts providing a rating of 3 or 4/number of experts

b pc (probability of chance occurrence) = [N!/A!(N-A)!] × 0.5N, N = number of experts; A = number of experts agreeing on a rating of 3 or 4

c k* (modified kappa) = (I-CVI-pc)(1-pc)

d Evaluation criteria for level of content validity: relationship between I-CVI and k*; excellent validity = I-CVI ≥ 0.78 and k* >0.74 (****); good validity I-CVI < 0.78 and ≥ 0.60 and k* ≤0.74 (***); fair validity I-CVI < 0.6 and ≥ 0.40 and k* ≤0.59 (**); poor validity I-CVI < 0.4 and k* <0.40 (*)

For the challenging behaviour dimension, all 18 questions showed good validity regarding relevance and 16 regarding clarity. One item was considered fair and one poor in terms of clarity. For the answers, 16 of 18 items showed excellent validity in terms of meaningfulness and 14 in terms of completeness. Two answers were considered good regarding meaningfulness and 3 regarding completeness. One answer item showed fair validity in terms of completeness. Altogether, 5 complete items (question plus answer options, no. 12 and no. 17) needed modification (Table 5).

Six of 7 questions for the dimension cognitive status were considered excellent (relevance and clarity). Two items had good validity (no. 4 for relevance; no. 4a for clarity). One answer (no. 3) showed fair values for meaningfulness, and two items were considered good in terms of completeness (no. 1 and no. 4a). In total, 6 complete items (questions and answers) needed some modification (Table 6).

All questions and all answers for the physical health status dimension were considered relevant or meaningful (excellent validity). All questions but one showed excellent validity in terms of clarity (no. 5a showed good validity). Three of 4 answers were considered excellent in terms of completeness, and one showed good validity (no. 5a). Overall, only one complete item (5a) needed revising (Table 7).

Eleven of 13 questions for the dimension physical discomfort were considered excellent, one good and one fair regarding relevance. Ten questions were rated excellent, 2 good and one poor in terms of clarity. Ten of 12 answers were considered excellent, one good and one fair regarding meaningfulness, and 8 were rated excellent and 2 good for completeness. Altogether, 6 complete items needed modification (Table 8).

For the dimension independence in everyday life, 12 of the 17 questions were rated excellent regarding relevance, 3 good and 2 fair. Fifteen questions showed excellent validity in terms of clarity, one showed good validity, and one had fair validity. Only 7 answers were considered excellent in terms of meaningfulness, whereas 5 were good, 5 fair and one poor. Six answers were rated excellent for completeness, 10 good, and one fair. Overall, 14 of the 17 complete items needed some degree of modification (Table 9).

Five of 6 questions for the dimension communication were considered to have excellent validity in terms of relevance and clarity. One question showed good validity for relevance and one for clarity. Four answers had excellent ratings and two had good ratings for meaningfulness and completeness. In total, 2 complete items needed modification (Table 10).

For the dimension personality & lifestyle before the onset of dementia , all questions were considered relevant and clear (excellent validity). Additionally, all answers showed excellent values in terms of meaningfulness, and all but one in terms of completeness. One answer showed good validity and needed modification (Table 11).

The relevance of 6 of the 7 questions in the mood & emotions dimension were rated excellent and one good. In contrast, only one question showed excellent validity, two indicated good validity, two were fair and one poor in terms of clarity. For meaningfulness and completeness, 6 answers were rated excellent for both. One answer showed good validity for meaningfulness. One answer showed fair validity and one poor validity for completeness. Altogether, all of the items needed modification to different degrees (Table 12).

For the environmental influences dimension, 9 of 10 questions showed excellent, two good and one fair validity regarding relevance. Seven questions had excellent, two good and one fair validity values in terms of clarity. For meaningfulness, 7 answers were considered excellent and 3 good. For completeness, 5 answers were considered excellent, 4 good and 1 poor. Overall, 9 complete items had to be revised (Table 13).

The modifications to the instrument were made based on the CVI scores and the free text comments of the expert panel (Table 3). The changes were mainly related to a reduction in the questions and a specification and clarification of the answers. The items in part 1 that focused on the experiences and reactions of the social environment regarding challenging behaviour were the most criticized. These criticisms resulted in the rejection of two items and a merging of other items. In part 2 of the instrument (capturing the triggers of the behaviour), the subject area independence in everyday life was fundamentally revised. The main criticism was that the items were too rich in detail about care dependency and that the relation of the dimension to challenging behaviour was not clear. The experts recommended replacing the assessment of dependency with an assessment of the consequences or experiences of dependency resulting in the development of challenging behaviour. These consequences included the “stress/burden of the person with dementia due to being limited in self-care ability” and “stress/burden related to the care activities themselves”. Additionally, the subject area mood & emotions was further refined. The experts mentioned several times in the free text that because of the large number of questions, it seemed difficult to maintain an overview of all potentially important factors related to the development of challenging behaviour. Similarly, the objective of the assessment seemed to fade during the assessment process, and the reference to the challenging behaviour decreased. To maintain focus on the aim of the instrument during the assessment process and to illustrate the relationship between the items and challenging behaviour, overarching key questions were developed for all subject areas. These key questions introduced revised assessment dimensions (Table 14). The modified version of the IdA® (version 2.0) included 72 items and was presented for discussion during the subsequent workshop with the health care practitioners.
Table 14

Content of the IdA® and the guiding questions (Version 3)

Dimension

Guiding questions

Content of dimension

Part 1 Description of the behaviour

 Assessment of challenging behaviour

Which type of challenging behaviour is observed?

Naming and description of the behaviour; description of the circumstances under which the behaviour occurs; quantification of behaviour; denomination of the consequences of the behaviour

Part 2 Capturing the triggers of the behaviour

 Cognitive status

Could the challenging behaviour be explained by the type or stage of dementia?

Could the identified cognitive impairments explain the challenging behaviour?

Events from the past, information about oneself, present living situation, sense of time, orientation in important rooms, complete activities, recognition of important everyday items, recall of information received a day or less before, recognition of important everyday items

 Physical health status & discomfort

Could the identified physical impairment or discomfort somehow be related to the challenging behaviour?

Mobility, fluid and food intake, excretion functions, sleep, vital physical functions, depression, pain, delusions/hallucinations, medications with adverse side effects

 Independence in everyday life

Could the identified stressful/burdening dependencies in everyday life activities have provoked the challenging behaviour?

Emotional burden/stress of care dependencies and care interventions

 Communication

Could the identified comprehension/communication difficulties have triggered/provoked the challenging behaviour? Is it possible that the behaviour itself presents a form of communication and to explain the behaviour accordingly?

Hearing/seeing well, language of communication, comprehensibility of speech, quality of verbal communication, understanding of verbal/written communication, contact with others, communication of personal wishes/desires

 Personality & lifestyle before the onset of dementia

Could the challenging behaviour be an expression of the resident’s personality? Could the challenging behaviour be related to past life events or the person’s former lifestyle? Could the challenging behaviour be a reaction to stress?

Personality before the onset of dementia, stress tolerance, frustration tolerance, management of stressful situations before dementia;

Events connected with negative emotions or threatening events, events associated with positive impact/emotions, leisure time before the onset of dementia, occupation, daily rhythm/daily rituals with unique importance

 Mood & emotions

Could the challenging behaviour be an expression of certain moods and emotions?

Could the challenging behaviour serve as self-stimulation?

Fear, tiredness/exhaustion, closer relationship to the resident, relationship showing confidence, safety, times without occupation, boredom, occupational activities/leisure time activities/structure of the day not matching the residents´ personal preferences

 Environmental influences

Could the challenging behaviour be related to certain environmental characteristics?

Could the challenging behaviour be related to a lack of sense of security and familiarity?

Could staff structure have an impact on the challenging behaviour?

Lighting, noise in the surroundings, smells, furnishing, privacy, contact with others, positive stimuli/stimulations, preference of caregiver, continuity of primary caregivers

2nd evaluation: Workshop with health care workers

Sample

Seventeen persons from 10 nursing homes participated in the workshop. Most of them were nurses, and a few had a geriatric background, were leaders of the unit or nursing home, or were quality managers. No further information about the participants was collected.

Results

The discussion with the health care workers was very helpful for determining how the items were understood and interpreted. The language was simplified based on the results of the workshop. The subject areas “communication” and “independence in everyday life” again required further clarification. Some items were perceived to be too broad; the participants expressed a need for more differentiated and detailed information. Difficulty summarizing the large amount of information obtained was the strongest piece of criticism. Two main points were discussed. First, the requirements needed to answer the items required further consideration. Some items could only be answered with more in-depth observations of the person with dementia, e.g., their cognitive abilities, or further information was needed from family members or doctors. For example, the domain “Personality & lifestyle before the onset of dementia” contained questions that could not be answered without knowing the person well and for a long period of time. The nursing home care staff was often dependent on information provided by family members or significant others, if present. In some cases, these informants were not available, and the necessary information to answer the IdA® questions could not be obtained. Other items led to actions such as speaking with the occupational therapist about activities or a change in care routines. Second, the participants wanted to have an overview of all relevant aspects that could be associated with the challenging behaviour in the particular individual. To address the first criticism, each item received a new assessment criterion called “what has to be done?”. This question included three selection options: a) further clarification needed, b) action needed to plan, and c) item remains important. Referring to the second point of criticism, key questions with the corresponding significant content were presented on a separate page through bullet points. These modifications resulted in version 3.0 of the IdA®, which contained 7 domains with 89 items and three parts.

Stage III evaluation

Sample

Sixty of the 229 questionnaires were returned. One ward unit withdrew from the study because of being transferred from one location to another. The response rate varied considerably between nursing homes (7% to 58%). The reasons for non-response included holiday time, illness, and a lack of time. The respondents’ characteristics are shown in Table 15. The characteristics of the sample reflect the population of the care staff in German nursing homes.
Table 15

Characteristics of the sample in the evaluation study

Characteristics of the sample n = 60

 

Women, n [%] n = 56

50 [89]

Age, n [%], years, n = 51, 8 missing

 Up to 19

1 [2]

 20–24

5 [10]

 25–29

8 [16]

 30–34

3 [6]

 35–39

1 [1]

 40–44

10 [20]

 45–49

8 [16]

 50–54

9 [17]

 55–59

3 [6]

 60 and over

3 [6]

Profession n [%], n = 55, 5 missing

 Registered nurses

23 [42]

 Nursing aides

22 [40]

 Other

10 [18]

Working hours per week, x [SD], n = 53, 7 missing

28.8 [9.3]

Results

Structure of the IdA®

The majority of the respondents positively rated 9 of the 13 structural aspects of the IdA®. Three aspects were rated positively by 40% to 50% of the respondents. However, 71% of the respondents rated the time aspect negatively. The IdA® did not have to be applied by registered nurses only (67%), but the interpretation of information from the IdA® required broad knowledge of dementia (Table 16).
Table 16

Evaluation of the structural aspects of the IdA®

Structural aspects

Items

Agreement

n

Clarity

Very clear

68%

57

Meaningfully constructed

70%

57

Not too inconvenient

66%

56

Allows prompt access to resident information

61%

56

Meaningfulness

Contains only information relevant to care

42%

57

No double documentation needed

49%

56

Completeness

Complete

81%

57

Length/scope

Not too detailed

53%

57

Not time consuming

29%

57

Not too extensive

42%

57

Comprehensibility

Clear language

75%

56

Does not require comprehensive training

53%

58

Not too complicated

69%

57

Requirements

Requires knowledge of dementia

60%

58

Completed only by registered nursing staff

33%

58

The detailed ratings of each domain of the IdA® showed that the majority of the respondents provided mostly positive assessments of the completeness (71–80%) of the information, the relevance of the information for daily care (64–76%) and the contribution of the IdA® domains for internal (71–82%) and external communication (59–67%).

Benefits of the IdA® for daily care

Six aspects, asked with 8 items, pertained to the benefit of the IdA® for the daily care of residents with dementia. The majority of the respondents (min. 50%) rated six items positively. However, most of the respondents perceived the IdA® as not capable of indicating quality of care and as less supportive of daily communication with other health care workers (Table 17).
Table 17

Evaluation of the benefit of the IdA® for daily care of residents with dementia and challenging behaviour

Aspects of daily care

Items

Agreement

n

Time

Does not reduce valuable time for care

53%

58

Information aid

Provides information about residents’ bio-psycho-social information

80%

56

The IdA® can be read by others

55%

58

Communication aid

Is helpful for communication with colleagues

60%

58

Is helpful for communication with other health care workers

46%

57

Planning aid

Supports planning of care

64%

58

Evaluation aid

Indicates good care

36%

58

Memory aid

Is a good memory aid

67%

58

The overall rating of the IdA® was positive. Two of three respondents (64%) rated the IdA® as good to very good. When asked “Would you continue using the assessment tool after the testing phase?”, 28 of 47 (56%) reported that they would use the IdA®, whereas 19 said they would not. If the IdA® continued to be used together in a team, 84% of the users said that they would keep using the tool.

The IdA® as a tool for decision making

For decision making regarding the causes of challenging behaviour, it was important for the care staff to have individualized information about the residents. In total, 74% of the respondents confirmed the statement “The IdA® gives me the opportunity to express the individuality of the resident with dementia”. The IdA® seemed to mostly stimulate reflection on the residents’ behaviour. The IdA® included aspects related to behaviour that the majority of the respondents had not considered before; furthermore, it helped describe the residents’ behaviour to others and to provide another perspective on the behaviour. Two-thirds of the respondents thought that the IdA® was relevant to their daily work. Regarding the statements of whether the IdA® had changed their own behaviour, helped plan interventions, provided unknown information or improved understanding of the behaviour, between 44% and 47% of the respondents agreed with the statements. Unfortunately, the respondents did not experience an improvement in the behaviour of the residents (Table 18).
Table 18

Benefits of the IdA® for decision making

Benefit for decision making

Agreement

n =

The IdA® has encouraged me to think more intensively about the residents’ behaviour

71%

59

The IdA® contains factors that I had never thought about in the context of residents’ problem behaviour

66%

59

I see a relationship between the IdA® and my daily work

64%

58

By using the IdA®, I see the behaviour of the residents with dementia from another perspective

63%

59

With the IdA®, I can better describe the residents’ problem behaviour to others

58%

59

Planning interventions to address the residents´ behaviour is easier with the information provided by the IdA®

47%

58

My behaviour towards the residents has changed since the introduction of the IdA®

47%

58

The IdA® provides information about the residents that I did not know before

45%

58

Since the introduction of the IdA®, I have a better understanding of the behaviour of the residents with dementia

44%

57

I feel that the behaviour of the residents has changed positively since the introduction of the IdA®

26%

58

The IdA® gives me the opportunity to express the individuality of the resident with dementia

74%

58

Discussion

The purpose of this study was to develop a nursing assessment of the triggers and causes of challenging behaviour among residents with dementia as a diagnostic guideline and to assess its content validity, feasibility and practicability. We used the approach suggested by Lynn [26], which is broadly applied in health care and nursing research [4749]. Although Beckstead [50] questioned the meaning of content validity for establishing the validity of an instrument, the two-stage approach is widely accepted in the methodological literature and is considered a requirement for the development of new instruments [31, 5153]. The main criticism refers to the calculation of the agreement indices and the significance of the number of experts for the risk of error. We used the modified kappa as an index that adjusts for chance agreement. This index provides information about the degree of agreement beyond chance. Questions concerning whether the use of other approaches, such as multirater kappa, would result in more correct agreement cannot be answered here. However, the agreement indices are only one part of the process of assessing content validity and should not be the only reason for the rejection or modification of items. The comments of the experts helped form judgements about the type of problems that existed for specific items. Thus, CVI together with modified kappa and written comments of the experts added rigour to the validation process of the content of IdA®.

Content validity expresses the representation of current available knowledge in the construct of interest [28]. Together with face validity, content validity represents the minimum quality requirement for an instrument. However, despite its limited value within the validity hierarchy [50, 54], content validity is an important quality indicator of an instrument’s validity and provides insight into its feasibility and practicability [32, 51, 54]. In this study, the development process of the instrument and the evaluation of its content validity supported the validity of the IdA® and provided a strong foundation from which to begin further examination of its validity and reliability.

To assess the content validity of the IdA®, it was important to assess different attributes of both the questions and the answer options. The experts perceived the majority of the questions as relevant; however, more weaknesses with the answer options were identified. The experts’ judgements together with the comments provided detailed information about the strengths and weaknesses of each item and led to well-reasoned modifications.

The decision to assess more than only the relevance of an item was shown to be very helpful in the development process. The experts distinguished between the relevance of the content (particular trigger or cause of behaviour) and the clarity of the wording. Most of the questions were considered relevant, but the phrasing was criticized. This differentiation facilitated the modifications and improved their comprehensibility. The responses received more negative criticisms than the questions. For example, the relevance of the item asking about pain was confirmed, and the answer options were meaningful. However, in the experts´ opinion, some important aspects relevant to pain assessment were not considered. Additionally, in some cases, the written comments indicated that there were problems with comprehension – the intended content was somewhat misunderstood. This information led to a more precise phrasing of the item.

The importance of the competency of the expert is crucial. Very different factors can be used to confirm a person as an expert, and there are no rules on how to define an expert. Usually, an expert is defined as a person who represents the content of interest in science or practice. In the context of assessment instruments, knowledge about the methodology of assessment is very helpful, and input from stakeholders from additional application fields can be useful [52]. In the first expert panel of this study, we focused on experiences and knowledge in dementia care, in nursing care in the context of nursing homes and in using or developing assessment instruments. The strong focus on nursing was requested, as we expected that those experts would help to develop an instrument with content that is meaningful, understandable and practicable for nurses and the nursing field. We could not clarify the extent to which the narrow focus on nursing field influenced the rating of relevance of the IdA® items and dimensions in favour of more physical and ADL topic than psycho-social aspects. We assume a small risk of bias since most the experts had additional qualifications and professions beyond nursing.

The disadvantage of the development process was the time and effort needed to assess the substantial number of items, as the process could not be replicated after the modifications suggested by the first expert panel had been made. Because the second review process occurred through a workshop with health care workers, individual ratings of the assessment items were not feasible. Thus, the effect of modifying the IdA® items on the CVI obtained in the first assessment could not be quantified, which is a limitation of the study.

The health care practitioner’s comments and discussion contributed significantly to the item clarity and adaptability to nursing practice. In this second modification, aspects such as problems obtaining the information needed to answer the questions were strongly discussed. For example, the domain “Personality & lifestyle before the onset of dementia” contained questions that could not be answered without knowing the person well and for a long period of time. The nursing home care staff was often dependent on information provided by family members or significant others, if present. In some cases, these informants were not available, and the information needed to answer the IdA® questions could not be obtained. This is not a problem specific to IdA® but is a general problem of assessments of complex and multifaceted information. The creation of a professional environment in which the observation, collection, communication, reflection and appraisal of essential information is an obvious part of the nursing process can overcome this challenge. IdA® should not be the start of this process, but it supports this process for a specific topic (dementia or challenging behaviour) and in a specific phase of this process (mainly complementary information collection, communication, reflection and appraisal of information). Although the CVI could not be calculated in this second expert round, the advantage of this round was the clear practice perspective, the simplification of the wording and the willingness of the carers to collect, provide and share information.

The majority of the IdA® users reported that the tools were well structured, clear, complete and comprehensive. The content appeared to be ambiguous. For almost 60% of the users, the relevance of the IdA® information for dementia care seemed to be unclear. This finding was consistent with the evaluation that the IdA® required comprehensive knowledge about dementia and challenging behaviour. This feedback was important for the future implementation of the IdA®. The findings highlighted the insufficient knowledge of the staff working in nursing homes about the triggers of challenging behaviour. Particular emphasis should be placed on expressing the relationships between the IdA® variables and their relevance to the development of challenging behaviour. As previously found, the qualification of the staff caring for people with dementia is known to influence the effectiveness of dementia-specific approaches [55]. However, how the staff experiences challenging behaviour and how they place this behaviour in context also influence their understanding of causes and triggers. In their study, Dupuis et al. [56] showed a complex process of interpretation through which nurses use several filters. The lens of pathology and the influence of cognition on the construction of meaning of challenging behaviour are very dominant. Thus, it is important that nursing home staff are supported in developing more person-centred attitudes towards persons showing challenging behaviour. The combination of evidence-based knowledge about triggers, supported by IdA®, and the reinforcement of dementia-friendly attitudes are a promising basis for the best care for those people.

The IdA® was time consuming because it covered a wide range of information on potential causes, and these causes are multifactorial. The care staff needed time to answer the questions, gather any missing information, discuss potential hypotheses on the causes of behaviours, and then plan interventions. This process required a substantial amount of time at a point when nursing care is characterized by a lack of time [57]. This conflict could not be resolved, as the broadness of the themes and items was necessary to trace the triggers and causes of behaviour [58]. Changes in routine processes, changes in the prioritization of tasks and facilitation of a professional self-understanding that incorporates analytic thinking processes into nursing tasks could help address this conflict. An important form of preparation for the implementation of IdA® is to analyse current practices of nursing assessment and documentation. Most of the information needed for IdA® already exists. Nurses should ensure that the existing information is valid and can be easily linked with the questions on IdA®. This process has the potential to save time. We also expect that the frequent use of IdA® will result in better management of the instrument and better prioritization of the parts of IdA® relevant to a particular person with dementia. We analysed this aspect in a recently completed intervention study [21, 59].

The IdA® should be viewed as a tool that offers support and assistance for carers in understanding the challenging behaviour exhibited by persons with dementia. This tool provides guidelines for navigating a complex network of widely varying potential causes and triggers of these behaviours. It is important to mention that the IdA® does not provide exact solutions for the problem but rather helps generate a hypothesis (one or more) about the causes of challenging behaviour. This type of support is important in Germany, as nurses act practically independently in nursing homes. In Germany, a general practitioner (GP) spends an average of less than 2 h peer week in nursing homes, caring for 20 residents during this time period [60]. Because of the limited time and the high number of residents with challenging behaviours, contact with the GP is mainly restricted to acute or severe situations and focuses mainly on medical and pharmacological questions [61]. Bartholomeyczik et al. [15] found that nurses are only moderately satisfied with GP consultation times for residents with dementia, and only 13% of GPs wanted to participate in case conferences very often or often. In that study, only one GP participated and contributed to one case conference. Consultations with psychologists are unusual. This situation differs from the situation in the Netherlands, for example, where an interdisciplinary team is responsible for managing challenging behaviour [16], and in the UK, where the psychologist has a key role in this context [17].

The IdA® asks for a substantial amount of information about the person with dementia, and therefore different sources of information are important to obtaining a comprehensive understanding of the person, e.g., reports from different shift workers and carers, family and friends, therapists and persons with dementia themselves. Hypotheses regarding the causes or triggers of a behaviour are not formed as a result of a deduction of “hard facts” but rather a construction of plausible and preliminary presumptions. The effects of the action deduced from the hypotheses verify the accuracy of these estimates. Discussing the pros and cons of a hypothesis is therefore necessary to properly apply the IdA®. Favourable opportunities for these discussions include team meetings or case conferences [22]. The initial experiences in which the IdA® was used during case conferences in nursing homes showed that the care staff benefitted from the exchange of individual views and experiences. They were able to realize that residents showed different behaviours with different staff members, and this realization facilitated the recognition of helpful and unhelpful approaches to the interactions between residents and staff. Self-reflection was supported. Residents´ behaviour was no longer considered a personal attack but could be perceived from a broader context. The IdA® helped to more precisely describe the behaviour of residents with dementia as well as the circumstances of behaviour and supported communication with physicians. Through the IdA®, gaps in knowledge regarding the residents became apparent [55]. Thus, the IdA® has the potential to support carers’ self-efficacy and thus reduce their burden in managing challenging behaviour [18]. The effectiveness of using the IdA® within case conferences is currently being examined in a cluster randomized controlled trial in Germany [21, 59].

The IdA® is not a typical measurement instrument that can be easily tested using all routine psychometric properties (e.g., inter-rater reliability, criterion validity, structure analysis). The outcome of the IdA® is not a risk status or physical status that can be prospectively validated or compared to a gold standard. The results of the IdA® as a whole include the formulation of hypotheses about the relationship between the information obtained from the IdA® and challenging behaviour. The “truth” can be verified only if the derived conclusions (hypothesis) lead to the expected effects. The rejection or confirmation of the hypothesis also depends on the choice of the right intervention (e.g., ways to communicate or changes in medication or enjoyable activities) and on the correct execution of the intervention. This requires knowledge about effective interventions and skills and technique in delivering the interventions.

In addition, the effects in each person and situation could differ: challenging behaviour could be reduced or stabilized, the right interactions could be implemented when the behaviour occurs, the behaviour could be integrated into daily living in the nursing home unit, the burden could be relieved for persons in the patient’s environment, the quality of life of the person showing challenging behaviour could be maintained or the challenging behaviour could be better understood or re-interpreted. When the conclusion does not lead to the desired effects, the reasons can vary from incorrect hypotheses, incorrect interventions, or both; incorrect, missing or unimportant items on the IdA®; or a lack of availability of significant information. The evaluation of reliability also has some challenges. For IdA®, as a team instrument, the evaluation of inter-rater reliability is less meaningful, except for some specific aspects such as pain, cognitive status or some behavioural characteristics. A form of intergroup reliability could be an alternative, but the time needed for conducting repeated team measurements should be taken into account. Evaluation of test-retest for the formulated hypotheses could make more sense and help to overcome the previously discussed problems with the interpretation of validity.

Limitations

The study has several limitations. Due to the many items included in the IdA®, it was not possible to conduct additional expert rounds, as suggested by Lynn [26], or to calculate CVIs after each evaluation round. Thus, whether the modifications led to improvements in the content validity remains unknown. Content validity is the weakest form of validity, and therefore this study is a first step in establishing the validity of the IdA®.

We used the CVI and modified kappa for the assessment of agreement, the merit of which has been criticized. Here, we do not know whether e.g. the multirater kappa statistic would provide more valid agreement indices. Therefore, different indices were used to decide the item modifications.

We included in the development process mainly professionals from the nursing field, since our priority was an instrument that covers information relevant to nursing care and can be managed by nurses. The involvement of other disciplines might influence the content and the content validity evaluation of IdA®.

The size of the care staff sample was small, and thus the interpretation of the results was limited.

Additionally, the delivery time was very short (6 weeks); this brief duration might have influenced the correct application of the IdA® and led to over- or underestimations of the IdA® quality. Conclusions regarding the experiences with the IdA® over time cannot be made.

Conclusion

The IdA® is a nursing assessment tool that focuses on the triggers and causes of challenging behaviour. The comprehensive literature review, strong theoretical foundation, and the two evaluations representing specific content expertise as well as practical perspectives contribute to the content validity of the IdA®. The IdA® is recommended to be used as a team instrument, as the information collected with the IdA® requires observations from different caregivers, situations and time periods to create a comprehensive understanding of the persons exhibiting challenging behaviour.

Declarations

Funding

The study was co-funded by DAN Produkte GmbH, a company developing paper- and software-based documentation for health care. The funders had no role in the design or analysis of the study nor in the decision to submit this publication.

Availability of data and materials

No additional data are available.

Authors’ contributions

MH developed the study design and the IdA®, collected and analysed the data, and wrote the manuscript. The paper was part of MH’s doctoral thesis (Halek, M. Entwicklung und Testung eines strukturierten Assessmentbogens zur Erfassung der Auslöser für herausforderndes Verhalten von Menschen mit Demenz in der stationären Altenhilfe. (Doctoral Thesis), Witten/Herdecke University, Witten. https://portal.dnb.de/opac.htm?method=simpleSearch&cqlMode=true&query=idn%3D1005029563. 2010). DH was involved in drafting the manuscript and providing critical revisions. SB supervised the conception, analysis and interpretation of the data and contributed to the development of the IdA®; additionally, she revised the manuscript critically. MH and DH supervised the translation process of IdA® into English (IdA-E®) (Additional files 1 and 2). All authors read and approved the final manuscript.

Ethics approval and consent to participate

The study obtained an ethics committee approval from the German Society of Nursing Association Science (15.10.2007). The caregivers participating in the study were provided with written and oral information on the study. Data protection issues and voluntary participation were ensured.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
German Center for Neurodegenerative Diseases (DZNE)
(2)
School of Nursing Science, Witten/Herdecke University

References

  1. Moniz Cook ED, Swift K, James I, Malouf R, De Vugt M, Verhey F. Functional analysis-based interventions for challenging behaviour in dementia. Cochrane Database Syst Rev. 2012;2012:CD006929.Google Scholar
  2. Wetzels R, Zuidema S, Jansen I, Verhey F, Koopmans R. Course of neuropsychiatric symptoms in residents with dementia in long-term care institutions: a systematic review. Int Psychogeriatr. 2010;22:1040–53.View ArticlePubMedGoogle Scholar
  3. Selbaek G, Engedal K, Bergh S. The prevalence and course of neuropsychiatric symptoms in nursing home patients with dementia: a systematic review. J Am Med Dir Assoc. 2013;14:161–9.View ArticlePubMedGoogle Scholar
  4. Schmidt KH, Diestel S. Job demands and personal resources in their relations to indicators of job strain among nurses for older people. J Adv Nurs. 2013;69:2185–95.View ArticlePubMedGoogle Scholar
  5. Edvardsson D, Sandman PO, Nay R, Karlsson S. Associations between the working characteristics of nursing staff and the prevalence of behavioral symptoms in people with dementia in residential care. Int Psychogeriatr. 2008;20:764–76.View ArticlePubMedGoogle Scholar
  6. Vasse E, Vernooij-Dassen M. Psychosocial interventions in dementia care. In: McNamara P, editor. Dementia - treatments and developments. Santa Barbara: Praeger; 2011. p. 151–71.Google Scholar
  7. Cohen-Mansfield J, Golander H, Ben-Israel J, Garfinkel D. The meanings of delusions in dementia: a preliminary study. Psychiatry Res. 2011;189:97–104.View ArticlePubMedGoogle Scholar
  8. Bedard A, Landreville P, Voyer P, Verreault R, Vezina J. Reducing verbal agitation in people with dementia: evaluation of an intervention based on the satisfaction of basic needs. Aging Ment Health. 2011;15:855–65.View ArticlePubMedGoogle Scholar
  9. Koopmans RT, Zuidema SU, Leontjevas R, Gerritsen DL. Comprehensive assessment of depression and behavioral problems in long-term care. Int Psychogeriatr. 2010;22:1054–62.View ArticlePubMedGoogle Scholar
  10. Bartholomeyczik S. Some critical remarks on standardised assessment instruments in nursing. Pflege. 2007;20:211–7.View ArticlePubMedGoogle Scholar
  11. Reisberg B, Borenstein J, Salob SP, Ferris SH, Franssen E, Georgotas A. Behavioral symptoms in Alzheimer's disease: phenomenology and treatment. J Clin Psychiatry. 1987;48(Suppl):9–15.PubMedGoogle Scholar
  12. Wood S, Cummings JL, Hsu MA, Barclay T, Wheatley MV, Yarema KT, et al. The use of the neuropsychiatric inventory in nursing home residents. Characterization and measurement. Am J Geriatr Psychiatry. 2000;8:75–83.View ArticlePubMedGoogle Scholar
  13. Cohen-Mansfield J, Marx MS, Rosenthal AS. A description of agitation in a nursing home. J Gerontol. 1989;44:M77–84.View ArticlePubMedGoogle Scholar
  14. Moniz-Cook E, Woods R, Gardiner E, Silver M, Agar S. The challenging behaviour scale (CBS): development of a scale for staff caring for older people in residential and nursing homes. Br J Clin Psychol. 2001;40:309–22.View ArticlePubMedGoogle Scholar
  15. Bartholomeyczik S, Holle D, Halek M. Herausforderndes Verhalten bei Menschen mit Demenz verstehen - Die Verbesserung der Versorgung Demenzkranker durch Qualitätsinstrumente. Weinheim und Basel: Beltz Juventa Verlag; 2013.Google Scholar
  16. Zwijsen SA, Smalbrugge M, Eefsting JA, Twisk JW, Gerritsen DL, Pot AM, et al. Coming to grips with challenging behavior: a cluster randomized controlled trial on the effects of a multidisciplinary care program for challenging behavior in dementia. J Am Med Dir Assoc. 2014;15:531.e1–10.View ArticleGoogle Scholar
  17. Jackman LJ, Wood-Mitchell A, James IA. Micro-skills of group formulations in care settings. Dementia (London). 2014;13:23–32.View ArticleGoogle Scholar
  18. Holle D, Halek M, Mayer H, Bartholomeyczik S. The influence of understanding diagnostics on perceived stress of nurses caring for nursing home residents with dementia. Pflege. 2011;24:303–16.View ArticlePubMedGoogle Scholar
  19. Lichtwarck B, Selbaek G, Kirkevold Ø, Rokstad AMM, Benth JŠ, Myhre J, et al. TIME–ttargeted interdisciplinary model for evaluation and treatment of neuropsychiatric symptoms: protocol for an effectiveness-implementation cluster randomized hybrid trial. BMC Psychiatry. 2016;16:233.View ArticlePubMedPubMed CentralGoogle Scholar
  20. Turner J, Snowdon J. An innovative approach to behavioral assessment and intervention in residential care: a service evaluation. Clin Gerontol. 2009;32:260–75.View ArticleGoogle Scholar
  21. Holle D, Roes M, Buscher I, Reuther S, Muller R, Halek M. Process evaluation of the implementation of dementia-specific case conferences in nursing homes (FallDem): study protocol for a randomized controlled trial. Trials. 2014;15:485.View ArticlePubMedPubMed CentralGoogle Scholar
  22. Reuther S, Dichter MN, Buscher I, Vollmar HC, Holle D, Bartholomeyczik S, et al. Case conferences as interventions dealing with the challenging behavior of people with dementia in nursing homes: a systematic review. Int Psychogeriatr. 2012;24:1891–903.View ArticlePubMedGoogle Scholar
  23. Gitlin LN, Winter L, Dennis MP, Hauck WW. A non-pharmacological intervention to manage behavioral and psychological symptoms of dementia and reduce caregiver distress: design and methods of project ACT3. Clin Interv In Aging. 2007;2:695.View ArticleGoogle Scholar
  24. Gitlin LN, Winter L, Dennis MP, Hodgson N, Hauck WW. Targeting and managing behavioral symptoms in individuals with dementia: a randomized trial of a nonpharmacological intervention. J Am Geriatr Soc. 2010;58:1465–74.View ArticlePubMedPubMed CentralGoogle Scholar
  25. Desai AK, Desai FG. Management of behavioral and psychological symptoms of dementia. Curr Geriatr Rep. 2014;3:259–72.View ArticleGoogle Scholar
  26. Lynn MR. Determination and quantification of content validity. Nurs Res. 1986;35:382–5.View ArticlePubMedGoogle Scholar
  27. DeVet H, Terwee C, Mokkink L, Knol D. Measurement in medicine: a practical guide. Cambridge: Cambridge University Press; 2011.View ArticleGoogle Scholar
  28. Streiner DL, Norman GR, Cairney J. Health measurement scales. A practical guide to their development and use. Oxford: Oxford University Press; 2014.Google Scholar
  29. Halek M, Bartholomeyczik S. Description of the behaviour of wandering in people with dementia living in nursing homes - a review of the literature. Scand J Caring Sci. 2012;26:404–13.View ArticlePubMedGoogle Scholar
  30. Waltz C, Stickland OL, Lenz E, Soeken KL. Validity of measures. In: Waltz C, Stickland O, Lenz E, editors. Measurement in nursing and health research. New York: Springer Publishing Company; 2005. p. 154–89.Google Scholar
  31. Polit DF, Beck CT. The content validity index: are you sure you know what's being reported? Critique and recommendations Res Nurs Health. 2006;29:489–97.View ArticlePubMedGoogle Scholar
  32. Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations Res Nurs Health. 2007;30:459–67.View ArticlePubMedGoogle Scholar
  33. Wynd CA, Schmidt B, Schaefer MA. Two quantitative approaches for estimating content validity. West J Nurs Res. 2003;25:508–18.View ArticlePubMedGoogle Scholar
  34. Fleiss J. Statistical methods for rates and proportions. New York: John Wiley & Sons; 1981.Google Scholar
  35. Cicchetti D, Sparrow S. Developing criteria for establishing interrater reliability of specific items: application to assessment of adaptive behaviour. Am J Ment Defic. 1981;86:127–37.PubMedGoogle Scholar
  36. Kunik M, Martinez M, Snow LA, Beck CK, Cody M, Rapp CG, et al. Determinants of behavioral symptoms in dementia patients. Clin Gerontol. 2003;26:83–9.View ArticleGoogle Scholar
  37. Smith M, Gerdner LA, Hall GR, Buckwalter KC. History, development, and future of the progressively lowered stress threshold: a conceptual model for dementia care. J Am Geriatr Soc. 2004;52:1755–60.View ArticlePubMedGoogle Scholar
  38. Cohen-Mansfield J. Nonpharmacological management of behavioral problems in persons with dementia: the TREA model. Alzheimer’s Quarterly. 2000;1:22–34.Google Scholar
  39. Algase DL, Beck CK, Kolanowski AM, Whall AL, Berent S, Richards KC, et al. Need-driven dementia-compromised behavior: an alternative view of disruptive behavior. Am J Alzheimers Dis Other Demen. 1996;11:10–9.View ArticleGoogle Scholar
  40. Kovach CR, Noonan PE, Schlidt AM, Reynolds S, Wells T. The serial trial intervention: an innovative approach to meeting needs of individuals with dementia. J Gerontol Nurs. 2006;32:18–25.PubMedGoogle Scholar
  41. Volicer L, Hurley AC. Management of behavioral symptoms in progressive degenerative dementias. J Gerontol A Biol Sci Med Sci. 2003;58:M837–45.View ArticlePubMedGoogle Scholar
  42. Finnema E, Dröes R, Ribbe M, Tilburg van W. A review of psychosocial models in psychogeriatrics; implications for care and research. Alzheimer Dis Assoc Disord. 2000;14:68–80.View ArticlePubMedGoogle Scholar
  43. Kolanowski AM, Litaker M, Buettner L. Efficacy of theory-based activities for behavioral symptoms of dementia. Nurs Res. 2005;54:219–28.View ArticlePubMedGoogle Scholar
  44. Cohen-Mansfield J, Agitation in the elderly. Definitional and theoretical conceptualizations. In: Hay D, Klein D, Hay L, Grossberg G, Kennedy J, editors. Agitations in patients with dementia: a practical guide to diagnosis and management. Washington: American Psychiatric Publishing; 2003. p. 1–21.Google Scholar
  45. Cohen-Mansfield J, Billig N. Agitated behaviors in the elderly. A conceptual review. J Am Geriatr Soc. 1986;34:711–21.View ArticlePubMedGoogle Scholar
  46. Bartholomeyczik S, Halek M, Sowinski C, Besselmann K, Dürrmann P, Haupt M. Rahmenempfehlungen zum Umgang mit herausforderndem Verhalten bei Menschen mit Demenz in der stationären Altenhilfe. Berlin: Bundesministerium für Gesundheit; 2007.Google Scholar
  47. Palm R, Kohler K, Bartholomeyczik S, Holle B. Assessing the application of non-pharmacological interventions for people with dementia in German nursing homes: feasibility and content validity of the dementia care questionnaire (DemCare-Q). BMC Res Notes. 2014;7:950.View ArticlePubMedPubMed CentralGoogle Scholar
  48. Dikken J, Hoogerduijn JG, Kruitwagen C, Schuurmans MJ. Content validity and psychometric characteristics of the "knowledge about older patients quiz" for nurses using item response theory. J Am Geriatr Soc. 2016;64:2378–83.View ArticlePubMedGoogle Scholar
  49. Hawkins RJ, Swanson B, Kremer MJ, Fogg L. Content validity testing of questions for a patient satisfaction with general anesthesia care instrument. J Perianesth Nurs. 2014;29:28–35.View ArticlePubMedGoogle Scholar
  50. Beckstead JW. Content validity is naught. Int J Nurs Stud. 2009;46:1274–83.View ArticlePubMedGoogle Scholar
  51. Haynes S, Richard D, Kubany E. Content validity in psychological assessment: a functional approach to concepts and methods. Psychol Assess. 1995;7:238–47.View ArticleGoogle Scholar
  52. McGartland R, Berg-Weger M, Tebb S, Lee S, Rauch S. Objectifying content validity: conducting a content validity study in social work research. Soc Work Res. 2003;27:94–104.View ArticleGoogle Scholar
  53. Squires A. A valid step in the process: a commentary on Beckstead (2009). Int J Nurs Stud. 2009;46:1284–5.View ArticlePubMedGoogle Scholar
  54. DeVon HA, Block ME, Moyle-Wright P, Ernst DM, Hayden SJ, Lazzara DJ, et al. A psychometric toolbox for testing validity and reliability. J Nurs Scholarsh. 2007;39:155–64.View ArticlePubMedGoogle Scholar
  55. Holle D, Kruger C, Halek M, Sirsch E, Bartholomeyczik S. Experiences of nursing staff using dementia-specific case conferences in nursing homes. Am J Alzheimers Dis Other Demen. 2015;30:228–37.View ArticlePubMedGoogle Scholar
  56. Dupuis SL, Wiersma E, Loiselle L. Pathologizing behavior: meanings of behaviors in dementia care. J Aging Stud. 2012;26:162–73.View ArticleGoogle Scholar
  57. Kolanowski A, Fick D, Frazer C, Penrod J. It's about time: use of nonpharmacological interventions in the nursing home. J Nurs Scholarsh. 2010;42:214–22.View ArticlePubMedGoogle Scholar
  58. Brechin D, Murphy G, James IA, Codner J. Alternatives to antipsychotic medications: psychological approaches in managing psychological and behavioural distress in people with dementia. 2013. http://www.bps.org.uk/content/alternatives-antipsychotic-medication-psychological-approaches-managing-psychological-and-be. Accessed 11 Nov 2013.Google Scholar
  59. Reuther S, Holle D, Buscher I, Dortmann O, Muller R, Bartholomeyczik S, et al. Effect evaluation of two types of dementia-specific case conferences in German nursing homes (FallDem) using a stepped-wedge design: study protocol for a randomized controlled trial. Trials. 2014;15:319.View ArticlePubMedPubMed CentralGoogle Scholar
  60. Schneekloth U, Wahl HW. Möglichkeiten und Grenzen selbständiger Lebensführung in privaten Haushalten (MUG III). Repräsentativbefunde und Vertiefungsstudien zu häuslichen Pflegearrangements, Demenz und professionellen Versorgungsangeboten. MUG III. 2005. https://www.bmfsfj.de/bmfsfj/selbststaendigkeit-im-alter%2D-mug-iii-%2D-%2Donline-publikation/79358?view=DEFAULT. Accessed 14 Nov 2005.
  61. Mueller CA, Tetzlaff B, Theile G, Fleischmann N, Cavazzini C, Geister C, et al. Interprofessional collaboration and communication in nursing homes: a qualitative exploration of problems in medical care for nursing home residents - study protocol. J Adv Nurs. 2015;71:451–7.View ArticlePubMedGoogle Scholar

Copyright

© The Author(s). 2017

Advertisement