Development and validation of the Diabetes Numeracy Test (DNT)
BMC Health Services Research volume 8, Article number: 96 (2008)
Low literacy and numeracy skills are common. Adequate numeracy skills are crucial in the management of diabetes. Diabetes patients use numeracy skills to interpret glucose meters, administer medications, follow dietary guidelines and other tasks. Existing literacy scales may not be adequate to assess numeracy skills. This paper describes the development and psychometric properties of the Diabetes Numeracy Test (DNT), the first scale to specifically measure numeracy skills used in diabetes.
The items of the DNT were developed by an expert panel and refined using cognitive response interviews with potential respondents. The final version of the DNT (43 items) and other relevant measures were administered to a convenience sample of 398 patients with diabetes. Internal reliability was determined by the Kuder-Richardson coefficient (KR-20). An a priori hypothetical model was developed to determine construct validity. A shortened 15-item version, the DNT15, was created through split sample analysis.
The DNT had excellent internal reliability (KR-20 = 0.95). The DNT was significantly correlated (p < 0.05) with education, income, literacy and math skills, and diabetes knowledge, supporting excellent construct validity. The mean score on the DNT was 61% and took an average of 33 minutes to complete. The DNT15 also had good internal reliability (KR-20 = 0.90 and 0.89). In split sample analysis, correlations of the DNT-15 with the full DNT in both sub-samples was high (rho = 0.96 and 0.97, respectively).
The DNT is a reliable and valid measure of diabetes related numeracy skills. An equally adequate but more time-efficient version of the DNT, the DNT15, can be used for research and clinical purposes to evaluate diabetes related numeracy.
In 1992, the National Adult Literacy Survey (NALS) estimated that over 90 million adult Americans have inadequate literacy and numeracy skills . More recently, the National Assessment of Adult Literacy (NAAL) showed that 36% of American adults had basic or below basic health-related literacy and quantitative skills . In that survey, 22% of adults demonstrated below basic quantitative skills and could only perform simple, concrete, single-step math operations, such as addition. Another 33% demonstrated basic quantitative skills, solving single-step math problems where the operation was either given or easily inferred. Patients with low literacy can have difficulty following medical instructions, understanding health information and performing self-management tasks . Low literacy is associated with worse disease knowledge and poorer clinical outcomes in patients with diabetes [4, 5]. The broad construct of literacy consists of several components: print literacy, oral literacy and quantitative skills . Many of the recent studies evaluating the relationship between literacy and health have focused on print literacy with little attention given to quantitative or numeracy related skills. While common for patients with low literacy to have inadequate quantitative skills, patients with adequate literacy may still have difficulty with numeracy.
Numeracy is an important component of literacy that can be simply defined as the ability to understand and use numbers in everyday life [6–10]. Numeracy does not refer solely to basic mathematical skills (arithmetic, calculations, fractions, algebra and geometry) but also to one's ability to understand time, currency, measurement, graphic representations, logic, hierarchies and probability. To apply numeracy to a specific problem or task, one must know which numerical skill(s) to apply, deduce and interpret the result(s) and then appropriately apply the result(s) to the situation [7, 11]. For a patient with diabetes, numeracy is needed to interpret blood glucose meter data, properly administer medications and follow nutritional recommendations. For example, a patient with diabetes may need numeracy to calculate their carbohydrate intake and adjust their insulin based on carbohydrates and/or current blood glucose level. Poor numeracy skills in patients with diabetes could lead to suboptimal glycemic control, increased hypoglycemic episodes or widely varying glucose values.
Scales have been developed to measure literacy in the health care setting [12–14]. One of these scales, the TOFHLA, did include 17 items to measure general quantitative skills . These items primarily focused on reading prescriptions and other materials that tested simple mathematical skills such as understanding dates and timing of medication dosage. These items were highly dependent on reading ability and, not surprisingly, they are highly correlated to reading ability. These numeracy items are not included in the shortened form, the S-TOFHLA . The TOFHLA was not designed to measure the range of numeracy skills needed in diabetes management and does not give a provider clinically useful information in the care of a patient with diabetes .
There are no current scales designed specifically to assess diabetes related numeracy skills required to perform daily diabetes self-management. Numeracy skills may be particularly critical in the management of diabetes and the currently available health literacy scales may not adequately identify patients with low numeracy. In order to better understand numeracy in diabetes, the Diabetes Numeracy Test (DNT) was developed. The DNT was specifically designed to evaluate the wide range of numeracy skills used by patients with diabetes.
Research design and methods
The first phase of development included item generation by a group of experts in diabetes, literacy and numeracy. This group included diabetologists, certified diabetes educators, primary care providers, registered dietitians, behavioral researchers in diabetes, and literacy and numeracy experts. The initial item pool contained 70 items that addressed all mathematical skills required in the daily management of diabetes. Those 70 items covered five diabetes self-management areas (nutrition, exercise, glucose monitoring, oral medication and insulin use) and the general mathematical skills required in the management of diabetes including addition, subtraction, multiplication, division, fractions/decimals, numerical hierarchy, inference skills and multi-step calculations. The 70 items were administered to 40 individuals without diabetes to assess understandability. To eliminate redundancy, the expert panel reduced the measure to 45 items that represented the five self-management areas. The panel agreed that the 45 item scale had adequate content validity to address the range of numeracy skills required in the management of diabetes.
The objective of the second phase of development of the DNT was to address the clarity of items for patients with diabetes. Ten cognitive response interviews were conducted with patients with diabetes to assess each item. Interviewees were asked specific questions about each item to assess the understandability of the wording. If an item was unclear, the interviewee was told the purpose of the item and then encouraged to suggest an alternate format or wording. In response to the interviews, the scale was reformatted and slightly reduced to the final 43 items .
The DNT was designed to assess nutrition, exercise, glucose monitoring, oral medication and insulin skills that patients may encounter during daily diabetes self-management. There are nine nutrition items focusing on nutrition label interpretation and carbohydrate counting. Four exercise items assess carbohydrate intake and insulin adjustment for exercise time. Blood glucose monitoring skills are evaluated by four items about number hierarchy, glycated hemoglobin and calculating supplies needed. Five items assess oral medication use, refill patterns and dates, and oral titration schemes. The remaining 21 items determine numeracy associated with insulin use including interpretation of syringes, correction or sliding scale insulin use, insulin adjustment for carbohydrate intake and titration instructions. See Table 1 for a description of the diabetes care domains and numeracy skills assessed in the DNT by item number. Items are scored as binary outcomes – correct or incorrect – and no partial credit is given. There is no time limit for the administration of the scale. Many patients with diabetes use calculators; therefore, participants were allowed to use calculators during the administration of the DNT to emulate real-life circumstances. DNT scores are reported as percent correct (with a possible range of 0% to 100%).
Phase 3 of development sought to assess the reliability and provide support for the construct validity of the DNT. Reliability was evaluated through internal consistency testing with the Kuder-Richardson 20 formula. There is no gold standard for numeracy in diabetes; hence an a priori model of correlations was determined by the expert panel to assess construct validity (see Figure 1). We hypothesized that increased education, income, literacy, math skills, and diabetes knowledge would all be moderately associated (rho of 0.3 – 0.6) with improved DNT scores. We also hypothesized that insulin use would be moderately associated with higher DNT since many of the DNT questions focused on insulin administration.
In the final phase of the study, the scale was shortened to a more clinically useful 15 items and then verified through random split sample analysis. To perform the split sample analysis, the sample data were randomly split into two smaller sub-samples. Sub-sample 1 was used for the development of the shortened scale and sub-sample 2 was used for confirmation of the results. In order to determine the items to include in the DNT15, principal components analysis was performed on sub-sample 1. Items with > 0.6 loading on the primary component in sub-sample 1 were included. In addition, those items with > 80% mean correct score were discarded. Three items with high face validity, as determined by practicing diabetologists and diabetes educators, were added back in to bring the total number of items to 15. Reliability was evaluated by internal consistency (Kuder-Richardson 20) and validity was assessed through correlation testing using Spearman's correlations between the DNT15 and the full DNT and comparing the DNT15 to the a priori construct validity model for both sub-samples.
Participants in this study were recruited from four sites: general medicine clinics at two academic health centers, a diabetes clinic at an academic health center and an endocrinology clinic at a VA health center. Participants were paid $20 to participate. A convenience sample of 398 participants was recruited at clinic visits. Inclusion criteria included diagnosis of type 1 or 2 diabetes, age 18 to 80 years and English speaking. Potential participants were excluded if their corrected visual acuity was > 20/50 using a Rosenbaum Pocket Vision Screener or if they had a diagnosis of significant dementia, psychosis or blindness.
Trained research assistants collected patient characteristics, blood glucose monitoring frequency data from glucose meters, and administered a series of surveys to the patients. Measures included a validated literacy measure (REALM), a validated general numeracy scale (WRAT-3R) and a validated diabetes knowledge test (DKT) [12, 17, 18].
Internal consistency was determined by the Kuder-Richardson 20 coefficient of reliability. Spearman's rank order correlations were performed for the a priori model testing. Principal component analysis with oblique rotation was performed to determine the presence of multiple factors and item loadings on the factors. For development of the DNT15, Spearman's correlations were used in the random split sample analysis. Stata 9.2 (StataCorp, College Station, TX) was used for all statistical analysis.
All work was approved by the institutional review boards of Vanderbilt University and the University of North Carolina.
A group of 398 patients with diabetes completed the DNT and the construct validation scales including the WRAT, REALM and DKT. Patient characteristics are presented in Table 2. The average age was 54.2 years and 51% of the participants were female. Eighty-six percent had type 2 diabetes. The majority (83%) of participants had completed high school but only 69% had greater than 9th grade literacy by the REALM and only 31% had greater than 9th grade numeracy by the WRAT. The 43-item DNT took an average (± SD) of 33 ± 13 minutes to complete. The average score (± SD) on the DNT was 61% ± 25.
Problem areas for participants included titration schemas, food label interpretation, insulin adjustment instructions, dates for refills and items that required multi-step math (e.g., calculating insulin dosage based on carbohydrate intake and glucose level). Figure 2 illustrates the importance of how the framing of diabetes education can impact patient performance. Two commonly used methods for sliding scale insulin adjustment instructions are displayed. Question 1, with a simple table to interpret, was answered correctly by 85% of participants in this study. However, question 2, which required patients to interpret a word problem and apply multiple numerical steps to determine their insulin dosage, was only answered correctly by 37% of the participants.
The 43-item version of the DNT is highly reliable as determined by internal consistency (KR-20 = 0.95). Construct validity was examined through the a priori construct model and expected correlations as determined by the expert panel (see Table 3). The DNT was moderately correlated with education, income and literacy (REALM) and strongly correlated with numeracy (WRAT-3R) and diabetes knowledge (DKT). One portion of the model did not correlate as expected. The a priori model predicted a moderate correlation with insulin use; however, minimal correlation with insulin use was observed (See Table 3). The DNT was also significantly correlated with frequency of blood sugar testing (rho = 0.15, p = 0.0025), patient adjustment of insulin for carbohydrate intake (rho = 0.51, p < 0.0001), and insulin adjustment for blood glucose (rho = 0.28, p < 0.0001)). Principal component analysis revealed one primary factor that accounted for 65% of the total test variance.
The DNT15 showed similar internal consistency and validity in both sub-samples as the full DNT. The KR-20 of the DNT15 was 0.90 in the development sample (sample 1) and 0.89 in confirmation sample (sample 2). Correlations with the a priori model were similar to the full DNT in both population sub-samples (see Table 4). The DNT15 was highly correlated with the full DNT (see Table 4). In a separate study, the approximate time of administration is 10–15 minutes.
Psychometric analysis of the 43 item version of the DNT shows that it has good reliability and validity in testing numeracy skills in patients with diabetes. Of the six a priori construct hypotheses, five demonstrated significant correlations in the expected directions. The DNT was correlated with diabetes knowledge, education, socioeconomic status, literacy and general numeracy. Insulin use was not correlated with the DNT scores as initially predicted. This is likely due to the fact that many patients on insulin may be placed on long-acting insulin in which one or two doses may be administered with no adjustment for blood glucose or carbohydrate intake. When complexity of the insulin regimen was taken into account, more complex regimens including adjustment for carbohydrate intake and blood glucose level were significantly associated with the DNT.
The shortened version, the DNT15, also showed good reliability and construct validity. The DNT15 was designed to retain the items that discriminated diabetes related numeracy skills while also keeping the items that would be most useful to a diabetes educator or clinician. The five diabetes self-care areas are retained in the DNT15, including three items on nutrition, one item about exercise, three items regarding blood glucose monitoring, one item on oral medications and seven items about insulin administration. A diabetes educator or clinician may use the DNT15 to help target education or guide therapy.
Patients with low literacy also showed low numeracy skills on both the WRAT and the DNT. However, there were several patients with literacy skills above the ninth grade level who had low numeracy skills. Although our population was, on average, highly educated, the mean score on the DNT was only 61% correct. This suggests that numeracy should be evaluated separately from literacy. Patients with low literacy need special instructions and interventions, and patients with inadequate numeracy skills may also require modified counseling and education to improve health outcomes.
This study has several limitations. The DNT was initially developed without input from patients to determine what numeracy skills patients viewed as important. We used a convenience sample of patients recruited from academic and VA clinics, and it is possible that patients who did not participate in the study may have had different literacy and numeracy skills than those that participated. The DNT has been tested in English speaking participants only. The DNT was, in part, validated against a commonly used literacy assessment tool (the REALM), but most patients in the study scored at the highest level that the REALM assesses. A more refined assessment of reading ability at the higher levels may have been more useful.
The DNT and DNT-15 are primarily research tools for understanding the role of diabetes specific numeracy in the management of diabetes. The clinical utility of these scales will be the subject of future research. The treatment of diabetes requires the application of many literacy and numeracy skills. As the disease progresses, the complexity of the regimen may also progress. Patients need ongoing education to appropriately treat their diabetes. However, little is known about the benefits of targeting education to a patient's level of diabetes specific numeracy to improve health outcomes. Other studies have identified the role of literacy and low-literacy techniques in the improvement of health outcomes in diabetes and congestive heart failure [19, 20]. Patients with low numeracy may benefit from similar interventions that address numeracy, particularly in the setting of diabetes management. The DNT and the DNT15 can provide measurement of diabetes specific numeracy and provide more information about the role of disease specific numeracy in future studies. More studies are needed to further understand the role of numeracy tailored interventions in the management of diabetes.
However, there are also clinical implications that can be learned from this study. We learned that the framing of instructions was very important in predicting patient performance. For example, study participants had a difficult time with the multi-step math required to calculate a correction dosage of insulin when instructions were presented as a sequence of sentences. This item was included to mirror clinical practice regarding how patients are currently instructed to take their insulin. Patients clearly had a much easier time calculating the insulin dosage when the material was presented in an easy to read table. This example provides an important lesson for health care providers and educators in effective communication styles for all clinical care recommendations.
Numeracy, a component of literacy, is important in the daily management of many chronic diseases. Even patients with good literacy skills may have marginal numeracy skills. The full DNT and the shortened version, the DNT15, are reliable and valid scales that may be useful in identifying patients that may have deficits in diabetes related numeracy skills.
This study identified several potential problem areas for patients. These areas included food label interpretation and carbohydrate calculations, understanding of when to refill medications, understanding of insulin measurement and insulin adjustment based on blood glucose and/or carbohydrate intake, and application of medication titration instructions. Framing of the instructions impacted the understandability and may impact health outcomes. Simplification of the instructions greatly improved the participant's ability to answer the questions correctly.
To help determine the clinical utility of these scales, we have designed a new educational intervention for patients with diabetes and low literacy and numeracy skills. This new intervention is based on simple instruction, pictorial representation and color-coding for ease of use, and other accommodations to help patients, particularly those with poor literacy and numeracy skills. This intervention is currently being evaluated in a prospective randomized control trial.
Kirsch I, Jungeblut A, Jenkins L, Kolstad A: Adult Literacy in America: A First Look at the Findings of the National Adult Literacy Survey. NCES 93275. 1993, National Center for Education Statistics, US Department of Education
Kutner M, Greenberg E, Jin Y, Paulsen C: The Health Literacy of America's Adults: Results from the 2003 National Assessment of Adult Literacy. 2006, National Center for Education Statistics
Schillinger D, Grumbach K, Piette J, Wang F, Osmond D, Daher C, Palacios J, Sullivan GD, Bindman AB: Association of Health Literacy With Diabetes Outcomes. JAMA. 2002, 288 (4): 475-482. 10.1001/jama.288.4.475.
Rothman R, Malone R, Bryant B, Horlen C, DeWalt D, Pignone M: The Relationship Between Literacy and Glycemic Control in a Diabetes Disease-Management Program. The Diabetes Educator. 2004, 30 (2): 263-273. 10.1177/014572170403000219.
Rothman RL, Malone R, Bryant B, Wolfe C, Padgett P, DeWalt DA, Weinberger M, Pignone M: The Spoken Knowledge in Low Literacy in Diabetes Scale: A Diabetes Knowledge Scale for Vulnerable Patients. The Diabetes Educator. 2005, 31 (2): 215-224. 10.1177/0145721705275002.
Nielsen-Bohlman L, Institute of Medicine (U.S.), Committee on Health Literacy, NetLibrary Inc: Health literacy a prescription to end confusion. 2004, National Academies Press, Washington, D.C, xix: 345.
Golbeck AL, Ahlers-Schmidt CR, Paschal AM, Dismuke SE: A definition and operational framework for health numeracy. American journal of preventive medicine. 2005, 29 (4): 375-376. 10.1016/j.amepre.2005.06.012.
Rothman RL, Housam R, Weiss H, Davis D, Gregory R, Gebretsadik T, Shintani A, Elasy TA: Patient understanding of food labels: the role of literacy and numeracy. Am J Prev Med. 2006, 31 (5): 391-398. 10.1016/j.amepre.2006.07.025.
Ancker JS, Kaufman D: Rethinking Health Numeracy: A Multidisciplinary Literature Review. J Am Med Inform Assoc. 2007
Rothman R, Montori V, Cherrington A, Pignone M: Perspective: The Role of Numeracy in Health Care. Journal of health communication.
Montori VM, Rothman RL: Weakness in numbers. The challenge of numeracy in health care. J Gen Intern Med. 2005, 20 (11): 1071-1072. 10.1111/j.1525-1497.2005.051498.x.
Davis TC, Long SW, Jackson RH, Mayeaux EJ, George RB, Murphy PW, Crouch MA: Rapid estimate of adult literacy in medicine: a shortened screening instrument. Fam Med. 1993, 25 (6): 391-395.
Williams MV, Parker RM, Baker DW, Parikh NS, Pitkin K, Coates WC, Nurss JR: Inadequate functional health literacy among patients at two public hospitals. Jama. 1995, 274 (21): 1677-1682. 10.1001/jama.274.21.1677.
Parker RM, Baker DW, Williams MV, Nurss JR: The test of functional health literacy in adults: a new instrument for measuring patients' literacy skills. J Gen Intern Med. 1995, 10 (10): 537-541. 10.1007/BF02640361.
Baker DW, Williams MV, Parker RM, Gazmararian JA, Nurss J: Development of a brief test to measure functional health literacy. Patient Educ Couns. 1999, 38 (1): 33-42. 10.1016/S0738-3991(98)00116-5.
Vanderbilt Diabetes Research and Training Center Assessment Scales and Educational Materials. [http://vanderbiltdrtc.org/preventionandcontrol/tools.php]
Wilkinson G: The Wide Range Achievement Test – Third Edition (WRAT-3). 1993, Wilmington, DE: Wide Range, Third
Fitzgerald J, Funnell M, Hess G, Barr P, Anderson R, Hiss R, Davis W: The reliability and validity of a brief diabetes knowledge test. Diabetes Care. 1998, 21 (5): 706-710. 10.2337/diacare.21.5.706.
Rothman RL, DeWalt DA, Malone R, Bryant B, Shintani A, Crigler B, Weinberger M, Pignone M: Influence of Patient Literacy on the Effectiveness of a Primary Care-Based Diabetes Disease Management Program. JAMA. 2004, 292 (14): 1711-1716. 10.1001/jama.292.14.1711.
Dewalt DA, Malone RM, Bryant ME, Kosnar MC, Corr KE, Rothman RL, Sueta CA, Pignone MP: A heart failure self-management program for patients of all literacy levels: A randomized, controlled trial [ISRCTN11535170]. BMC Health Serv Res. 2006, 6 (1): 30-10.1186/1472-6963-6-30.
Bennett I, Switzer J, Aguirre A, Evans K, Barg F: 'Breaking it down': patient-clinician communication and prenatal care among African American women of low and higher literacy. Ann Fam Med. 2006, 4 (4): 334-340. 10.1370/afm.548.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/8/96/prepub
This research is funded with support from the American Diabetes Association (Novo Nordisk Clinical Research Award), the Pfizer Clear Health Communication Initiative, and the Vanderbilt Diabetes Research and Training Center (NIDDK 5P60DK020593). Dr. Rothman is also currently supported by an NIDDK Career Development Award (NIDDK 5K23DK065294).
The authors declare that they have no competing interests.
MMH performed the analysis of the data, developed the DNT15 and was the primary author of the manuscript. TAE and KAW participated in the development of the DNT, the design of validation study and the analysis of the data. KC assisted with the analysis of the data and the preparation of the manuscript. DD and RPG participated in the development of the DNT and in the design of the validation study. LSF provided expertise in numeracy and participated in development of the DNT. RM, AC, DD, JB and MP also participated in the development of the DNT and in the design of the validation study. RLR conceived of the study, participated in the development of the DNT, the design of the validation study, oversaw the analysis of the data, developed the DNT15, made final changes to the manuscript and provided mentorship to MMH. All authors have read and approved the final manuscript.
About this article
Cite this article
Huizinga, M.M., Elasy, T.A., Wallston, K.A. et al. Development and validation of the Diabetes Numeracy Test (DNT). BMC Health Serv Res 8, 96 (2008). https://doi.org/10.1186/1472-6963-8-96