Table 1 Summary of Included Studies
First author | Year | Country | Discipline | Synopsis of review | Reliability and validity | Evaluative comments |
|---|---|---|---|---|---|---|
Abda A [16]. | 2018 | Canada | Psychology | A systematic review that investigated the psychosocial outcomes of children and adolescents with severe congenital heart disease. | This tool was selected for its good inter-rater reliability (k ¼ 71.5%) and its ability to highlight the methodological strengths and weaknesses of studies (Sirriyeh et al., 2012). | - |
Adam A [17]. | 2016 | Denmark | Health Sc | A systematic review that investigated the effectiveness of obesity related interventions at retail grocery stores and supermarkets. | Transparent and validated tool. | |
Albutt A.K [18]. | 2016 | UK | Psychology | A systematic review that investigated the role of patients and their relatives in escalating clinical deteriotation in hospital settings. | Strong and significant correlation between 1st and 2nd reviewer’s quality assessments, r = .73,P.039. | |
Alsawy S [19]. | 2017 | UK | Psychology | A mixed-methods systematic review that investigated what good communication is for people living with dementia. | Agreement of 96.0 and 94.4% was achieved between the first researcher (SA) and independent raters 1 and 2 respectively. Statistically significant (p < 0.00) inter-rater reliability of quality assessments was achieved across all three raters (the first researcher and two independent). | The outcomes suggest overall agreement in the quality ratings of each study and robustness of the QATSDD tool. |
Arbour-Nicitopoulos K.P [20]. | 2018 | Canada | Allied Health | A scoping review investigating the inclusive out-of-school time Physical activity programs for children and youth with physical disabilities. | This quality assessment tool has demonstrated good face validity and interrater and test–retest reliability for examining study quality across diverse methodologies. | Three reviewers independently appraised the quality of each of the included studies (53.5% agreement) using a 16-item quality assessment tool that can be applied to quantitative, qualitative, and mixed-method studies. Consensus was obtained on rating discrepancies through group discussion. - Selection of this tool was based on its consideration of additional elements that are often not taken into account in study quality yet are critical to external validity of the study findings e.g. application of theoretical frameworks and/or constructs to the research, evidence of user involvement in the study design and discussion of strengths and limitations. |
Augestad L.B [21]. | 2017 | Norway | Psychology | A systematic review to investigate self-concept and self-esteem among children and young adults with visual impairment. | The tool, which was developed to assess the quality of studies on one topic but using different approaches or designs, has been found to have good reliability and validity (Sirriyeh et al., 2012). | - |
Augestad L.B [22]. | 2017 | Norway | Medicine | A systematic review to investigate mental health among children and young adults with visual impairments. | The tool has been found to have good reliability (Cohen’s kappa, 71.5) and good face validity (Sirriyeh et al., 2011). - The weighted kappa was 0.5 (indicating moderate agreement), and the Spearman correlation was 0.75 (indicating a strong association or relationship). | - |
Aztlan-James E.A [23]. | 2018 | America | Medicine | A systematic review investigating multiple unintended pregnancies in U.S. women. | The validity and reliability of QATSDD is established and has been reported. In case of disagreements, the study was discussed until agreement was reached on quality score. | - |
Band R [24]. | 2015 | UK | Psychology | A systematic review investigating patient outcomes in assosication with significant other responses to chronic fatigue syndrome. | The measure has demonstrated adequate reliability (Sirriyeh et al., 2012), although normative values associated with study quality are not currently available. | - |
Batten G [25]. | 2014 | UK | Psychology | A systematic review investigating the factors associated with social interactions between deaf children and their hearing peers. | Inter-rater reliability scored at 0.743(using Spearman’s Correlation) at < 0.01 significance level(for a 43- question devised checklist combining QATSDD with 3 other QA checklists). | |
Baxter R [26]. | 2016 | UK | Health Sc | A systematic review investigating the methods used to apply positive deviance within healthcare organisations. | Validated tool that standardises the quality assessment of research with heterogenous study-designs. | |
Blackwell J.E [27]. | 2017 | UK | Psychology | A systematic review investigating the cognitive function and psychosocial well-being in school-age children with narcolepsy. | Substantial inter-rater agreement (89.3%),remaining differences solved by discussion. | Particularly suited as QATSDD involves qual and quant aspects both. |
Blake D.F [28]. | 2018 | Australia, New Zealand, Canada | Medicine | A systematic review investigating the effects of helicopter retrieval on injured divers. | Studies identified were of diverse designs so the modified QATSDD tool was used to better compare the levels of evidence. | |
Bradford N [29]. | 2019 | Australia | Nursing | An integrative review investigating the evaluation frameworks in health services. | - | We added a fifth item—(Not Applicable) for articles that were narrative discussions rather than research per se. Two authors (NB and SC) independently appraised the included articles with high agreement (92%). Despite the QATSDD being designed for disparate study designs, many criteria were not applicable to the type of papers included in this review. |
Braun S.E [30]. | 2019 | USA | Psychology | An integrative review investigating mindfulness in health care professionals and its relation to patient care. | - | Although this tool was not developed to evaluate cross-sectional studies, it can easily be applied to cross-sectional designs without omitting domains or adapting the tool; furthermore, it has been used in previous systematic reviews to assess cross-sectional research. |
Burton A [31]. | 2016 | UK | Psychology | A systematic review and meta-analysis investigating mindfulness-based interventions for reducing stress among healthcare professionals. | QATSDD combines previously validated tools to produce a comprehensive list of indicators of good quality research. | |
Carrara A [32]. | 2018 | Switzerland | Health Sc | A systematic review investigating the role of health literacy in predicting adherence to nutritional recommendations. | QATSDD has shown good reliability and validity in evaluating the quality of methodologically diverse studies in the contexts of psychology, sociology and nursing. | |
Clausen C [33]. | 2017 | Canada | Health Sc | An integrative review investigating educational interventions that enhance competencies for interprofessional collaboration among nurse and physician managers. | Quality Assessment Tool (QAT) tool was chosen for quality appraisal of the included studies. This tool, tested by the authors for reliability and validity, was chosen for its rigor in the assessment of qualitative, quantitative, and mixed method studies. | Although the QAT tool was transferable across studies, all qualitative articles scored poorly. One could question whether the tool was well adapted and reliable to provide sufficient comparison amongst studies. |
Connolly F [34]. | 2017 | Ireland | Health Sc | A systematic review of the barriers and facilitators related to the implementation of a physiological track and trigger system. | Validated tool for assessing study quality. | Inconsistencies in scoring were resolved through discussion. |
Curran C [35]. | 2018 | Ireland | Medicine | A systematic review investigating the primary care safety climate survey instruments. | This tool allows standardized evaluation of studies with varying research designs. | |
Deming A [36]. | 2019 | US | Health Sc | A study investigating the absence of evidence-based practices (EBPs) in the treatment of sexual abusers. | Several systems for scoring and rating research studies of diverse designs and methodologies have been developed and described (including QATSDD). Each approach recommends somewhat different methods or systems for determining the overall strength of research, but none have been developed specifically for use with research relating to individuals with a history of sexual offending. | |
Dias C.C [37]. | 2013 | Portugal | Medicine | A systematic review and meta-analysis investigating the clinical prognostic factors for disabling Crohn’s disease. | Allows comparison of different study designs | |
Emerson L.M [38]. | 2017 | UK | Psychology | A systematic review and narrative synthesis investigating the teaching of mindfulness to teachers. | Additional Item “Clarity of Intervention” added to QATSDD, initial agreement between the researchers was 91.6%, calculated using Cohen’s Kappa. Discrepancies were resolved through discussion. | |
Fenton L. [39] | 2016 | Canada | Health Sc | An integrative review investigating the benefits of recreation for the recovery and social inclusion of individuals with mental illness. | QATSDD has been evaluated for validity & reliability. | Only tool specifically designed to evaluate diverse research approaches. |
Fenton L. [12] | 2015 | Canada | Health Sc | A comments and critiques paper investigating the QATSDD critical appraisal tool. | Potential value but a number of aspects for clarification: -Unclear meaning of language -Further definition of language in each indicator and inclusion of explicit examples for each criterion recommended - Needs outlining of clear parameters around the use of tool, stating that the tool should be used in synthesis work for studies of mixed methods or work that includes qual and quant research informed by a positivist paradigm - Tool is subjective in nature - Dropping the scoring system recommended - “Evidence of User Involvement in design” is inappropriate - No indicator addressing bias included | |
Filmer T [40]. | 2018 | Germany | Medical Education /Medicine | A systematic review investigating the effectiveness of interventions teaching cross-cultural competencies to health-related professionals with work experience. | - For all criteria ratings, the unadjusted two-way random single-measure intraclass correlation coefficient (2,1) was 0.93, confirming a very good reliability. | Any discrepancies in ratings were discussed and a consensus was achieved. |
Fylan, B [41]. | 2015 | UK | Health Sc | A thesis paper that investigated medicines management after hospital discharge. | It was chosen as a suitable tool because of the heterogeneity of research designs in the literature | |
Graham-Clarke E [42]. | 2018 | UK | Psychology | A systematic review and thematic synthesis investigating the facilitators and barriers to non-medical prescribing. | A validated quality-assessment tool. | Two reviewers independently assessed the studies using the tool; resolving any disagreement in the scores through discussion |
Gillham R [43]. | 2015 | UK | Psychology | A systematic review investigating the outcomes for women admitted to a mother and baby unit. | Interrater reliability was very good (k = 0.91). | |
Gkika S [44]. | 2017 | UK | Psychology | A systematic review investigating social cognition and metacognition. | To examine potential assessment bias,25% of papers were independently assessed by a colleague and good agreement between 2 raters was observed. | |
Hardy M [45]. | 2016 | UK | Medicine | A systematic review investigating if radiography advanced practice improves patient outcomes and health service quality. | High interrater agreement (K = 0.89). | Components fulfil the criteria for quality assessment within “Centre of Reviews and Dissemination Guidance”. |
Harris K [46]. | 2016 | UK | Psychology | A systematic review investigating distress in significant others of patients with chronic fatigue syndrome. | There may be disadvantages of using a quality assessment tool scored from 0 to 3 as opposed to a dichotomous (yes/no) rating scale. One potential disadvantage is that a greater number of response options in the scale may increase the subjectivity of the ratings. In contrast, a dichotomous scale could have been rated on the absence or presence of key information, which would have provided fewer opportunities for bias. The study design should be taken into account when interpreting the study’s findings. | |
Harrison R [47]. | 2015 | Australia | Health Sc | A systematic review investigating the patient safety and quality of care in developing countries in Southeast Asia. | Disagreements were resolved through discussion resulting in substantial agreement (k = 65.8%) on a random sample of 30% papers. | |
Harrison R [48]. | 2015 | Australia | Health Sc | A systematic review investigating patients’ experiences of adverse events in health care. | Disagreements between 2 reviewers resolved by discussion resulting in substantial agreement (k = 61.6%). | |
Harrison R [49]. | 2014 | UK | Nursing | A narrative review investigating the contribution of nurses to incident disclosure. | Disagreements between reviewers resolved through discussion and substantial agreement (k = 73.7%) reached. | |
Hawkins R.D [50]. | 2017 | UK | Social Sc. | A systematic review investigating the psychological risk factors for childhood nonhuman animal cruelty. | The publications were scored by 2 authors independently (x = 0.78),with Cohen’s kappa demonstrating a substantial strength of agreement. | Case studies could not be easily assessed using these criteria. |
Heath G [51]. | 2016 | UK | Health Sc | A mixed-methods study that investigated the development of a tool that support communication of parental concerns when a child is in hospital. | The QATSDD has in a preliminary assessment been shown to have good face validity, as well as good interrater and test-retest reliability in evaluating qualitative as well as quantitative studies. | - |
Hesselstrand M [52]. | 2015 | Sweden | Allied Health | A systematic review investigating occupational therapy interventions in chronic pain. | The QATSDD has in a preliminary assessment been shown to have good face validity, as well as good inter‐rater and test–retest reliability in evaluating qualitative as well as quantitative studies (Sirrieyh et al., 2012) | - |
Hill S [53]. | 2015 | UK | Health Sc | A rapid review investigating the conduction of contingent valuation studies in older and young populations. | Quality assessment was considered; however following examination of potential tools available for the process, a decision was made not to progress with quality assessment. Although this review included studies of diverse design, it was felt that the items in the tool devised by Sirriyeh et al. (2012) did not allow sufficient focus on the methods used (i.e. contingent valuation) | |
Holl M [54]. | 2015 | Netherlands | Social Sc | A systmetic review investigating the interventions to prevent tenant evictions. | Weighted kappa was 0.70 (substantial agreement) | QATSDD does not provide cut-off points for quality rating of individual studies as good, fair or poor. |
Iddon J.E [55]. | 2016 | UK | Psychology | A systematic review investigating positive psychological interventions and chronic non-cancer pain. | QATSDD has shown good reliability and validity when assessing the risk of bias and quality of diverse study designs. | Although there are implications and subsequent limitations to consider when applying a more general tool for quality assessment,the QATSDD allowed for cross-comparison between differing methodologies. Whilst this has its advantages, QATSDD total scores should be interpreted with some discretion as particular areas of significant methodological weaknesses may be concealed by perhaps less influential strengths in other areas, and vice versa. For example, a poor score on the item assessing the appropriateness of the study sample size may be obscured by a higher score merited from describing the study research setting in detail. |
Jaarsma E.A [56]. | 2018 | UK | Psychology | A systematic review investigating the promotion of physical activity for disabled people who are ready to become physically active. | Used with the exception of Criteria 14 (Reliability of analytical process-qualitative only) because this check is flawed and now known to be ineffective for reliability purposes in terms of qualitative research (Smith & McGannon 2017)***, all included studies were assessed for quality based on the tool by Sirriyeh et.al. | |
Jackman P [57]. | 2019 | England | Health Sc | A systematic review investigating flow states in exercise. | - | Used with the exception of criterion 14, which was omitted due to recent criticism of reliability strategies for qualitative research (Smith & McGannon, 2018). - During these critical discussions, the authors highlighted some concerns with the study quality scores as the QATSDD was unable to detect many of the conceptual and methodological issues identified by this review |
Jackson-Blott K [58]. | 2019 | UK | Psychology | A narrative literature review investigating recovery-oriented training programmes for mental health professionals. | An inter-rater reliability of 71% was obtained between reviewers (two of the authors) on a random sample of four papers (25%). | The scoring system of methodological quality (QATSDD) accounted for the diversity of study designs and inter-rater reliability checks provided assurance of its rigorous application. |
Johnson D [59]. | 2017 | Australia | Psychology | A systematic review investigatingthe resilience to emotional distress in response to failure, error or mistakes. | QATSDD used to assist in the development of coding matrix for this study using “iterative” process. | |
Jones N [60]. | 2018 | UK | Health Sc | A systematic review investigating visual impairment on nutritional status. | QAT has been reported to have good validity and reliability. | |
Khajehaminian M.R [61]. | 2018 | Iran | Health Sc | A systematic review investigating the criteria and models for casualty distribution in trauma related mass casualty incidents. | In the case of quality appraisal of the included literature an extensive search to find appropriate tool was unsuccessful. Although there were some tools for appraising diverse design studies, [including QATSDD], they are applicable for medical studies. In this case, there is a need to develop a new quality appraisal tool to assess studies with diverse design in a multidisciplinary research environment. | |
Klingenberg O [62]. | 2019 | Norway | Social Sc | A systematic review investigating digital learning in mathematics for students with severe visual impairment. | - | For example, it is not defined or clearly explained how the indicators ‘theoretical framework’ or ‘the statement of aims’ should be scored, which may imply a change in construct validity. The QATSDD should therefore be used with caution. Studies scoring above 75% were considered ‘high quality’, 50–75% ‘good quality’, 25–50% ‘moderate quality’, and those scoring below 25% ‘poor quality’. |
Kolbe A.R [63]. | 2015 | Haiti, USA | Social Sciences | A qualitative study investigating transactional sex between UN peacekeepers and Haitian citizens. | - | ‘‘Redundancy” as ideal when little existing data available about the extent of phenomenon. |
Kumar M.B [64]. | 2012 | Canada | Health Sc | A literature review investigating trends in Métis-related health research from 1980 to 2009. | Inter-rater agreement was substantial (k = 0.67). | - |
Lambe, K [65]. | 2019 | Ireland | Patient Safety | A systematic review investigating hand hygiene compliance in the ICU. | The tool has been assessed by its authors for reliability and validity and was used by two authors for each study in this review. | The evaluation is subjective and concerns have been raised about the tool’s structure, particularly around the equal weighting of all items for all studies. |
Lamore K [66]. | 2017 | France | Psychology | A systematic review investigating treatment decision-making in chronic diseases and family members’ roles, needs and attitudes. | QATSDD was selected for its reliability and validity when assessing the quality of diverse study designs | QATSDD is the only tool which can be applied to mixed study designs Can be improved with better defined criteria,some items can be clearer by adding examples. Other items could be weighted as more indicative of a rigorous methodology than others (e.g. sample size vs user involvement in design) |
Levy I [67]. | 2017 | Israel | Medicine | A systematic review and narrative synthesis investigating the use of complementary medicine for treatment of agitation and delirium in older persons. | QATSDD has been validated in previous studies | We omitted indicator 15 (evidence of user involvement in design),which was determined as irrelevant in a recent comment and critique of this scale (Fenton et.al.,2015)**** |
Madden C [68]. | 2018 | Ireland | Medicine | A systematic review investigating the potential value of patient record review to assess and improve patient safety in general practice. | The QATSDD has been previously used in other systematic reviews, with high levels of agreement reported. Two reviewers completed the quality assessment and disagreements were resolved through discussion. | |
Martins-Junior P.A [69]. | 2017 | Brazil | Health Sc | A systematic review investigating dental treatment under general anaesthetic and children’s oral health-related quality of life. | Authors used the Quality Assessment Tool for Studies of Diverse Design (QATSDD) with some modifications to assess the quality of studies. They set up three teams of two investigators to independently extract data for each paper, which lessened subjectivity in article selection and analysis. Also, an inter-reviewer agreement was performed, enhancing the reliability of the results. | |
McClelland G [70]. | 2019 | UK | Medicine | A narrative review investigating the frequency, characteristics and aetiology of stroke mimic presentations. | A quality assessment tool tailored for cohort studies may have been more appropriate than the QATSDD tool that was chosen before study identification | |
McPherson A.C [71]. | 2016 | Canada | Medicine | A scoping review investigating best practices when communication with children and families about obesity and weight related topics. | Cut-offs not provided by QATSDD (Low/moderate/high quality ratings) | |
Medford E [72]. | 2017 | UK | Health Sc | A systematic review investigating the demographic and psychological influences on treatment adherence for children and adolescents with PKU. | QATSDD has shown good reliability and validity and was chosen due to the diverse methodologies of included studies. Inter-rater reliability was good (K = 0.71) | Only studies reporting statistical analyses were included in order to identify the factors most robustly linked with metabolic control and QATSDD was found to be a valid tool for assessing the methodological quality of the studies included in the current review. |
Medway M [73]. | 2016 | Australia | Psychology | A qualitative meta-synthesis investigating young people’s experience of family therapy for anorexia nervosa. | Discrepancies in rating resolved by discussion. | |
Miller L. [74] | 2019 | Australia | Medicine | A systematic review investigating epidemiology, risk factors and measures for preventing drowning in Africa. | The scores of the criteria were summed up to assess the methodological quality of included studies with a maximum score of 36. For ease of interpretation, the scores were converted to percentages and were categorised as excellent (> 80%), good (50–80%) and low (< 50%) quality of evidence based on the overall score | |
Mimmo L. [75] | 2018 | Australia | Health Sc | A systematic review and narrative synthesis investigating patient safety vulnerabilities for children with intellectual disability in hospital. | Disagreements were resolved through discussion resulting in substantial agreement (κ = 0.75) between reviewers on a random sample of 25% of the papers. | |
Nghiem T [76]. | 2017 | Canada | Medicine | QAT has been validated and found to be reliable for assessing the quality of studies - ICC between pairs of reviewers were 0.840 and 0.703 with CI of 0.776–0.887 and 0.612–0.774,resp. | ||
Nghiem T [77]. | 2018 | Canada | Medicine | An integrative review investigating pain experience of adults with osteogenesis imperfecta. | Chosen a priori because the tool permits appraisal of studies across a range of designs (i.e., quantitative, qualitative, or mixed method) and would allow the findings to be compared to the review on pain in children and adolescents with Osteogenesis imperfecta. | |
Noblet T [78]. | 2017 | Sydney | Allied Health | A mixed methods systematic review investigating barriers to and facilitators of independent non-medical prescribing in clinical practice. | Good validity, inter-rater reliability and test-retest reliability have been established for the QATSDD across a variety of study designs, demonstrating its value for consistent quality assessment in mixed methods designs . | - |
O’Dowd E [79]. | 2019 | Ireland | Medical Education | A systematic review investigating 7 years of research on entrustable professional activities in graduate medical education from 2011 to 2018. | This assessment tool has been shown to produce good agreement and has been used in a number of different reviews pertaining to health services and medical education research The variability in the quality scores of the development studies is interesting. Although it can be difficult to balance methodological quality and practical success, it is important that researchers developing EPAs give consideration to the quality of their approach. This variability may indicate a need to broadly examine methods in medical education research or to develop methodological quality assessment tools better suited to the field of research. | |
Orr, K [80]. | 2019 | Canada | Health Sc | A scoping review investigating children and youth with impairments in social skills and cognition in out-of-school time inclusive physical activity programs. | The QATSDD provides a percentage score to compare reporting quality across studies; however, there are no guidelines to suggest values of high or low reporting quality. Thus, based on the guidelines applied in an earlier scoping review (Arbour-Nicitopoulos et al.) the following cut-points were used: less than 60% (low-quality reporting), 60–80% (moderate-quality reporting), and greater than 80% (high-quality reporting). | |
Pini S [81]. | 2011 | UK | Psychology | A systematic review investigating the effect that cancer diagnosis has on the educational engagement and school life of teenagers. | Absence of official “cut-off ‘’score for quality-assessment. | |
Powney M [82]. | 2014 | UK | Psychology | A systematic review investigating the attachment and trauma in people with intellectual disabilities. | The QATSDD is a 16-item quality assessment tool, which has shown good inter-rater reliability (k = 71.5%) and validity for the assessment of studies with diverse designs (Sirriyeh, et al., 2012). | For accuracy, the QATSDD was designed to produce an overall quality rating expressed as a percentage. |
Quinn C [83]. | 2018 | UK | Medicine | A systematic review investigating the influence of positive aspects of dementia caregiving on caregivers’ wellbeing. | The QATSDD was selected because the items seemed appropriate for the types of papers included; however, there have been criticisms that the QATSDD is too subjective. Although there were clearly benefits in using the QATSDD, there were also challenges to implementing the tool. For instance, we found that studies with smaller word counts (because of journal requirements) risked having a lower score because there is less scope to explain the study in-depth. This suggests that quality-rating tools would benefit from more flexibility: for instance, the ability to take into account the length of the paper. | |
Rosella L. [84] | 2016 | Canada | Health Sc | A study investigating the development and validation of a meta-tool for quality appraisal of public health evidence: Meta Quality Appraisal Tool (MetaQAT). | Most tools(including QATSDD) are design-specific and focus only on “risk of bias”. The model of “one tool fits all” approach does not make best use of the existing science of quality appraisal, nor does it adapt in the way articulated in the aforementioned goals**. | |
Salman Popattia A [85]. | 2018 | Australia | Health Sc | A systematic review investigating the ethical responsibilities of pharmacists when selling complementary medicines. | The tool was selected because it has demonstrated good reliability and validity when applied to a methodologically diverse set of research articles | - |
Sibley A.M. [86] | 2017 | UK | Health Sc | A mixed-methods study investigation diabetes patients’ beliefs about their medicines from a nurse prescribers’ perspective. | The ‘quality assessment tool’ reported inter-rater reliability as ‘substantial’ to ‘very substantial’ (kappa ranging from 0.69 to 0.91) for question level agreement (Sirriyeh et al. 2012). | - |
Ten Hoorn S [87]. | 2016 | Netherlands | Medicine | A systematic review investigating communication with conscious and mechanically ventilated critically ill patients. | Test-retest and inter-rater reliabilities range from “good” to “substantial”(K 0.698–0.901). | Can be applied to diverse study designs Clearly defined scales |
Tomlin E.M [88]. | 2018 | UK | Psychology | A thesis investigating the experience-based co-design approach within the NHS with patients at the centre of design to improve quality of care. | This validated tool has demonstrated good internal reliability and is seen as a pragmatic approach to providing a robust, transparent and standardized method to assess quality across different research methodologies. | Owing to the heterogeneity of study designs included within the inclusion criteria, the Quality Assessment Tool for Studies with Diverse Designs (QATSDD) was initially pre-specified as an appropriate method to assess study quality. The criteria did not map onto the reported content of the EBCD QI projects. This meant that relevant data was not accurately assessed and ultimately made the comparison between the EBCD QI projects and evaluative and research papers problematic. For instance: evidence of sample size considered in terms of analysis, representative sample of target group of a reasonable size and fit between research questions and method of analysis were not considered to be routinely reported aspects of EBCD QI projects. Therefore, a dual approach was taken to assess the quality of included papers within the review. |
Tuominen O [89]. | 2018 | Australia, Finland | Health Sc | A scoping review investigating the rescheduling of nursing staff with information technology-based staffing solutions. | Agreement of scoring between the two reviewers was calculated using a Cohen’s kappa. Kappa values varied between the two reviewers from 0.78 to 1.00 for these ten papers, showing good (0.60 to 0.80) or very good (> 0.80 to 1.00) agreement. | - |
Vyth E.L. [90] | 2012 | Netherlands | Allied Health | A review investigating the methodological quality of front-of-pack labelling studies. | Initial coder disagreement of 55 of the 496 scores, resolved completely after discussion. | Applicable to diverse research designs. |
Wallace A [91]. | 2016 | UK | Medicine | A systematic review investigating the traumatic dental injury research. | Inter examiner agreement for QA was poorer (54–82%)than for general data-extraction; Intra-examiner agreement after 8-weeks interval was good for QA (64–76%) | - QATSDD is a unique quality assessment tool can be applied to diverse study designs, unlike majority of other tools which evaluate a single methodological approach - No authors referred to an explicit theoretical framework or model to underpin their work. This is a difficult concept and may not have been relevant to all study designs. Indeed, the developers of the QATSDD acknowledge that some of quality criteria may not be suitable for certain study designs. Theoretical frameworks may be more appropriate to studies with a qualitative component, and they may help to inform the study design and explain findings. The absence of any qualitative studies therefore makes the results less remarkable. - Standard deviation for the scores ascribed to studies in the quality assessment exercise was high (18.7%). One explanation for this may be a true marked difference in quality of the papers. Characteristics of the QATSDD rather than the study quality per se may explain the low scores for some papers, and hence the high standard deviation. - The QATSDD uses a 4‐point scoring system to try and provide a more accurate representation of the paper’s quality. However, this scaled scoring system means there is a wider margin for disagreement between reviewers compared to a 2‐point scoring scale - The research experience of the reviewers in this present review was quite varied and may account for the inconsistencies in inter‐ and intra‐examiner agreement. Furthermore, it was felt that the QATSDD would benefit from the provision of greater detail in some of the descriptions to improve inter-examiner agreement. However, the QATSDD was found to be generally applicable to the range of study designs encountered in this review. |
Walton M [92]. | 2015 | Australia | Health Sc | A systemtatic review and narrative sysnthesis investigating workplace training for senior trainees. | Substantial agreement (k = 62.5%)confirmed between reviewers | |
Wells, E [93]. | 2016 | UK | Health Sc | A thesis study investigating the role of parenting interventions in promoting treatment adherence in cystic fibrosis. | An independent researcher rated 7 of the 15 included papers (47%) and any discrepancies were resolved through discussion. Cohen’s Kappa was 0.71 indicating ‘substantial’ interrater agreement. | - |
Wright C.J [94]. | 2017 | Australia | Philosophy | The QATSDD has adequate face validity, inter-rater reliability (κ = 71.5%; indicating substantial agreement), and good to substantial agreement for test-retest reliability | Although the tool has been empirically tested, concerns have been raised regarding the use of scales (i.e., thought to affect the establishment of inter-rater reliability) and its ease of application |