Skip to main content

Table 5 ISPOR task force checklist for quality assessment of retrospective database studies [29]

From: Minimum volume standards in day surgery: a systematic review

Study reference/ID

Degan et al. [10] (2017)

Evers et al. [11] (2018)

Jain et al.

[12] (2005)

Jain et al.

[13] (2004)

Liu et al.

[14] (2018)

Ayala and Yencha [15] (2015)

Chen et al. [16] (2014)

Keay et al. [17] (2012)

1. Relevance: Have the data attributes been described in sufficient detail for decision makers to determine whether there was a good rationale for using the data source, the data source’s overall generalizability, and how the findings can be interpreted in the context of their own organization?

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

2. Reliability and Validity: Have the reliability and validity of the data been described, including any data quality checks and data cleaning procedures?

No

Yes

Yes

Yes

No

No

Yes

Yes

3. Linkages: Have the necessary linkages among data sources and/or different care sites been carried out appropriately, taking into account differences in coding and reporting across sources?

Yes

NA

Yes

Yes

Yes

NAa

NAa

Yes

4. Eligibility: Have the authors described the type of data used to determine member eligibility?

Yes

Yes

Yes

Yes

Yes

No

No

Yes

5. Data analysis plan: was a data analysis plan, including study hypotheses, developed a priori? Was the study conducted prospectively?

Partialb

Partialb

Partialb

Partialb

Partialb

Noc

No

Partialb

6. Design selection: has the investigator provided a rationale for the particular research design?

No

No

No

No

No

No

No

Partiald

7. Research design limitations: did the author identify and address potential limitations of that design?

Yes

No

No

No

Yes

No

No

Partial

8. Treatment effect: for studies that are trying to make inferences about the effects of an intervention, does the study include a comparison group and have the authors described the process for identifying the comparison group and the characteristics of the comparison group as they relate to the intervention group?

NA

NA

NA

NA

NA

NA

NA

NA

9. Sample selection: have the inclusion and exclusion criteria and the steps used to derive the final sample from the initial population been described?

Partiale

Partialf

Yes

Yes

Partialg

No

Partialf

Yes

10. Eligibility: are subjects eligible for the time period over which measurement is occurring?

NA

NA

NA

NA

NA

NA

NA

NA

11. Censoring: were inclusion/exclusion or eligibility criteria used to address censoring and was the impact on study findings discussed?

Partialh

Partialh

Partialh

Partialh

Partialh

No

Partialh

Yes

12. Operational definitions: are case (subjects) and end point (outcomes) criteria explicitly defined using diagnosis, drug markers, procedure codes, and/or other criteria?

Yes

Yes

Yes

Yes

Yes

Partiali

Yes

Yes

13. Definition validity: have the authors provided a rationale and/or supporting literature for the definitions and criteria used and were sensitivity analyses performed for definitions or criteria that are controversial, uncertain, or novel?

NA

NA

NA

NA

NA

NA

NA

NA

14. Timing of outcome: is there a clear temporal (sequential) relationship between the exposure and outcome?

NA

NA

NA

NA

NA

NA

NA

NA

15. .Event capture: are the data, as collected, able to identify the intervention and outcomes if they actually occurred?

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

16. Disease history: is there a link between the natural history of the disease being studied and the time period for analysis?

NA

NA

NA

NA

NA

NA

NA

NA

17. Resource valuation: for studies that examine costs, have the authors defined and measured an exhaustive list of resources affected by the intervention given the perspective of the study and have resource prices been adjusted to yield a consistent valuation that reflects the opportunity cost of the resource?

NA

NA

NA

NA

NA

NA

NA

NA

18. Control variables: if the goal of the study is to examine treatment effects, what methods have been used to control for other variables that may affect the outcome of interest?

Yes

Yes

Yes

Yes

Yes

Nonej

Nonek

Yes

19. Statistical model: have the authors explained the rationale for the model/statistical method used?

No

No

No

No

No

Partiall

No

Yes

20. Influential cases: have the authors examined the sensitivity of the results to influential cases?

NA

NA

NA

NA

NA

NA

NA

NA

21. Relevant variables: have the authors identified all variables hypothesized to influence the outcome of interest and included all available variables in their model?

Yes

Yes

Yes

Yes

Yes

Nom

No

No

22. Testing statistical assumptions: do the authors investigate the validity of the statistical assumptions underlying their analysis?

Yes

Yes

Yes

Yes

Yes

No

No

No

23. Multiple tests: if analyses of multiple groups are carried out, are the statistical tests adjusted to reflect this?

NA

NA

NA

NA

NA

NA

NA

NA

24. Model prediction: if the authors utilize multivariate statistical techniques in their analysis, do they discuss how well the model predicts what it is intended to predict?

No

No

No

No

No

NA

NA

Yes

25. Theoretical biases: have the authors provided a theory for the findings and have they ruled out other plausible alternative explanations for the findings?

Partialn

Partialn

Partialn

Partialn

Partialn

Partialn

No

Yes

26. Practical versus statistical significance: have the statistical findings been interpreted in terms of their clinical or economic relevance?

No

No

No

No

No

Yes

Partialo

Yes

27. Generalizability: have the authors discussed the populations and settings to which the results can be generalized?

Partialp

No

No

No

Partialp

No

No

Yes

Overall level of quality

Very low

Very low

Very low

Very low

Very low

Very low

Very low

Very low

  1. aSingle data source
  2. bIt is indicated in the text that a hypothesis was created a priori, but no more information is revealed
  3. cIt is not explicitly stated in the text that a hypothesis was created a priori
  4. dThey state that population-based studies can be generalized to the community more easily than center-specific studies because they represent the broad range of conditions under which surgeries are conducted on diverse populations in a wide range of settings by many surgeons with varying levels of experience
  5. eOnly inclusion criteria were described
  6. fThe inclusion/exclusion criteria were described only in part
  7. gOnly exclusion criteria were described
  8. hCriteria were mentioned, but the impact on findings was not discussed
  9. iNo explicit procedure codes were used
  10. jOnly bivariate statistical analyses were conducted – no multivariate regression model was applied
  11. kOnly causative factors are listed descriptively
  12. lReasons for using the conducted statistical tests were given
  13. mOnly a low volume hospital with outpatient thyroidectomy was the object of analysis
  14. nThe authors did not rule out other possible interpretations
  15. oAuthors refer only to one paper that also tests the respective hypothesis
  16. pIt is stated that the current results are of limited generalizability