Table 2 Quality criteria and associated analyses
Criterion | Data Requirement | Analysis | Result | Advantages / Disadvantages |
|---|---|---|---|---|
Congruence | Item phrasing and open comments | Compare whether comments written in an open-ended comment field match with what the question/item is asking | •Congruent (on-topic) •Incongruent (off-topic) •Unclear (cannot be determined) •Not applicable | +Congruence provides anindicator the question was understood as intended +Easy to administer +Only marginally increases questionnaire completion time +Respondents with more to say are given the opportunity; all others can simply skip the comments +Allows respondents to clarify quantitative responses for every item +Simple analytic procedure +More scalable to large samples than cognitive interviewing +Can provide indications of quality issues missed in cognitive interviewing -May require adjustments to questionnaire layout -Open comments not always interpretable -Incongruence is not a definitive indicator that the item was misunderstood |
Convergence | Response on Likert-type scale and open comments | Check whether quantitative ratings and comments in comment field agree (e.g., high quantitative rating and positive comments) | •Convergent •Divergent •Unclear (cannot be determined) •Neutral (neither convergent nor divergent) •Not applicable | +Convergence provides an indicator of convergent validity +Easy to administer +Only marginally increases questionnaire completion time +Respondents with more to say are given the opportunity; all others can simply skip the comments +Allows respondents to clarify quantitative responses for every item +Simple analytic procedure +More scalable to large samples than cognitive interviewing +Can provide indications of quality issues missed in cognitive interviewing -May require adjustments to questionnaire layout -Open comments not always interpretable -Convergence requires articulate and interpretable comments |
Credibility | Open comments | 1.1.1.1.Classify what the comment is trying to convey, e.g. conveying a clarification, confirmation, disconfirmation of the quantitative rating, expressing comprehension difficulty or inability to judge | •Clarifying statement •Confirming statement •Disconfirming statement •Cannot judge •Comprehension difficulty | +Credibility uses multiple procedures to provide a summary evaluation of instrument quality +Provides additional indicators of whether the question was understood as intended +Easy to administer +Only marginally increases questionnaire completion time +Respondents with more to say are given the opportunity; all others can simply skip the comments +Allows respondents to clarify quantitative responses for every item +More scalable to large samples than cognitive interviewing +Can provide indications of quality issues missed in cognitive interviewing +Complements analytic procedures for congruence and convergence -More complex analytic procedure -May require adjustments to questionnaire layout -Open comments not always interpretable -Credibility criterion does not have clear cut-off |
Response on Likert-type scale and open comments | 2.2.2.2.Compare quantitative response and qualitative comment. Infer whether the respondent is answering the question as intended | Inference of whether the respondent answered the question as intended | ||
Field notes | 3.3.3.3.Draw inferences from field notes of staff administering paper-and-pencil questionnaires | Inference of whether the respondent answered the question as intended |