This investigation shows that selected single terms and combinations of search terms can reach high levels of performance in the retrieval of high quality literature in the area of economic analysis and in the retrieval of literature focusing on the cost of health care services. By assisting in the retrieval of relevant cost literature, clinicians and researchers will be able to find the information they need more dependably and quicker, perhaps improving evidence-based decisions. Single term and combination search strategies have been shown to be highly sensitive and specific in the areas of cost and economics. Although there was little difference when comparing single and combination strategies for cost articles, the economics articles saw much better performance in terms of sensitivity for the single term strategy than the combination strategy when optimizing for specificity. Finally, when optimizing for both specificity and sensitivity, the combination strategies for the cost searches saw a slightly more sensitive return than the single strategies, while there was no real difference when comparing the type of search strategies when applied to the economics searches.
It is important to note that several top performing terms are exploded index terms and many are text words. In the event that new index terms relevant to cost and/or economic studies are added to Emtree, it is likely that our reported search strategies will perform similarly in terms of sensitivity and specificity. Text word searching involves only the title and abstract of the article so additions to Emtree will have no effect on the performance of these terms. Additionally, if new index terms are added and if they are closely related to the exploded index term included in the search strategy, the articles indexed with the new term will be retrieved.
In all points of comparison, the investigated search strategies performed well in terms of the accuracy of their returns. In fact, all accuracy values were over 92%. Even so, the precision of searches, that is, the proportion of retrieved articles that are on target, is suboptimal. This is simply a reflection of the very low concentration of cost and economics in the huge EMBASE database; for sound economics studies, the concentration was less than 0.1%. Precision is dependent on the concentration of target articles (in this case, cost or economic studies) in the entire database. We tested our search strategies in a subset of EMBASE records. Therefore, the precision figures reported are included only as an illustration of search strategy performance. When searching in the entire EMBASE database, precision will be lower.
While two of the single term cost strategies achieved precision levels of 39.4% and 24.0%, none of the economics single term strategies achieved better than 8% precision, meaning that only 8% of the retrieved articles were on target. The overall precision decreased further for the combination search strategies when compared to the single term strategies. Thus the somewhat higher sensitivity of the combination strategies is at the expense of decreased precision and accuracy. In addition to precision being dependent on the concentration of target articles in the entire database, low precision returns could also point to a potential problem of over indexing, that is, index terms that appear to be specific to good quality economic studies are not used solely for those types of articles resulting in the retrieval of many false positive articles (i.e., studies that are not evaluating the cost or economics of a health care situation).
Finally, the methodologic criteria for economic studies are fairly rigorous as noted by the low number of pass economic studies (n = 31) in the database. Since pass and fail economic studies are a subset of cost studies, searchers could use the cost strategies if they fail to find relevant articles when searching using the economic strategies. This is also true for economic studies based on models. Our definition of a pass economic study required that the study be based on data from real patients. Therefore, those that were based on models would only be retrieved as "false positives" when using our economic search strategies but have high likelihood of being retrieved when using the cost strategies.
We recently published economics and cost search strategies to use when searching in MEDLINE in the context of retrieving literature relevant to health services research (HSR) [6]. The HSR strategies were developed in a subset of journals (n = 68) that are indexed in MEDLINE and that publish HSR literature (there is some overlap with the EMBASE journal list). When comparing the EMBASE search strategies with those reported for use in MEDLINE we find some similarities. Optimal search strategies for use in EMBASE and MEDLINE are made up of both index terms and text words. Cost.tw. and/or a variation thereof (e.g., costs.tw., cost:.mp) is a top performer in both databases as is cost effectiveness.tw. and/or a variation thereof (e.g., cost effective:.tw.). Sensitivity analys:.tw. is a top performer in both EMBASE and MEDLINE when performing a highly specific search for economic articles. The index terms that were top performers in EMBASE and MEDLINE are quite different. This difference is partially due to the fact that some of the top performing index terms are not supported in the other database. For example, the index term "cost effectiveness analysis" is a top performer in EMBASE but this is not an index term in MEDLINE. Overall, although the search strategies developed for EMBASE and MEDLINE are different there are many similarities when comparing the text words that are top performers.
Multivariate statistical techniques may yield better results than we observed. However, when we tested a logistic regression approach to developing search strategies for MEDLINE, we found no improvement on the same Boolean approach used in the EMBASE study [9]. Even if such techniques did improve yield, the increase would be marginal at best, given how well the strategies shown here work, except perhaps for increasing sensitivity for strategies optimizing specificity. Such strategies would also likely have the disadvantage of being more complex, and thus harder to implement.
Machine learning methodologies may yield better results than we observed. We are currently exploring this possibility through collaborative ventures with two research groups in the United States.