Response to Kim et al. “When conducting a systematic review, can one trade search efficiency for potential publication bias?”

Abstract

Dear Editor, We sincerely thank Kim et al. for their interest in our study entitled “A search of only four key databases would identify most randomized controlled trials of acupuncture: A meta-epidemiological study.” Our study found that combined retrieval from two Chinese databases (CNKI and WanFang) and two English databases (PubMed and CENTRAL) identified most randomized controlled trials (RCTs) of acupuncture and highlighted the importance of searching for both Chinese and English RCTs when performing a systematic review (SR) on acupuncture. Our findings were based on acupuncture SRs in Chinese and English owing to the limitations of language, as Kim et al. indicate. Nevertheless, the SRs in Chinese and English were based on searches that were not restricted to Chinese and English databases but included databases in other languages, thus RCTs originally published in nonChinese and non-English languages may have been included in our analyses. It would have been difficult and unnecessary for the study to include acupuncture SRs in all languages. The quality of research published in English was higher and a sensitivity analysis including only acupuncture SRs in English was performed. All 1840 RCTs were extracted from 119 acupuncture SRs in English, of which 34 (1.8%) RCTs were not recalled by searching the four key databases (CNKI, WanFang, PubMed, and CENTRAL). The 34 unrecalled RCTs were from 25 SRs, including 17 SRs each with 1 unrecalled RCT, 7 with 2 unrecalled RCTs and 1 with 3 unrecalled RCTs. The unrecalled rate of RCTs per SR ranged from 2.5% to 33.3%. A search of the four key databases produced at least 90% recall of included RCTs per SR in 93.3% (95% confidence interval [CI] 88.8%–97.8%) of SRs (Figure 1), meaning that the combined retrieval of four key databases might achieve a 90% recall of the RCTs included in an SR. The limited number of acupuncture SRs published in English led to the wide 95%CI of the proportion of 90% recall SRs which meant that a firm conclusion could not be drawn. Therefore, more acupuncture SRs published in English are needed to validate our findings. The selection of target databases was initially based on prior experience and also on the search frequency of the databases used in acupuncture SRs. We assumed that these target databases would retrieve the majority of acupuncture RCTs. This assumption was later confirmed since 99.3% of acupuncture RCTs were recalled by searching the six target databases, CNKI, WanFang, VIP, PubMed, CENTRAL, and EMbase. It was not the intention of our study to assess the impact of non-Chinese and non-English RCTs in an acupuncture SR. Kim et al. considered that searching acupuncture RCTs in Korean and Japanese was important when performing an SR on acupuncture and we sought to validate their idea by analyzing our existing data. We found that 92 out of 1227 SRs searched Korean or Japanese databases but only 24 SRs searched the four key databases in addition to Korean or Japanese databases. These 24 SRs included a total of 312 RCTs, only two (0.6%) of which were published in Korean and could be retrieved by CENTRAL. No RCT in Japanese or any other language was included. Ma et al. analyzed the number of acupuncture-related publications between 1995 and 2014 and also found that those in English and Chinese accounted for 94.4% of all and only 0.5% were published in Korean or Japanese. Kim et al. assessed the impact of trials recalled by Korean databases and journals in Cochrane reviews on acupuncture and found the inclusion of these trials did not change the direction of effect in limited relevant meta-analyses and the effect size was almost unchanged. Current evidence has shown that the number of acupuncture RCTs in Korean and Japanese may be limited and some are retrievable from English databases, such as CENTRAL. The omission of the limited number of Korean and Japanese studies may not affect the results of meta-analyses. Therefore, we advocate searching the four key databases to enable retrieval of most acupuncture RCTs when time and resources are limited. When time permits, an exhaustive search for all RCTs on acupuncture, including Korean, Japanese and other non-Chinese and non-English databases is advisable. Received: 17 August 2022 Revised: 29 September 2022 Accepted: 4 October 2022