[5]. Research studies, especially the Randomized Controlled Trials (RCTs) for positive interventions and Cohort studies, where interventional studies are not possible, are the major study designs employed to collect information on the effect of interventions and exposures on health-related outcomes [6]. Yet, the effect estimates from multiple studies may vary in intensity, direction of the effect, sample sizes and methodological intricacies. Systematic reviews (SR) and meta-analysis (MA) were introduced to address these concerns, to review, collate, analyse, estimate the pooled effects, and interpret and critically analyse all the existing studies under specific topic of interest [7,8]. In the level of evidence, meta-analysis of well-conducted RCTs has been placed at Level IA, the highest level of evidence [5], for answering any clinical or public health questions. Meta-analysis can be done on observational analytical studies (case-control, cohort studies) as well as on cross-sectional prevalence studies. This review enumerates the applications of SR and MA, steps to be followed while undertaking SR and MA, and common methodological issues encountered during the meta-analysis, which will be helpful for beginners in the evidence synthesis. Also, the review elaborately discusses the concept of heterogeneity, its impact on the meta-analysis estimates, techniques to measure heterogeneity and the strategies to address heterogeneity.

Meta-analysis and its applications
Meta-analysis has been defined as "The statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings" [9]. Meta-analysis is the statistical extension of the systematic review [10]. By applying appropriate statistical methods, MA helps to estimate the pooled effect size from multiple studies answering the same research question. It improves the statistical power with increased precision of the effect estimates [11,12]. A single estimate will always be better in terms of decision-making in the clinical and public health domains. By pooling the data from multiple studies on the same research question, meta-analysis informs the researchers, clinicians and policymakers whether there was any significant effect of the exposure/interventions on the outcomes, and if so, what is the direction and strength of the effect [13]. Such pooling of the data can also be done while estimating the burden of the diseases in terms of prevalence, indicating the application of meta-analysis in descriptive studies. Thus, meta-analysis can assess the distribution and determinants of health states and diseases, which are the essential components of epidemiology. The outcomes of a meta-analysis may enhance the accuracy of impact estimates, address issues that were not addressed by the individual studies, resolve disputes resulting from seemingly incongruent studies, and provide new hypotheses [14]. Overall, meta-analysis, preceded by systematic review, is the most valuable tool under EBM and EBPH

Steps in meta-analysis

All SR and MAs should follow the "Preferred Reporting Items for Systematic Reviews and Meta-Analyses" (PRISMA) or "Meta-analyses Of Observational Studies in Epidemiology" (MOOSE) guidelines for maintaining the objectivity and quality in the process [15,16]. The steps involved in conducting an SR and MA, including the tools to assist the process, are discussed in the subsequent section. [Figure 1]

Figure 1: Major steps in conducting the meta-analysis

Building a team:
Systematic review and meta-analysis require forming a group with a minimum of three researchers (reviewers), to enable independent data extraction and adjudication of the contradictions. The group should consist of expert(s) of the domain in which the research is being undertaken. For instance, SR and MA on the effect of a clinical intervention on neonatal mortality should include a neonatologist or a paediatrician. At-least one of the members should be well-versed with the stepwise process of the SR and MA and the statistical methods employed under various aspects of meta-analysis. These domain experts and methodological experts should have a consultative working through all SR and MA stages to achieve methodologically robust and policy-influencing outcomes.

Identifying and refining the research question
Research question is one of the most critical steps in the SR and MA since it drives the whole exercise. Answering the research question is the overall objective of the SR and MA. The research question may be initiated in a broader sense. A preliminary literature search and review should be undertaken to refine and ensure the research question is ethical, specific, feasible and novel [17]. The answer to this research question should be interesting for clinicians and policymakers, as it will have practical implications. While the domain expert can bring in the broader idea for the research question, the methodological expert can enable refining the research question by providing relevant literature and review techniques.

Determining the inclusion and exclusion criteria
Once the research question is finalised, specific inclusion and exclusion criteria for the studies to be included in the review shall be listed out [18]. These inclusion and exclusion criteria shall be made according to the components in the research questions. For interventional studies, the acronym PICO is used to denote the components [19,20]. It included Population in which the primary studies are conducted and on whom the investigators want to apply their findings, Intervention which needs to be evaluated, Comparator group which needs to be compared with the intervention (placebo-control or active control) and the Outcome, which is the effect of the intervention that needs to be assessed. For instance, El Dib et al. in their SR and MA on the effect of probiotics on depression and anxiety, defined their research questions in terms of PICO [21]. The criteria can be modified as PECO (Population, Exposure, Control, Outcomes) for the observational, analytical studies and only PO (Population and outcomes) for the descriptive, prevalence studies [22]. Other important aspects to be prespecified under the inclusion/exclusion criteria are the study designs, language of the articles, geography and the time period of the eligible studies [23,24].

Registration of the SR and MA protocol
The SR and MA protocol should contain the plan the reviewers intend to follow for the exercise. Protocol should consist of rationality for the SR and MA, research question, inclusion and exclusion criteria, databases included, search strategy, screening and data extraction plan, risk of bias assessments, data analysis and dissemination plans. The protocol needs to be registered prospectively with one of the registering organisations. Multiple registries such as International Prospective Register of Systematic Reviews (PROSPERO), Cochrane Collaboration, INPLASY, Registry of Systematic Reviews/Meta-Analyses in Research Registry, Open Science Framework Registries and protocols.io are available [25,26]. PROSPERO is the most widely used open-access database for registering the SR and MA protocols [27,28]. While the SR and MA on the interventional studies must be prospectively registered, it is also strongly recommended to register the SR and MA protocols on the observational studies. Guidelines on the reporting of SR and MA protocols are also available [26].

Systematic Search for the studies
The relevant and potentially eligible studies to answer the research question can be identified from the multiple databases. With 35 million citations, PubMed is the largest open-access search engine for studies undertaken in medical sciences [29]. It enables a search of studies indexed in MEDLINE and those deposited with PubMed Central. Other important databases include Scopus, EMBASE, Cochrane Central Register of Controlled Trials (CENTRAL), PsycInfo, CINAHL, OVID, Web of Science, ProQuest and EBSCOHost [30–37]. At least three primary databases appropriate for the research question need to be searched to identify the relevant studies [38]. Based on the pre-determined PICO criteria, a suitable search strategy needs to be formulated for each database, and the search must be undertaken. Google Scholar can be used as a secondary search source. Pre-print servers such as medRxiv, arXiv, bioRxiv, BioRN, ChemRxiv, and SSRN can also be searched for unpublished literature.[39–43] Hand search of the references in eligible primary research papers, reviews and relevant speciality journals can also be done [44,45]. MEDLINE, Embase and Web of Science as the primary databases with Google Scholar as the secondary search source has been reported to be the best combination of databases for retrieving the relevant studies [46]. Duplicates from multiple databases shall be deleted to have a unique list of studies [47,48]. Various tools are available to enable the systematic search process and removal of duplication. Ovid platform facilitates systematic search and downloading results from multiple databases such as Medline, Embase, Cochrane etc., and also assists in deduplication of the results obtained [49,50]. Other tools useful in the deduplication of the studies from multiple databases are Mendeley, EndNote, Rayyan and COVIDENCE. Among these, Mendeley and Rayyan are free-of-cost applications.

Selection of the studies and meta-analysis extraction
The screening process in SR and MA is done in two stages: Title-abstract screening and full-text review. Each unique study obtained after deduplication must be screened independently by at least two reviewers based on the title and abstract by applying the pre-determined eligibility criteria and identifying potential studies for full-text review. The full text of these eligible studies obtained from the title-abstract should be reviewed by two reviewers independently, applying the precise PICO eligibility criteria. Finally, the studies eligible for the data extraction need to be short-listed [51,52]. Studies for which full text could not retrieved should be recorded and excluded from the review. Online tools such as Covidence, Rayyan and {NESTED} Knowledge are available to enable the screening of eligible studies [53,54]. A data extraction sheet should be made according to the variables required for the research question. It should also include the basic demographic details reported by the studies. The data from the final eligible studies shall be extracted according to the data extraction sheet. There should be a plan to remove the contradictions and differences of opinion between the reviewers during the screening and data extraction process. Such strategies include adjudication by the third reviewer and consensus meetings between the independent extractors. With the advent of Artificial Intelligence (AI), platforms such as {NESTED} Knowledge facilitate the screening and data extraction process [55].

Other Machine Learning (ML) tools have also been explored in search, screening and extraction steps of the SR and MAs [56–62].

Risk of bias assessment
The quality of the eligible and included studies in the SR and MA shall be assessed using design-specific quality assessment tools. Cochrane risk-of-bias tool for randomised trials (RoB 2) for RCTs, New Castle Ottawa quality assessment scales for cohort and case-control studies, JBI quality assessment tool and NHLBI tools for prevalence and case series studies are a few of the most applied and recommended quality assessment tools. Based on the evaluation, the studies are declared as good or poor quality to be included/excluded in the sensitivity analysis.

Data analysis and software for analysis
Outcomes assessed and the type of outcome (rate, ratio, proportion, time to event, mean difference etc.,) should be mentioned [63,64]. Primary analysis plan, statistical methods used to determine the heterogeneity and the pooled estimate, any sensitivity analysis [65,66], and sub-group analysis planned, methods and tools to assess the risk of bias [67,68], and publication bias in the included studies shall be explicitly mentioned before initiating the SR and MA [69–72], and the same shall be executed at the data analysis stage. Software available for meta-analysis include R studio, STATA, RevMan 5 and Comprehensive Meta-Analysis (CMA). While RevMan 5 and CMA are menu-based applications, R Studio is a command-based application, and STATA is a combination of menu-based and command-based application. Depending on the heterogeneity levels in the study, fixed-effect model (FEM) or random-effects model (REM) can be applied [73]. As a rule of thumb, if the heterogeneity is substantially high (I² >50%), then REM can be applied. Publication bias in the MA can be explored by means of the funnel plot and Egger's test [69]. Recently, novel and better measures of publication bias, such as the Doi plot and LFK index, have been introduced [74,75]. The certainty assessment of the effect estimate from the meta-analysis needs to be assessed using standard tools such as "Grading of Recommendations, Assessment, Development, and Evaluations" (GRADE[76]) [77–79].

Presenting the results
The eligible studies and their characteristics (author, place of study, year of study, age, sex, outcomes of interest, total sample and key findings) shall be tabulated. A PRISMA flow chart denoting the reasons for the exclusion of studies at each stage of the screening needs to be presented. The pooled estimate of the primary outcome with its 95% confidence interval and heterogeneity statistics shall be reported along with a forest plot [80,81]. Risk of bias assessment of each domain must be presented in a table or figuratively, along with the overall quality of the study. All sensitivity and sub-group analyses must be reported. Publication bias depiction as funnel plots or Doi plots shall be included in the results [82,83]. The overall certainty of the pooled estimates from the SR and MA must be presented as very low, low, moderate or high [76].

Implications and recommendations
The pooled estimate, quality of the studies included, and the generalisability of the findings must be critically discussed with the existing evidence and practices. The strengths and limitations of the SR and MA must be articulated. Recommendations based on the pooled estimates and the certainty of the estimate obtained shall be made to the clinicians, public health professionals, policymakers and researchers.

Methodological heterogeneity comprises variations in the tools used to assess the outcomes, risk of bias and the research design used in the studies. Statistical heterogeneity refers to the variation in the magnitude of the effects measured between the studies [89]. Statistical heterogeneity is caused due to the clinical and/or methodological heterogeneity present in the study [64].

Effect of heterogeneity on the study estimates and interpretation
The meta-analysis aims to club the samples and effects obtained from multiple studies, critically analyse, and estimate a single effect with improved power. The single pooled estimate obtained from the meta-analysis is only as good as its generalisability to all populations, interventions and tools employed in the included studies. Heterogeneity poses a challenge in interpreting the pooled estimate from the meta-analysis [90]. Meta-analysis undertaken with studies that harbours high heterogeneity may be less interpretable in a broader sense [91].

Addressing heterogeneity
However, the mere presence of heterogeneity will not make the meta-analysis unnecessary or useless. The other school of thought is that heterogeneity is inevitable in meta-analysis, since the objective of meta-analysis itself is to pool the effects of multiple studies from different settings. This is more so when the meta-analysis addresses a broader question eg., the effect of a class of drug on a particular disease, rather than focussing on a single drug within the class [64]. In the past, almost one-third of the SR and MAs did not consider or address the problem of high heterogeneity between the included studies [92]. It is imperative that the investigators recognise the potential factors responsible for the heterogeneity at various levels, followed by adopting strategies to address the heterogeneity [92,93].

Clinical and methodological heterogeneity can be identified and addressed by restriction at the time of planning and identifying the eligible studies. This can be achieved by clearly defined research questions, and inclusion and exclusion criteria. For instance, "What is the effect of physician-administered cognitive behavioural therapy (CBT) on depression in adolescents without any comorbidities?". In this, the tool used for measuring depression should be the same and objective (Eg PHQ9), across the studies. However, restriction may be an overtly exclusionary concept. Another method to address this type of heterogeneity is sub-group analysis, which is planned a priori in the protocol. Sub-group analysis involves dividing and clubbing the studies into groups according to the potential and known factors causing heterogeneity [73,94,95]. For instance, grouping the study according to geography, which may be a proxy for the biological factor (genetics) and environmental factors, grouping the studies according to the age of the population (adults/adolescents/children), grouping the studies according to the type of intervention used (CBT by psychologist/CBT by primary care physician), grouping the studies according to the tool used for measuring the outcomes (PHQ 9/DASS) are few sub-group analyses which can be done to explain the heterogeneity.

Meta-regression is another method to identify and explain the heterogeneity. But, sub-group analysis and meta-regression can only be done for the known/potential factors of heterogeneity. Also, in the meta-analysis of RCTs, these methods suffer from the limitation that the explanatory factors based on which sub-group analysis and meta-regression are conducted were not randomised, unlike the actual intervention under study [88].

Statistical heterogeneity is the most mentioned heterogeneity while discussing the meta-analysis. It denotes the variations in the outcomes between the studies more than which can occur by chance. Various tests are available to determine the levels of heterogeneity between the studies. Cochran's Q test employs the chi-square distribution and applies a null hypothesis that outcomes across the included studies are same with only chance being the reason for any variations [96]. A p value of <0.05 is taken as the level of significance for rejecting the null hypothesis. However, it does not apply to meta-analysis involving few studies [97]. Tests to quantitatively measure the heterogeneity among the studies are also available. These have come from the assumption that heterogeneity always exists between the studies of a meta-analysis qualitatively. Hence, there is a necessity to quantify the heterogeneity [64], and determine the levels as low and high heterogeneity. Based on the following cut-offs of 25%, 50% and 75% of I² values, a low, moderate, or high heterogeneity is assigned to the studies included in the study. When the I² value is >50%, considerable heterogeneity can be attributed to the analysis [96]. In those meta-analyses with high heterogeneity, techniques such as sub-group analysis and meta-regression can be applied to identify and address the heterogeneity.

In a simulation of a model analysis created by the authors, the pooled risk ratio of an outcome following an intervention from multiple studies was estimated. The heterogeneity level was high (I²=87%) [Figure 2].

In order to identify the variable causing the heterogeneity, we undertook a sub-group analysis by clubbing the studies according to the age group (≥18 years and <18 years) included in the study. Sub-grouping by age led to the elimination of heterogeneity (I²=0%). [Figure 3]

Dropping the few studies with the effect estimate that appears to be outliers can also reduce the heterogeneity [64]. In order to identify the outliers, various tests such as Baujat plot, leave-one-out analysis, Graphical display of study heterogeneity (GOSH) plots, radial plot and diagnostic tests can be applied [98–102]. The outliers identified can be excluded, and the pooled estimate with the remaining studies can be determined, which might reduce the heterogeneity. In the following model, heterogeneity between the studies was found to be significantly high (I²=91%) [Figure 4].

Figure 2: Pooled risk ratio of an outcome among the study population

Figure 3: Pooled estimates and heterogeneity of the studies grouped according to the age

Figure 4: Pooled risk ratio of the outcomes from 14 studies

Based on the above methods, studies C and J were identified as outliers [Figures 5 and 6].

Figure 5: Outlier analysis: a) Leave one out analysis; b) Baujat plot; c) Radial plot

Figure 6: Graphical display of study heterogeneity (GOSH) plots

Sensitivity analysis conducted after removing the outliers eliminated the heterogeneity (I²=0%) [Figure 7].

Figure 7: Sensitivity analysis after removing the outliers (studies C & J)

Meta-regression is another way to explain the heterogeneity depicted by the bubble plots [94,103,104]. A model analysis depicts the pooled prevalence of a condition with high heterogeneity(I²=84%) [Figure 8].

Figure 8: Pooled prevalence of the outcomes from 29 studies

Meta-regression was undertaken where the prevalence was plotted against the mean age of the study participants and the proportion of males in the included studies [Figures S1 and S2]. While the mean age showed no significant relationship (beta=-0.03, p=0.41), the proportion of males have a direct relationship with the prevalence of the condition (beta=0.04, p=0.01) i.e., studies with a higher proportion of males had a higher prevalence of the condition. Thus, the proportion of male patients in the studies is a potential factor causing the heterogeneity in the pooled estimates from the studies.

Alternatively, when these techniques (sub-group analysis, meta-regression and sensitivity) cannot identify or eliminate the heterogeneity considerably, or the number of studies is not large enough to allow for sub-group analysis, then the random effects model can be applied in the meta-analysis to estimate the pooled outcome [96,105]. Yet, the random effects model is not an elixir for the heterogeneity problem. Caution must be taken while generalising the results to the broader population [91], with robust measures and statistics provided by the investigators to further address the high heterogeneity in their study. The most commonly used method to undertake the random-effects model is the DerSimonian-Laird method (DL method) [105]. However, it is not an optimal method, especially when there is a low number of studies and high heterogeneity. Alternative and better approaches such as the Knapp-Hartung, Likelihood estimates, and Bayesian models can be applied to estimate the pooled outcomes by the random effects model [106,107]. Inthout et al., in their simulation, found that Hartung-Knapp-Sidik-Jonkman method of the random effects model had more adequate error rates than the DL method [108]. An increasingly large number of meta-analyses are applying the Bayesian models to improve the robustness of the random effects analysis [109].

Additionally, statistics describing the quantum of variation between the studies, tau² (τ²), may be calculated, and the prediction interval (PI) can be provided along with the pooled outcome estimate and other heterogeneity statistics. PI can assist in better presenting the pooled effect estimates in the meta-analysis of studies that harbour high heterogeneity [110]. It helps the readers understand and interpret the extent of the heterogeneity in the same measure as the study's outcome (prevalence, risk ratio etc.,), unlike I² [110]. PI also estimates the range within which the outcomes of 95% of the studies (similar to the ones included in the index meta-analysis) conducted in future will occur [111]. Sahoo et al. in their SR and MA estimated the pooled prevalence of suicidal ideation among adolescents in India to be 11% but with a high heterogeneity (I²=98%). This level of heterogeneity was expressed as PI in terms of the outcome measure (proportion of suicidal ideation), ranging from 2% to 45% [22]. This will enable the readers to understand the extent of variation existing in the studies and arrive at a meaningful interpretation of whether to take the pooled estimate as a reliable measure or with caveats [64,105].

7. Kang H. Use, application, and interpretation of systematic reviews and meta-analyses. Korean J Anesthesiol [Internet]. 2021 Oct 1 [cited 2023 Jan 18];74(5):369. Available from: /pmc/articles/PMC8497910/

8. Methods - Application of Systematic Review Methodology to the Field of Nutrition - NCBI Bookshelf [Internet]. [cited 2023 Jan 18]. Available from: https://www.ncbi.nlm.nih.gov/books/NBK44075/

9. Glass G V. Primary, Secondary, and Meta-Analysis of Research. Educ Res. 1976 Nov;5(10):3.

10. Paul M, Leibovici L. Systematic review or meta-analysis? Their place in the evidence hierarchy. Vol. 20, Clinical microbiology and infection : the official publication of the European Society of Clinical Microbiology and Infectious Diseases. England; 2014. p. 97–100.

11. Ahn E, Kang H. Introduction to systematic review and meta-analysis. Korean J Anesthesiol. 2018 Apr;71(2):103–12.

12. Kang H. Statistical considerations in meta-analysis. Hanyang Med Rev. 2015;35(1):23–32.

13. George BJ, Aban IB. An application of meta-analysis based on DerSimonian and Laird method [Internet]. Journal of Nuclear Cardiology. Springer; 2016. Available from: https://link.springer.com/article/10.1007/s12350-015-0249-6

14. Haidich AB. Meta-analysis in medical research. Hippokratia [Internet]. 2010 [cited 2023 Jan 6];14(Suppl 1):29. Available from: /pmc/articles/PMC3049418/

15. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000 Apr;283(15):2008–12.

16. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ [Internet]. 2021 Mar [cited 2021 Dec 27];372:n71. Available from: https://www.bmj.com/content/372/bmj.n71

17. Tawfik GM, Dila KAS, Mohamed MYF, Tam DNH, Kien ND, Ahmed AM, et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health. 2019;47:46.

18. Uman LS. Systematic Reviews and Meta-Analyses. J Can Acad Child Adolesc Psychiatry [Internet]. 2011 [cited 2023 Jan 18];20(1):57. Available from: /pmc/articles/PMC3024725/

19. Huang X, Lin J, Demner-Fushman D. Evaluation of PICO as a Knowledge Representation for Clinical Questions. AMIA Annu Symp Proc [Internet]. 2006 [cited 2023 Jan 18];2006:359. Available from: /pmc/articles/PMC1839740/

20. Abbade LPF, Wang M, Sriganesh K, Mbuagbaw L, Thabane L. Framing of research question using the PICOT format in randomised controlled trials of venous ulcer disease: a protocol for a systematic survey of the literature. BMJ Open [Internet]. 2016 Nov 1;6(11):e013175. Available from: http://bmjopen.bmj.com/content/6/11/e013175.abstract

21. El Dib R, Periyasamy AG, de Barros JL, França CG, Senefonte FL, Vesentini G, et al. Probiotics for the treatment of depression and anxiety: A systematic review and meta-analysis of randomised controlled trials. Clin Nutr ESPEN. 2021 Oct 1;45:75–90.

22. Sahoo S, Yadav S, Shamim MA, Nisha S, Sandeep M, Padhi BK, et al. Prevalence of suicidal behavior in adolescents in India: A Systematic review and Meta-analysis. Asian J Psychiatr [Internet]. 2023;86(August):103661. Available from: https://www.sciencedirect.com/science/article/pii/S1876201823002174

23. Gandhi AP, Gupta PC, Padhi BK, Sandeep M, Suvvari TK, Shamim MA, et al. Ophthalmic Manifestations of the Monkeypox Virus: A Systematic Review and Meta-Analysis. Vol. 12, Pathogens. 2023.

24. Gandhi P A, Patro SK, Sandeep M, Satapathy P, Shamim MA, Kumar V, et al. Oral manifestation of the monkeypox virus: a systematic review and meta-analysis. eClinicalMedicine [Internet]. 2023 Feb 1 [cited 2023 Jan 9];56:101817. Available from: http://www.thelancet.com/article/S2589537022005466/fulltext

25. Pieper D, Rombey T. Where to prospectively register a systematic review. Syst Rev [Internet]. 2022;11(1):8. Available from: https://doi.org/10.1186/s13643-021-01877-1

26. Systematic Review Protocols and Protocol Registries | NIH Library [Internet]. [cited 2023 Jun 16]. Available from: https://www.nihlibrary.nih.gov/services/systematic-review-service/systematic-review-protocols-and-protocol-registries

27. PROSPERO [Internet]. [cited 2023 Jan 6]. Available from: https://www.crd.york.ac.uk/prospero/

28. Booth A, Stewart L. Trusting researchers to use open trial registers such as PROSPERO responsibly. Vol. 347, BMJ (Clinical research ed.). England; 2013. p. f5870.

29. PubMed [Internet]. [cited 2023 Jan 18]. Available from: https://pubmed.ncbi.nlm.nih.gov/

30. Welcome - Embase [Internet]. [cited 2023 Jan 18]. Available from: https://www.embase.com/landing?status=yellow

31. ProQuest | Better research, better learning, better insights. [Internet]. [cited 2023 Jan 18]. Available from: https://www.proquest.com/

32. EBSCOhost Research Platform | EBSCO [Internet]. [cited 2023 Jan 18]. Available from: https://www.ebsco.com/products/ebscohost-research-platform

33. CINAHL Complete | EBSCO [Internet]. [cited 2023 Jan 18]. Available from: https://www.ebsco.com/products/research-databases/cinahl-complete

34. Scopus preview - Scopus - Welcome to Scopus [Internet]. [cited 2023 Jan 18]. Available from: https://www.scopus.com/

35. Ovid: Welcome to Ovid [Internet]. [cited 2023 Jan 18]. Available from: https://ovidsp.ovid.com/

36. Clarivate [Internet]. [cited 2023 Jan 18]. Available from: https://access.clarivate.com/login?app=wos&alternative=true&shibShireURL=https:%2F%2Fwww.webofknowledge.com%2F%3Fauth%3DShibboleth&shibReturnURL=https:%2F%2Fwww.webofknowledge.com%2F&roaming=true

37. APA PsycInfo [Internet]. [cited 2023 Jan 18]. Available from: https://www.apa.org/pubs/databases/psycinfo

38. Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017 Sep;358:j4008.

39. arXiv.org e-Print archive [Internet]. [cited 2023 Jan 18]. Available from: https://arxiv.org/

40. Homepage | ChemRxiv | Cambridge Open Engage [Internet]. [cited 2023 Jan 18]. Available from: https://chemrxiv.org/engage/chemrxiv/public-dashboard

41. Home :: SSRN [Internet]. [cited 2023 Jan 18]. Available from: https://www.ssrn.com/index.cfm/en/

42. medRxiv.org - the pre-print server for Health Sciences [Internet]. [cited 2023 Jan 18]. Available from: https://www.medrxiv.org/

43. bioRxiv.org - the pre-print server for Biology [Internet]. [cited 2023 Jan 18]. Available from: https://www.biorxiv.org/

44. Handsearching - Systematic Reviews in the Health Sciences - Research Guides at Rutgers University [Internet]. [cited 2023 Jan 18]. Available from: https://libguides.rutgers.edu/c.php?g=337288&p=2269575

45. Vassar M, Atakpo P, Kash MJ. Manual search approaches used by systematic reviewers in dermatology. J Med Libr Assoc [Internet]. 2016 Jan 5 [cited 2023 Jan 18];104(4):302–304–302–304. Available from: https://jmla.pitt.edu/ojs/jmla/article/view/145

46. Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev [Internet]. 2017;6(1):245. Available from: https://doi.org/10.1186/s13643-017-0644-y

47. Rathbone J, Carter M, Hoffmann T, Glasziou P. Better duplicate detection for systematic reviewers: evaluation of Systematic Review Assistant-Deduplication Module. Syst Rev [Internet]. 2015;4(1):6. Available from: https://doi.org/10.1186/2046-4053-4-6

48. Deduplicating - Systematic Reviews, Scoping Reviews, and other Knowledge Syntheses - Guides at McGill Library [Internet]. [cited 2023 Jan 18]. Available from: https://libraryguides.mcgill.ca/knowledge-syntheses/deduplicating

49. Etchells E. Ovid. BMJ Evidence-Based Med [Internet]. 2000;5(3):70. Available from: https://ebm.bmj.com/content/5/3/70

50. Ovid is the world's most trusted medical research platform | Wolters Kluwer [Internet]. [cited 2023 Jun 19]. Available from: https://www.wolterskluwer.com/en-in/solutions/ovid

51. Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol [Internet]. 2019;19(1):132. Available from: https://doi.org/10.1186/s12874-019-0782-0

52. Polanin JR, Pigott TD, Espelage DL, Grotpeter JK. Best practice guidelines for abstract screening large‐evidence systematic reviews and meta‐analyses. Res Synth Methods [Internet]. 2019 Sep 1 [cited 2023 Jan 18];10(3):330. Available from: /pmc/articles/PMC6771536/

53. Covidence - Better systematic review management [Internet]. [cited 2023 Jan 6]. Available from: https://www.covidence.org/

54. Rayyan – Intelligent Systematic Review - Rayyan [Internet]. [cited 2023 Jan 6]. Available from: https://www.rayyan.ai/

55. Nested Knowledge – Transforming the systematic review and meta-analysis paradigm from static out-of-date PDFs to dynamic living, interactive web-based visuals. [Internet]. [cited 2023 Jun 19]. Available from: https://about.nested-knowledge.com/

56. Marshall IJ, Wallace BC. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. [Internet]. Systematic reviews England: BioMed Central; Jul 11, 2019 p. 163. Available from: /pmc/articles/PMC6621996/

57. Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. J Am Med Inform Assoc. 2016 Jan;23(1):193–201.

58. Marshall IJ, Noel-Storr A, Kuiper J, Thomas J, Wallace BC. Machine learning for identifying Randomized Controlled Trials: An evaluation and practitioner's guide. Res Synth Methods. 2018 Dec;9(4):602–14.

59. Wallace BC, Noel-Storr A, Marshall IJ, Cohen AM, Smalheiser NR, Thomas J. Identifying reports of randomised controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach. J Am Med Inform Assoc. 2017 Nov;24(6):1165–8.

60. Soto AJ, Przybyła P, Ananiadou S. Thalia: semantic search engine for biomedical abstracts. Bioinformatics. 2019 May;35(10):1799–801.

61. Mo Y, Kontonatsios G, Ananiadou S. Supporting systematic reviews using LDA-based document representations. Syst Rev. 2015 Nov;4:172.

62. Cohen AM, Smalheiser NR, McDonagh MS, Yu C, Adams CE, Davis JM, et al. Automated confidence ranked classification of randomised controlled trial articles: an aid to evidence-based medicine. J Am Med Inform Assoc. 2015 May;22(3):707–17.

63. Saldanha IJ, Lindsley KB, Money S, Kimmel HJ, Smith BT, Dickersin K. Outcome choice and definition in systematic reviews leads to few eligible studies included in meta-analyses: a case study. BMC Med Res Methodol [Internet]. 2020;20(1):30. Available from: https://doi.org/10.1186/s12874-020-0898-2

64. Chapter 10: Analysing data and undertaking meta-analyses | Cochrane Training [Internet]. [cited 2023 Jan 6]. Available from: https://training.cochrane.org/handbook/current/chapter-10

65. Riley RD, Sutton AJ, Abrams KR, Lambert PC. Sensitivity analyses allowed more appropriate and reliable meta-analysis conclusions for multiple outcomes when missing data was present. J Clin Epidemiol [Internet]. 2004 Sep 1 [cited 2023 Jan 18];57(9):911–24. Available from: http://www.jclinepi.com/article/S0895435604001143/fulltext

66. Marušić MF, Fidahić M, Cepeha CM, Farcaș LG, Tseke A, Puljak L. Methodological tools and sensitivity analysis for assessing quality or risk of bias used in systematic reviews published in the high-impact anesthesiology journals. BMC Med Res Methodol [Internet]. 2020;20(1):121. Available from: https://doi.org/10.1186/s12874-020-00966-4

67. Kennedy CE, Fonner VA, Armstrong KA, Denison JA, Yeh PT, O'Reilly KR, et al. The Evidence Project risk of bias tool: assessing study rigor for both randomised and non-randomised intervention studies. Syst Rev [Internet]. 2019;8(1):3. Available from: https://doi.org/10.1186/s13643-018-0925-0

68. Frampton G, Whaley P, Bennett M, Bilotta G, Dorne J-LCM, Eales J, et al. Principles and framework for assessing the risk of bias for studies included in comparative quantitative environmental systematic reviews. Environ Evid [Internet]. 2022;11(1):12. Available from: https://doi.org/10.1186/s13750-022-00264-0

69. Lin L, Chu H. Quantifying publication bias in meta-analysis. Biometrics. 2018 Sep;74(3):785–94.

70. Sedgwick P. What is publication bias in a meta-analysis? BMJ [Internet]. 2015 Aug 14 [cited 2023 Jan 18];351. Available from: https://www.bmj.com/content/351/bmj.h4419

71. Thornton A, Lee P. Publication bias in meta-analysis: its causes and consequences. J Clin Epidemiol [Internet]. 2000;53(2):207–16. Available from: https://www.sciencedirect.com/science/article/pii/S0895435699001614

72. Mathur MB, VanderWeele TJ. Sensitivity analysis for publication bias in meta-analyses. J R Stat Soc Ser C Appl Stat. 2020 Nov;69(5):1091–119.

73. Borenstein M, Higgins JPT. Meta-analysis and subgroups. Prev Sci. 2013 Apr;14(2):134–43.

74. Gandhi P., Satapathy P, Sarvesh R, Hermis AH, Sah R, Padhi BK. Comments on "Shigellosis in Southeast Asia: A systematic review and meta-analysis." Travel Med Infect Dis. 2023;54:102593.

75. Furuya-Kanamori L, Barendregt JJ, Doi SAR. A new improved graphical and quantitative method for detecting bias in meta-analysis. Int J Evid Based Healthc [Internet]. 2018;16(4):195–203. Available from: https://journals.lww.com/ijebh/fulltext/2018/12000/a_new_improved_graphical_and_quantitative_method.3.aspx

76. What is GRADE? | BMJ Best Practice [Internet]. [cited 2023 Jan 6]. Available from: https://bestpractice.bmj.com/info/toolkit/learn-ebm/what-is-grade/

77. Murad MH, Mustafa RA, Schünemann HJ, Sultan S, Santesso N. Rating the certainty in evidence in the absence of a single estimate of effect. Evid Based Med [Internet]. 2017 Jun 1 [cited 2023 Jan 18];22(3):85. Available from: /pmc/articles/PMC5502230/

78. Assess Certainty - Systematic Reviews and Meta-Analyses - Research Guides at Virginia Tech [Internet]. [cited 2023 Jan 18]. Available from: https://guides.lib.vt.edu/SRMA/assesscertainty

79. Assessing certainty of evidence | NHMRC [Internet]. [cited 2023 Jan 18]. Available from: https://www.nhmrc.gov.au/guidelinesforguidelines/develop/assessing-certainty-evidence

80. Forest Plot - an overview | ScienceDirect Topics [Internet]. [cited 2023 Jan 18]. Available from: https://www.sciencedirect.com/topics/medicine-and-dentistry/forest-plot

81. Dettori JR, Norvell DC, Chapman JR. Seeing the Forest by Looking at the Trees: How to Interpret a Meta-AnalysisForest Plot. Glob Spine J [Internet]. 2021 May 1 [cited 2023 Jan 18];11(4):614. Available from: /pmc/articles/PMC8119923/

82. Funnel Plot - an overview | ScienceDirect Topics [Internet]. [cited 2023 Jan 18]. Available from: https://www.sciencedirect.com/topics/medicine-and-dentistry/funnel-plot

83. Sterne JAC, Sutton AJ, Ioannidis JPA, Terrin N, Jones DR, Lau J, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ [Internet]. 2011 Jul 22 [cited 2023 Jan 18];343(7818). Available from: https://www.bmj.com/content/343/bmj.d4002

84. Siddaway AP, Wood AM, Hedges L V. How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses. Annu Rev Psychol. 2019 Jan;70:747–70.

85. Greco T, Zangrillo A, Biondi-Zoccai G, Landoni G. Meta-analysis: pitfalls and hints. Hear Lung Vessel [Internet]. 2013 [cited 2023 Jan 6];5(4):219. Available from: /pmc/articles/PMC3868184/

86. Veginadu P, Calache H, Gussy M, Pandian A, Masood M. An overview of methodological approaches in systematic reviews. J Evid Based Med [Internet]. 2022 Mar 1 [cited 2023 Jan 6];15(1):39–54. Available from: https://onlinelibrary.wiley.com/doi/full/10.1111/jebm.12468

87. Gandhi PA, Patro SK, Sandeep M, Satapathy S, Shamim MA, Kumar V, et al. Oral manifestation of the monkeypox virus: A systematic review and meta-analysis. eClinicalMedicine [Internet]. 2023;56:101817. Available from: https://www.thelancet.com/journals/eclinm/article/PIIS2589-5370(22)00546-6/fulltext

88. Murad MH, Montori VM, Ioannidis JPA, Jaeschke R, Devereaux PJ, Prasad K, et al. How to Read a Systematic Review and Meta-analysis and Apply the Results to Patient Care: Users' Guides to the Medical Literature. JAMA [Internet]. 2014;312(2):171–9. Available from: https://doi.org/10.1001/jama.2014.5559

89. Fletcher J. Clinical Epidemiology Notes: What is heterogeneity and is it important? BMJ Br Med J [Internet]. 2007 Jan 1 [cited 2023 Jan 18];334(7584):94. Available from: /pmc/articles/PMC1767262/

90. Sandercock P. The Authors Say: 'The Data Are Not So Robust because of Heterogeneity' – So, How Should I Deal with This Systematic Review? Cerebrovasc Dis [Internet]. 2011 May [cited 2023 Jan 18];31(6):615–20. Available from: https://www.karger.com/Article/FullText/326068

91. Imrey PB. Limitations of Meta-analyses of Studies With High Heterogeneity. JAMA Netw Open [Internet]. 2020;3(1):e1919325–e1919325. Available from: https://doi.org/10.1001/jamanetworkopen.2019.19325

92. Schroll JB, Moustgaard R, Gøtzsche PC. Dealing with substantial heterogeneity in Cochrane reviews. Cross-sectional study. BMC Med Res Methodol [Internet]. 2011 Feb [cited 2023 Jan 18];11:22. Available from: /pmc/articles/PMC3056846/

93. Ruppar T. Meta-analysis: How to quantify and explain heterogeneity? Eur J Cardiovasc Nurs. 2020 Oct;19(7):646–52.

94. Spineli LM, Pandis N. Exploring heterogeneity in meta-analysis: Meta-regression analysis. Am J Orthod Dentofac Orthop [Internet]. 2020 Oct 1 [cited 2023 Jan 18];158(4):623–5. Available from: http://www.ajodo.org/article/S088954062030408X/fulltext

95. Jansen JP. Heterogeneity and Subgroup Analysis in Network Meta-Analysis BT - Design and Analysis of Subgroups with Biopharmaceutical Applications. In: Ting N, Cappelleri JC, Ho S, Chen (Din) Ding-Geng, editors. Cham: Springer International Publishing; 2020. p. 369–85. Available from: https://doi.org/10.1007/978-3-030-40105-4_18

96. Sedgwick P. Meta-analyses: what is heterogeneity? BMJ [Internet]. 2015;350. Available from: https://www.bmj.com/content/350/bmj.h1435

97. Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002 Jun;21(11):1539–58.

98. Viechtbauer W, Cheung MW-L. Outlier and influence diagnostics for meta-analysis. Res Synth Methods [Internet]. 2010 Apr;1(2):112–25. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26061377

99. Willis BH, Riley RD. Measuring the statistical validity of summary meta-analysis and meta-regression results for use in clinical practice. Stat Med. 2017 Sep;36(21):3283–301.

100. Baujat B, Mahé C, Pignon J-P, Hill C. A graphical method for exploring heterogeneity in meta-analyses: application to a meta-analysis of 65 trials. Stat Med. 2002 Sep;21(18):2641–52.

101. Bowden J, Spiller W, Del Greco M F, Sheehan N, Thompson J, Minelli C, et al. Improving the visualisation, interpretation and analysis of two-sample summary data Mendelian randomisation via the Radial plot and Radial regression. Int J Epidemiol. 2018 Aug;47(4):1264–78.

102. Matsushima Y, Noma H, Yamada T, Furukawa TA. Influence diagnostics and outlier detection for meta-analysis of diagnostic test accuracy. Res Synth Methods. 2020 Mar;11(2):237–47.

103. Baker WL, White CM, Cappelleri JC, Kluger J, Coleman CI. Understanding heterogeneity in meta-analysis: the role of meta-regression. Int J Clin Pract. 2009 Oct;63(10):1426–34.

104. Introduction - Meta-regression Approaches - NCBI Bookshelf [Internet]. [cited 2023 Jan 18]. Available from: https://www.ncbi.nlm.nih.gov/books/NBK43897/

105. Serghiou S, Goodman SN. Random-Effects Meta-analysis: Summarising Evidence With Caveats. JAMA [Internet]. 2019;321(3):301–2. Available from: https://doi.org/10.1001/jama.2018.19684

106. Cornell JE, Mulrow CD, Localio R, Stack CB, Meibohm AR, Guallar E, et al. Random-effects meta-analysis of inconsistent effects: a time for change. Ann Intern Med. 2014 Feb;160(4):267–70.

107. Aert RCM Van, Jackson D. A new justification of the Hartung‐Knapp method for random‐effects meta‐analysis based on weighted least squares regression. Res Synth Methods [Internet]. 2019; Available from: https://onlinelibrary.wiley.com/doi/abs/10.1002/jrsm.1356

108. IntHout J, Ioannidis JPA, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Med Res Methodol [Internet]. 2014;14(1):25. Available from: https://doi.org/10.1186/1471-2288-14-25

109. Hackenberger BK. Bayesian meta-analysis now–let's do it. Croat Med J. 2020;61(6):564.

110. IntHout J, Ioannidis JPA, Rovers MM, Goeman JJ. Plea for routinely presenting prediction intervals in meta-analysis. BMJ Open [Internet]. 2016;6(7). Available from: https://bmjopen.bmj.com/content/6/7/e010247

111. Higgins JPT, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. J R Stat Soc Ser A Stat Soc. 2009 Jan;172(1):137–59.

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of Evidence Journals and/or the editor(s). Evidence Journals and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.