The Policy Studies Institute (PSI) organized a research seminar and practical training on meta-analysis at the Policy Studies Institute in Addis Ababa on 30 October 2026. The session was led by Gebretsadik Teshager Siltanu, a PhD Fellow at the University of Copenhagen and an Erasmus+ trainee at PSI. The seminar was entitled “Social Capital and Climate Change Adaptation: Evidence from a Meta-Analysis” and was followed by a practical training on how to conduct meta-analysis in the social sciences using recently developed techniques.
The programme was designed to introduce participants to the methodological foundations of meta-analysis, present the research paper, and provide practical experience in conducting the different stages of a meta-analysis. The training combined a research presentation, seminar discussion, methodological explanations, software-based exercises, and a final practical assignment.
The morning session began with a welcome and an introduction to the objectives of the seminar. Gebretsadik subsequently presented his research paper, “Social Capital and Climate Change Adaptation: Evidence from a Meta-Analysis.” The presentation was followed by an interactive seminar discussion and question-and-answer session, during which participants had an opportunity to engage with the research and discuss its methodological and substantive dimensions.
In the methodological session, the presenter explained that meta-analysis is the statistical analysis of an existing body of research literature, in which each reported estimate from the reviewed studies is treated as an observation in a new dataset. He noted that applied research often generates numerous findings on the same research question, but these findings may differ in their direction, magnitude, and statistical significance.
He explained that meta-analysis provides a systematic approach to synthesizing these findings and examining the factors that may explain differences across studies. It also enables researchers to investigate how research estimates are generated and how the selection of findings for publication may influence the available evidence.
The presenter further discussed the position of meta-analysis within the hierarchy of research evidence, referring to experimental and randomized controlled trials, cohort and panel studies, cross-sectional and case-control studies, case reports, and expert opinions. He emphasized that meta-analysis helps bring together findings from different types of studies to provide a broader assessment of the available evidence. He also highlighted its growing importance as an evidence standard increasingly required in policy research and decision-making, drawing on the works of Stanley and Doucouliagos (2012) and Irsova et al. (2023).
3. Overview of the Meta-Analysis Workflow
The presenter introduced a structured workflow consisting of twelve stages, beginning with research design and ending with reporting and archiving. He explained that these stages could be grouped into three broad phases: design, analysis, and synthesis.
3.1. Design Stages: Defining the Evidence Base
The first phase covered Steps 0–3, which determine what evidence will be included in the analysis.
Step 0: Protocol and Preregistration
The presenter explained that a meta-analysis should begin with a clearly defined research protocol. This involves specifying the research question, objectives, inclusion and exclusion criteria, methodological approach, and planned analytical procedures. Preregistration was presented as an important measure for establishing transparency and documenting the intended research process before the analysis begins.
Step 1: Developing the Search Query
Participants were introduced to the process of developing a systematic search strategy. The presenter explained that an effective search query should be designed to identify relevant studies addressing the research question. He emphasized the importance of using appropriate search terms and structuring search queries to ensure that the literature search is systematic and sufficiently comprehensive.
Step 2: Screening Studies Using PRISMA
The presenter discussed the screening process and introduced the PRISMA approach as a framework for documenting the identification, screening, eligibility assessment, and inclusion of studies. He explained that this stage helps researchers apply predetermined selection criteria consistently and transparently, while documenting the number of studies retained or excluded at each stage.
Step 3: Coding All Estimates
The next stage involved extracting and coding relevant information from the selected studies. The presenter explained that researchers should systematically record the estimates and other relevant information required for the analysis. He emphasized that coding all relevant estimates is essential for constructing the dataset used in the subsequent statistical procedures.
3.2. Analysis Stages: Estimating and Interpreting the Evidence
The second phase covered Steps 4–9, which focus on the statistical analysis and interpretation of the collected estimates.
Step 4: Effect Sizes and Conversion Using R
The presenter introduced the concept of effect sizes and explained the importance of selecting an appropriate effect-size measure based on the type of literature and research question. Participants were also introduced to the process of converting between different effect-size measures. The practical exercises were designed to help them understand how these procedures could be undertaken using R.
Step 5: Pooling Estimates and Constructing Forest Plots
Participants were introduced to the process of combining estimates through different statistical approaches, including fixed-effects (FE), random-effects (RE), and unrestricted weighted least squares (UWLS) models. The presenter explained how pooled estimates could be interpreted and how forest plots could be used to display individual study estimates alongside the overall synthesized result.
Step 6: Examining Heterogeneity
The presenter explained that findings from different studies may vary considerably and that researchers need to assess the extent of this variation. The session introduced heterogeneity analysis as a means of examining differences among study estimates and understanding the extent to which the findings were consistent or varied across the reviewed literature. Participants were also introduced to the interpretation of prediction intervals using R.
Step 7: Assessing Publication Bias
The training addressed publication bias and its potential implications for the conclusions of a meta-analysis. The presenter introduced the funnel plot as a tool for examining the distribution of study estimates and discussed statistical approaches for testing and correcting publication bias. Particular attention was given to the FAT-PET and PET-PEESE methods. He also introduced other approaches, including MAIVE, selection models, and p-based tests, explaining that these methods contribute to the assessment and correction of potential publication bias.
Steps 8 and 9: Dependence, Meta-Regression, and Model Averaging
The presenter further explained that meta-analysis may involve multiple estimates originating from the same study, creating potential dependence among observations. He introduced approaches for addressing this issue, including the use of cluster-robust methods (CHE). The session also covered meta-regression and Bayesian Model Averaging (BMA), highlighting their relevance to examining factors that may explain variations in research findings and assessing alternative model specifications.
The final phase covered Steps 10 and 11, which focus on producing and communicating the overall findings.
Step 10: Developing a Best-Practice Estimate
The presenter introduced the concept of a best-practice estimate as part of the synthesis process. This stage was presented as an important step in identifying and reporting estimates that account for methodological considerations and the quality of the available evidence.
Step 11: Reporting and Archiving
The final stage focused on reporting the findings and archiving the research process. The presenter emphasized the importance of documenting the methods, data, analytical procedures, and results in a transparent and accessible manner so that other researchers can examine and assess the work.
Practical Training Using R and RStudio
The afternoon session focused on the practical application of the methods introduced in the morning. Participants were guided through the use of R and RStudio and the relevant software packages for conducting meta-analysis.
During the coffee break, participants were asked to verify their software installations rather than undertake new installations. They were instructed to open the shared training file, training_lab.R, and run the command library(metafor) to confirm that the required package was functioning properly. They were also asked to check whether the dataset, lab10.csv, could be opened using the command read.csv("lab10.csv").
Participants who encountered technical difficulties were encouraged to seek assistance from the R support person during the break. A backup cloud-based project, with the necessary software and packages already installed, was also made available for those whose systems were not functioning.
Following the lunch break, the presenter reviewed the morning's sessions before proceeding with the practical exercises. Participants then worked through the stages of coding estimates, calculating and converting effect sizes, pooling estimates, constructing forest plots, examining heterogeneity, and assessing publication bias using R.
The practical exercises were intended to help participants move beyond theoretical understanding and develop the technical capacity to implement the different stages of a meta-analysis using statistical software.
The presenter outlined several competencies that participants were expected to acquire by the end of the training. These included the ability to design a meta-analysis from the initial research question and protocol through to the development of search queries, screening procedures, and data coding.
Participants were also expected to develop the ability to select appropriate effect-size measures for their respective research areas and convert between different effect sizes. In addition, they were introduced to the procedures for pooling estimates using fixed-effects and random-effects models, interpreting forest and funnel plots, and assessing heterogeneity and prediction intervals using R.
The training further aimed to equip participants with the knowledge to test and address publication bias using FAT-PET and PET-PEESE, while understanding the contributions of other approaches such as MAIVE, selection models, and p-based tests. Participants were also expected to understand how dependence among estimates, meta-regression, model averaging, and best-practice estimates contribute to completing a rigorous meta-analysis.
The training concluded with a practical exercise in which participants were asked to develop a preliminary meta-analysis protocol. This exercise was intended to consolidate the knowledge and skills acquired throughout the day by requiring participants to apply the principles of research-question formulation, protocol development, search strategy design, and methodological planning.
Overall, the seminar and practical training provided participants with a structured introduction to the principles, procedures, and recently developed techniques of meta-analysis in the social sciences. By combining a research seminar with hands-on exercises, the programme demonstrated how researchers can systematically synthesize existing empirical evidence, assess variations across studies, examine publication bias, and produce more reliable estimates.
The training also strengthened participants' practical exposure to R and its application in meta-analytical research. Its emphasis on transparent protocols, systematic literature searches, rigorous screening, appropriate effect-size selection, statistical synthesis, and reproducible reporting highlighted the importance of methodological quality in evidence-based research. These competencies are particularly relevant to researchers and policy institutions seeking to generate credible, comprehensive, and policy-relevant evidence to inform decision-making.
The practical training continued with an in-depth introduction to the initial stages of conducting a meta-analysis, focusing on protocol development and preregistration, the construction of systematic search queries, screening and recording studies using the PRISMA approach, and coding research estimates. The facilitator, Gebretsadik Teshager Siltanu, explained the methodological requirements of each stage and guided participants through practical group exercises to strengthen their understanding and application of the techniques.
The facilitator explained that the first step in conducting a meta-analysis is the development of a written and time-stamped research protocol. He noted that the protocol should establish the research question, inclusion criteria, planned moderators, and analytical methods before researchers examine the data. This approach helps distinguish between analyses specified in advance and those undertaken later as exploratory exercises.
He emphasized that researchers may make deviations from the original protocol when necessary, but such changes must be transparently disclosed. He further explained that preregistration helps minimize hindsight bias by preventing researchers from changing methodological decisions in response to observed results.
Regarding the practical implementation, the facilitator recommended structuring the protocol in line with PRISMA guidance and registering it on the Open Science Framework (OSF). He stressed that researchers should establish their methodological rules in advance and should never make inclusion or analytical decisions conditional on the results.
Following the explanation, participants undertook a 12-minute group exercise, working in groups of two to four, to formulate research questions using the PICO framework. They were instructed to develop a one-sentence research question incorporating four essential elements: Population, variable of Interest, Comparison, and Outcome. The exercise encouraged participants to formulate clear and focused research questions that could guide the subsequent stages of a meta-analysis.
The facilitator subsequently introduced the process of developing a systematic and reproducible literature search query. He explained that a search query should be a clearly documented search string that can be rerun by other researchers. For each concept in the research question, relevant synonyms should be connected using the Boolean operator OR, while the different conceptual blocks should be combined using AND.
He indicated that researchers could conduct their searches in major academic databases such as Scopus and Web of Science, while also noting the growing use of Google Scholar as a search platform. He emphasized that the search query determines the evidence base of a meta-analysis and that omitting relevant synonyms may unintentionally narrow the scope of the literature. Such omissions, he observed, cannot be corrected through subsequent statistical procedures.
To improve the comprehensiveness of the search strategy, the facilitator recommended identifying approximately five anchor studies that the search query must retrieve. Researchers should then refine the search terms until these studies appear in the search results. He also encouraged participants to identify relevant synonyms, use wildcard characters, examine the reference lists of relevant studies through snowballing, and rerun the search before submitting the final research report.
The facilitator further introduced a scale-check guideline, indicating that a meta-analysis should aim to include at least 30 estimates drawn from a minimum of 10 studies. He presented this as a practical consideration when assessing whether the available literature would provide a sufficient evidence base for the proposed analysis.
Using the research on social capital and climate change adaptation as an example, he demonstrated how a search query could combine concepts related to social capital, climate adaptation, climate-related events, and household or agricultural units of analysis. The illustrative query included terms such as social capital, social networks, trust, group membership, cooperation, adaptation, coping, risk management, diversification, climate, drought, flood, rainfall, households, farms, and pastoralists.
Practical Exercise: Constructing a Search Query
Participants then undertook a 15-minute group exercise to develop their own search queries based on the PICO research questions formulated in Step 0. They were instructed to:
1. Use the research question developed during the previous exercise.
2. Develop one OR-block for each element of the question, incorporating three to six relevant search terms and wildcard variations.
3. Combine the conceptual blocks using AND to produce a single search query.
4. Identify two anchor studies that the proposed query should retrieve.
The facilitator emphasized that a search query should be logically structured and defensible in every component, rather than merely appearing technically sophisticated.
Screening and Recording Studies Using PRISMA
The next session focused on screening and recording studies using the PRISMA approach. The facilitator explained that screening involves making systematic decisions about whether individual research records should be included in or excluded from the meta-analysis. He stressed that researchers must document the reason for every exclusion.
He explained that transparent and neutral screening is essential for reducing selection bias during the design stage. The screening record also provides the information required to construct a PRISMA flow diagram, which is commonly expected in academic publications.
The facilitator outlined two main stages of screening: the initial assessment of titles and abstracts, followed by a full-text assessment of the studies that pass the first stage. He emphasized that the inclusion and exclusion criteria must be established in writing before screening begins.
He also cautioned participants against excluding studies based on their findings, perceived quality, or publication outlet. Instead, he recommended coding methodological quality and examining it as a potential moderator during the analysis. This approach helps avoid subjective selection decisions and preserves relevant evidence for subsequent assessment.
Practical Exercise: Screening Four Research Abstracts
To demonstrate the application of the screening criteria, participants undertook a 12-minute group exercise involving four sample research abstracts. The facilitator provided five criteria for assessing eligibility: the study should focus on the household or farm level, use quantitative methods, include at least one social capital variable as a regressor, examine a climate adaptation outcome, and report a usable statistical estimate.
The four abstracts represented different research designs and levels of analysis:
· Abstract A examined the relationship between cooperative density and irrigation investment using district-level panel data.
· Abstract B investigated the association between farmer-group membership and the adoption of drought-tolerant seeds using survey data from 412 farm households and a probit model.
· Abstract C explored how trust influenced recovery following the 2019 floods through 25 key-informant interviews.
· Abstract D examined the relationship between a social capital index and the number of climate-smart practices adopted by 1,742 households using a panel dataset and a Poisson model.
Participants were asked to discuss each abstract in their groups and decide whether it should be included or excluded, providing the specific eligibility criterion supporting each decision.
The facilitator subsequently reviewed the exercise and clarified the appropriate decisions. Abstract A was excluded because it used district-level rather than household- or farm-level data. Abstract B was included because it met the relevant quantitative and substantive criteria. Abstract C was excluded because it relied on qualitative interviews rather than quantitative analysis. Abstract D was included because it examined household-level climate adaptation outcomes using a quantitative approach. He also noted that the confidence interval reported for Abstract D could be inverted to recover the standard error required for the analysis.
Coding All Estimates Verifiably
Following the lunch break, the training resumed with a review of the morning's sessions before moving to the practical stages of meta-analysis using R. The facilitator began by revisiting the screening exercise and confirming the inclusion and exclusion decisions for the four abstracts.
He then introduced the third step, which focused on systematically coding all estimates extracted from the selected studies. He explained that the meta-analysis dataset should contain one row for each reported estimate and one column for each relevant characteristic being coded. These characteristics should be clearly defined in a written codebook to ensure consistency and transparency.
The facilitator emphasized the importance of coding all relevant estimates rather than selecting only the estimates that researchers consider preferable. He explained that preferred estimates could be reweighted or examined through alternative analytical procedures at a later stage, whereas estimates excluded during coding would no longer be available for subsequent analysis. Furthermore, retaining variation within individual studies is important for conducting moderator analysis and understanding differences in research findings.
Regarding the coding procedure, he recommended preparing the codebook before beginning data extraction and including at least ten moderators. He also emphasized that every coded value should be traceable to its original source through the corresponding page number. To improve reliability, he recommended using two independent coders or conducting a blind re-extraction of a sample of the coded data.
As an example of coding reliability, the facilitator referred to the case study dataset, which contained 283 estimates and achieved 99.3% agreement between coders. This example demonstrated the importance of systematic coding procedures and verification in producing a reliable meta-analysis dataset.
Practical Exercise: Extracting and Coding Two Estimates
Participants subsequently undertook a 12-minute group exercise to practice extracting statistical information and preparing coding entries from research studies.
In the first exercise, participants were provided with an example in which a study reported that group membership had a coefficient of 0.14, a standard error of 0.056, and a sample size of 412, with nine regressors. They were asked to prepare the corresponding coding row and identify the relevant t-statistic and degrees of freedom.
In the second exercise, participants were presented with another study reporting a coefficient of 0.09, marked with two asterisks, based on a sample of 200 observations and five regressors. They were asked to identify the appropriate conversion class and determine the conservative t-statistic.
These exercises were designed to familiarize participants with extracting statistical information from published studies, interpreting reported estimates, and organizing the information in a standardized dataset for subsequent meta-analysis.
The sessions provided participants with practical knowledge of the first three critical stages of conducting a meta-analysis: developing a transparent research protocol, constructing a reproducible literature search strategy, and screening and coding research estimates systematically.
Through the combination of methodological explanations and group exercises, the facilitator demonstrated how researchers can reduce selection bias, improve the transparency of research procedures, and establish a reliable evidence base for statistical synthesis. Particular emphasis was placed on preregistration, the use of PRISMA, systematic application of inclusion and exclusion criteria, comprehensive coding, and verification of extracted estimates.
The exercises also encouraged participants to apply the techniques to their own research questions and to develop practical skills that could be used in future evidence-synthesis studies. These competencies are important for strengthening the methodological rigor, reproducibility, and policy relevance of social science research.
The practical training continued with an in-depth examination of the analytical stages of meta-analysis, focusing on data coding, the selection and conversion of effect sizes, statistical pooling, heterogeneity analysis, and the assessment of publication bias. Gebretsadik guided participants through the methodological principles underlying each stage and demonstrated their practical application using R and the metafor package. The sessions combined technical explanations, illustrative research data, statistical formulas, and computer-based exercises.
The facilitator began by demonstrating how information extracted from individual research studies should be organized in a meta-analysis dataset. Using Abstract B from the earlier screening exercise, he presented an example of a coded row containing the study identification number, estimate identification number, statistical results, methodological characteristics, and source references.
He explained that each reported estimate should occupy one row in the dataset, while the columns should capture the relevant statistical and study characteristics. For example, the estimate from Abstract B was assigned study ID 214 and estimate ID 3. Its reported coefficient, standard error, sample size, and number of regressors were recorded as 0.14, 0.056, 412, and 9, respectively, based on Table 3 on page 11 of the original study.
The presenter demonstrated how the t-statistic and degrees of freedom could be calculated from the reported information. The t-statistic was obtained by dividing the coefficient by its standard error, producing 2.50, while the degrees of freedom were calculated by subtracting the number of regressors and one from the sample size, resulting in 402.
He further explained that the dataset should capture substantive and methodological characteristics, including the form of social capital, the type of climate adaptation outcome, the estimation method, the data structure, the study setting, and the country and income context. In the illustrative case, social capital was classified as bridging social capital because the explanatory variable concerned group membership, while the adaptation outcome was classified as incremental adaptation because it involved the adoption of improved seeds. The study used a probit estimator and cross-sectional data and examined drought-related adaptation in Ethiopia, classified as a low-income setting.
The facilitator placed particular emphasis on recording the exact page reference for every coded value. He explained that traceability enables researchers to verify extracted information and strengthens the reliability of the dataset. Referring to the case study, he indicated that the full dataset contained 2,886 estimate-level observations and 35 moderators, with every coded value traceable to its original page. He linked this systematic documentation to the 99.3% verification agreement achieved during the coding process.
The facilitator then introduced the fourth stage of meta-analysis, which involves selecting an appropriate effect-size measure. He explained that an effect size provides a common statistical metric into which estimates from different studies are converted before they can be pooled and compared.
He emphasized that the appropriate effect-size measure depends on the type of data and statistical results reported in the literature. He presented several examples:
· For two-group experiments involving continuous outcomes, Cohen's d or Hedges' g is commonly used, particularly in psychology and education.
· For binary outcomes and binary exposures presented in 2×2 tables, the log odds ratio is commonly used, particularly in medicine and epidemiology.
· For associations between continuous variables, Pearson's correlation coefficient (r) and Fisher's z are appropriate measures, particularly in psychology.
· For log-log regression models, elasticity is a preferred measure in economics.
· For mixed regression models, including probit, logit, OLS, and structural equation modelling, partial correlation coefficients (PCC) may be used as a last resort.
Using the social capital and climate change adaptation case study, the facilitator explained why partial correlation coefficients were selected. Since the reviewed literature included different estimation approaches, such as probit, logit, structural equation modelling, and count models, PCC provided a common measure that could be recovered across the 376 studies included in the case study.
He also stressed that researchers must convert the variance alongside every point estimate. In other words, converting an effect-size estimate without converting its corresponding variance would compromise the subsequent statistical analysis. He cautioned participants against transferring interpretive benchmarks from one effect-size family to another, explaining that a value considered moderate for Cohen's d might have a different interpretation when expressed as a correlation coefficient.
Converting Between Effect-Size Measures
The facilitator proceeded to demonstrate how researchers could convert different statistical measures into a common effect-size metric. He presented several conversion procedures and explained the assumptions underlying each method.
For estimates reported as t-statistics, he demonstrated that the partial correlation coefficient could be calculated using the t-statistic and degrees of freedom. The corresponding variance could then be derived from the partial correlation coefficient and degrees of freedom.
He also introduced the conversion of Cohen's d into a correlation coefficient, explaining that this procedure assumes equal group sizes in a two-group design with a continuous outcome. In addition, he demonstrated how Cohen's d could be converted into a log odds ratio under an assumed logistic outcome distribution.
Other conversion procedures included approximating a correlation coefficient from an odds ratio, converting Pearson's correlation coefficient into Fisher's z, and converting Spearman's rank correlation into Pearson's correlation under a bivariate normality assumption.
The presenter also discussed the approximation of a correlation coefficient from standardized regression coefficients, referring to the Peterson-Brown approach. He cautioned that such conversions involve assumptions and should be applied carefully.
He compared these conversions to exchange rates, emphasizing that researchers must understand the assumptions and limitations associated with each conversion before relying on the resulting estimates. He further indicated that the different conversion procedures would be applied during the practical R session.
The Five Routes for Extracting and Converting Estimates
To help participants understand how different types of reported statistical information could be incorporated into a common dataset, the facilitator introduced five extraction routes for converting reported estimates into partial correlation coefficients.
The first route applied to studies reporting a regression coefficient and its standard error, t-statistic, or p-value. Participants were shown how to calculate the t-statistic from the coefficient and standard error and subsequently convert it into a partial correlation coefficient using the relevant degrees of freedom. The case study included 2,436 estimates in this category.
The second route applied to studies reporting exact p-values or statistical significance stars. The presenter explained that researchers could recover the corresponding t-statistic by inverting the reported p-value and then follow the first conversion route. This category contained 162 estimates.
The third route involved studies reporting standardized beta coefficients. The facilitator introduced the Peterson-Brown approximation for converting standardized beta coefficients into correlation coefficients. This category contained 144 estimates.
The fourth route covered standardized path coefficients reported in structural equation modelling studies, which were treated as correlation coefficients for the purposes of the case study. This category contained 111 estimates.
The fifth route involved studies reporting zero-order correlation coefficients or odds ratios derived from contingency tables. The presenter explained that zero-order correlations could be used directly, while odds ratios could be converted into approximate correlation coefficients. This category contained 36 estimates.
He emphasized three important rules for the extraction process. First, researchers should record the conversion route used for every estimate because this allows them to conduct robustness checks restricted to particular extraction classes. Second, the variance calculation must correspond to the specific conversion route rather than applying a single formula to all estimates. Third, publication-bias tests should be conducted using the first class, which consists of directly reported estimates, while the first and second classes could be used to establish assumption-free bounds.
Returning to Abstract B, the facilitator confirmed that it belonged to the first extraction class because the study directly reported the regression coefficient and standard error. The previously calculated t-statistic of 2.50 and degrees of freedom of 402 therefore provided the basis for converting the estimate into a partial correlation coefficient.
Practical Exercise in R: Computing and Converting Effect Sizes
The facilitator then guided participants through an 18-minute practical exercise using their laptops. They were instructed to run the Step 4 section of the shared training_lab.R script, which reproduced the calculations they had previously completed manually during the data-coding exercise.
Participants were subsequently asked to adapt the R commands to calculate the partial correlation coefficient for a second estimate belonging to the second extraction class, with a t-statistic of 1.97, sample size of 200, and five regressors. They were also asked to calculate the effect size for Abstract D using the standard error derived from its reported confidence interval.
The facilitator provided R commands to demonstrate how the t-statistic and degrees of freedom could be used to calculate the partial correlation coefficient and its standard error. He also demonstrated the conversion of Cohen's d into a correlation coefficient and the approximation of a correlation coefficient from an odds ratio.
During the exercise, participants were encouraged to compare the three resulting partial correlation coefficients and discuss why the ratio of the partial correlation coefficient to its standard error should reproduce the original t-statistic. They were also asked to consider which degrees of freedom should be used when a study reports only the sample size.
Interpreting the Effect-Size Conversion Results
Following the practical exercise, the facilitator presented the expected output and explained its interpretation. For Abstract B, the original t-statistic of 2.50 was converted into a partial correlation coefficient of approximately 0.124, with a standard error of approximately 0.0495.
He explained that the partial correlation coefficient indicated a relatively small positive association between farmer-group membership and the adoption of drought-tolerant seeds, holding the study's nine control variables constant. He related the value to the interpretive benchmarks used in economics, placing it between the illustrative small and moderate effect-size thresholds.
The presenter further explained that the standard error represented the precision associated with the estimate and would determine the weight assigned to it during subsequent statistical pooling. He emphasized that the variance must always accompany the point estimate throughout the conversion process.
He also demonstrated that dividing the calculated partial correlation coefficient by its standard error reproduced the original t-statistic of 2.50. This result confirmed that the conversion procedure had been correctly implemented. Conceptually, he explained that the squared partial correlation coefficient represents the proportion of residual outcome variance explained by the explanatory variable, holding the other controls constant. This makes the measure unit-free and comparable across different estimation approaches.
Pooling Estimates Using Fixed-Effects and Random-Effects Models
The next session focused on the statistical pooling of estimates from different studies. The facilitator explained that pooling involves combining individual estimates into a weighted average and that the choice of statistical model determines how much weight each study receives.
He introduced two principal approaches: the fixed-effects model and the random-effects model.
Fixed-Effects Model
The presenter explained that the fixed-effects model assumes that all studies estimate one common underlying effect and that differences between their reported estimates arise only from sampling error. Under this model, each estimate is weighted by the inverse of its sampling variance. Consequently, studies with smaller standard errors and greater precision receive substantially higher weights.
He noted that this model may be appropriate when studies are essentially identical replications, although such conditions are relatively uncommon in social science research.
Random-Effects Model
The facilitator then introduced the random-effects model, explaining that it assumes the true effects may vary across studies because the studies are conducted in different settings and under different conditions. In addition to sampling error, the model accounts for variation in the underlying true effects.
He explained that the random-effects model incorporates both the sampling variance and the between-study variance, represented by tau-squared (τ²), when calculating weights. As the between-study variance increases, the differences in weights across studies become less pronounced, allowing studies with different levels of precision to contribute more evenly to the pooled estimate.
He noted that random-effects models are particularly relevant in social science research because findings often vary across institutional, geographical, and socioeconomic contexts.
Unrestricted Weighted Least Squares
The presenter also introduced unrestricted weighted least squares (UWLS), explaining that it preserves precision weighting without rewarding imprecise studies. He noted that this approach is also reported in economics alongside random-effects estimates.
He demonstrated that the R function rma() from the metafor package could be used to estimate the different pooling models.
Understanding Between-Study Variance and Heterogeneity
The facilitator subsequently explained how researchers estimate the between-study variance, τ², and assess the extent of heterogeneity in the literature.
He introduced Cochran's Q statistic as a measure of the disagreement among study estimates relative to what would be expected from sampling error alone. Using the practical training dataset, he reported a Q statistic of 47.6 with nine degrees of freedom, indicating substantial variation among the estimates.
He then demonstrated how the DerSimonian-Laird approach could be used to estimate the between-study variance by relating the excess Q statistic to a function of the study weights. In the training dataset, the estimated τ² was 0.0153.
The presenter further explained the I-squared (I²) statistic, which expresses the proportion of observed variation attributable to genuine between-study heterogeneity rather than sampling error. The practical dataset produced an I² value of approximately 81%, while the full case study had an I² of approximately 86%.
He noted that the metafor package also provides the restricted maximum likelihood (REML) estimator, which refines the estimation of between-study variance while retaining the same underlying concept.
Practical Exercise in R: Comparing Fixed-Effects and Random-Effects Models
Participants then undertook a 22-minute laptop-based exercise to compare fixed-effects and random-effects models using ten studies from the case study dataset. Before running the R commands, they were asked to predict how much weight a particularly precise study, with a standard error of 0.017 compared with 0.06–0.11 for the other studies, would receive under the fixed-effects model.
The facilitator guided participants through the process of loading the dataset, estimating the fixed-effects model and the REML random-effects model, and comparing the weights assigned to individual studies. They were also instructed to construct a forest plot to visualize the study-level estimates and pooled results.
In an additional exercise, participants were asked to remove the particularly precise study, identified as Batool et al. (2018), and rerun the fixed-effects model to assess how much the pooled estimate changed. They were encouraged to compare the sensitivity of the two models and discuss which studies gained weight under the random-effects model and why this might matter when considering publication selection.
Interpretation of the Model Results
The facilitator presented the results of the exercise, reporting a fixed-effects pooled estimate of 0.0267 with a standard error of 0.0142 and a random-effects pooled estimate of 0.0695 with a standard error of 0.0458.
The random-effects model produced a between-study variance of 0.0153 and an I² value of 81.1%, while Cochran's Q statistic was 47.59 with nine degrees of freedom and a p-value below 0.0001.
He explained that the two pooled estimates differed because the models assigned different weights to the studies. Under the fixed-effects model, Batool et al. (2018), which had a particularly precise estimate close to zero, received approximately 69.4% of the total weight. Under the random-effects model, its weight declined to approximately 13.5%, while the other studies received more balanced weights, ranging from approximately 8% to 11%.
The presenter explained that the fixed-effects model allowed the highly precise study to exert substantial influence on the pooled result, whereas the random-effects model gave greater consideration to the diversity of research settings represented in the literature.
He cautioned participants against treating either pooled estimate as automatically definitive. Given the substantial heterogeneity, he recommended reporting the random-effects estimate alongside its prediction interval and investigating the sources of heterogeneity before drawing substantive conclusions.
Interpreting Forest Plots
The facilitator then explained how to interpret the forest plot generated during the practical exercise. He drew participants' attention to the distribution of individual study estimates and their confidence intervals.
He noted that the plot contained one highly precise study with an estimate close to zero, six studies whose confidence intervals crossed zero, one significantly negative study identified as Nath et al., and three significantly positive studies.
He explained that the fixed-effects pooled estimate appeared as a relatively narrow diamond positioned close to the highly precise study, whereas the random-effects pooled estimate appeared as a wider diamond shifted towards the positive majority of the studies.
The presenter emphasized that researchers should examine the overall pattern of the forest plot before relying on a single pooled estimate. The plot provides a visual understanding of the direction, magnitude, precision, and variation of individual study findings and helps explain why different pooling models may produce different results.
Assessing Heterogeneity and Interpreting Prediction Intervals
The next session focused on heterogeneity and the distinction between confidence intervals and prediction intervals. The facilitator explained that heterogeneity refers to the extent to which true effects vary across studies and settings. He introduced three important measures: I², which represents the proportion of observed variation attributable to genuine heterogeneity; τ², which measures the variance of the true effects; and the prediction interval, which indicates the range within which the true effect of a new setting may plausibly fall.
Using the full case study, he reported an I² value of approximately 86.5% and emphasized that interpreting the pooled mean alone could be misleading when heterogeneity is substantial.
He explained that the confidence interval indicates the uncertainty surrounding the estimated average effect, whereas the prediction interval provides information about the range of effects that might be observed in a new setting. He stressed that this distinction is particularly important for policy research because policy interventions are implemented in specific geographical and institutional contexts rather than in an abstract average setting.
The facilitator presented the formula for calculating a 95% prediction interval, explaining that it incorporates the estimated between-study variance alongside the standard error of the pooled mean. He demonstrated how the R functions confint(re) and predict(re) could be used to obtain the relevant heterogeneity statistics and prediction intervals.
Practical Exercise: Interpreting Confidence and Prediction Intervals
Participants undertook a nine-minute group exercise to interpret the two types of intervals. They were first asked to verify the prediction interval manually using the random-effects estimate of 0.0695, the between-study variance of 0.0153, and the standard error of 0.0458. The resulting prediction interval was approximately −0.19 to 0.33.
They were then asked to interpret the full-study result, which reported a pooled mean of 0.040 and a 95% prediction interval ranging from −0.12 to 0.20. Participants were instructed to formulate a sentence explaining what this result meant for policymakers planning a climate adaptation programme in a particular district.
The exercise reinforced the importance of interpreting pooled estimates in relation to the uncertainty and contextual variation surrounding them.
Understanding and Assessing Publication Bias
The facilitator proceeded to the seventh stage, focusing on publication bias and the methods used to assess and correct it. He explained that publication bias arises when studies reporting statistically significant or favourable results have a greater likelihood of being published than studies reporting insignificant or less favourable findings.
He emphasized that publication bias can distort the overall conclusions of a meta-analysis by making the pooled effect appear larger than the underlying association. Referring to the case study, he indicated that approximately half to three-quarters of the published mean could reflect selection effects rather than the underlying association.
He explained that several statistical methods used to assess publication bias are based on the relationship between reported effect sizes and their standard errors. Under selection, reported effects may be correlated with their standard errors, and different methods examine this relationship using alternative statistical specifications.
FAT-PET
The facilitator introduced the Funnel Asymmetry Test and Precision Effect Test (FAT-PET) approach. He explained that the method examines the relationship between the reported effect size and its standard error. In the regression specification, the slope is used to assess selection effects, while the intercept provides the PET estimate of the corrected mean.
PET-PEESE
He subsequently introduced the PET-PEESE approach, explaining that it examines whether selection bias increases with the variance of the estimate when a genuine effect exists. He noted that when the PET intercept is statistically different from zero, the PEESE estimate may be reported as the corrected mean.
Other Methods for Assessing Publication Bias
The facilitator also introduced a broader set of methods, including kink, WAAP, MAIVE, selection models, p-uniform, and caliper approaches. He explained that these methods examine publication selection from different methodological perspectives and provide alternative ways of assessing and correcting potential bias. He recommended that researchers understand the different methodological families and select appropriate approaches rather than relying on a single test.
Practical Exercise in R: Testing for Publication Selection
The final practical exercise focused on constructing funnel plots and estimating FAT-PET and PEESE regressions using R. Participants were given 15 minutes to complete the exercise using the metafor package and the training dataset.
The facilitator demonstrated how to generate a funnel plot to examine the distribution of study estimates and how to run regression models assessing the relationship between partial correlation coefficients and their standard errors. Participants were also introduced to a weighted regression specification and a PEESE model using the squared standard errors.
They were encouraged to examine where publication selection might appear in the funnel plot and interpret the regression results. The group discussion also considered how researchers should interpret a finding of no detected publication bias in a meta-analysis containing only 12 studies.
The facilitator cautioned that statistical tests for publication bias have limited power when the number of studies is small. Using the training example, he reported that the FAT slope standard error was approximately 2.15 for the ten-study dataset, compared with a slope of 0.76 and a standard error of 0.29 in the full study. He emphasized that the failure to detect publication bias should not be interpreted as proof that publication bias is absent.
The sessions provided participants with practical exposure to the analytical procedures required to transform extracted research findings into a statistically comparable and systematically synthesized body of evidence. Through the use of R, the facilitator demonstrated how researchers can code and verify estimates, convert different effect-size measures, compare fixed-effects and random-effects models, interpret forest plots, assess heterogeneity, and examine potential publication bias.
A central message throughout the training was that the reliability of a meta-analysis depends not only on the statistical techniques employed but also on the quality, transparency, and traceability of the underlying data and methodological decisions. The facilitator therefore encouraged participants to retain all relevant estimates, document the conversion route for each estimate, match variances to their corresponding effect-size measures, and interpret pooled findings in light of heterogeneity and potential publication selection.
The practical exercises further demonstrated the value of using statistical software to make the analytical process more transparent, reproducible, and accessible. These skills are particularly important for social science researchers and policy institutions seeking to synthesize existing research and generate credible evidence to inform policy formulation and implementation.
The training continued with an examination of the advanced stages of meta-analysis, focusing on additional methods for assessing publication bias, addressing dependence among estimates, conducting moderator analysis and model averaging, developing best-practice estimates, and reporting research findings transparently. Gebretsadik Teshager Siltanu explained the methodological principles underlying these stages and demonstrated their application using the full case-study dataset and R-based analytical procedures.
Further Publication-Bias Estimators
The facilitator introduced additional methods for assessing and correcting publication bias, organizing them into five methodological families. He explained that researchers should not rely on a single statistical test because different methods examine potential selection effects from different perspectives.
The first family consisted of funnel-regression methods, including FAT-PET, PET-PEESE, and the kink estimator. He identified R's lm() function and metafor::regtest() as relevant entry points for implementing these methods. The case-study results reported estimates of 0.005, 0.025, and −0.001, respectively.
The second family involved instrumented estimation, particularly the Meta-Analytic Instrumental Variable Estimator (MAIVE), which uses the inverse sample size as an instrument for the squared standard error. The facilitator identified the MAIVE R package as the relevant analytical tool and presented a case-study estimate of −0.001, with a standard error of 0.014 and a first-stage F-statistic greater than 93,000.
The third family comprised methods designed to ensure adequate statistical power, including WAAP and Top10. These approaches assess estimates using considerations of statistical power and study precision. The facilitator reported that, for extraction class 1, 42 estimates out of the total 2,886 met the powered-estimate criterion in the case study.
The fourth family involved selection models, particularly the Andrews-Kasy approach, implemented through the metastudies and RobustVariance packages. He reported a case-study estimate of 0.048 expressed as a Fisher's z value and indicated that the assumption of independence was rejected.
The fifth family covered p-value-based approaches, including p-uniform* and caliper tests. The facilitator identified the puniform package and customized code as possible implementation routes. He reported a p-uniform* estimate of 0.126, noting that it was inflated by heterogeneity, and also referred to the application of caliper tests.
Drawing on Irsova et al., the facilitator emphasized a practical rule for publication-bias assessment: researchers should always undertake bias correction, apply at least one method from each methodological family, and report the results of all the methods used. He stressed that the overall conclusion should be based on the convergence of findings across different approaches rather than on any single statistical estimate. The R commands and packages introduced during the session were presented as starting points for applying these methods to researchers' own datasets.
Addressing Dependence Among Estimates
The facilitator subsequently introduced the issue of dependence among estimates, using the full case-study dataset to demonstrate why this methodological consideration is important.
He explained that the reviewed studies reported an average of 7.7 estimates each, with many estimates derived from the same households or underlying samples. Consequently, the estimates could not be treated as statistically independent observations.
He cautioned that treating dependent estimates as independent would lead to underestimated standard errors and confidence intervals that were artificially narrow. This could make research conclusions appear more certain and statistically robust than they actually were.
To address the problem, he introduced several approaches. The first was clustering estimates by study and using a wild bootstrap when the number of studies was fewer than 40. He then discussed a more comprehensive approach based on a correlated and hierarchical effects (CHE) working model, which accounts for variation at both the study and estimate levels.
The model incorporated study-level and estimate-level random effects, with the within-study correlation assumed to be 0.5. The facilitator also introduced CR2 robust inference as a method for obtaining more reliable statistical inference in the presence of clustered estimates.
Using R, he demonstrated how the variance-covariance structure could be constructed with the vcalc() function, specifying the standard errors, study identifiers, estimate identifiers, and assumed within-study correlation. He then showed how the multilevel model could be estimated using rma.mv() and how CR2 robust coefficient tests could be obtained through the clubSandwich package.
He emphasized that this procedure required the complete multi-estimate dataset, with one row for each estimate and a corresponding study identifier.
Participants undertook a brief two-minute exercise in which they were asked to vote on whether treating all 2,886 dependent estimates as independent observations would produce confidence intervals that were too narrow or too wide. The facilitator then invited participants to explain the reasoning behind their responses.
The exercise reinforced the importance of accounting for dependence among estimates to avoid overstating the precision and reliability of meta-analytical findings.
Moderator Analysis and Model Averaging
The next session focused on moderator analysis and model averaging using the full case-study dataset. The facilitator explained that meta-regression extends the basic publication-bias regression by incorporating additional study characteristics or moderators that may explain differences in reported effect sizes.
He presented a meta-regression framework in which the effect size was explained by its standard error and a set of additional explanatory variables. In the case study, the analysis included 35 moderators organized into seven thematic families and covered all 2,886 estimates, with standard errors clustered by study.
The facilitator explained that subgroup comparisons may be misleading when differences in study characteristics are confounded with the contexts in which the studies were conducted. For example, an observed difference between two groups of studies may reflect geographical, institutional, or methodological differences rather than the particular characteristic being examined. Meta-regression helps address this problem by conditioning on relevant study characteristics.
He also introduced Bayesian Model Averaging (BMA) as a method of accounting for uncertainty about which explanatory variables and model specifications should be included. Rather than relying on a single selected model, BMA assigns weights to alternative models according to their statistical fit. He explained that posterior inclusion probabilities (PIPs) are obtained by summing the posterior probabilities of models that include a particular variable.
The facilitator further introduced frequentist model averaging (FMA) as a complementary approach for checking the robustness of the results.
For the practical demonstration, he showed how a simple meta-regression could be estimated in R using a binary variable identifying studies published from 2020 onwards. Participants were introduced to the rma() function and shown how to include the moderator in the model through the mods specification.
Developing a Best-Practice Estimate
The facilitator then discussed the development of a best-practice estimate, explaining that researchers should consider what the estimated effect would be under an ideal research design.
He demonstrated this by evaluating the meta-regression at a set of idealized study characteristics, including the absence of publication selection and the highest level of methodological quality. Using the case study, he reported that the published estimate was 0.049, with a standard error of 0.004. Under the ideal-study assumptions, the estimated effect declined to 0.028, with a standard error of 0.027. He also reported an estimate of 0.007 for transformational research designs.
The facilitator emphasized that the assumptions used to define an ideal study must be clearly specified and made open to scrutiny. He explained that the best-practice estimate should not be presented as an unquestionable result; rather, it should be understood as a conditional estimate whose validity depends on the assumptions and methodological choices adopted by the researcher.
Reporting and Publication
The facilitator highlighted the importance of transparent and reproducible reporting as the final stage of a rigorous meta-analysis. He explained that researchers should make their analytical procedures, data, and findings sufficiently accessible for other researchers to verify and reproduce the results.
He recommended following the PRISMA 2020 guidelines and MAER-Net reporting standards. He further emphasized the importance of ensuring that all reported numerical results could be reproduced through the statistical analysis. In the case-study approach, both Stata and R were used to reproduce the reported results, while the data and analytical code were archived on the Open Science Framework (OSF).
He noted that transparent reporting and data archiving strengthen the credibility of research findings, facilitate independent verification, and enable other researchers to build on existing evidence.
Practical Exercise: Defining an Ideal Study
Participants were invited to undertake a brief two-minute group discussion on how they would define an ideal study for their own research questions. They were asked to identify two methodological or design characteristics they would consider essential and one characteristic that other researchers might reasonably challenge.
The exercise encouraged participants to reflect critically on the assumptions underlying best-practice estimates and recognize that methodological choices should be transparent and open to discussion.
Key Take-Home Checklist for Conducting a Meta-Analysis
Towards the conclusion of the methodological sessions, the facilitator presented a practical checklist based on Irsova et al. (2023), which summarized the essential requirements for conducting a rigorous meta-analysis. The checklist covered the following 18 recommendations:
1. Select a research topic that the researcher understands well and ensure that the study offers a clear contribution to the existing literature.
2. Develop a research protocol and preregister it before commencing the analysis.
3. Construct and refine a systematic search query and ensure that the identified anchor studies appear in the search results.
4. Use reference-list snowballing and conduct additional searches to capture recent publications.
5. Avoid excluding studies based on their methodological quality or publication outlet.
6. Extract all relevant estimates together with their corresponding standard errors.
7. Use two independent coders or implement a verification procedure to ensure coding reliability.
8. Convert reported estimates into comparable effect sizes, using partial correlation coefficients as a last resort where appropriate.
9. Examine outliers and apply winsorization only with appropriate disclosure.
10. Identify and plan at least ten moderators in advance.
11. Use unrestricted weighted least squares as a straightforward summary approach.
12. Always undertake publication-bias correction and apply at least one method from each methodological family.
13. Account for dependence by clustering estimates at the study level and use wild bootstrap procedures when the number of studies is small.
14. Apply study fixed effects where possible.
15. Use Bayesian Model Averaging with a dilution prior and conduct a frequentist model-averaging check.
16. Report conditional estimates based on best-practice study characteristics.
17. Follow PRISMA and MAER-Net reporting standards.
18. Make the research data and analytical code publicly accessible online.
The facilitator explained that the checklist was intended to serve as a practical reference for participants when designing and implementing their own meta-analysis projects.
Final Exercise: Developing a Preliminary Meta-Analysis Protocol
The training concluded with an individual practical exercise in which participants were asked to prepare a preliminary meta-analysis protocol. The exercise was allocated 20 minutes and could also be completed as a home assignment.
Participants were instructed to develop a short protocol incorporating five essential components: a one-sentence research question formulated using the PICO framework; identification of three anchor studies; a proposed search query developed using the techniques introduced in Step 1; selection of an appropriate effect-size measure supported by the available literature, together with an explanation of the choice; and identification of the first two publication-bias correction methods they intended to apply.
The facilitator explained that the exercise was designed to consolidate the methodological knowledge acquired throughout the training and encourage participants to translate the concepts into practical research plans.
The programme concluded with a vote of thanks from the facilitator, who acknowledged the institutional arrangements and collaboration that had made the seminar and practical training possible. The training was organized under the Erasmus+ Student Mobility for Traineeships grant and hosted by the Policy Studies Institute in collaboration with the University of Copenhagen.
The advanced sessions expanded participants' understanding of the methodological and analytical requirements of conducting rigorous meta-analysis in the social sciences. In particular, the training demonstrated the importance of addressing publication bias, accounting for dependence among estimates, identifying sources of heterogeneity through moderator analysis, and developing conditional best-practice estimates.
The emphasis on model averaging, robust statistical inference, transparent reporting, and public archiving further highlighted the importance of methodological accountability and reproducibility in evidence synthesis. Through the final protocol-development exercise, participants were encouraged to apply the knowledge gained to their own research areas.
Overall, the seminar and practical training provided a comprehensive introduction to the design, analysis, and reporting stages of meta-analysis, strengthening participants' capacity to synthesize empirical evidence systematically and generate more credible and policy-relevant research findings.