Correcting calculation and data errors reveals that the original conclusions were incorrect in “The best drug supplement for obesity treatment: a systematic review and network meta-analysis”
Diabetology & Metabolic Syndrome volume 15, Article number: 163 (2023)
The goal of this study was to reproduce and evaluate the reliability of the network meta-analysis performed in the article “The best drug supplement for obesity treatment: A systematic review and network meta-analysis” by Salari et al. In recent years, it has become more common to employ network meta-analysis to assess the relative efficacy of treatments often used in clinical practice. To duplicate Salari et al.‘s research, we pulled data directly from the original trials and used Cohen’s D to determine the effect size for each treatment. We reanalyzed the data since we discovered significant differences between the data we retrieved and the data given by Salari et al. We present new effect size estimates for each therapy and conclude that the prior findings were somewhat erroneous. Our findings highlight the importance of ensuring the accuracy of network meta-analyses to determine the quality and strength of existing evidence.
In the publication by Salari et al. , “The best drug supplement for obesity treatment: A systematic review and network meta-analysis,” the authors used a network meta-analysis to analyze the effectiveness of several anti-obesity medications. In the history of obesity pharmacotherapy, several promising treatment candidates have arisen only to be discontinued owing to unacceptably high safety concerns . The potential value of obesity pharmacotherapy encourages the publication of meta-analysis papers that combine and rigorously compare studies to determine the quality and strength of the existing evidence.
A network meta-analysis uses both direct and indirect data from a network of trials to compare three or more treatments at once in a single study. The use of network meta-analysis to evaluate the relative efficacy of therapies often used in clinical practice has gained popularity recently . There are several advantages to using network data for meta-analytical purposes, including deriving more exact estimations of the relative impact of each intervention in the network. Additionally, using network meta-analysis allows investigators to rank the interventions included in the analysis . Credible inference in a network meta-analysis is premised on the assumption that the different studies included in the analysis are similar in terms of all major features that might affect the relative effects  and that the analysis was properly conducted. It is therefore essential to ensure that meta-analyses and network meta-analyses are conducted correctly.
Upon inspection of Salari’s et al. paper, we uncovered several data reporting and extraction errors that render the results invalid. We brought these errors to the attention of the authors, leading them to publish a correction , but the correction did not address all the errors that we discussed. The correction addressed the following: in Table 1, the treatment reported for the Davies et al. study  was incorrect (liraglutide 0.3 mg should be liraglutide 3.0 mg) and the wrong supplement was reported for the Greenway et al. study  (naltrexone + bupropion 16.0 mg and naltrexone + bupropion 32.0 mg; should be naltrexone 16.0 mg + bupropion and naltrexone 32.0 mg + bupropion).
Because the correction issued by the authors did not address all the errors we previously identified, we redid the network meta-analysis. Note that the analysis by Salari et al. involved 11 parallel studies which we refer to as the “original studies”: Apovian et al. , Aronne et al. , Davies et al. , Fidler et al. , Gadde et al. , Greenway et al. , Le Roux et al. , Lu et al. , O’Neil et al. , Pi-Sunyer et al. , and Smith et al. . Here we report the discrepancies between what Salari et al. reported and what we obtained by extracting data directly from the original studies.
Data extraction and evaluation
We first attempted to collect the original datasets and code from the corresponding author. We reached out to the corresponding author of the original research manuscript on October 27, 2021, asking them to share their data and the R code used to generate their results. Dr. Mohammadi provided us with two materials: the appendix of the preliminary results (a Microsoft Excel file) and an R script file. Salari et al. carried out their systematic review and network meta-analysis, which we refer to as the “original research”, by conducting a systematic database search, categorizing documents for evaluation, applying inclusion and exclusion criteria, extracting data, and conducting the network meta-analysis. The data reported in the original research paper were from participants who completed post-treatment assessments.
We sought to recapitulate this analysis by extracting the same data from completers in the 11 original studies. We extracted data (sample size, mean, standard deviation, treatment name, etc.) directly from the original studies listed in Tables 1 and 2 of the Salari et al. paper. The sample size, mean, and standard deviation refer to each arm of every study. The data extracted were verified by two or more researchers.
We used the extracted data to calculate the effect size for each treatment using Cohen’s D metric to estimate the weight difference between groups due to changes from baseline. Upon review of the original studies, however, we found that the data in some studies were reported differently from what Salari et al. stated in their original publication1 and in the correction . Some of the original studies did not report the actual final weight value but instead reported the mean change in weight in kilograms (kg). We thus calculated the final weight values or mean change in weight for both the original studies and number reported by Salari et al. Similarly, we computed the number of participations based on the sex distribution (percentage) reported in the original studies. We then recapitulated the network meta-analysis in R studio using the data we extracted to explore the most effective drug treatment for obesity. We compared our results with the data provided by the corresponding author and the data reported by Salari et al. in their original research.
In Table 1 (age and treatment), Table 2 (sex and number of participants), and Table 3 (weight and weight change), we compare the data we extracted from the original studies with those reported by Salari et al. Our new estimates of the effect sizes are included in Fig. 1.
We found several discrepancies between the data we extracted and the data in the spreadsheet we received from the corresponding author. For example, their dataset did not include all the studies listed in the article as having been analyzed. Specifically, Aronne et al.’s study, Le Roux et al.’s study, and Lu et al.’s study were completely missing from the dataset but were reported in the published paper.
When we carefully reviewed the information reported in Salari et al.’s Tables 1 and 2 and compared it with the information reported in the original studies, we found several data extraction errors or discrepancies. For example, in Table 1 of the original research paper, the mean age and standard deviation are completely missing for the Davies et al. study in the interventions column . However, these data were reported in the original study by Davies et al.: the mean (SD) ages were 55 (10.8) years in the liraglutide 3.0 mg group and 54.9 (10.7) years in the liraglutide 1.8 mg group. We also found data extraction errors for initial average weight and average weight change in Table 2 of the Salari et al. paper. Salari et al. did not disclose whether the variance reported (e.g.: ± 6.4 in Fidler et al.’s study) in Table 2 was a standard deviation or a standard error. Standard deviation of weight change is used to calculate the treatment effect standard error. If the authors reported the treatment effect estimates and their uncertainty using standard error for some and standard deviation for others, these effects would not be directly comparable. They should not be directly used as input for the network. We found that some of the original studies used standard deviation, and other used standard error.
We believe the aforementioned discrepancies and errors in addition to the others reported in Tables 1, 2 and 3 influenced the results of the network meta-analysis performed by Salari et al. After we calculated the differences in weight loss between each drug vs. placebo and conducted a random-effects model, we arrived at different values for the effect size (and 95% CI) compared with those reported in Fig. 6 of Salari et al. Salari et al. claimed to show the standardized mean difference (SMD). However, they reported mean differences between the groups that differed from our recapitulated results (Figs. 1 and 2) using the re-extracted values. For example, the effect size of phentermine 15.0 mg + topiramate 92.0 mg should be -8.8 [-10.72, -6.88], not − 9.10 [-10.37, -7.83], and the effect size of pramlintide should be -1.5 [-4.17, 1.17], not − 6.50 [-13.46, 0.46].
Discussion and conclusion
According to the Committee on Publication Ethics’ Retraction Guidelines, retraction should be considered if there is “clear evidence that the findings are unreliable… as a result of a major error (e.g., miscalculation or experimental error)” . When the errors in the paper by Salari et al. are corrected, we find a substantially different rank order of drugs in terms of the most effective weight-loss medications. We respectfully believe that Salari et al.’s network meta-analysis should be retracted because the conclusion drawn is inaccurate owing to miscalculation and inaccuracy of the data reported.
Additionally, as one reviewer noted, “It is also problematic that only completers were analyzed. A sensitivity analysis using the pattern-mixture model to model the missing participants would have elucidated whether missingness may threaten the validity of the results.” . We agree. Our purpose herein was to evaluate whether the results could be reproduced (as defined by the National Academy of Sciences , They could not be. Future research should determine the answers obtained when the analyses are conducted optimally. Obtaining such answers will require a new full-scale endeavor including sensitivity analyses to respond to concerns around treatment of missing data.
Analysis data may be obtained from the corresponding author at firstname.lastname@example.org.
Salari N, et al. The best drug supplement for obesity treatment: a systematic review and network meta-analysis. Diabetol Metab Syndr. 2021;13:110.
Kakkar AK, Dahiya N. Drug treatment of obesity: current status and future prospects. Eur J Intern Med. 2015;26:89–94.
Rouse B, Chaimani A, Li T. Network meta-analysis: an introduction for clinicians. Intern Emerg Med. 2017;12:103–11.
Riley RD, et al. Using individual participant data to improve network meta-analysis projects. BMJ Evid Based Med. 2022. https://doi.org/10.1136/bmjebm-2022-111931.
Chaimani A, Caldwell DM, Li T, Higgins JPT, Salanti G. Chapter 11: Undertaking network meta-analyses. (Cochrane Handbook for Systematic Reviews of Interventions version 6.3, 2022).
Salari N, et al. Correction to: the best drug supplement for obesity treatment: a systematic review and network meta-analysis. Diabetol Metab Syndr. 2022;14:68.
Davies MJ, et al. Efficacy of liraglutide for weight loss among patients with type 2 diabetes. JAMA. 2015;314:687.
Greenway FL, et al. Effect of naltrexone plus bupropion on weight loss in overweight and obese adults (COR-I): a multicentre, randomised, double-blind, placebo-controlled, phase 3 trial. Lancet. 2010;376:595–605.
Apovian CM, et al. A randomized, phase 3 trial of naltrexone SR/bupropion SR on weight and obesity-related risk factors (COR-II). Obes (Silver Spring). 2013;21:935–43.
Aronne LJ, Halseth AE, Burns CM, Miller S, Shen LZ. Enhanced weight loss following coadministration of pramlintide with sibutramine or phentermine in a multicenter trial. Obes (Silver Spring). 2010;18:1739–46.
Fidler MC, et al. A one-year randomized trial of lorcaserin for weight loss in obese and overweight adults: the BLOSSOM trial. J Clin Endocrinol Metab. 2011;96:3067–77.
Gadde KM, et al. Effects of low-dose, controlled-release, phentermine plus topiramate combination on weight and associated comorbidities in overweight and obese adults (CONQUER): a randomised, placebo-controlled, phase 3 trial. Lancet. 2011;377:1341–52.
le Roux CW, et al. 3 years of liraglutide versus placebo for type 2 diabetes risk reduction and weight management in individuals with prediabetes: a randomised, double-blind trial. Lancet. 2017;389:1399–409.
Lu C-W, Chang C-J, Yang Y-C, Lin W-Y, Huang K-C. Multicentre, placebo-controlled trial of lorcaserin for weight management in chinese population. Obes Res Clin Pract. 2018. https://doi.org/10.1016/j.orcp.2018.07.007.
O’Neil PM, et al. Randomized placebo-controlled clinical trial of lorcaserin for weight loss in type 2 diabetes mellitus: the BLOOM-DM study. Obes (Silver Spring). 2012;20:1426–36.
Pi-Sunyer X, et al. A randomized, controlled trial of 3.0 mg of liraglutide in weight management. N Engl J Med. 2015;373:11–22.
Smith SR, et al. Multicenter, placebo-controlled trial of lorcaserin for weight management. N Engl J Med. 2010;363:245–56.
Editorial A. COPE retraction guidelines. Version 2. November 2019. Sci Ed Publ. 2022;6:148–54.
Mavridis D, White IR, Higgins JPT, Cipriani A, Salanti G. Allowing for uncertainty due to missing continuous outcome data in pairwise and network meta-analysis. Stat Med. 2015;34:721–41.
National Academies of Sciences, Engineering, and, Medicine, et al. Reproducibility and replicability in Science. National Academies Press; 2019.
Supported in part by National Institutes of Health grants R25HL124208, R25DK099080, and R01DK132385. The opinions expressed are those of the authors and do not necessarily represent those of the NIH or any other organization.
Ethics approval and consent to participate
Consent for publication
DBA has disclosed several personal payments or promises from various sources as well as funds or donations from various sources to support the work of the School of Public Health and the University more broadly, but they had no influence in the drafting or scientific content of this manuscript. PLC is part owner of Biochemical Renovations LLC, but this company had no influence in the drafting or scientific content of this manuscript. The remaining authors declare that they have no competing interests as defined by BMC, nor other interests (financial or personal) that might be perceived to influence the results and/or discussion reported in this manuscript.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Yu, X., Capers, P.L., Zoh, R.S. et al. Correcting calculation and data errors reveals that the original conclusions were incorrect in “The best drug supplement for obesity treatment: a systematic review and network meta-analysis”. Diabetol Metab Syndr 15, 163 (2023). https://doi.org/10.1186/s13098-023-01134-6
- Network meta-analysis
- Drug supplement
- Obesity treatment