President Donald Trump revealed on May 18, 2020 that he was taking hydroxychloroquine to prevent contracting COVID-19. News media reports run the gamut from saying hydroxychloroquine is 91% effective to it being both ineffective and dangerous. How do people know what to believe?

Our Health Outcomes, Policy, and Evidence Synthesis (HOPES) group at the University of Connecticut has conducted many high profile projects in the past, studying conditions such as cystic fibrosis and asthma. We published an assessment on May 27, 2020, of all of the controlled studies that have been conducted around the world through May 8, 2020, in the Annals of Internal Medicine. Like our other projects, we used search strategies and methods that are sanctioned by international bodies like the Cochrane Group to determine whether hydroxychloroqune was effective and safe. Based on the media reports you have heard, the results may surprise you.

What did we find?

We focused our assessment on controlled studies, those that compare the effects of a drug in one group versus those who did not receive therapy in another. This is critical when establishing the effectiveness of a therapy because without it, the results are meaningless. If 100 people took hydroxychloroquine and 10 died, it could mean that therapy was great if 20 were going to die without it but horrible if two were going to die.

The first thing we found is that there are no controlled studies assessing the use of hydroxychloroquine to prevent getting COVID-19. So there is no evidence one way of the other that you can prevent COVID-19 by taking hydroxychloroquine.

We did find 14 controlled studies that were published in medical journals or in pre-print where hydroxychloroquine was used to treat people who had COVID-19 and we were able to independently evaluate both the methods and results.

We found that even these studies had serious methodological weaknesses. None of the studies had placebo control or blinding of patients, clinicians and investigators, as is typical in clinical trials and considered the “gold standard.” Some studies had major baseline differences in patient age, gender and severity of disease between the two groups and did nothing to correct for that. And one study just decided to not analyze the data for some of their patients receiving hydroxychloroquine.

These and other weaknesses reduced our confidence that the results we would find would be believable and not due to these biases. Many of the studies had a very small number of patients, so there was high risk that differences between studies could be due to chance.

Some of the studies suggested that people receiving hydroxychloroquine were less likely to die, and more likely to clear the virus faster, or improve the look of the lung scans better than those not receiving it. However, a majority of the studies suggested hydroxychloroquine does not provide any of these benefits and some studies were stopped early because independent ethics boards felt the lack of benefit was not worth the adverse effects that were occurring. This is exactly the type of dispersion in study results you would expect to see in this poor quality literature base. The main adverse effect seen so far was something doctors call severe QTc interval prolongation – an EKG finding used to determine arrhythmia risk – which, if excessive and not caught early, can lead to a serious arrhythmia, which can be fatal.

When we applied recommended methods for determining the strength of evidence, the only conclusion is that the current studies are insufficient to say whether hydroxychloroquine works in COVID-19. That means it is impossible for anyone to know whether or not it works given the studies that currently exist.

An observational study came out on May 11 in the Journal of the American Medical Association and another on May 22 in The Lancet. Both were too late to be included in our systematic review. The first is one of the larger ones with 1,438 patients and was conducted in New York City hospitals. It found that hydroxychloroquine use, with or without azithromycin, had no mortality benefits at all.

The second multinational study is, by far, the largest, with 96,032 patients, and it did a good job of controlling for differences between groups at baseline. Use of hydroxychloroquine alone or hydroxychloroquine plus a macrolide like azithromycin increased mortality substantially in this study.

When added to the studies we already reviewed, it is becoming less likely that a mortality benefit will result from the use of hydroxychloroquine and more likely that a neutral or harmful effect on survival will occur. If hydroxychloroquine doesn’t hasten death, then it could still be useful should it be found to reduce other endpoints of importance, such as duration of illness and decrease in severity of symptoms. Only future studies will be able to say for sure.

Where do we go from here?

There is a study with strong research methods currently underway, but it is not coming out until 2021. No one knows what they will find.

People currently prescribing hydroxychloroquine and those taking it should do so with the full knowledge that it is completely experimental for COVID-19 and not without risks. Cherry picking a few studies that fit your preconceived idea about hydroxychloroquine’s benefits or safety and disregarding the rest isn’t good science and shouldn’t guide health decisions.

Originally published in The Conversation.