How to Discredit Research on MDMA's Benefits: First, Misread the Abstract

Kent Sepkowitz, an internist and infectious disease specialist at Memorial Sloan-Kettering Cancer Center in New York City, thinks people have too readily accepted the idea that MDMA can be useful as a psychotherapeutic catalyst for soldiers and others diagnosed with posttraumatic stress disorder. In particular, he complains in a Daily Beast essay, the press and public should be more skeptical of research sponsored by the Multidisciplinary Association for Psychedelic Studies (MAPS) in which subjects who took MDMA were much more likely to improve (as measured by scores on the Clinician Administered PTSD Scale) than subjects who took a placebo. That result, Sepkowitz claims, was "suggestive but not statistically significant," because "the likelihood that the result was due to the drug was 83%," as opposed to 95 percent, the usual standard for statistical significance.

But as MAPS Executive Director Rick Doblin points out in the comments below Sepkowitz's article, the doctor, a professor of medicine at Weill Cornell Medical College, seems to have made an embarrassing error: Eighty-three percent is the share of subjects in the treatment group who improved, compared to 25 percent in the control group. The difference was indeed statistically significant, with a 1.3 percent probability that it would occur purely by chance, as opposed to the 17 percent that Sepkowitz claims. Doblin comments:

Kent seems to have been so motivated to critique our initial study that he misread our abstract and reported an incorrect claim that our statistical analysis was not significant….If Kent had taken more than a few minutes to read the abstract and thought about what it actually reported, or if he had decided to read the actual paper, or even part of it, he would have learned that we reported that the difference between a response rate of 83% in the MDMA group and 25% in the placebo group (whose subjects received our extensive psychotherapy but with an inactive placebo) was significant ([p =] 0.013).

What about Sepkowitz's more general claim, that people are making too much of a suggestive but small and inconclusive study, partly because they are attracted by "the hipness factor" and a story line in which "Ecstasy leaves the gutter to save the day"? First of all, MDMA was arbitrarily consigned to "the gutter" by the Drug Enforcement Administration, which imposed an "emergency" ban on the substance in 1985 after people began using it for fun (shudder) at clubs and dance parties. But before MDMA emerged in that recreational context, it was a hit among psychotherapists who found that it enhanced empathy and candor (and who opposed its prohibition for that reason). While their testimonials are not the sort of evidence that gets pharmaceuticals approved by the Food and Drug Administration (FDA), it's not as if the idea of using MDMA in psychotherapy came out of the blue.

Now that MAPS is jumping through the hoops necessary to win FDA approval, Sepkowitz warns that more evidence is needed. No kidding. The title of the first report on this study, which was published two years ago in the Journal of Psychopharmacology, was "The Safety and Efficacy of 3,4-Methylenedioxymethamphetamine Assisted Psychotherapy in Subjects With Chronic, Treatment-Resistant Posttraumatic Stress Disorder: The First Randomized Controlled Pilot Study." The authors (who included Doblin as well as the lead investigator, South Carolina psychiatrist Michael Mithoefer) concluded that "MDMA-assisted psychotherapy can be administered to posttraumatic stress disorder patients without evidence of harm, and it may be useful in patients refractory to other treatments." They added, "The promising results of this initial pilot study suggest that further research is warranted to con?rm our ?ndings, distinguish and re?ne the essential elements of this approach, enhance the methodology, and elucidate the mechanisms involved."

A follow-up article published in the same journal last month (which generated the press coverage that apparently offended Sepkowitz) reported that "the majority of these subjects with previously severe PTSD who were unresponsive to existing treatments had symptomatic relief provided by MDMA-assisted psychotherapy that persisted over time, with no subjects reporting harm from participation in the study." The authors added, "Should further research validate our initial findings, we predict that MDMA-assisted psychotherapy will become an important treatment option for this very challenging clinical and public health problem." Or as Sepkowitz puts it, "If confirmed by larger trials, this finding would be welcomed."

Contrary to Sepkowitz's implication, the researchers did not suggest this was anything more than a pilot study with a small number of subjects (a total of 20 initially). In fact, they emphasized this was the first study of its kind to test the psychotherapeutic potential of this prohibited substance (a precedent that helps explain why it attracted press attention), and they emphasized the need for further research to replicate their findings (as Doblin does again in his comment at The Daily Beast). Perhaps some news reports on this research exaggerated its import, but Sepkowitz does not cite a single specific example.

Sepkowitz correctly warns that bias can lead to flawed research that confirms an investigator's preconceived notions, and he himself illustrates that danger with his inaccurate description of the initial MDMA study. Certainly it's true that Doblin and Mithoefer want to see certain kinds of results, but that is hardly an unusual situation among researchers. As Doblin observes, scientists have developed safeguards aimed at minimizing the impact of bias, including randomization, double-blind procedures, transparent descriptions of their methods, and pre-publication peer review. Post-publication criticism is another important safeguard against bias, but it should be done more carefully and thoughtfully than Sepkowitz manages here.

Speaking of bias, my own view is that MDMA, like other currently illegal drugs, should be available to any adult who wants it for any reason, including fun as well as self-improvement, regardless of what research indicates about its benefits and whether or not the user has managed to obtain a psychiatric diagnosis. But I admire the efforts of Doblin and his colleagues to win a bit more pharmacological freedom by working within the system, a quest that seemed quixotic when MAPS was founded 26 years ago, right after the DEA banned MDMA. Despite my sympathy for their cause, I agree with Sepkowitz that their research should be judged by its scientific merits and should not get a pass simply because it may help loosen restrictions on irrationally proscribed substances. But by the same token, researchers should not have to jump a higher hurdle simply because their work casts doubt on our government's irrational bias against certain psychoactive chemicals.

Update, January 7: Responding to Doblin's complaint, Kent Sepkowitz has apologized for erroneously stating that the results reported by Mithoefer et al. in 2010 were statistically insignificant. In a correction appended to the end of his piece sometime between last Thursday (when I last looked) and today, he says, "Their claim that MDMA with psychotherapy led to a significantly better overall outcome as measured by lower CAPS score in the cohort of 20 patients evaluated at 3 points after treatment, including 2 months after last dose MDMA 8-hour session, is indeed true and the finding is extremely interesting."