A New Scientist editorial poses this thought experiment:
IMAGINE you are seated at a table with two bowls in front of you. One contains peanuts, the other tablets of the illegal recreational drug MDMA (ecstasy). A stranger joins you, and you have to decide whether to give them a peanut or a pill. Which is safest?
You should give them ecstasy, of course. A much larger percentage of people suffer a fatal acute reaction to peanuts than to MDMA.
…that on all tests except for verbal memory, ecstasy users performed just as well as before [they used it] and on a par with abstainers….the effect on [verbal memory] was so small—a difference of a quarter of a word on average from a list of 15—the real world implications are questionable.
All recreational drugs cause neurological changes—that's kind of the point of taking them. Long term downsides to ecstasy use may emerge, but so far its main effect has been, as the New Scientist editorial notes, "to drive politicians out of their minds." And to be used as an excuse to toss thousands of users into jail. The editorial ends with a plea:
We need a rational debate about the true damage caused by illegal drugs—which pales into insignificance compared with the havoc wreaked by legal drugs such as alcohol and tobacco. Until then, we have no chance of developing a rational drug policy.