New at Reason

Julian Sanchez | January 6, 2006

Ronald Bailey surveys the struggle for the soul (or at least brain) of science.

Comments

|1.6.06 @ 11:04AM|#

Scientists are self-selected for the problems they study. No one devotes a career to studying things they don't think are a problem. Obviously.

Likewise, scientists are discouraged from taking up one-off experiments, because a successful (funded) lab is one where one project naturally leads to another. So, you choose a field of interest that you think is interesting and important, and therefore fundable.

Therefore, by definition, all the experts on a topic are people who think this topic is a problem worth studying, and funding. If a scientist didn't think a topic was worth funding, and by extension regulating, then he would study something else.

|1.6.06 @ 11:14AM|#

bubba, Ph.D.:

Good points. But if that is the case, how is it that scientists whose research is funded by pharmaceutical companies tend to find positive results whereas scientists who are not, don't?

Also, do agency scientists have the ability to "self-select" problems?

Ron Hardin|1.6.06 @ 11:19AM|#

Peer review, when it works, works by curiosity.

You can get huge things past peer review by making it boring enough, even though the motions remain.

Warren|1.6.06 @ 11:25AM|#

Good stuff Ron. Looks like you are boning up for the big debate on the 10th. Rip Mooney a new one for me will ya.

|1.6.06 @ 11:32AM|#

Bailey's description of the complaints about the "Federal Data Quality Act" leaves out an important point:

the act requires that all researchers who receive federal grants make the entirety of their research notes, correspondance, and all other paperwork associated with a study available to Congress whenever they ask. This is a recipe for fishing expeditions and a strenuous red tape burden, that can be brought to bear on, say, any researcher who ticks off James Inhofe by writing a paper that concludes that there is evidence for global warming.

The bill could have been written to establish some ground rules for when all this paperwork can be demanded, or for the request to go through an actual scientist (say, the CDC or NSF) to make sure there is a legitimate basis for the demand, but it wasn't. It's just a license to impose administrative burdens on scientists that a politician wants to squash.

Larry A|1.6.06 @ 11:35AM|#

If Congress really wants to enhance science's public image for veracity, it should establish a tracking process for scientific pronouncements.

Call it the Correct Prognostication Average.

Let's say a pundit predicts, "Passing concealed carry in the state of <name> will lead to blood in the streets!" The publisher would then insert, at the end of the article, "This expert's CPA is 0/28."

|1.6.06 @ 11:39AM|#

joe: But think of all the heartache that would have been avoided had stem cell reseacher Hwang Woo Suk been required to show all his notes, data, and other paperwork.

Are you saying that you're against "transparency" when it comes to government funded science? Should regulations be made on the basis of "secret" science? Just asking.

|1.6.06 @ 11:52AM|#

Ron,

A wise man once said, "The bill could have been written to establish some ground rules for when all this paperwork can be demanded, or for the request to go through an actual scientist (say, the CDC or NSF) to make sure there is a legitimate basis for the demand."

No, I don't support secret science. I would have loved to see the magazine that published Sook's paper request all of his background material, when they discovered there was a reason to do so.

I would not have wanted to see a South Korean legislator who opposed stem cell research on religious grounds demand all the background paperwork of everyone who has published a paper on stem cell research. It would have shut down the field of research for months.

The differences here are:

1. The request is coming when there is a legitimate reason to check the researcher's work, not when there is a political reason to squash research in the field, and

2. The request is coming from a credible scientific source, who is capable of distinguishing between the two reasons in #1, above.

3. As such, the "oversight agent" will be more likely to request only that information necessary to answer the legitimate question that needs answering, rather than simply imposing a mandate for a time-intensive document dump and convincing other researchers to avoid the field.

Warren|1.6.06 @ 11:53AM|#

the act requires that all researchers who receive federal grants make the entirety of their research notes, correspondence, and all other paperwork associated with a study available to Congress whenever they ask.

I don't think this is problematic. A scientist working on a project should as a matter of course be keeping everything in a computer file. Burning that file to disc at the time of publication and handing over a copy to congress seems a small imposition to me.

Sure, people who don't like a studies findings can go fishing for doubt. But that's the point of Bailey's article, science is always political. The science doesn't matter much to the politics.

However, the science does matter to many of us and transparency is essential to the method.

|1.6.06 @ 11:54AM|#

Ron,

Is this the first time you've encountered this objection to the bill? I found it odd that you would purport to describe the argument of its opponents, and not state one of the two major planks of that argument.

|1.6.06 @ 11:56AM|#

Showing every single shred of data and correspondence is like asking for the entire haystack because you want the needle.

Most of the recent high profile frauds seem to involved groups that weren't even transparent to the participants. If even the participants aren't entirely sure what somebody else did, good luck to the outsider who wants to make sense of a researcher's own private notation, sloppy handwriting, personal file sorting system (we all have our own ways of organizing notebooks, folders, digital files, etc.).

It would be far more useful to simply look for the key points of the paper and then ask the participants "Did you actually collect this piece of the data/take this image/etc.?" When he or she says no, you ask who did collect the image. From what we've seen of the recent high profile frauds, most of the suspect stuff will quickly lead to a single person. And nobody else will be able to recall actually seeing his samples or raw data or whatever it may be.

Then you ask that single person to produce the raw data, followed by the processed data, followed by the publication quality graph of that data. If you get something like "Um, well, actually, we only took one image, and then assumed that the others..." you know you've got the perpetrator.

Recent issues of Science and Nature Medicine discuss the possibility of requiring that every author on a paper describe his or her contribution to the work. That wouldn't mean every author has to hand over every scrap of raw data, just that everybody has to give a short paragraph saying which parts of the paper (especially graphs, tables, and pictures) he or she directly contributed to. If it becomes obvious that all of the key stuff hinges on one or two people, and nobody else ever witnessed the key steps, you can be immediately suspicious.

For instance, a few years ago it turned out that the Bell Labs fraud guy worked almost entirely alone. Other people would make materials for him, but he was the one who did the key optical and electrical measurements, as well as a few key processing steps. Nobody ever saw the measurements, they just saw him appearing to be busy in his lab.

Me, if I were a contributor to something so allegedly groundbreaking, at some point I'd probably want to see his experiments. Not out of mistrust, just because I'd be like "Hey, that's awesome, can I see it?" And if he said "Um, well, the equipment is being calibrated today" I'd be like "That's cool, just let me know when it's fixed. It sounds like a neat experiment!"

I wouldn't be suspicious at first, but if I kept getting excuses I might become suspicious. The consensus in my group at the university, when we all read about it, is that his collaborators were not corrupt but definitely had a shocking lack of curiosity. Nobody ever said "Hey, that sounds neat, let me see it!"

So, anyway, don't ask for the haystack. Just ask everybody what they actually did. If it turns out that everything hinges on one guy, you still don't have to ask for the entire haystack of notebooks and files. Just ask him to show a few steps. You'll get to the bottom of things pretty quickly.

|1.6.06 @ 11:56AM|#

Don't you think that the very existence of government research grants guarantees that science will be "politicized"?

All we can hope is that the myriad of strings that come attached to federal dollars will make private sponsorship a more attractive option.

I just hope that those strings relate to basic principles of fairness, objectivity and transparency, rather than cronyism, environmentalism, theocracy or political correctness.

|1.6.06 @ 12:07PM|#

Russ R,

I think it guarantees that there will be some political influence in allocating which subjects get how much funding, but no, it does not guarantee that the research itself, or the conclusions the researchers draw, will be politicized.

Does the federal funding of the National Weather Service make your local news's weather forecasts politicized?

There are many ways to wall off the decision making and administrative operations of a government-funded agency from the politics of the people who fund that agency. You often see City Councillors bitch and moan on camera when the Parking Authority raises its rates at the Municipal Garage, then explain that they can't do anything, because the legal status of the Authority makes it impossible for them to compel a change.

In my experience, people claiming that such a wall cannot be erected are seeking in some way to pierce that wall, and bring a too-independent administrator to heel.

Ed|1.6.06 @ 12:09PM|#

Russ is correct. Public money corrupts scientific research and necessarily politicizes it. It's important to mention once in a while that there is such a thing as private research. Not that good work can't be done with expropriated funds (taxes) but the economic principle of bad money driving out good holds true in the scientific sphere as well.

|1.6.06 @ 12:10PM|#

BTW, one reason why I caution against asking for the whole haystack is that most of what a researcher generates is useless. You're working on a problem, you're trying lots of things. You record everything that you do, but most of it won't end up in a publication. In the end, only a handful of it will work out and end up in a paper. If the work is honest, the researcher should be able to cough up the key protocols, notes, files, etc.

My approach has always been that once I get it to work I start again from the beginning, but I skip all the bullshit trials that ended in error. If this thing works, I should be able to produce a clear and fast trail from start to finish, and I should have no qualms about showing it to the world.

If everything leads to one guy and he can't cough up the key steps, you don't need the haystack. There is no needle in there.

|1.6.06 @ 12:14PM|#

Ron said,

Good points. But if that is the case, how is it that scientists whose research is funded by pharmaceutical companies tend to find positive results whereas scientists who are not, don't?

This is quite a statement. Have you done the meta-analysis of the field that would support this conclusion? If so how well did funding correlate to the conclusion's polarity? How many studies did you survey? What was the reliability of your findings by blinded judges?

There are trends emerging in science in general and health care science in particular that address your concerns, and they are not driven by legislation but by scientific concerns. Science works despite bias because there is no single bias that dominates. My bias one way motivates me to test your results that disagree and vice versa. Funding bias is pretty easy to claim. It is also pretty easy to prove. Biased science is harder to sustain given the existence of opposing views.

Rex Maximus, Adventurer|1.6.06 @ 12:15PM|#

I love Secret Science! Back in the day, Prof. Challenger, Doc Savage and I used forbidden technology to defeat the Iron Sultan and his Dino-sassins, and arcane methods to drill through the Earth's crust to save the Reptile Queen from the MagmaMen. They didn't ask to see my research notes when I blasted the atomic horror of Insecticus off the Bikini Atoll with my sonic cannon!

|1.6.06 @ 12:21PM|#

Ronald did an excellent job of equating these two fundamentally unequatable sides. Good work, my protege!

Warren|1.6.06 @ 12:35PM|#

foul

|1.6.06 @ 12:39PM|#

"Does the federal funding of the National Weather Service make your local news's weather forecasts politicized?"

I'm not concerned about the local weather forecast (since I've learned that I'm better off looking out my bedroom window to decide whether or not to carry an umbrella).

What concerns me is what the National Weather Service might have to say about climate change (or any other sensitive issue) especially when they take into consideration the political affiliation and agenda of the individual who chairs the committee that decides how much funding they get next fiscal year.

|1.6.06 @ 12:57PM|#

Russ,

The National Weather Service is set up so that they don't have to take into consideration what the Chair of the Yadda Yadda Commmittee thinks. That's the point.

|1.6.06 @ 1:01PM|#

Also, the national weather service doesn't have the money and careers of its researchers tied up in either Sunny or Rainy.

If they got more money by proving one of the other of these is a problem and that they needed refunding every two years to look at the problem you can bet there would be some type of statistically significant results that showed Rainy or Sunny were getting worse.

I can't claim to know this is true for all aspects of governement funded scientific research, but it IS true for anxiety disorder research, genetic predisposition to cancer research, and drug treatment alternative to prison research.

|1.6.06 @ 1:06PM|#

Ron Bailey -

What about publication bias for the non-industry-funded drug studies? If I am doing a data dredge and I find no relationship between nail polish usage and hypothermia induced death, I can't publish that.

|1.6.06 @ 1:11PM|#

NYer said,

I can't claim to know this is true for all aspects of governement funded scientific research, but it IS true for anxiety disorder research, genetic predisposition to cancer research, and drug treatment alternative to prison research.

Although there is a grain of truth here, it is just as easy to make a science career out of proving the anthithesis of any particular stance. It ends up being just like the weather. If I can show it plausible that you don't need to worry about genetic disposition, or that anxiety disorder isn't real, or that drug treatment is not an alternative to prison, I can write a grant and get funding to continue work along those lines. Rainy or Sunny, I still get funding... if I have the science chops and the grant-writing skills to convince the monetary source. Again, bias exists, but the direction of bias is not necessarily tilted in a single direction by government funding, private funding, or some combination. It is important that the funding be available so that the science can be done. It is important that there are multiple sources available, with different political and monetary interests to balance the various predilictions against each other.

|1.6.06 @ 1:23PM|#

Science: I didn't do the meta-analyses, but others have. I did link to a news article which cited one such and there are others. From the New Scientist:

"Researchers analysed 30 previous reports examining pharmaceutical industry-backed research and found the conclusions of such research were four times more likely to be positive than research backed by other sponsors."

Transparency Lovin' Dep't|1.6.06 @ 1:41PM|#

Why is transparency in tone quotes? Is stransparency being mocked? if so, harumph.

Transparency Lovin' Dep't|1.6.06 @ 1:41PM|#

Why is transparency in tone quotes? Is transparency being mocked? if so, harumph.

|1.6.06 @ 1:44PM|#

the act requires that all researchers who receive federal grants make the entirety of their research notes, correspondance, and all other paperwork associated with a study available to Congress whenever they ask.

and this is a bad thing? This should be complied anyway and made public in the first place.

|1.6.06 @ 1:49PM|#

joshua-

Making the haystack available seems to be of minimal benefit but significant cost.

OTOH, there are proposals out there to make all publications of federally funded research available to the public. Making the final results public will produce much greater benefit at a lower cost. I don't know what the benefit will be from countless pages of "This trial didn't work. Adjust protocol and try again tomorrow." I do, however, see significant benefits from easy to access to the final results.

|1.6.06 @ 1:55PM|#

In my experience, people claiming that such a wall cannot be erected are seeking in some way to pierce that wall, and bring a too-independent administrator to heel.

what the hell joe...why is this a bd thing? Why on earth would we want and unelected administrator immune from the preview of the elected government?

|1.6.06 @ 2:00PM|#

Science,

True, but I can't forget how one lead research said that a younger investiogsator had to make his grant sexy- meaning incorporate whatever the political issue of the day was - so he'd have a better chanced of getting it funded and therefore the school would get it's 50% cut.

I'm not saying that the research wasn't worthwhile (since I'm on the articles I think it was the best study ever carreid out!), but the prior results weren't viewed as the all or nothing factor of getting the grant approved.

|1.6.06 @ 2:06PM|#

joshua, "Why on earth would we want and unelected administrator immune from the preview of the elected government?"

Because we - or perhaps I should say, some of us - think that science, like the judiciary, should be insulated from political pressure, and scientists allowed to pursue their research without pressure from politicians pushing political agendas.

You don't see why it would be a bad idea to empower Rick Santorum to lean on biologists dating dinosaur bones? What the hell, dude?

|1.6.06 @ 2:06PM|#

Making the haystack available seems to be of minimal benefit but significant cost.

i guess this depends on the feild...and what is germain to the specific problem. I don't know if you follow the climate audit real climate debate but much of it has to do with data that appears to have been withheld by multiproxy climatology studies...and they appear to be withholding on the grounds that you make...not that your grounds are unfounded but it seems easy to me for a researcher to simply claim the haystack defence in order to hide problems in the study.

Anyway the price and time needed to generate all scaps is comimg down...much of it is digitized or can be easily digitized and the price is dropping fast.

|1.6.06 @ 2:13PM|#

You don't see why it would be a bad idea to empower Rick Santorum to lean on biologists dating dinosaur bones? What the hell, dude?

If those biologists are federaly funded with the tax dollars paid by the voters who put ID lunatic Rick into power then no, I don't see a problem.

Your arguement seems to parrallel those of PBS and public funding...if PBS really wants to be independant then get off the public dime.

If scientists what to be independant then they should get off the public dime as well.

|1.6.06 @ 2:20PM|#

joshua-

There are solutions intermediate between handing over the haystack and handing over just the published results. Handing over a large raw data set for analysis is one possibility. I don't know about climate research, but it's done in particle physics, astrophysics, many areas of genetics, and other areas that generate huge data sets. In these fields the giant data set can be acquired, subjected to some minimal processing to make it intelligible, and then be made available. After the initial publications on how the data was acquired, subsequent work usually focuses on detailed analysis of it, looking for patterns and trends and whatnot.

When people start talking about handing over all notes and records, I think about haystacks. If you're talking about benchtop work that generates lots of small experiments, lots of failed experiments, and lots of pieces in a puzzle while the individual researcher tries to figure out what the hell he's doing, the records are likely to be unintelligible to anybody else. I make a habit of periodically writing synopses of what I have thus far, to impose some order on the chaos. Otherwise it would be useless even to my close co-workers, let alone the general public.

In that sort of small scale work, where you piece together several different bench top experiments to get a complete picture, there are still intermediate solutions short of handing an incomprehensible lab notebook. Some journals now feature the main article and then bonus material, comprising raw data, more details of methods, etc. This information is ancillary to the central point of the article, but necessary for replication.

I vigorously oppose handing over haystacks. To be honest, I'm ashamed of how messy my haystack is. But I'd be happy to hand over supplemental materials that go into much greater depth. Basically, if you want to inspect my work at least let me impose some order on the chaos. It will be less embarrassing for me, and more intelligible for you.

|1.6.06 @ 2:23PM|#

I see, joshua. You're not saying that researchers can't carry out their work in an independent manner when funded by the govenrment. You're not saying that they can't be shielded from political pressure. You're not saying that there's no way to stop Rick Santorum from using the power of his office to intimidate scientists who carry out research he disapproves of.

You're just saying you don't want these things done. Well that's just great.

|1.6.06 @ 2:29PM|#

Thoreau -

Nobody is asking for *all* your notes. If you make 10 different versions of intermediate data sets, 9 of them wrong, just keep the right one and the script required to get it. If you ran your pet statistical test, say something financial fraud related to your optical measurements, feel free to keep it to yourself. But please show all the R^2s you took, even the ones that were counterindicative to your conclusion.

If you have been funded by the taxpayers for your research, you shouldn't be telling other researchers, after you publish, that your data is your property and the your statistical calculations are a secret. Especially if you implemented yourself in FORTRAN IV something you could have done with a SAS procedure.

And you don't need to save every email. But if you threw out half of your data because someone told you that the operator was unreliable on Mondays and Wednesdays, please save that message.

You don't need to write a "make" file for every paper. Though that's not a bad idea. But if it's impossible for you or anyone else to recreate the result you published given the same data you started with then something has gone very wrong.

|1.6.06 @ 2:38PM|#

johnl-

Much of what you say seems fair. I take it you're a scientist, from the way you write?

|1.6.06 @ 2:39PM|#

Isn't an unelected administrator an even worse political flak than an elected one? They have to be sucking a politician's toes without even having the personal abilites to get elected in the first place.

Maybe if the administrator was appointed for life, sort of like a supreme court justice.

|1.6.06 @ 2:40PM|#

johnl,

The complaint I raised about the proposed law is that it does, in fact, allow a researcher to be ordered to produce every scrap of paper, and that it allows such demands to be made by Congress.

|1.6.06 @ 2:43PM|#

BTW, john, I get the impression that your message is referencing some particular incidents. Something that relies heavily on statistical analysis.

|1.6.06 @ 2:46PM|#

Native NYer.

Here is the link to the homepage of the Board of the National Science Foundation.

http://www.nsf.gov/nsb/

If you click on the picture, you can read the bios of the members. They don't seem like toe suckers to me.

|1.6.06 @ 3:09PM|#

Do they really list their piccadillos? Thanks for the link!

kgsam|1.6.06 @ 3:09PM|#

I have thought that science/engineer types in government are most likey to produce what we expect of them, if only because they are motivated by curiousity rather than poitical sentiment. On the other hand, one tends to avoid biting the hand that feeds one.

I have a friend who once worked for the Justice Dept. She and a collegue did a bunch of research and produced a large report pertaining to criminal justice. As the results were rather opposed to current justice theory, only a small number of (like 22) of the origninal report made it into the final report produced by their bureacratic superiors.

kgsam|1.6.06 @ 3:23PM|#

I once visited the HQ of the National Geological Survey (is the right?) in Menlo Park.
There were guys sitting at computers vectorizing terrain maps by hand, clicking points along contours by mouse. Seems to me they would have better spent their time developing a program to do that.

|1.6.06 @ 3:31PM|#

Joe,

I have to hand it to you, you are pretty good at analysis..and i think you accentuate the differences between libertarian and democrat very well.

One note. Although I think that federally funded science is pretty much open to anything an elected official want to throw at it...no matter how crazy. I will say that if a privately funded scientist wishes to keep his notes secret, although probably bad science, I will as a libertarian defend her right to do so...and the government should keep its friggin nose out of it and short of a court order has no right to it.

|1.6.06 @ 3:32PM|#

kgsam,

That's the difference between a department and an institute/foundation/authority/center. The former is designed to be subject to political pressure, becuase it is performing work that is explicitly political. It is important to have people setting policy be answerable to the people's representatives.

The latter model is designed for the purpose of allowing its workers to be free of political pressure, and function independently. Questions of science need to be studied dispassionately and objectively. The so-called "Data Quality Act" is an attempt to breach this wall, and allow political pressure to be brought to bear on researchers that should be pursuing research without being pressured by the politicians.

Seriously, what improvement to the quality of data is going to be achieved by having Congressmen and Hill Staffers review it?

|1.6.06 @ 3:33PM|#

Thoreau,

You are a scientist...do you recive any federal funds for your research?

|1.6.06 @ 3:43PM|#

Seriously, what improvement to the quality of data is going to be achieved by having Congressmen and Hill Staffers review it?

Well in the case of climatology it has allowed data that was being withheld (data that was generated on the publics dime) to be reviewed by skeptics.

|1.6.06 @ 3:46PM|#

joshua-

I work at a federal research institute. Yes, I know, I flunk the purity test. I probably seem like a slacker, but I generate ideas quickly and test them quickly. I do a lot between posts, and breaking up my time is the best way to get ideas and really work them around from different angles. To give you some perspective, I'm about to send a paper off to a journal for an idea that I got in late October. That's fairly respectable turnaround time.


johnl-

I've been a coauthor on a theoretical paper with nearly a dozen authors. The paper brought together several different ideas. I was heavily involved in one stage of the work, and I was given an opportunity to comment on a draft before the lead author sent it to a journal. Some of these long author lists are perfectly respectable, and reflect the significant ideas of many people. Others, well, others I take a dimmer view of.

That's why I like the idea of journals requiring that authors describe their contributions. The standards for getting your name on a paper vary by field, and I'm OK with differing standards as long as the standards are understood. In very large collaborations with huge author lists, it's understood that most of the people listed made small technical contributions, and the essential intellectual contributions came from the first author and the author with the asterisk by his name. (The asterisk is usually the boss of the first author, and the one you direct all correspondence to if you want to ask questions about the work.)

I'm fine with those low standards as long as they're understood. Where it gets fuzzy is in smaller groups, where people have widely different standards for how much contribution you need to make to be an author. The standard of first author and asterisk author remains, so the stakes aren't too high for dubious decisions on who else to include. Still, I don't see that as an excuse for sloppy or inconsistent standards. Sadly, not everybody shares my view.

|1.6.06 @ 5:01PM|#

Those of you asking for the primary research notes have no idea what you're asking for. There are two types of research. There is the GLP (Good Laboratory Practices) research conducted for FDA approval. These things are rigorously and thoroughly documented. They are often handled by a contract research organization that has specialized in this type of record keeping.

The primary research, conducted in academic labs, and in my own corporate lab, is well, lets just say it's not as tightly documented. How many bags of post-it notes does congress really want? Publication is when these post-it notes are compiled into a coherent story. My company has a database of all our research, but even this is one step removed from the "primary" data. However, because we have a patent lawyer, our notebooks are infinitely more detailed than those in an academic setting.

I have trouble deciphering my own notebooks from prior years. You have no chance.

This is why science is based on replication. No matter how presuasive a paper is, it doesn't count until it is replicated in another lab. The people who were "disappointed", or whatever, in the Korean fraud were just dumb. Sure, it's a bummer that he was a fraud, but you didn't need his primary notes to figure this out.

Meanwhile, there are thousands upon thousands of new scientific publications each year. Most of them are wrong, even if not fradulent. What's the point of a congressional review? When you stumble across an act of fraud, you prosecute him, but it's no reason to hassle all the other scientists.

To summarize, here's how it works. Scientists try a bunch of different things, and document most of this on napkins. When they get an idea that works, they repeat it in their own lab and write it up. It is foolish to expect real documentation of all the things that didn't work. Let me repeat that. It's absolutely freakin' nuts to expect a scientist to waste any time writing up his failures in a form that someone else would understand.

|1.6.06 @ 5:19PM|#

bubba, Ph.D.:

Good points. But if that is the case, how is it that scientists whose research is funded by pharmaceutical companies tend to find positive results whereas scientists who are not, don't?

Also, do agency scientists have the ability to "self-select" problems?

Negative results are difficult to publish, regardless of funding. However, there may very well be an ascertainment bias at work. When a company funds a trial, they are trying to show an effect. So, the study is set up with statistical power to show this effect. It is much, much more difficult to show an ABSENCE of effect with statistical power. Maybe the effect is small. Maybe you were just unlucky. You can't publish a failure to find an effect, if you don't have good statistical power to actually exclude an effect. It is a common belief among scientists that they could prove their theory if only they had more data.

I have not worked for a government agency, but academic scientists at Universities and at National Labs pick their projects based on their interests, and based on what is trendy within the department/institution. There are simply more resources available if you have the support of your Department Chairman and/or Institution. There are separate pools of money that encourage "interdisciplinary" work, or work that leverages the unique abilities of National Labs, for example.

A phone call to a colleague who has worked with an agency assures me that the scientists there are every bit as self-serving as any scientist or government bureaucrat. It might not be obvious to outside observers what the motive is, but agency scientists have reputations to build, consulting gigs to get, whatever.

When it comes to scientific pecking order, agencies are not exactly at the top of the heap. This is because they get some academic freedom to select problems, but not nearly as much as those at academic instutions like universities or even national labs. Maybe the agency studies are more likely to have statistical power to exclude an effect because they are more independent. Or, a more cynical thought is that these scientists aren't as good, and have trouble distinguishing between absence of evidence, and evidence of absence.

Regardless, the original point, I think, is that corporate funding of research is corrupting, based on the observation that corporate-funded research is more likely to show the desired effect. I think there are much more benign explanations for this observation. There is certainly no evidence to suggest that one should ignore the studies that are published as a result of guilt by association. Each study should be judged based on the evidence presented, and based on reproducibility.

|1.6.06 @ 5:29PM|#

bubba-

Thank you for backing me up on documentation.

As to corporate funding of university research, a third of my thesis was on a corporate funded project. It involved materials science, and they simply wanted an answer one way or the other. They wanted me to either tell them how to do something, or tell them that it can't be done and demonstrate why. I did exactly that: I showed them what could be done, I showed them how to do it, I showed them what couldn't be done, and I showed them why that part couldn't be done. I demonstrated that the lack of a solution wasn't simply due to limited imagination; rather, it was due to fundamental facts of optics.

I had a very positive experience working with corporate researchers. However, it may very well be different in other fields.

Robert|1.7.06 @ 12:13AM|#

If you haven't yet read "The Myth of Scientific Public Policy" by Robert L. Formaini, do so. It's short.

|1.7.06 @ 3:32AM|#

Ron,

From the article you link.

"But of the 13 studies that looked at the scientific methods used, NONE reported that the industry-backed trials were of lower quality. Lexchin says this is because standard scales used to examine scientific quality do not cover all issues, such as what drug was used in a trial as a comparison.Industry-backed studies often compare a new drug to placebo or a second line agent, he says, which makes it easier to get a positive result for a new drug. "My preference would be to compare with what's recommended as the current best therapy," he says."

Although this news article does not provide enough information about study methods to make any real conclusions, just this statement by the author demonstrates the importance of looking carefully at results. The conclusions of a study that compares a new drug to a placebo are answering one type of question. Studies comparing the new drug to the current best therapy are answering another. The first may be more likely to get "positive" results compared to the second, but the comparison between the two is meaningless.

The first asks "Does the drug work (is it safe)?"
The second asks "Is it an improvement?"
The first may be positive and the second negative without indicating a bias (or a problem) -- no matter who is asking the question. I imagine (not my field) that the first question is the one a drug developer is interested in the answer for, while the second is the one that institutes, doctors, & clinicians are interested in.

As bubba pointed out. It is evidence and replication that matter here. But to replicate, you need to be asking the same question. Comparing studies that ask different questions is not a fair comparison and cannot reveal bias.

Politicians will put pressure on science always. Science will continue on its scientific way. That's its beauty and the source of its success.

Thanks for the link, I'll have to take the time to read the source study.

|1.7.06 @ 9:20AM|#

There is no intellectually serious argument for withholding data.

|1.7.06 @ 9:53AM|#

Of the two arguments against allowing Congress to inspect all research notes on demand, I find the pragmatic to be more compelling. Let's face it, Congressmen aren't very good at much, that's why they're politicians in the first place. Pontificating on steroids is minor league compared to wading through the scores of research notes (mostly useless) that go into a published paper. While transparency and accuracy must be paramount, trying to enforce such through one of the least transparent and accurate groups in America is folly.

On the more philosophical side, Joe's right that such a measure would lead to "fishing expeditions and a strenuous red tape burden". Unfortunately, it probably would be common for the most politically contentious scientific debates of the day. Laying ground rules and requiring the consent of a scientific organization are fine ideas, but I wonder whether that would really alleviate the bureaucratic burden. If anything, it just might add one more set of tape.

All that said, it's hard to really argue against full transparency for publicly funded research, even if such may be a logistcal and political nightmare. There really is no good solution here. The government will continue to make public policy decisions based on scientific research, some of it faulty. Completely divorcing science from public funding will not change this, and might exacerbate it. Allowing politicians to review politically funded research won't help either, even with stricter guidelines.

|1.7.06 @ 11:22AM|#

Stretch-

Transparency can be accomplished without having untrained people come in and go through all your stuff. Scientific discovery does have its rigorous, systematic, documented side. It also has its creative, intuitive side that relies in large part on dumb luck. You act on hunches, try different stuff, and hope for a little luck, until you find something. Up till there, the process looks pretty messy, and anybody who dislikes the final results could easily pick it apart and make it look like the scientists are a bunch of fools.

But once you find something you investigate it systematically. That's where the nicer paper trail is generated.

If somebody comes in and simply takes the whole haystack, they'll think that the whole thing is lunacy and we have no clue what we're doing. That's what I want to avoid.

If somebody asks me to show them every step of the process that I used to test a finding, I can do that. I have no problem doing that. But if somebody asks me to hand him the haystack so he can go and make of it what he will, that I have
a problem with.

Somebody who wanted to discredit a finding could no doubt find lots of stuff to make hay with. "Look at how random these evolutionists are in their work! Why, they just try all sorts of crazy shit until they find something that says there is no God!" Um, no, they try lots of ideas until they find one that actually passes muster with the data. And once they find an idea that passes muster, they subject it to a more systematic battery of experiments.

Ask me to show you what I've done and I'll be happy to. Ask me to hand stuff over so you can twist it and I'll do it if the law requires it, but I hope you can understand why I might not be a happy camper.

|1.7.06 @ 1:37PM|#

I don't disagree, thoreau. As I said, from a purely pragmatic perspective having the government go through every scrap of information is ridiculous. I also agreed that having the government comb through mounds of research notes to distort conclusions to suit its own needs is a problem. I don't claim that this is the best, or even a good way to increase transparency in science, only that at bottom if the government wants to look at everything so be it. Again, it's foolish and can lead to some bad things, but has it really changed anything?

The government has always selectively chosen the science it supports in order to make the policies it wants. A method to challenge policy based on inaccurate science isn't such a bad thing. There are certainly problems with the guidelines and the way it's been implemented, but the ability to verify key scientific data that influence policy in response to a public petition is fine by me. It's been 5 years now and I don't think the relationship between government and science had been altered fundamentally.

|1.8.06 @ 10:31AM|#

So, here's something I just thought of if the entire haystack were handed over: Somebody finds an old version of a climate modeler's code. A version full of bugs. Bugs that violate the laws of physics.

But those bugs produce a result that is more pleasing to certain people. So we get the following announcement:

"When forced to hand over ALL of their research results, climate modelers admitted that many of their simulations actually reveal global cooling, not warming. But they only published the results that predict warming."

If you get an entire haystack you can take anything out of context and make the whole work look bad. Ask us to document more steps and we'll do it. Ask us for answers and we'll give them.

But don't ask us for the haystack so you can build a strawman out of it.

|1.8.06 @ 11:06AM|#

thoreau:

Of course the risk exists that haystack combers will manipulate the data to serve their own agendas even as the haystack creator may have done. But that's the point. Unless all the data is available for inspection by multiple parties it is impossible in principle to determine whether any conclusion drawn from that data is reliable. As with free speech, the proper response to misrepresentation by any party is more (ore at least equal) access to information, not less.

|1.8.06 @ 11:45AM|#

D. A. Ridgely-

I largely agree that the solution to bad speech is more speech. But scientific results are considered valid when reproduced in other labs, not when one lab has thoroughly inspected another lab's notes and files.

|1.8.06 @ 12:07PM|#

Were scientists not mere mortals and if no policy implications were involved, such validation within the scientific community would suffice. But the evidentiary requirements where policy issues are involved transcend the standards of the scientific community because the absence of complete data will always give rise to further doubt and suspicion of concealment of significant contrary data. That is not, mind you, an argument that scientists should change the way they do science, but only an argument that the epistemology of evidence for public policy requires attention to other concerns when scientific results are proffered for or against any particular policy.

(BTW, I see I posted "ore" above. I really need a smarter spell checker!)

|1.8.06 @ 5:03PM|#

D. A. Ridgely-

I see what you're getting at. But I think that competition and open debate between independent researchers is good enough to get to the bottom of most things. Handing over stacks of hay doesn't seem like a good way to arrive at truth.

Now, of course it is different if there is legitimate reason to suspect fraud. In that case, lab notebooks may be necessary to prove that the researchers were deliberately dishonest rather than merely wrong. But unless there is reason to suspect fraud, I see little good that can come from handing over haystacks. At least little good that can't be accomplished by more modest means: Requesting summaries of methods that are somewhat more exhaustive than you find in professional journals, requesting that authors state their contribution to a work, and placing in the public domain articles on work performed at public expense.

One area where it makes sense to go further is for large data sets that took considerable time and/or expense to acquire. (e.g. data from space telescopes, particle accelerators, climate records, large gene sequencing projects, protein crystallography data, etc.). This data would be difficult to replicate, unlike the more common scientific research that happens in small groups, and often a single data set will generate dozens or hundreds of papers. So it would make sense to put the raw data out there for everybody to see (especially if it was acquired at public expense).

But even there, you wouldn't need the researchers' notebooks. If they say "We did these tests on the data and it clearly shows such-and-such trend..." then you can go do the same test yourself. You don't need their notebooks to do that.

So, in short, what I advocate is ways to put more information out there, but leave the embarrassing noise of failed ideas in the safe anonymity of incomprehensible lab notebooks. And with more information out there, let open debate and competition between independent researchers take care of the rest. Save the fishing expeditions for times when there is good reason to suspect fraud.

If open debate between independent competitors works so well in other areas of life (that is a libertarian notion, right?) why not in science?

|1.9.06 @ 10:22AM|#

Thoreau, I don't know who it is you are arguing with over the discarded runs. In climate science, the problem is that researchers wont turn over *final* runs, the ones they used to make the stats they published, and that those can't be reproduced. The researchers then play the "I've got a secret" game when other people can't reproduce their results, and then refuse to turn over their data and code so that people can see what it is that they did.

|1.9.06 @ 10:41AM|#

johnl-

If the climate scientists are as bad as you say, then that is wrong. I would not trust results shrouded in such secrecy until a more transparent group achieved the same results.

I originally started this in response to something that joe said about requiring researchers to hand over all notes. I said that if such a thing is required that would be a bad idea. Other people have disputed me on that, so I'm arguing with them.

|1.9.06 @ 12:04PM|#

Thoreau -

Well you can hardly expect Congress to solve one problem without creating another. That's why the best outcome would be premptive good behavoiur by scholars, and reasonable standards from universities and journals.

|1.9.06 @ 12:13PM|#

johnl-

You'll be pleased to know that many of the things I've advocated (supplemental methods materials in journals, descriptions of author contributions, sharing of large data sets, open source code) are being implemented. Change is slow, but it's happening, and some of it has been happening for a long time. I would like to know more about the climate scientists before I conclude that the field is as opaque as you say. There may be more to the story (or there may not be). A lot of people have axes to grind against climate scientists. Yeah, I know, climate scientists have their own axes to grind. But they also have competitors.

Another positive trend is increasing private funds for university research. The share is still small, but it's growing.

Don't conclude that the whole of science is rotten.

|1.9.06 @ 1:05PM|#

Back to the original topic of ascertainment bias: Corporate research is more positive than Agency research.

Are we really surprised that companies are faster to discover benefits to their drugs than agencies are? Don't we expect companies to have better access to their own drugs, and if there is a benefit, this will be found by the company before it is found by an agency? Once an FDA-approved study is published, it is unlikely that an Agency will spend time and money replicating this. These studies, especially human trials, are very expensive. In other words, Corporate trials are conducted on drugs with a prior probability of success, based on animal and biochemical studies.

It's not until there is a perception that there is a problem, or there is some desperation to find a cure for an uncommon disease, is there motivation for an Agency to conduct a trial. This would obviously bias the Agency trials toward failure, in that Agency trials would not have the same prior probability of success.

Either of these scenarios could explain an observed different in success rates between Companies and Agencies, without any data bias by either group.

In other words, wouldn't you be surprised, stunned even, if Agencies were better at identifying therapeutic benefits than the Companies that created the drugs?

|1.9.06 @ 6:26PM|#

OK Ron Bailey, we have two sources of publication bias. Good news is no news for established drugs, hence a bias in favor of negative independent articles. And (thanks bubba) for drugs that need establishing, the industry is going to come up with more publications than for their failures. Can we put this one to bed yet?

Thoreau -

Nobody thinks all of science is rotten. There are a few rotten apples and if they don't shape up, it will be bad for everyone, no matter what Congress does. Right now, the rotten apples are congregating in climate science and reproductive medecine.

|1.9.06 @ 8:49PM|#

Climateaudit has a recent post that covers a lot of controversy trying to get data to replicate MBH98.