November 6th, 2008
The other day I linked to this story from the Economist called “Publish and Be Wrong.” They were making the criticism that scientists may be overselling their results to get into the most prestigious journals, and hence the top journals were more likely to be publishing “wrong” results.
Well, I have been thinking more about this and have some criticisms of the article. Sorry. I know criticizing economists can be like kicking a dead horse, but…sometimes you just feel the urge to rear back and kick the hell out of the thing.
Most science publications are called “journals.” I see papers published in them as the official public version of the research journal of mankind. It is totally normal in a journal to get excited about preliminary results, to look at small samples before examining larger data sets, and to engage in some informed speculation to help motivate future work.
A paper is wrong, and only wrong, when mistakes are made.
A spurious correlation is not a mistake. A speculation that turns out to be wrong after additional observations or experiments is not a mistake. Sometimes these things turn out to be right, and fast and high-profile publication directs people to work on them right away and check them out.
In astronomy, we have a journal exactly for this: Astrophysical Journal Letters. Short papers, fast publication, more timely, may be more speculative:
Timeliness — A Letter should have a significant immediate impact on the research of a number of other investigators or be of special current interest in astrophysics. Permanent, long-range value is less essential. A Letter can be more speculative and less rigorous than an article for Part 1 but should meet the same high standard of quality.
Related to this post is the sloppy nomenclature of writers talking about “failed” experiments. Experiments test ideas, giving a result in support of, or contradictory to, a hypothesis or theory. An experiment is only a failure when it doesn’t provide a test. Science is about learning things, not confirming things. Engineering is about making things work. Don’t confuse the two.
You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.
I wanted to add that ApJ Letters, as it is called, is a prestigious place to publish. Science and Nature, as discussed in the Economist article, also, and those magazines also require very brief papers. New, exciting, and timely results don’t always stand up to scrutiny, and that’s ok. We still want to have them visible rapidly, yes?