I was on an NIH panel a couple of years ago with about 25 other scientists, reviewing something like 90 grants. It was pointless. 25 people is just too many to make a decision. What happened was that there were 3 or 4 people who were experienced in the process, who ended up guiding the entire discussion.
The highlight–or, I should say, lowlight–was when we were reviewing a proposal involving the study of the carcinogenic effects of hookah (water pipe) smoking. I asked if this was really such a big deal, and one of the panel members told me that smoking tobacco through a hookah is something like 10 times worse than smoking a cigarette. If so, the public health consequences could be pretty serious, even if not so many people did it. I said this sounded like a reasonable point to me. Then this guy across the table from me spoke up and said that he knew somebody who was 80 years old, had been smoking with a hookah all his life and was none the worse from it. At this point, I blew up. I couldn’t believe that the “my elderly aunt smokes and she didn’t get cancer” argument could be brought up at an NIH panel!
The NYTimes article on research funding is aptly classified under sociology. Funding small, "safe" projects is a very human failing and I have seen it even in my own area of financial services related modeling.
What happens is that there is a tendency to a. show activity and b. not to be seen as too radical. Hence the desire to fund projects that end up being very incremental in their impact. The project regarding dietary habits that got funding at the NIH, for instance. The impact of diet on cancer prevention is well known. So there was really little need to "reinvent" the wheel, by funding a project in this area. But that is precisely the failing that a lot of the prioritization process suffers from. The best way to be safe (i.e. not back a failure) and at small incremental investments is by funding a project in a space where the big leap has already happened and is less likely to fail.
The NIH doesn't really get enough credit for recognizing issues like these and combating them. I can think of several reforms that have been recently implemented or are soon to be implemented that address a lot of the issues brought up here. There's been a lot done by the NIH and elsewhere to fund high-risk/high-reward projects in the past few years, and those programs have ramped up a bit with the stimulus. There's also an effort to reign in the increasing age of folks receiving their first R01, a new scoring system to make program managers actually do their job and manage things rather than picking the highest score and saying that's that, and some work on getting more discussion between reviewers and applicants while the review is in process… speeding up the appeals system, etc. Work on the last point would address the hookah problem; I'd hope if you had 25 people that argument didn't go very far!
A better argument would be that it'd be a lot cheaper to field a study of hookah users in countries where a larger fraction of the population smokes out of one.
The article mentions things like the challenge grants, but the reforms going on are deeper than that.
One of the issues, with NIH grants in particular, is that there seems to be a real bias toward proposals that _look_ like conventional proposals.
That newspaper article discussed scientists who consciously hide their innovative work in order to get funding, but I don't think that's the major issue. What's more important, I think, is that there are many researchers for whom NIH grants ("R01's," in that horrible jargon) are the basis of their salaries. Most of these researchers do "normal science," because that's what most people do.
On the other side, at the reviewers' table, I see a big, big problem with reviewers demanding a level of predictability in the proposal writing that non-professional-grantwriters can't always do. Again, it's not that the reviewers are biased toward boring or incremental _science_, it's that they're biased toward proposals that look a certain way–and the people who know how to write proposals a certain way are those who write proposals all the time.
A (possibly) revealing anecdote: A few years ago, I met some colleagues in another department for a research meeting. We exchanged ideas, and afterward, they said, Hey, maybe we can write a proposal together. (Not, maybe we can write a paper together.) For them, the proposal was the basic unit. Having published a few hundred papers–most of which never seem to get noticed–I can understand this attitude, but it still disturbed me a bit.
And don't get me started on the whole power-calculation thing.
Back to the review panels: In my experiences on NIH review panels, I was repeatedly disturbed by the number of reviewers who wanted to ding proposals based on silly paperwork issues (for example, not fully filling out the human subjects forms for a study that involved no risk to the subject) or who liked proposals for little other reason that that they did all they were supposed to do. I didn't see this sort of paint-by-numbers attitude at other granting agencies.
Regarding the hookah story: I'd say my only positive contribution at that two-day meeting was to shout at that guy. Usually I'm pretty restrained, but stick me in a room with a bunch of people for two days, and my impatience starts to come out.
My favorite parts in the article were the references to the relationship between NIH grant funding patterns and monies made available to the NIH. They never came out and said, "this administration funded us, and that administration chipped away at our funding," but that is what happened.
Your anecdote about writing a proposal together brings up a recent problem (in my view) with funding agencies that isn't generally seen as a problem yet. I think huge, collaborative grants are a horrible idea, and they're sucking a ton of money out of the system. Very few of these grants appear to require being funded in this way (the proposed product requires work from all parties). There are about a million incarnations of "Institute for Buzzword, Buzzword, and Buzzword" that have sprung up in the past few years with little to show for it that couldn't have been accomplished by the individual labs.
Your thoughts about positive feedback in the review cycle (learn to create a type of proposal, then require that others write the same when you start reviewing) are interesting. Hopefully involving the applicant more in the review process with virtual reviews and things will help out with failure by clerical error. I was once a student in a department with an absolutely fantastic person on staff who ruthlessly edited the hundreds of proposals and papers produced every year … probably a better investment than anything/anyone else in the department, but it shouldn't be that way.
"High risk" research by definition is research that we don't know in advance whether it will pay off in meaningful results. Kolata's article is advocating for granting agencies to adopt a strategy that would result in less-frequent but (one hopes) bigger payoffs compared to the current system. That leads to a couple of questions:
1. How do we know that the net result (risk X payoff) would be better than the current system? I'm not saying that it wouldn't be, just that it seems to be an unchallenged assumption in the discussion.
2. How should we choose which risky proposals to fund, since "high risk" by definition means "uncertain merit"? Presumably there is an even larger pool of potential risky proposals than the current pool of safe ones. Given a fixed pool of money to hand out, wouldn't we have to turn to non-merit-based criteria for funding?
Note: this is all separate from the discussion of whether review panels are using the right criteria (like in the hookah example — an example of what Paul Meehl colorfully labeled the "Uncle George's pancakes fallacy" 35 years ago). I'm sure we could get panels to make better use of valid predictive information. But that's an incremental improvement in the granting system. Kolata's article seems to be suggesting something more radical — that we should fund proposals where the fundamental scientific merit is less predictable.
Another unstated assumption in the article is (or seems to me to be) that funding for high risk research should come from the NIH. Perhaps the NIH should fund boring incremental research, while funding for high-risk research should come from somewhere else.
The pharmaceutical industry has a system for handling high risk: profits from a few really successful drugs pay for development of that drug and all the others that never made it to market. Of course the profits come after the development, but they deal with that by borrowing and issuing stock.
Another alternative source of funding is private foundations. Some foundation grants have less demanding criteria than the NIH (e.g. no requirement for preliminary data) and are, at the same time, less generous, so we try to use them as seed money, to fund high risk "pilot" studies. If the pilot study works out, then the research is no longer quite so high-risk, and the pilot data's used for an NIH grant application.
The name of the game, as everyone (including program officers) knows, is to apply for an NIH grant when the research is already half-completed, then use the money to half-complete the research for the *next* grant.
The "F&A" (formerly: "indirect cost") rate for NIH grants is typically 50% or more (because these grants, while ostensibly paying for specific pieces of research, are, in reality, subsidizing institutions). $1 of NIH grant money costs the NIH $1.50. $1 of private foundation money might cost the foundation as little as $1 The more expensive the investment, the less risk you'll tolerate.
The NIH has invented funding mechanisms for high-risk research. I'm told by awardees, though, that "creeping conservatism" has infected these as well.
PS: nice that a statistician was on a study section. There was none on mine. (Though I shouldn't complain: it would probably have hurt me.)