Once again I’m on an NIH review panel. I just read my post from the first time I did this, and I’m a little amazed at how similar the process was. Ok, so we had a lot of the same reviewers, and the science review officer (he who must not be named) was also the same, and the funding mechanisms that we reviewed were all the same. Still, it’s good to know there really isn’t much in the way of variation from one study section to the next.
Once again I am impressed with how NOT random the process seems, at least from this side. But it’s easy to see how random it appears to the grantees. Especially if their grant was Not Discussed. Maybe 1/3 of the grants or so are Not Discussed. They still receive summary critiques from the 3 or 4 assigned reviewers. And if any reviewer wants the grant to be discussed, it will be. So the idea that one reviewer having an off day can kill your grant isn’t really true. If two reviewers thought your grant was great, and one thought it wasn’t, and that caused your initial score to be low enough that it was designated Not Discussed, the two reviewers who thought it was great will speak up for it.
More likely though, is that two reviewers might think your grant is mediocre, and one might think it is abysmal, and no one speaks up for it.
Reviewers who are enamored with a grant will champion it. Others might voice valid objections, and the champion will argue for why this grant deserves a good score anyway.
What was most striking this time around was how many flaws even the strongest grants have. I can’t help but think that next time I submit a grant to NIH (still a far off event, as we are a long way from having enough preliminary data for that), my grant will be solid because I’ll know exactly what they’ll look for.
Then I laugh at myself for being naive. I’ll find new ways to make mistakes!
A common mistake is how people respond to reviewers. If a grant is rejected, it can be resubmitted once. The resubmission is a good opportunity. Most of the reviewer comments are fairly sound and can only improve the project. Statistics are frequently omitted from first time submissions, for example. Some reviewer comments show the grantee that something was misunderstood. That’s easy to respond to: explain (respectfully) what you really meant. And rephrase it so that it can’t be misunderstood again. Some reviewer comments you just disagree with. That’s ok too! Just explain (respectfully) why you disagree.
What I was seeing today was more than one resubmission that ignored reviewers’ comments, disagreed but not respectfully, or disagreed but didn’t explain why.
For example, suppose the reviewer said “This project is too much to be completed in three years.”
Wrong response: “No it isn’t.”
Correct response: “Aim 2 is now Aim 2 and 3. I removed the original Aim 3.”
Correct response: “I have included a timeline that illustrates that it is reasonable to complete this in three years.”
One thing that has really bothered me about grants I have submitted is when the reviewer completely misunderstands something. Did he even read it? Well, these grants are read surprisingly thoroughly. If one reviewer misunderstood something, other reviewers will catch that and point it out. We even look things up for ourselves to better understand the qualities of a proposal.
Last time, I said I was going to try to read all the grants, not just the ones assigned. I didn’t quite succeed. Of the ones I did read, I didn’t read them nearly as thoroughly as the ones I critiqued, but I was looking for specific things that I could find or fail to find easily. I still didn’t make it through all of them. I decided not to read the conference grants, or the educational program grants. They have a different critique form so I don’t really know what I’m looking for on those. (Maybe next time I’ll go ahead and critique one even though I’m not assigned to, just so I know what to look for.) There were still a few I hadn’t read by the time we started this morning. (I read them while we were reviewing the conference grants and the educational program grants.)
As we were reviewing, I came up with a strategy for next time. I’ll read my assigned grants & critique them. Then I’ll wait until the initial scores are in and only read the ones that are likely to be discussed. Plus any others that catch my eye.
On the other hand, reading the ones that are Not Discussed is a good way to get a feel for the range of scores.