Previously I referred to my experience on NIH study sections as “From the inside” and “From the other side“. Since I designated our science review officer “He Who Must Not Be Named*”, it’s only appropriate that my latest study section experience be “The dark side”.
At this week’s study section, the SRO showed up with a completely shaved head. Coincidence?
After the last study section I reviewed for, I gave a talk to my colleagues back home about the AREA grants. These are special grants NIH reserves for smaller institutions that are less competitive in general for NIH funding. The goal of most NIH grants is to fund good research. AREA grants have three goals: 1) to fund good research, 2) to improve research at these underfunded institutions, and 3) to encourage students to pursue a career in biomedical or behavioral sciences. Many of the AREA applications we reviewed completely missed #2 and especially #3. In my talk, I emphasized the importance of knowing the goals of the funding agency or funding mechanism.
I was particularly interested in the AREA grants that we reviewed this week. I was assigned to a couple of these, but I took a look at the others, checking specifically for student involvement. During the study section, I brought that up repeatedly whether I was an assigned reviewer on that grant or not. We’d had a new document this time to help us review these grants, but I’m not sure all the reviewers had noticed and read it.
The last time I reviewed, I made an effort to read all the grants, not just the ones I was assigned to review. I mostly succeeded. This time, life got ahead of me. I managed to get four of my five assigned grants reviewed before I left for a week on Biking Across Kansas, but I was hard pressed to get my fifth grant reviewed before the deadline, the day after I returned from Kansas. I had no chance to read the other grants, except for looking through the AREA grants for student involvement.
Let’s talk again about scores. I gave an overview of how we assign scores in an earlier post. After discussion, scores may change. We had a couple applications with a wide range of scores, which meant that some of the reviewers thought highly of the application and others thought less of it. The discussions were intense and almost heated, and ended with the reviewers maintaining their original opinion. That was, in my experience, a little unusual. Also it was fun.
The scoring system is 1-9, with 1 being the best and 9 the worst. Mostly the full range is used, with 9 being the rarest. For the best applications, the ones that will be funded, the scores are appropriate. I did feel that there was a tendency to gravitate toward the middle of the range, so that a wide range of quality of applications could get a 5 whereas some might deserve a 7. However, neither 5 nor 7 is a typically a fundable score so I don’t think that would make a difference in whether the grant was funded.
On one grant, the initial scores were 2, 2, 3, and 4. After discussion the scores were revisited: 4, 4, 4, and– the fourth reviewer said “1. No, just kidding, 4.” This is what passes for reviewer humor.
Although some grants are designated “ND” (not discussed), so as to not waste our time on grants that are not competitive, any reviewer who wants to discuss a grant can request that it be discussed. That happened for one grant this time.
I realized recently, talking to someone who has served on study sections for another NIH institute and for other types of grants (I have not reviewed R01s), that some of the details I’ve described are specific to the study section I serve on. However, the general operation applies across the board.
On the training grants, I feel a little frustrated by what I see as unrealistic expectations of K01 (a training grant designed to provide a “bridge to independence”) candidates. If a candidate was too new a postdoc, reviewers suggested the candidate should be applying for a different mechanism. If a candidate was too long in the tooth, reviewers suggested the candidate should be applying for an R21 or an R01. I wondered what the expected experience level of a postdoc is supposed to be for one of these. It seemed to me an applicant must be 3 and no more than 3 years from their PhD. However, no candidate was “perfect” so these factors affected their scores fairly equally.
Once again, I enjoyed reviewing the grants, but the distraction of Biking Across Kansas and a spot of poor health immediately following that trip and during the study section attenuated my enjoyment of the process.
I had one minor complaint, to which I now devote an entire paragraph, which is eerily like our reviews of the applications. (“This application is fantastic! Well done! Just one teeny tiny thing…”)
I was a little resentful that NIH will no longer provide refreshments during the meeting. I found the availability of fresh fruit all morning essential in helping me concentrate. The hotel generously provided free breakfast and I stocked up on fresh fruit, so I got by. But it seems like a short sighted, bureaucratic, and ineffective attempt to save money. A few months ago when I heard about this change, I filed my objections as far up the ladder as I could reach. My opinion was noted, but the policy is too general–it applies far beyond NIH. So it’s political. It’s too bad because I think the review process suffers. But that’s bureaucracy for you, and NIH is nothing if not bureaucratic!
*My SRO appreciates his new title.