Research Eyes

Previously I referred to my experience on NIH study sections as “From the inside” and “From the other side“. Since I designated our science review officer “He Who Must Not Be Named*”, it’s only appropriate that my latest study section experience be “The dark side”.

At this week’s study section, the SRO showed up with a completely shaved head. Coincidence?

After the last study section I reviewed for, I gave a talk to my colleagues back home about the AREA grants. These are special grants NIH reserves for smaller institutions that are less competitive in general for NIH funding. The goal of most NIH grants is to fund good research. AREA grants have three goals: 1) to fund good research, 2) to improve research at these underfunded institutions, and 3) to encourage students to pursue a career in biomedical or behavioral sciences. Many of the AREA applications we reviewed completely missed #2 and especially #3. In my talk, I emphasized the importance of knowing the goals of the funding agency or funding mechanism.

I was particularly interested in the AREA grants that we reviewed this week. I was assigned to a couple of these, but I took a look at the others, checking specifically for student involvement. During the study section, I brought that up repeatedly whether I was an assigned reviewer on that grant or not. We’d had a new document this time to help us review these grants, but I’m not sure all the reviewers had noticed and read it.

The last time I reviewed, I made an effort to read all the grants, not just the ones I was assigned to review. I mostly succeeded. This time, life got ahead of me. I managed to get four of my five assigned grants reviewed before I left for a week on Biking Across Kansas, but I was hard pressed to get my fifth grant reviewed before the deadline, the day after I returned from Kansas. I had no chance to read the other grants, except for looking through the AREA grants for student involvement.

Let’s talk again about scores. I gave an overview of how we assign scores in an earlier post. After discussion, scores may change. We had a couple applications with a wide range of scores, which meant that some of the reviewers thought highly of the application and others thought less of it. The discussions were intense and almost heated, and ended with the reviewers maintaining their original opinion. That was, in my experience, a little unusual. Also it was fun.

The scoring system is 1-9, with 1 being the best and 9 the worst. Mostly the full range is used, with 9 being the rarest. For the best applications, the ones that will be funded, the scores are appropriate. I did feel that there was a tendency to gravitate toward the middle of the range, so that a wide range of quality of applications could get a 5 whereas some might deserve a 7. However, neither 5 nor 7 is a typically a fundable score so I don’t think that would make a difference in whether the grant was funded.

On one grant, the initial scores were 2, 2, 3, and 4. After discussion the scores were revisited: 4, 4, 4, and– the fourth reviewer said “1. No, just kidding, 4.” This is what passes for reviewer humor.

Although some grants are designated “ND” (not discussed), so as to not waste our time on grants that are not competitive, any reviewer who wants to discuss a grant can request that it be discussed. That happened for one grant this time.

I realized recently, talking to someone who has served on study sections for another NIH institute and for other types of grants (I have not reviewed R01s), that some of the details I’ve described are specific to the study section I serve on. However, the general operation applies across the board.

On the training grants, I feel a little frustrated by what I see as unrealistic expectations of K01 (a training grant designed to provide a “bridge to independence”) candidates. If a candidate was too new a postdoc, reviewers suggested the candidate should be applying for a different mechanism. If a candidate was too long in the tooth, reviewers suggested the candidate should be applying for an R21 or an R01. I wondered what the expected experience level of a postdoc is supposed to be for one of these. It seemed to me an applicant must be 3 and no more than 3 years from their PhD. However, no candidate was “perfect” so these factors affected their scores fairly equally.

Once again, I enjoyed reviewing the grants, but the distraction of Biking Across Kansas and a spot of poor health immediately following that trip and during the study section attenuated my enjoyment of the process.

I had one minor complaint, to which I now devote an entire paragraph, which is eerily like our reviews of the applications. (“This application is fantastic! Well done! Just one teeny tiny thing…”)

I was a little resentful that NIH will no longer provide refreshments during the meeting. I found the availability of fresh fruit all morning essential in helping me concentrate. The hotel generously provided free breakfast and I stocked up on fresh fruit, so I got by. But it seems like a short sighted, bureaucratic, and ineffective attempt to save money. A few months ago when I heard about this change, I filed my objections as far up the ladder as I could reach. My opinion was noted, but the policy is too general–it applies far beyond NIH. So it’s political. It’s too bad because I think the review process suffers. But that’s bureaucracy for you, and NIH is nothing if not bureaucratic!

*My SRO appreciates his new title.

At least in the life sciences, research involves an awful lot of trying the same thing over and over again hoping for different results. Albert Einstein said the same thing about insanity. Insanity = science? It could be.

Here is a brief background of the experiment I’m attempting. In our grant we proposed the hypothesis that bending the leg back and forth would cause changes in gene expression in the spinal cord. (The rationale for this…never mind, let’s not go there.) Before I joined the project, a genome microarray of 50,000 genes showed changes in gene expression in 180 genes. (Again, let’s not go into the probability that three rats would have changes in gene expression if you looked at 50,000 genes even if you did NOTHING to them. Like I said, all this was done before I joined the team.)

Microarray results have to be confirmed with quantitative RT-PCR. RT-PCR stands for Reverse Transcriptase-Polymerase Chain Reaction. Basically that means you take a strand of RNA, create a DNA copy, then copy that. When you copy DNA, each cycle doubles the  DNA. The “quantitative” part means that a bit of fluorescence labels each DNA molecule and a machine reads how bright the fluorescence is at the end of each cycle. All this is converted back into relative numbers so we can compare the amount of RNA that was in the original samples.

Ideally, in the end we are able to say, “This sample had more of this RNA than that sample did.”

My explanation is probably too complicated for most of my readers to follow and too simple (ie inaccurate) for the biosci readers.

All that is still a long way away for me. As each experiment failed to show any RNA at all, I peeled it back step by step. Frustrated at inconsistent RNA yields, I finally gave up using the procedures that had been used by the team, and ordered stuff I’ve used before successfully. As in, I’ve published on it.

I got good RNA. (Qiagen’s RNeasy. Great stuff.)

I moved on to RT-PCR, just the regular RT-PCR, no “q”. Nothing. I tried different primers. At this point I’m not even trying primers for the genes I’m trying to confirm. I’m just trying to get control primers to work.

I ordered control RNA, control primers, and GAPDH primers. I don’t care about GAPDH, and I don’t want to use GAPDH as an internal standard. I want to use a ribosomal RNA, such as Rpl27, as an internal standard. The reason I ordered GAPDH primers is because that is one thing I know every cell in every rat has. If I can find GAPDH in my rats, then I can focus on what’s wrong with the primers for the genes I care about.

(Sorry for the non-researchers who don’t know what primers are. Don’t worry about it. I think you’ll get the point of this article anyway.)

After repeating an RT-PCR over and over again, it finally happened. I got different results.

That’s right, ladies and gentleman. Three lovely bands. Lane 1 is the new RNA with GAPDH primers. Lane 2 is the old RNA with GAPDH primers. Lane 3 is control template and control primers.

Today I did another RT-PCR, testing 5 Rpl27 primers, GAPDH, and a negative control that had GAPDH primers but no RNA.

I got two, and only two, lovely bands. One in the GAPDH lane, and one in the negative control lane.

This probably isn’t clear to non-scientists. “Negative control” means there is NOT supposed to be a band. Clearly I have offended the Flying Spaghetti Monster.

To be quite honest, I don’t exactly do the same thing over and over again expecting different results. In one of the experiments I suspected one of the pipettors was sometimes malfunctioning. I had been doing this upstairs in the lab that has the thermocyclers. I switched to doing it in my own lab with pipettors that I trust. I checked every time I pipetted to make sure there was an appropriate volume of liquid in the pipet tip. And that it expelled.

Another time I thought the gel was questionable. One of the labs upstairs uses TBE. The other one uses TAE. I had made the gel with TBE, and I suspected that the electrophoresis chamber had TAE in it. I made up new TAE, emptied and rinsed the electrophoresis chamber, made a new gel with the new TAE and filled the chamber with the new TAE.

This latest result, where both my positive and negative control had bands? Although I thought I had been extremely careful, I don’t remember with 100% certainty that I put water and NOT TEMPLATE in the negative control tube. So I’ll do it again…and hope for different results.

On the other hand I’m pretty darn confident now that NONE of the five primer sets for Rpl27 work. I have a couple options. I can try to monkey around with the Mg concentration and the primer concentration. I’ve used all of PrimerQuest’s 5 suggested primer sets, so I could try another primer generator software and see what it comes up with. I can assume the published Rpl27 sequence is wrong, perhaps my rats have a different allele than the published sequence (I don’t know how highly conserved it is, although I’d think fairly conserved since it is a ribosomal RNA), and pick a different gene, like Rpl13. I’ll probably order primer sets for Rpl13, and while waiting for them to arrive I’ll play with the Mg and Rpl27 concentrations.

After I eat my spaghetti and repeat the positive and negative controls.

January 2009, a lab accident resulted in the death of a lab tech. She and the postdocs who came to her aid were poorly trained to cope with accidents, and her university, UCLA, and the primary investigator, Dr. Harran, are facing criminal charges for willful negligence. This is a landmark case because it could change the way science is done.

Whether the outcome is in favor of UCLA and Dr. Harran or not, there is potential here for major change throughout academia. The case will probably come to plea bargain, and Dr. Harran and UCLA probably won’t experience in jail time.

Dr. Harran’s defense is that he was acting like any other professor would act. That is true. My opinion is that does not exonerate him. It exposes an enormous employee safety gap. Academia sorely needs to be held accountable in its labor transgressions. Safety is perhaps its most egregious violation. Paying its administrators ridiculously large salaries while paying adjuncts a pittance is another labor violation. Sheri Sangji’s death will be a wake up call to universities and professors for safety procedures, but not for the exploitation of adjuncts, postdocs, and grad students.

UCLA, and other universities, haven’t yet reacted to the wake up call by implementing more rigorous safety oversights. Individual professors would be well advised to do anything in their power to change how they operate. Not only will that save the lives of their lab personnel and protect the professor herself in the event of a tragedy like Sangji’s, but it will also prevent others like Dr. Harran from using the excuse, “This is what all the professors are (not) doing.”

Once again I’m on an NIH review panel. I just read my post from the first time I did this, and I’m a little amazed at how similar the process was. Ok, so we had a lot of the same reviewers, and the science review officer (he who must not be named) was also the same, and the funding mechanisms that we reviewed were all the same. Still, it’s good to know there really isn’t much in the way of variation from one study section to the next.

Once again I am impressed with how NOT random the process seems, at least from this side. But it’s easy to see how random it appears to the grantees. Especially if their grant was Not Discussed. Maybe 1/3 of the grants or so are Not Discussed. They still receive summary critiques from the 3 or 4 assigned reviewers. And if any reviewer wants the grant to be discussed, it will be. So the idea that one reviewer having an off day can kill your grant isn’t really true. If two reviewers thought your grant was great, and one thought it wasn’t, and that caused your initial score to be low enough that it was designated Not Discussed, the two reviewers who thought it was great will speak up for it.

More likely though, is that two reviewers might think your grant is mediocre, and one might think it is abysmal, and no one speaks up for it.

Reviewers who are enamored with a grant will champion it. Others might voice valid objections, and the champion will argue for why this grant deserves a good score anyway.

What was most striking this time around was how many flaws even the strongest grants have. I can’t help but think that next time I submit a grant to NIH (still a far off event, as we are a long way from having enough preliminary data for that), my grant will be solid because I’ll know exactly what they’ll look for.

Then I laugh at myself for being naive. I’ll find new ways to make mistakes!

A common mistake is how people respond to reviewers. If a grant is rejected, it can be resubmitted once. The resubmission is a good opportunity. Most of the reviewer comments are fairly sound and can only improve the project. Statistics are frequently omitted from first time submissions, for example. Some reviewer comments show the grantee that something was misunderstood. That’s easy to respond to: explain (respectfully) what you really meant. And rephrase it so that it can’t be misunderstood again. Some reviewer comments you just disagree with. That’s ok too! Just explain (respectfully) why you disagree.

What I was seeing today was more than one resubmission that ignored reviewers’ comments, disagreed but not respectfully, or disagreed but didn’t explain why.

For example, suppose the reviewer said “This project is too much to be completed in three years.”
Wrong response: “No it isn’t.”
Correct response: “Aim 2 is now Aim 2 and 3. I removed the original Aim 3.”
Correct response: “I have included a timeline that illustrates that it is reasonable to complete this in three years.”

One thing that has really bothered me about grants I have submitted is when the reviewer completely misunderstands something. Did he even read it? Well, these grants are read surprisingly thoroughly. If one reviewer misunderstood something, other reviewers will catch that and point it out. We even look things up for ourselves to better understand the qualities of a proposal.

Last time, I said I was going to try to read all the grants, not just the ones assigned. I didn’t quite succeed. Of the ones I did read, I didn’t read them nearly as thoroughly as the ones I critiqued, but I was looking for specific things that I could find or fail to find easily. I still didn’t make it through all of them. I decided not to read the conference grants, or the educational program grants. They have a different critique form so I don’t really know what I’m looking for on those. (Maybe next time I’ll go ahead and critique one even though I’m not assigned to, just so I know what to look for.) There were still a few I hadn’t read by the time we started this morning. (I read them while we were reviewing the conference grants and the educational program grants.)

As we were reviewing, I came up with a strategy for next time. I’ll read my assigned grants & critique them. Then I’ll wait until the initial scores are in and only read the ones that are likely to be discussed. Plus any others that catch my eye.

On the other hand, reading the ones that are Not Discussed is a good way to get a feel for the range of scores.

Research isn’t done in a vacuum. We rely on the knowledge already out there to decide what questions to ask and how to find answers. While it is possible for a research idea to spring up out of nowhere like Athena bursting forth from Zeus’ head, more commonly even if the question is novel, we use established methods to find answers. During today’s morning run, I was thinking about the many times I’ve based a large project on published evidence and methods, and nothing worked ever.

A responsible mentor will encourage grad students to use established methods and not try to do anything ground breaking and risky, because they risk never graduating. Not just use  established methods, but methods that have been used successfully in his or her lab, so the grad student can learn it from people instead of from a paper. A researcher further along in his or her career can afford to take that risk.

When I was a grad student my mentor suggested a project that required a method we hadn’t done in our lab, but a collaborator was doing successfully. She had published a couple papers on it. We spent a day in her lab and went home and I tried to get it going in our lab. We had a phone meeting or two with her, and some emails, so she was continuing to help us. Then in the middle of it all she quit her job. We never did get the experiment quite the way we wanted it, although we did manage to get some data from that project. Fortunately that wasn’t my only project, so I had other data to fall back on for my dissertation.

As a postdoc I came up with another research question and dug into the literature to find out how best to answer it. There were a few somewhat questionable models of this disorder, nothing well established and they didn’t model the disorder very well. I found a couple reports, from a German lab, that seemed promising. I tried to contact them but they never responded. I designed a project around their method anyway, got a grant, and started trying to test it. It didn’t work at all.

Next I tried another method to test a different disorder. A researcher at a university I had been at had developed this one. I knew of her, if I didn’t know her personally. I tried it out exactly as she described in her paper (which had been published in a high-impact journal). It didn’t work as she described. I called her and asked if she had any suggestions. Her response: “We’ve been having difficulties replicating our results.” Her own lab couldn’t repeat the study they had published in a highly respected journal!

When that sort of thing happens, should the authors retract their original paper? I don’t think so, because the data are the data. Statistics aren’t perfect, they are 95% perfect, or 99% (depending on where you set your level of confidence). They didn’t do anything wrong in the original study. But they should publish the follow up study too. Unfortunately, no journal is going to want to publish that. Journals will publish “Hey we’ve got this new thing!” or “We have evidence that the other guys were wrong”. But they won’t publish “We have evidence that shows the last paper we published is wrong.” It suggests that you didn’t know what you were doing last time, and if you didn’t know what you were doing then, why should they trust that you know what you are doing now? Even though it happens to everyone.

You’d think after 3 papers like that I’d have learned. I certainly felt burned, and felt like I’d learned something. I’m very cautious about other papers. But nonetheless I got in a situation where I was taking over a project that was designed long ago, based on published data and with the cooperation and assistance and advice of the lab who published that. Should be safe enough, but the original method they had published on hadn’t worked in our team’s hands. The team came up with another method, one that hadn’t really been used for this. (In fact I’m not sure where the idea came from.) It seemed, in theory, a much better method for many reasons. But it hadn’t been published on in this context. The first experiment, with just 8 animals per treatment, worked great. The treated rats performed better than the controls. Then the next set of rats came in and the effect was the opposite. They repeated the experiment on 5 sets of rats. It was at that point I joined the team and started crunching the data. There was absolutely no effect of treatment.

We never did get what we hoped for out of that study. We couldn’t find a valid positive control, among other things.

My initial reaction is to wonder how I can avoid this in the future. The German group that refused to speak to me is a big red flag. And always consulting with the authors long before I even start the project is another step, so long as they are honest with me about little things like “We haven’t been able to replicate our PNAS results”. But some of my experiences were with collaborators. They didn’t do anything wrong, but we still ended up investing a lot into projects that went no where.

My second reaction is to wonder if this isn’t part of a bigger picture. Something to do with too little rigor on publications. Not enough statistical training for authors or reviewers. A result of the pressure to publish or perish. When the result of a rejected paper is a dead career, authors fight hard against that rejection. I know, I’ve been there.