Evidence based medicine

Lies, Damned Lies, and Medical Science by David Freedman

We are taught that modern medicine is “evidence based medicine”. Evidence based medicine means rigorous scientific studies underlie every aspect of medical practice. Particularly drugs are supposed to be grounded in randomized, double-blind, placebo controlled trials. This means that the drug is tested against a placebo, something that looks, smells, feels and tastes just like the drug but without the drug. It means the participants are randomized to receive either drug or placebo. And it means that neither the participants nor the data collector know whether a participant is receiving drug or placebo.

This article reveals that there is great incentive to bias a study, either knowingly or unknowingly, that studies are frequently biased, and that most of medicine is pretty far from being evidence based. Including, and especially, the drugs. This quote is particularly shocking: “[John Ioannides] charges that as much as 90 percent of the published medical information that doctors rely on is flawed.”

One incentive is that it is simply nearly impossible to publish negative results. “X doesn’t work” isn’t nearly as interesting as “X cures cancer”. It’s difficult to publish confirmatory results. “Breaking news–X cures cancer” is more interesting than “Dr. Lab confirms Dr. Superstar’s findings last year that X cures cancer”.

So how unbiased is science? I’m sure no one is really surprised to find out that although we are taught the scientific method, that actual science is pretty biased. It’s all a big religion. I don’t have anything against religion, but science claims to be different than religion.
Public Library of Science (PLoS) has seven journals which publish studies with sound study design, including negative results. Articles are peer reviewed and published online. That’s a step in the right direction.

Posted in Uncategorized | Tagged , , , | Comments Off on Evidence based medicine

Mao

The card game “Mao” is so much like academia it should be named Academia, or perhaps academia should be called Ruthless Bloodthirsty Dictator Land. Since I’m not allowed to talk about the rules of Mao, I will illustrate by way of example. Nell taught many of us how to play Mao over Christmas. Some of us had experienced the game before, some had not. The game went sort of like this. (I don’t remember every detail exactly.)

Nell announced the rules. “The first rule is: I can tell you the first three rules. The second rule is: We can’t talk about the rules. The third rule is: Every time a game is won, a new rule is added. So after 5 games, there will be 5 new rules.”

She dealt the cards. We were chatting and some people naturally picked up their cards. “PENALTY for touching your cards during Point of Order,” she said sternly, passing out penalty cards. They sheepishly put their cards down. “End Point of Order,” she announced, and picked up her cards.
“Can we–” Dad started to ask.
“PENALTY for speaking out of turn,” she sternly handed another penalty card over.
We waited. “PENALTY for failing to play within 5 seconds.” Another penalty card to Dad.
Jon was to Dad’s left. He shrugged. He tossed a card onto the pile.
“PENALTY for failing to play in suit.”
Katie to his left tossed a card out. This one was in suit. No penalty.
The game continued.
“PENALTY for failing to knock. PENALTY for playing out of turn (certain cards, it appeared, reversed turn order, or skipped the next player).” And, most often, “PENALTY for speaking out of turn!”

The game was won, and the next game began. Jon had caught on, and was (illegally) helping Katie by grabbing her arm when she started to play out of turn after a reverse or a skip. But a new rule had been added. No one knew what the new rule was until we heard, “PENALTY!”

In Mao, in Academia, and in dysfunctional families, players are routinely penalized for breaking rules they didn’t know existed. Players are expected to know rules without being told. Players can rant and rave against the system, or they can adapt, learning from their own and others’ infractions, accepting the penalties they can’t avoid and accumulating as few penalties as they can.

I wrote this a couple weeks ago, and delayed posting it until after a recent party where Nell got to teach several people how to play Mao. Only they didn’t cooperate. One threw down 6 cards at once. “Penalty,” Nell scolded and returned the 6 cards plus a penalty card. Only while she was doing that everyone else started throwing down cards. And talking. And breaking all of the rules at once. Showing that unionization and solidarity can beat even Mao. At least if Mao is a sweet little 15 year old girl without too many tanks and weapons of mass destruction.

Posted in Uncategorized | Tagged , | Comments Off on Mao

Computers as Doctors

IBM’s computer “Watson” won Jeopardy recently. IBM’s claim that Watson’s progeny could benefit the medical community led Carey Goldburg to speculate on what the future of medicine may hold in this article, ‘The McDonald’s Rx’: How Computers Can And Should Change Doctoring. While the prospect could put doctors out of jobs, she thinks many doctors won’t view this as a bad thing. Many doctors are dissatisfied, because after becoming highly trained their skills languish and instead they do paperwork. The field of medicine emphasizes standard of care. Ideally, a patient would receive the same treatment regardless of which doctor she sees.

This is another case of a field encountering the problem of overeducation.

According to the CIA, the US is 99% literate. That’s good. I wouldn’t trade that for anything. If having a 99% literacy rate means we also have unemployed PhDs, I’ll take it. But I don’t think it does.

We decry the state of our schools, and justly so. They are deplorable. We mock the college system in which astronomy students graduate from Harvard believing that the phases of the moon are caused by the earth’s shadow. And yet we have unemployable PhDs and physicians doing paperwork. People regret the time and money they wasted getting their BS, their MBA, their PhD which left them no more employable and sometimes LESS employable– and deeply in debt.

I don’t pretend to know how this has come about. (Probably it has something to do with education as a business, recruiting students who won’t benefit and can’t pay for school. Something to do with lawmakers nobly “improving” education, making it possible for more and more people to take out loans and go to school, when school isn’t necessarily the best thing for them.) I don’t pretend to know the solution either. I want everyone to have a chance, impoverished & disadvantaged kids, and lower middle class kids who can’t get a break.

My only argument is that there IS a problem. Education for everyone is idealistic, but in practice it means everyone getting an expensive piece of paper which does not translate into knowledge. Once upon a time a high school diploma was something special, something not everyone had or was expected to have. Now it is standard and a college degree is moving that direction too. Where does it stop? Is it possible to calculate an appropriate amount of education that will yield responsible citizens and voters but not dissatisfied janitors?

Posted in Uncategorized | Tagged , | Comments Off on Computers as Doctors

The Lab, Part II

I recently reviewed “The Lab”, an interactive video produced by the Office of Research Integrity, in which you can play one of four people who had opportunities to minimize the damage of another person’s misconduct. I had made a beeline for playing the part of the PI, who I believe has the most responsibility (other than the actual perpetrator), the most ability to influence the outcome, and the least risk if he chooses to do nothing.

Recently I spent another half-hour playing the part of the graduate student who catches the fraud. Again, the acting (for a training video) is good quality, the dialog is not forced and stilted. As far as the storyline, one scene actually brought tears to my eyes: her mother, who had not been supportive of her stepping forward with her accusations, has a change of heart after mom’s minister told the parents that their daughter, far from being a failure, is a hero.

I was not quite as satisfied with the video’s portrait of the grad student’s experience as I had been with the PI perspective. Realistically, there is no good outcome for the grad student, whether she comes forward or not. In fact, the outcome of her having her own lab years later is pretty unrealistic no matter what anyone else had or hadn’t done. In the worst case scenario, years later she is horrified because that postdoc had finally gotten caught and all of his publications, including the ones she was in, were under investigation. As a PI, is that really going to effect her much? Most graduate students in her situation, facing the decision of whether or not to report fraud in the lab, would NOT be swayed by some potential future repercussions, particularly as the odds of them actually landing a tenure track job are so slim in the first place.

The best case scenario, in which she reports the fraud, is also not compelling. She experiences retaliation for having come forward, everyone knows it was her even though the Research Integrity Officer claimed to protect her anonymity. She ends up leaving her lab and joining another, which sets her back a year because she has to switch projects. In this day and age, she can find a lab that has funding for her? She is set back ONLY a year?

Suppose the misconduct had never occurred. The implication was that the postdoc’s fraud made the lab shine, so it was well funded and prestigious, and everyone in the lab benefited. Without the misconduct, the lab would have been average, and her prospects not as shiny. Without funding, her project would suffer, toiling with old methods and making her own reagents (and mistakes) would delay her finishing. She wouldn’t get the prestigious postdoc and she wouldn’t get the tenure track position.

If I had written her story, I’d have had her get a job as a Research Integrity Officer after going through what she did to report the misconduct. The way that academia works right now, there is no good outcome for a graduate student in a lab where fraud occurs, whether or not the fraud is reported, whether or not the grad student is the one to report it.

One thing I found very interesting is that the storyline for “best possible outcome” changes if you play the grad student vs. the PI. As the PI, you had the chance to build a culture of trust, and in the best outcome your grad student came to you, you took her concerns seriously and you contacted the Research Integrity Officer. When you play the grad student, there is no culture of trust in place. If you talk to your PI first, nothing happens and you end up going to the Research Integrity Officer yourself. I like that the “default” situation is that there is no culture of trust. Because if you are in one of the few labs like that, you don’t need this training video.

I’m hooked now, I’ll be playing the other postdoc (not the perpetrator) and the Research Integrity Officer next!

Posted in Uncategorized | Tagged , , , , , , , | Comments Off on The Lab, Part II

The Lab, an interactive video on science misconduct

The Office of Research Integrity developed an interactive video for Responsible Conduct of Research (RCR) training, called “The Lab”. I am very impressed. The video was very well done. Only one actor seemed distractingly stilted, the dialog was not distractingly forced and stiff, and it’s a clever interactive video, like a Choose Your Own Adventure.

More importantly, although it is a postdoc who commits the misconduct in this video, you never play the part of that postdoc. The point of the training is that many people have the opportunity to prevent or minimize the damage of misconduct, other than the perpetrator. I was very pleased to see that one of the parts you can play is the principal investigator (PI) of the lab.

It took a good half-hour to play through the video, and being of the scientific mind that I am, I went through it making all the obviously wrong choices, the obviously right ones, and combinations. Therefore I’ve only played the PI, but I want to go back through it and play the other roles at some point.

There are some spoilers in the rest of this post, so if you want to try out the video yourself, go do that before reading any further.

I’m impressed that the video from the PI point of view emphasized strongly that publications, grants, and prestige are not the only metrics of success, and the PI in the video comes to the conclusion that the MOST IMPORTANT measure of success is the success of your students and postdocs. The only way to have successful students and postdocs is to devote your own time and resources to mentoring, even if it is at the expense of publications, grants, and prestige.

If you choose wrongly consistently, you ultimately get fired. If you make some good decisions and some bad, the postdoc gets away with it, and you don’t get fired. If you make all the right decisions, namely spending appropriate amounts of time on mentoring and developing a lab culture of openness, fairness and respect, you are able to catch the misconduct because your students felt comfortable coming to you with their concerns and you took them seriously.

Kudos to ORI for this great video.

Posted in Uncategorized | Tagged , , , , , , , | 2 Comments

Can old professors learn new tricks?

A matter of trust, by Christie Rizk, is about fraud and scientific misconduct. In many cases of science misconduct, trainees (postdocs and graduate students) are eventually “convicted”. One might conclude that trainees are therefore more likely than people further up the ladder to commit misconduct. One might reason this is because they have less to lose and more to gain. Universities and institutions tasked with addressing Responsible Conduct of Research (RCR) might therefore target trainees with their initiatives to prevent misconduct, by requiring ethics classes and RCR training.

Or is it that their role models are more skilled at getting away with it? It has always bothered me that only trainees are required to take these classes. Universities argue that trainees are the “low hanging fruit”. With so many demands on their time, they reason, it is impossible to require faculty to complete RCR training. The trainees will one day be faculty so eventually, the faculty will consist of people who have completed RCR training (but will they remember their training from 20 years ago?)

In “A Matter of Trust”, Ms. Rizk interviewed Aine Donovan, director of Dartmouth College Ethics Institute. I think we’re doing a really good job right now with undergrads, graduates, and postdocs,” she quotes Donovan, “but then someone goes into a lab with a 62-year-old lab director who’s kind of a moral reprobate, and all that goes out the window.” Donovan believes that principal investigators need the same RCR training that trainees get.

This seems obvious to me, but this is the first time I’ve seen the sentiment expressed by someone with authority. Thank you Dr. Donovan!

Posted in Uncategorized | Tagged , , | Comments Off on Can old professors learn new tricks?

CellCraft

From www.freewebarcade.com, my 15 year old daughter is playing CellCraft. She complains it insidiously educates her. She just yelled, “I need more ATP! Who makes ATP?”
“The mitochondria,” I answered.
“Thanks,” she said, and grumbled that she was running low on nucleic acids. And amino acids.
“This is a Super Cell,” she told me. “It has both chloroplast and mitochondria.” Her dad looked at us blankly as we chuckled at the joke.

Posted in Uncategorized | Tagged , , | Comments Off on CellCraft

Grant considerations

Some professors have funny attitudes about what they could do with their grant money. As long as an experiment is sort of related to the general topic of the grant, they reason, it was fair game. When ordering supplies, the factor that determines which grant pays for it is whichever grant needs spent out, that is, whichever one is closest to ending.

In writing a grant proposal, they have little concern over investigating too closely whether a project is feasible. If it doesn’t work out the way they imagine it, they reason, they can always try a new approach, or “tweak” the aims. They have a much broader definition of the word “tweak” than most folks.

The lesson I was taught as a trainee was that funding agencies weren’t too particular about what was done with the money after it was awarded. One professor told me that funding agencies NEVER want their money back. That they will bend over backwards to work with you! When I got my own grants I discovered that funding agencies were not quite as willing to work with you as that, and not so keen about you “tweaking” the specific aims. (I’m a little bitter about that. Sure I learned the correct lesson thoroughly, but I would have learned it well the easy way. I suppose as a result of having learned it the hard way I am uber-respectful of funding agencies, I’ll track every penny, justify every expense and never have to worry about weathering an audit.)

When it comes to eligibility requirements on grants, these are guidelines rather than unbendable rules. Don’t try to hide something that might make you ineligible, but talk to the program officer before you dismiss a grant as one you are ineligible for. I’ve found that often, the guideline is a simple statement, but what they really want is more complex, and they’re willing to work with you if you fit what they really want. For example, one postdoctoral fellowship was for postdocs who had received their PhD less than 2 years ago, according to the eligibility requirements. But when I spoke to the program officer, the purpose of the fellowship was to encourage trainee scientists who were new to cancer research. At that point I had only been in cancer research for a year, so she reckoned I qualified.

But when it comes to allowable expenses, if you have any uncertainties at all, talk to the program officer before spending the money. It’s far too stressful to worry about what is going to happen if a grant can’t cover an expense, there are no other options to cover it, and the money has already been spent.

Posted in Uncategorized | Tagged , , , | Comments Off on Grant considerations

A new NIH blog

Sally Rockey, Deputy Director of Extramural Research at NIH, has a new blog, “Rock Talk“. I’m pleased about this development, because any massive bureaucratic government institution like NIH needs as much transparency as it can muster, and because she is not your typical blogger.

Her Jan 25 post, “Looking to the Future of the Biomedical Workforce” describes the new subcommittee which will answer, among other things, “What is the right size of the workforce?” This is a critical question for the future of science and research in the US, and for postdocs. When the NIH budget doubled between 1998 and 2003, this turned into a disaster. The money wasn’t spent wisely but was thrown around a bit too freely, and the end result was an influx of newly minted PhDs, because graduate students were a cheap source of labor. Researchers never have enough money, so even with a surplus of money they were cutting corners, trying to get as much as possible out of it. Hiring cheap grad students instead of paying their postdocs and technicians more.

Coincidentally (not really), I started graduate school in 1998 and received my PhD in 2003. I had no idea that the NIH budget was doubling when this was occurring. No one talked about it. No one talked about what my career prospects might look like when the doubling was complete, and whether the next administrations might try to maintain the doubling. They couldn’t have, even if they wanted to, and the next administration was Republican so they didn’t want to.

I digressed a bit. My point is, PLANNING IS A GOOD THING. Whether you are planning an experiment, your career, or the future of biomedical research in the US. Why shouldn’t we just let supply and demand work itself out? While it is probably true we’ll reach a balance, it’s not a nice way to do it. There was some community, I think in Colorado or some state out that direction, that had a troubled deer population. Deer hunting was banned, but the hunting the predators was encouraged. The predators quickly declined and the deer population bounded back. The community didn’t like the idea of deer hunters killing Bambi, so deer hunting was still banned. And the deer increased. They got diseases. They starved. They looked horribly sick and miserable. Finally hunting was reinstated and balance restored.

I’m not saying that there should be hunting season on graduate students and postdocs. I’m saying it’s kinder to look ahead, see how many PhDs we actually need and target that number, rather than let them die of disease & starvation because there are too many for the market.

Posted in Uncategorized | Tagged , , | Comments Off on A new NIH blog

When I’m grown up

Less than a couple years ago I was still wondering what I wanted to be when I grew up. At my age, another question has to accompany this question: when will I be grown up? Now that I have a real grown up job, I’ve turned to wondering “What will success look like?” and “When will I be successful?” I’ve read two articles recently which address success. “Research Centered” describes her endeavors to avoid impostor syndrome in her approaching 40’s. Douglas Green defines what success means to him in the article “Opinion: Success!” in The Scientist.

To Green, success happened during the transition from him asking successful scientists what made them successful to him getting that question. He compared this to “The Princess Bride”, where every night the Dread Pirate Roberts told Wesley “I’ll probably kill you in the morning,” until one day the Dread Pirate Roberts announced he was retiring and that Wesley was now the Dread Pirate Roberts.

Shortly after starting my new job I had a list for myself. To be a successful researcher, I decided, I need three pretty decent publications and an R01 within 5 years. The R01 is a 5-year grant from the National Institutes of Health, and often at least one if not two or three simultaneous R01s are required for tenure. Having narrowly escaped a tenure track job, I came up with exactly the same criteria. And I only gave myself 5 years to do it, whereas most departments give their tenure track professors 6 or 7 years to do it.

“Research Centered” is a lot like me, in that she narrowly escaped a tenure track job. But her career is not as research focused as mine. Despite her blog name she describes herself as 75% administrative, 25% research. If she wants metrics for success she can’t copy and paste from tenure track definitions. That’s lucky for her. I want to do better than that too. There’s a reason I am no longer interested in a tenure track position. So what sense does it make to impose the same on myself? That’s no escape.

I still want good publications and an R01. But maybe I can come up with different definitions of success for myself.

Posted in Uncategorized | Tagged , , , | Comments Off on When I’m grown up