NIH is not spelled "NSF"


I've often wondered what it is like on an NIH study section, and how it differs from an NSF panel, so I happily agreed to serve this past week on my first Genetic Variation and Evolution (GVE) study section.  

What is an NIH study section like? 

Well, on the face of it, much like an NSF panel.  Before you come to the meeting, you are given a set of applications to review.  Like the NSF, your review of each individual application is based on certain criteria and you are asked to comment on these criteria.  In an NSF review you first score the proposal (Excellent, Very Good, Good, Fair, or Poor).  At NSF, criteria you should comment on or focus on in your review are "Intellectual Merit" and "Broader Impacts" (with strengths and weaknesses for each).   You also give a summary statement at the end of your review that should reflect your score.

 For the NIH, the scored criteria are "Significance", "Investigator", "Innovation", "Approach", and "Environment".  Unlike the NSF, the NIH has tried to make these scores quantitative and comparable across study sections. So,scores of 1-3 are "good" or "high" (contrary to the numerical trend), scores of 4-6 are supposed to be "average" (an application may be of high importance but weaknesses bring down the overall impact or the application topic is of moderate importance), and scores of 7-9 are reserved for applications with serious problems or of low or no importance.  Each of the criteria are scored in this way (so you can get a sense of whether or not the approach is something the panel didn't like or the significance was seen as lacking.  You also write statements reflecting the strengths and weakness for each and you write an "overall impact" section - akin to the summary statement of NSF.  Finally, you provide your "overall" score -- this is what will be used to rank the applications during the section meeting.

You submit your reviews electronically -- like the NSF.  One big deviation from that of the NSF panel review as that after the deadline, you actually get to see the reviews of others and alter your scores and reviews.  This was interesting to me for two reasons: I wondered if folks would generally get bullied into lowering their scores (giving worse scores) or raising their scores (giving better scores).  I also wondered if clearly good or clearly bad proposals would get unanimously consistent scores.  The Scientific Review Officer also assumes a role at this point, guiding folks to match their text (in their reviews) with their scores -- for example, if you only wrote negative comments, why did you give the application a "3"? Or, if you had only glowing things to say about the proposal, why did you rank it at "5"?

Then we all arrive in a hotel at a predetermined location.  Like NSF panels, a relatively large group (25 of us in the case of GVE) meet in one room over a few days to discuss a large set of proposals.  Unlike NSF, NIH sections don't discuss all applications! Let me repeat that -- in case you've never applied to the NIH - if you score poorly, your application won't be discussed.  The only feedback you will get is from the reviews and those scores (which, granted, can be significant).  Many colleagues have told me that if you are not discussed, it is very likely that you will never get that grant funded.  That is because of the "three two strikes" rule at NIH -- you can only submit the same application 2 times.  What's the likelihood you'll go from not-discussed, to discussed or funded in three rounds without the feedback actually provided by a discussion?  Who decides what gets discussed? The Scientific Review Officer -- and this is largely based on how the applications scored.  However, anyone on the study section can "rescue" an application from "triage" by asking for it to be discussed. 

There's a wonderful aspect to the triaging of "bad" applications -- as a reviewer, it means you don't have to rush through a huge pile of proposals, or waste time/energy on clearly poor applications, or spend over 2 days stuck in a room away from your work/family/friends/etc.  However, as an applicant, I rather like that NSF provides pretty substantial feedback even to applications that are not the top of the heap -- although I know that some NSF panels do effectively "triage" poor scoring proposals, most of the applications get discussed at NSF while a large fraction are being triaged at NIH.

Ok, so now we've submitted our reviews, we have looked over the reviews of our colleagues and are in the conference room bright and early awaiting the start of the section. What happens?

- Application discussion order generally follows scores (best up first)
- An application to be discussed is brought up 
- Everyone takes a minute to read the abstract and aims 
- Assigned reviewers present their preliminary scores
- Each reviewer discusses strengths and weaknesses
- The application is opened up to discussion by all
- Human subjects, vertebrate animals, and biohazards are discussed
- Assigned reviewers are asked again what their scores will be (often these change after discussion).  These scores provide a voting "range"  -- that is to say, if the three reviewers ranked your application overall as 2,4 and 5 then everyone on the panel can vote within the range (2-5) or, they have to raise their hand and announce that they will vote out of the range (either above or below -- although this fact remains private).  Although you could imagine a situation where folks feel odd about voting out of range, or somehow peer pressured into the range, this didn't seem to be the case on the GVE panel at all.  In addition, you don't have to say why you are voting out of range, but simply state that you are.
- Everyone casts a vote
- The budget and other non-scoring criteria are discussed

Remember: The only people reading your full proposal are those assigned to review it!!! That is, only THREE people.  Unless they go out of their way to download and read the full proposal <unlikely>, everyone else ONLY READS THE ABSTRACT AND AIMS!! Make those pages good, folks! Make them readable, stand on their own, and capture the imagination.

Come back tomorrow for some nuggets of advice I've learned from being on this panel (albeit from a current unfunded beginning investigator)

Comments

Post a Comment

Popular posts from this blog

The Wolbachia "holy grail"

Image
The once-obscure alpha-proteobacterium Wolbachia pipientis, was catapulted into medical relevance by the discovery that it inhibits the replication of RNA viruses .  However, it was first discovered for its odd quirk of manipulating insect reproduction . That's right, Wolbachia were called "reproductive parasites" and for a long time, no fitness benefit was attributed to them.  These intracellular masters do things like kill male offspring, feminize male offspring, induce parthenogenesis, and the most common phenotype, sperm-egg incompatibility (also known by its more complicated name: cytoplasmic incompatibility or CI) (Figure 1).   Figure 1. Modified from Werren et al., 2008 .  Wolbachia cause four distinct reproductive phenotypes in a range of arthropod orders (top).  The mechanisms behind CI - meaning how does  Wolbachia  induce it -  have been a mystery for some time.   When you make a cross between infected males and uninfected...
Read more

What's in a name? Would a Wolbachia by any other name, block RNA viruses as sweetly?

Image
I've ranted about bacterial nomenclature in the past (see h ere ), but this post is specifically about Wolbachia pipientis, and how we should name strain-level variants within a particular host-associated clade (such as w Mel, for example).  Currently, there seem to be multiple ways to write these names and to present them in databases or the literature.  For example: For variants within w Mel, we find examples of the following types of strain names: w MelPop-<insert specific substrain here after the dash> The stuff that comes after the strain name ( w MelPop) sometimes refers to the host insect from which the strain was isolated, or where it currently resides (such as in w MelPop-PYGP) or sometimes refers to how the strain was altered, through sequential passage (such as in w MelPop-CLA).  This variability in the naming schemas used in Wolbachia could lead to confusion down the road. We are currently adding strain level information to some of the stocks th...
Read more

ASM's MRA Journal Supports Undergraduate Research

Image
Disclaimer: Dr. Irene Newton is EiC of ASM's Microbiology Resource Announcements and Catherine Putonti is a Senior Editor. Undergraduate research changed my life. Truly. Along with many thousands of other undergraduate students, I started my career really thinking I was going to be an MD. That career path appealed for many reasons - my immigrant family understood it and approved, it was prestigious, and it meant my life was planned out for the next decade. But one semester of research with my undergraduate mentor was enough to get me hooked. I was privileged to have the opportunity to do one-on-one research at a small liberal arts institution but not all of our students have that opportunity. I now teach at one of the "Big 10" - with over 45,000 undergrads on campus there is no way we can provide all of them the research experience I had. But we can and are trying to integrate true scientific research into our undergraduate classes. Some good examples of this are the FI...
Read more