NIH is not spelled "NSF"

I've often wondered what it is like on an NIH study section, and how it differs from an NSF panel, so I happily agreed to serve this past week on my first Genetic Variation and Evolution (GVE) study section.  

What is an NIH study section like? 

Well, on the face of it, much like an NSF panel.  Before you come to the meeting, you are given a set of applications to review.  Like the NSF, your review of each individual application is based on certain criteria and you are asked to comment on these criteria.  In an NSF review you first score the proposal (Excellent, Very Good, Good, Fair, or Poor).  At NSF, criteria you should comment on or focus on in your review are "Intellectual Merit" and "Broader Impacts" (with strengths and weaknesses for each).   You also give a summary statement at the end of your review that should reflect your score.

 For the NIH, the scored criteria are "Significance", "Investigator", "Innovation", "Approach", and "Environment".  Unlike the NSF, the NIH has tried to make these scores quantitative and comparable across study sections. So,scores of 1-3 are "good" or "high" (contrary to the numerical trend), scores of 4-6 are supposed to be "average" (an application may be of high importance but weaknesses bring down the overall impact or the application topic is of moderate importance), and scores of 7-9 are reserved for applications with serious problems or of low or no importance.  Each of the criteria are scored in this way (so you can get a sense of whether or not the approach is something the panel didn't like or the significance was seen as lacking.  You also write statements reflecting the strengths and weakness for each and you write an "overall impact" section - akin to the summary statement of NSF.  Finally, you provide your "overall" score -- this is what will be used to rank the applications during the section meeting.

You submit your reviews electronically -- like the NSF.  One big deviation from that of the NSF panel review as that after the deadline, you actually get to see the reviews of others and alter your scores and reviews.  This was interesting to me for two reasons: I wondered if folks would generally get bullied into lowering their scores (giving worse scores) or raising their scores (giving better scores).  I also wondered if clearly good or clearly bad proposals would get unanimously consistent scores.  The Scientific Review Officer also assumes a role at this point, guiding folks to match their text (in their reviews) with their scores -- for example, if you only wrote negative comments, why did you give the application a "3"? Or, if you had only glowing things to say about the proposal, why did you rank it at "5"?

Then we all arrive in a hotel at a predetermined location.  Like NSF panels, a relatively large group (25 of us in the case of GVE) meet in one room over a few days to discuss a large set of proposals.  Unlike NSF, NIH sections don't discuss all applications! Let me repeat that -- in case you've never applied to the NIH - if you score poorly, your application won't be discussed.  The only feedback you will get is from the reviews and those scores (which, granted, can be significant).  Many colleagues have told me that if you are not discussed, it is very likely that you will never get that grant funded.  That is because of the "three two strikes" rule at NIH -- you can only submit the same application 2 times.  What's the likelihood you'll go from not-discussed, to discussed or funded in three rounds without the feedback actually provided by a discussion?  Who decides what gets discussed? The Scientific Review Officer -- and this is largely based on how the applications scored.  However, anyone on the study section can "rescue" an application from "triage" by asking for it to be discussed. 

There's a wonderful aspect to the triaging of "bad" applications -- as a reviewer, it means you don't have to rush through a huge pile of proposals, or waste time/energy on clearly poor applications, or spend over 2 days stuck in a room away from your work/family/friends/etc.  However, as an applicant, I rather like that NSF provides pretty substantial feedback even to applications that are not the top of the heap -- although I know that some NSF panels do effectively "triage" poor scoring proposals, most of the applications get discussed at NSF while a large fraction are being triaged at NIH.

Ok, so now we've submitted our reviews, we have looked over the reviews of our colleagues and are in the conference room bright and early awaiting the start of the section. What happens?

- Application discussion order generally follows scores (best up first)

- An application to be discussed is brought up 

- Everyone takes a minute to read the abstract and aims 

- Assigned reviewers present their preliminary scores

- Each reviewer discusses strengths and weaknesses

- The application is opened up to discussion by all

- Human subjects, vertebrate animals, and biohazards are discussed

- Assigned reviewers are asked again what their scores will be (often these change after discussion).  These scores provide a voting "range"  -- that is to say, if the three reviewers ranked your application overall as 2,4 and 5 then everyone on the panel can vote within the range (2-5) or, they have to raise their hand and announce that they will vote out of the range (either above or below -- although this fact remains private).  Although you could imagine a situation where folks feel odd about voting out of range, or somehow peer pressured into the range, this didn't seem to be the case on the GVE panel at all.  In addition, you don't have to say why you are voting out of range, but simply state that you are.

- Everyone casts a vote

- The budget and other non-scoring criteria are discussed

Remember: The only people reading your full proposal are those assigned to review it!!! That is, only THREE people.  Unless they go out of their way to download and read the full proposal <unlikely>, everyone else ONLY READS THE ABSTRACT AND AIMS!! Make those pages good, folks! Make them readable, stand on their own, and capture the imagination.

Come back tomorrow for some nuggets of advice I've learned from being on this panel (albeit from a current unfunded beginning investigator)