Payline Complexity Explained & CSR College of Reviewers Updated by NIAID
03.31.10 by Michelle Kienholz
As always, the latest issue of NIAID Funding News is a treasure trove of information and good advice.
First, for the hundreds of you out there wondering why your IC hasn’t set a payline yet, NIAID reports, shockingly, that the trend toward score clustering has increased and explains how score clustering causes jumps in assigned percentiles. An impact score of 20 seems to be the sweet spot thus far: “In the first two review cycles of this fiscal year, approximately 3% of applications reviewed by CSR received a score of 20.” NIAID gives an example in which a score of 20 in one study section might land at the 9th percentile, with a score of 21 in turn at the 11th percentile (payline at the 10th percentile).
NIAID also includes an update on the CSR College of Reviewers (discussed here previously), including the current membership roster. As a reminder, these folks (“editorial board members”) will be asked to provide written reviews only on up to 12 applications a year for 2 years as part of 2-stage reviews (with the second stage being the face-to-face meetings of “editors”).
Lots of other good intel and advice, so be sure to scroll through the entire newsletter and, no matter which IC is “yours”, sign up for delivery to your very own digital mail box.
Reviews of New NIH Review System
02.23.10 by Michelle Kienholz
Thanks to Jef Akst at The Scientist for his post earlier today entitled NIH Reviewers Praise New Rules (& SaG for the alert). Thanks too to those who in turn have commentend on the piece. I tend to agree with both sets of contributors – the 3 grant reviewers in the main article and the various commenters who have come along since.
Recognizing the learning curve associated with adopting a new system and inherent challenges in assessing significance and impact, reviewers felt that “changes to the reviewing guidelines have actually increased the validity and utility of the reviews.” Specifically:
For example, in the new system scoring is limited to whole numbers (1 through 9), whereas before, a reviewer could give a proposal a priority score anywhere from 1 to 5 in increments of tenths. But such a fine scale was counterproductive, Wiley noted. “[Y]ou cannot possibly discriminate grants on that kind of level,” he said, adding that the process was effectively “a crapshoot after you pick the top 25%.”
Another benefit of the new system is that reviewers are now required to justify their scores by listing strengths and weaknesses for each subcriterion…
Other changes to the review process specifically aim to cut down on the amount of time the process takes. … The template now provided by the NIH gives bullet points where reviewers are to write a couple of sentences summarizing the strengths and weakness of each category, limiting them to just half a page.
Finally, rather than reviewing the grant proposals in random order, the study section starts with the highest scored applications (based on preliminary scores) and works their way down the list. In addition to cutting the total number of grants the study section will review orally — low-ranked applications with no chance of getting funded won’t even be discussed — it also helps the reviewers to “recalibrate” their scores, Wiley said, by providing an excellent standard against which the others can be judged. “It was a very clever idea,” he said. “I found this has been very, very helpful.”
Of course, those fast, easy, bulleted comments are not without their shortcomings, as many commenters noted (e.g.):
The comments are vague and provide little meaningful guideline for improvement. Of course, that is not the primary goal-to be helpful. I think this is mainly a mechanism to more efficiently winnow down the stack of applications in this time of tight paylines.
The new format makes the critique more diffuclt to decipher and the applicant will have a hard time to get a handle on how to revise the application to get a better score.
Plus a reflection on the preliminary score-driven discussion order:
The new review order is truely a clever idea that helps to calibrate the scores and make the whole process more consistent. I delibrately avoid the wording “fairer” because I also see the downside of this new order. The more contentious and contraversial applications are more likely being discussed near the lunch breaks or late in the day and people tend to get really tired by that point. The debate on those contraversial grants may not be as vigorous as it used to be.
I dunno. Looking at summary statements coming out of this brave new world, I’m pretty happy with the way things are going, vague bullets and all. I especially like the ability (or perhaps increased tendency) of reviewers to comment directly on the advisability of resubmission. And I’m happy with the way SROs are preparing the Resume & Summary of Discussion paragraphs as well. I like the short applications. I like the reviews. The paylines, not so much.
NIH FY10 Fiscal Policies
01.8.10 by Michelle Kienholz
The NIH gave notice of the FY10 Fiscal Policy for Grant Awards now that its $31B budget (2.3% increase over FY09) is in place. It’s not entirely business as usual.
The goal is to fund 9,200 new and competing RPGs (average cost 2% higher than that in FY09) and at least 1,650 new investigators.
For FY10, only a 2% (vs 3%) inflation allowance will be permitted for non-competing (Type 5) awards. However, this policy does not apply to projects supported by ARRA or to Ks, Fs, Ts, or SBIR/STTRs. NRSAs (Fs, Ts) will receive a 1% increase in stipend levels.
The NIH Director’s Innovator Awards (i.e., Junior Pioneer or DP2) will continue with funding from the Common Fund, and K99s will be implemented as in previous years.
However, the NIH Directors Bridge Award (R56) will be suspended in FY10, though ICs will be permitted to use their own appropriated funds for this purpose (i.e., provide limited support to maintain projects just outside the payline).
A separate notice bumps up the FY10 salary cap to $199,700, and another lists the legislative mandates in the FY10 appropriations omnibus bill (e.g., no disseminating false data, no forgetting to acknowledge federal funding, no using federal funds for lobbying, distributing sterile needles, abortions [specified exceptions], human embryo research, promoting legalization of controlled substances, or contracts to contractors who have not paid all their taxes).
Oh, and you get 5 (vs 2) days to correct your electronic applications during the entire Cycle 1 submission period (Jan 25-May 7, 2010).
Remember too that this is the last calendar year to submit any grandfathered A2 applications (no A2s will be accepted after Jan 7, 2011).
ARRA Administrative Supplements
01.4.10 by Michelle Kienholz
When it rains, it pours … in odd and extravagant ways …
Administrative Supplements for Comparative Effectiveness Research Workforce Development
Receipt Date: March 1, 2010
The maximum sum that may be requested is $500,000, as total costs, in FY2010 only. Up to 16 awards may be made, likely by September 2010 and, where possible, shortly after their approval.
This administrative supplement provides the opportunity for eligible grantees to develop, expand, and/or increase CER training, education, and career development programs within existing U.S. NIH-supported grants. Some types of awards from NIH (eg, S10 awards and many R and P awards) are not eligible for administrative supplements. Institutional awards that support the scope of training envisaged in this supplement announcement include the K12, K30, T32, T35, and T90 mechanisms, together with their KL and TL equivalents. Questions about eligibility should be addressed to the program officer and/or the grants management specialist for the award.
To be eligible, the parent grant must be active, and the training, education, and career development activities proposed in the supplement must be accomplished within the current competitive segment. The proposed supplement MUST be within the general scope of the peer-reviewed activities and aims approved within the parent grant, including projects on a no-cost extension.
Some examples of the types of supplements that could be appropriate include, but are not limited to: adding scholar or training slots for CER education and training through short courses, Certificate programs, and advanced degree-awarding programs; creating a course curriculum for early or midcareer researchers to develop or enhance skills in CER; creating an outreach training or course for community based research to develop or enhance skills in CER; and developing a CER training or course to include related fields such as communication and information dissemination science, medical decision-making, and outcomes and evaluation research as long as the proposal’s specific aims supports Federal Coordinating Council-defined CER.
Grantsmanship Downloads Page Added
01.3.10 by Michelle Kienholz
Happy 2010, everyone!
I’ve added a page (see top list of links in the right margin) with downloadable grantsmanship files … some from the NIH and NSF and a couple of writedit originals. One summarizes the shorter application format and enhanced review/scoring procedures, and the other is a big catch-all introduction to NIH terminology, policies, and whatnot for young investigators as well as an overview of early career stage funding mechanisms (fellowships, career development, diversity supplements). I’ll add more as I find them online or refine my own – suggestions welcome (as are critiques of what I’ve put up). Cheers!
AHRQ Embracing Most Enhancements
12.22.09 by Michelle Kienholz
AHRQ Announces Changes to Peer Review Processes, Evaluation Review Criteria, and New Application Forms for Grant Applications
AHRQ will be adopting many, but not all, of the procedures currently being implemented by NIH under the NIH Enhancing Peer Review initiative. This includes implementation of enhancing review criteria for evaluating the scientific and technical merit of research grant applications submitted to AHRQ through the peer review system.
AHRQ will also use the new, restructured versions of the paper PHS 398 and electronic SF 424 application packages, including changes to the Research Plan (i.e., Research Strategy with sections for Significance, Innovation, and Approach), Resources (environment), and Biographical Sketch.
Original new and competing renewal applications that were submitted prior to January 25, 2010 will be permitted two amendments (A1 and A2). For these “grandfathered” applications, AHRQ expects that any A2 will be submitted no later than January 7, 2011, and AHRQ will not accept A2 applications after that date.
BUT …
AHRQ will not be using the new, significantly shorter page limits announced but will instead maintain the page limits it currently uses. However, each AHRQ FOA will specify the page limit requirements for the Research Strategy section of the grant application.
Also, AHRQ will not be using the designation of Early Stage Investigator (ESI) on grant applications.
Best Timing for NIH Applications
12.9.09 by Michelle Kienholz
My standard advice is always to apply when the application is ready – strong, competitive, and not submitted just to get feedback (especially now with just one resubmission). I always give caveats on the various standard receipt dates and review cycles, such as Cycle 1 carrying the highest risk of delayed or deferred funding. Now, NIAID (who else) has a nice table summarizing all this for me.
| New R01 Application Timing | Characteristics and Considerations |
|---|---|
|
Review Cycle 1 |
|
|
Review Cycle 2 |
|
|
Review Cycle 3 |
|
Other great articles in this week’s issue of the NIAID Funding News cover the roles and responsibilities of the Advisory Council , whether securing an R21 helps new investigators receive an R01 later (yes, by a 2:1 ratio), and conducting your own “peer review” prior to submitting an application to the NIH.
New NIH Instructions & Forms Available
11.24.09 by Michelle Kienholz
The NIH issued a notice summarizing the changes to the application packages for submissions on or after January 25, 2010. Most important is this reminder for those planning electronic submissions (i.e., most of you):
Applicants MUST return to the FUNDING OPPORTUNITY ANNOUNCEMENT (FOA), or the reissued Parent Announcement, to download the new application forms for due dates on or after January 25, 2010.
The sample biosketch in the PHS 398 form list is always convenient to have on hand though. The PHS 398 (paper submission) page includes instructions and forms, while you’ll of course find the instructions only for SF 424 electronic submissions (Adobe Forms B).
Have fun.
OER Strikes Back
11.4.09 by Michelle Kienholz
I took a pass in September on noting Les Costello’s piece in The Scientist entitled NIH R01s: No Longer the Best Science, in which he expresses concern over policies designed to increase funding to new/early stage investigators. This month, Walter Schaffer and Sally Rockey from OER (NIH’s Office of Extramural Research, which brings you the NIH Guide, Extramural Nexus, NIH Regional Grant Seminars, and all you need to know about grant application and management policies) respond with NIH Continues to Support the Best Science through R01s. {see Alison McCook’s comment below for the corrected subtitle to the Schaffer-Rockey piece.}
Essentially, Schaffer and Rockey lay out the history of NIH’s efforts to promote funding to new and early stage investigators (ESI) and the rationale for doing so:
When Dr. Costello [who "vehemently" objects to the new/ESI policy] received his first traditional NIH research grant (R01) in 1963, success rates were near 58%, and 35% of the competing R01s went to first-time recipients. … In 1977, the average age of new investigators was nearly 37, success rates had decreased to 28%, and the proportion of R01s going to new investigators had decreased to 33% … by 2006, less than 24% of the recipients of competing R01s were new investigators, success rates were below 21%, and the average age at first award of an R01 had increased to more than 42.
The first comment, however, keeps on with The Scientist theme about whether the NIH is funding the best science … not necessarily due to any potential discrimination favoring new investigators so much as penalizing amended applications by percentiling them separately (intended to reduce review burden by funding more A0s than A1s based on historical data showing that ~70% of A0s are eventually funded as A1s or A2s). The opening line probably sums up the scientific community’s mood though:
The angst over new and early stage scientists indicates a broader anxiety among established NIH investigators over what is seen as administrative meddling adding to an already capricious peer review process.
And then, of course, we have the forum over at Genome Technology asking Is Peer Review Broken? (review of grant applications and journal manuscripts). Clearly, CSR and OER need to keep the communication channels open for continued feedback on their enhancements to the application and review processes.
Is Peer Review Broken?
11.3.09 by Michelle Kienholz
So asks the cover story by Meredith Salisbury for Genome Technology.
She starts the ball rolling in her accompanying editorial:
Get three scientists together, and it’s almost a guarantee that the conversation will eventually turn toward the vagaries of the peer review process. Be it for winning grant funding or getting a paper published, this system of relying on a handful of fellow scientists to select the most promising and influential research shapes — at least to some degree — every single researcher’s career path.
And then she gets 3+ scientists together, with the tone set right off the bat by Ferric Fang:
“For something that is of and for scientists, the peer review process is very unscientific,” says Ferric Fang, a professor of laboratory medicine and microbiology at the University of Washington. Whether it’s for papers or grants, having just a handful of people review someone’s work is statistically unsound, he adds. “If these [reviews] were data that you generated in your lab, you would say, ‘I don’t know what the conclusion of this is.’”
And per the suggestions made on grant review processes, apparently efforts to enhance peer review at the NIH haven’t gone far enough. A sampling to get you over to Genome Technology for the full report:
One hope is that having a larger pool of reviewers could help reduce the impact of any individual review, says Fang. Under the current system, “one bad review can sink an application.”
Another take on the grant review system in general is that focus needs to shift away from today’s model of specific proposals for short-term periods. …
Lawrence [Peter Lawrence at the zoology department of the University of Cambridge] would prefer a system where reviewers considered the track record of the investigator more than the details of the new research proposal (with special dispensation for new investigators). …
According to Fang, this concept of awarding funds on a track record basis would also serve the purpose of weeding out people who are very skilled at writing proposals but are less competent at actually performing the science.
Oh no! What’s a writedit to do?! Well, I am the first to acknowledge that no amount of skilled grantsmanship can make up for poor science, so I think, to a certain extent, this last concern can be dispensed with.
