Feedback on NIH Scoring
07.16.10 by Michelle Kienholz
Update: Jeremy Berg has posted similar analyses of Approach and Innovation scores at the Feedback Loop … and now, regression analysis results, too!
My NIGMS Feedback Loop listserv alerted me to Jeremy Berg’s assessment Model Organisms and the Significance of Significance. Not much on model organisms (an interesting comment by Whimple – update: and others now), but Dr. Berg notes that:
To examine how reviewers apply the significance criterion in determining overall impact scores, I analyzed 360 NIGMS R01 applications reviewed during the October 2009 Council round. [he shows a plot, too]
As anticipated, the scores are reasonably strongly correlated, with a Pearson correlation coefficient of 0.63. Similar comparisons with the other peer review criteria revealed correlation coefficients of 0.74 for approach, 0.54 for innovation, 0.49 for investigator and 0.37 for environment.
Hmm. Not too surprising. Research is not likely to have much impact if it is not both significant (meaningful) and well designed/planned. I realized on reading his post that I do indeed tend to discount the scores (and, to some extent, the comments) under the other criteria and focus on the overall impact bullets plus Significance and Approach when reviewing Summary Statements.
I actually like this definition of Overall Impact from Sally Amero’s presentation on peer review at the June 2010 NIH Regional Grant Seminar:
Likelihood for the project to exert a sustained, powerful influence on the research field(s) involved
- Likelihood (i.e., probability) is primarily derived from the investigator(s), approach and environment criteria
- Sustained powerful influence is primarily derived from the significance and innovation criteria
Though I still focus on assessment of Significance and Approach in the review …
I’ll be interested to see if these data change with the just-completed reviews of the first short-format applications submitted during Cycle 1. If anything, I would expect them to become more tightly correlated, which is I’m sure what Toni Scarpa hopes as well. Then again, the Summary Statements from this round that I’ve already read invariably note something to the effect that details are lacking (in approach), so we’ll see.
(and, after ignoring the blogosphere for a few weeks due to travels & grant overload, I just thought to check, and, yes, DrugMonkey covered this as well … but in case there’s anyone here but not there who might be interested in the NIGMS Feedback …)
NIH Regional Grants Seminar
06.24.10 by Michelle Kienholz
Writedit is in Portland for the NIH Regional Grants Seminar (& I recommend everyone attend one of these or at least view the online presentations) – limited Web access and will be off the grid for a few days after. Have fun.
But … a few tidbits already. Later this afternoon, I’ll learn about plans to shorten/streamline/”enhance” the writing of FOAs. More on that later.
At one talk, an SRO shared a good rule of thumb for differentiating Impact from Significance: Significance is the hypothetical benefit to science/technology/clinical practice *if* the aims are achieved … Impact is the real-world impact, taking into account why the investigators & environment will really make this cool study work & shift a paradigmm or two.
Also, for resubmissions, SROs really want the reviewers to look at the A1 as a “new” application (reviewers don’t see old application in any case) evaluated based on its own merit – not in relation to how much it improved from the prior submission or whether all the reviewer critiques were met. Not news – but clearly laid out today.
And Sally Rockey (head of OER) confirmed that the NIH is rigorously sniffing out “new” applications that are not new. Rigorously (investing time & personnel needed). Please remember that just changing PAs does not make the application new (changing mechanisms, resubmission after failing at an RFA do qualify as new). She also noted that so far, there has only been a 10% bump in applications submitted. The next big jump will likely be in 2012, when everyone who had ARRA funding and asked for no-cost extensions comes back to the trough for more ….
NIH FY09 Success Rates
04.8.10 by Michelle Kienholz
So as the 2009 applicants await word as to whether they will be funded with FY10 dollars, I thought I’d post the lastest success rate data from the NIH.
The big NIH-wide scoreboard shows an overall success rate of 20.6%. This includes all competitive applications (e.g., new, renewals, supplements) for all mechanisms. As a reminder, NIH Success Rates:
include applications that are peer reviewed and either scored or unscored by an Initial Review Group. Success rates are determined by dividing the number of competing applications funded by the sum of the total number of competing applications reviewed and the number of funded carryovers [i.e., applications reviewed and scored but not funded the fiscal year prior]. Applications having one or more amendments in the same fiscal year are only counted once.
Grants funded jointly by 2 or more ICs are counted only by the IC footing the largest chunk of the bill.
On the master file, you can click on your favorite ICs to get their specific success rate stats.
Across the NIH, R21s (all Type 1s) have success rates well below R01s (which include Type 2/3 applications in their success rate). Among the big ICs, NHLBI is at 14.5% for the R21; NCI, 13.7%; NINDS, 12.8%; NIAID 11.8%; NIGMS, 7.9%; and NIDDK (which discourages applicants from using this mechanism except for specific types of work), 4.6%. NIMH is an outlier with an R21 success rate of 20.1%. In general, in fact, NIMH looks to be a nice place to go for money … except for R03s, which have a success rate of 9.6% (go to NIDDK, 58%, or NCI, 30.8%, for this mechanism … though these are probably mainly secondary data analyses awards).
No data on F, K, or T awards here … you need to scroll down the main success rate page and check the appropriate Excel spreadsheet for these data … or better yet, check the NIH Data Book for trends in Career Development Awards and Training Grants & Fellowships.
Payline Complexity Explained & CSR College of Reviewers Updated by NIAID
03.31.10 by Michelle Kienholz
As always, the latest issue of NIAID Funding News is a treasure trove of information and good advice.
First, for the hundreds of you out there wondering why your IC hasn’t set a payline yet, NIAID reports, shockingly, that the trend toward score clustering has increased and explains how score clustering causes jumps in assigned percentiles. An impact score of 20 seems to be the sweet spot thus far: “In the first two review cycles of this fiscal year, approximately 3% of applications reviewed by CSR received a score of 20.” NIAID gives an example in which a score of 20 in one study section might land at the 9th percentile, with a score of 21 in turn at the 11th percentile (payline at the 10th percentile).
NIAID also includes an update on the CSR College of Reviewers (discussed here previously), including the current membership roster. As a reminder, these folks (“editorial board members”) will be asked to provide written reviews only on up to 12 applications a year for 2 years as part of 2-stage reviews (with the second stage being the face-to-face meetings of “editors”).
Lots of other good intel and advice, so be sure to scroll through the entire newsletter and, no matter which IC is “yours”, sign up for delivery to your very own digital mail box.
Reviews of New NIH Review System
02.23.10 by Michelle Kienholz
Thanks to Jef Akst at The Scientist for his post earlier today entitled NIH Reviewers Praise New Rules (& SaG for the alert). Thanks too to those who in turn have commentend on the piece. I tend to agree with both sets of contributors – the 3 grant reviewers in the main article and the various commenters who have come along since.
Recognizing the learning curve associated with adopting a new system and inherent challenges in assessing significance and impact, reviewers felt that “changes to the reviewing guidelines have actually increased the validity and utility of the reviews.” Specifically:
For example, in the new system scoring is limited to whole numbers (1 through 9), whereas before, a reviewer could give a proposal a priority score anywhere from 1 to 5 in increments of tenths. But such a fine scale was counterproductive, Wiley noted. “[Y]ou cannot possibly discriminate grants on that kind of level,” he said, adding that the process was effectively “a crapshoot after you pick the top 25%.”
Another benefit of the new system is that reviewers are now required to justify their scores by listing strengths and weaknesses for each subcriterion…
Other changes to the review process specifically aim to cut down on the amount of time the process takes. … The template now provided by the NIH gives bullet points where reviewers are to write a couple of sentences summarizing the strengths and weakness of each category, limiting them to just half a page.
Finally, rather than reviewing the grant proposals in random order, the study section starts with the highest scored applications (based on preliminary scores) and works their way down the list. In addition to cutting the total number of grants the study section will review orally — low-ranked applications with no chance of getting funded won’t even be discussed — it also helps the reviewers to “recalibrate” their scores, Wiley said, by providing an excellent standard against which the others can be judged. “It was a very clever idea,” he said. “I found this has been very, very helpful.”
Of course, those fast, easy, bulleted comments are not without their shortcomings, as many commenters noted (e.g.):
The comments are vague and provide little meaningful guideline for improvement. Of course, that is not the primary goal-to be helpful. I think this is mainly a mechanism to more efficiently winnow down the stack of applications in this time of tight paylines.
The new format makes the critique more diffuclt to decipher and the applicant will have a hard time to get a handle on how to revise the application to get a better score.
Plus a reflection on the preliminary score-driven discussion order:
The new review order is truely a clever idea that helps to calibrate the scores and make the whole process more consistent. I delibrately avoid the wording “fairer” because I also see the downside of this new order. The more contentious and contraversial applications are more likely being discussed near the lunch breaks or late in the day and people tend to get really tired by that point. The debate on those contraversial grants may not be as vigorous as it used to be.
I dunno. Looking at summary statements coming out of this brave new world, I’m pretty happy with the way things are going, vague bullets and all. I especially like the ability (or perhaps increased tendency) of reviewers to comment directly on the advisability of resubmission. And I’m happy with the way SROs are preparing the Resume & Summary of Discussion paragraphs as well. I like the short applications. I like the reviews. The paylines, not so much.
NIH FY10 Fiscal Policies
01.8.10 by Michelle Kienholz
The NIH gave notice of the FY10 Fiscal Policy for Grant Awards now that its $31B budget (2.3% increase over FY09) is in place. It’s not entirely business as usual.
The goal is to fund 9,200 new and competing RPGs (average cost 2% higher than that in FY09) and at least 1,650 new investigators.
For FY10, only a 2% (vs 3%) inflation allowance will be permitted for non-competing (Type 5) awards. However, this policy does not apply to projects supported by ARRA or to Ks, Fs, Ts, or SBIR/STTRs. NRSAs (Fs, Ts) will receive a 1% increase in stipend levels.
The NIH Director’s Innovator Awards (i.e., Junior Pioneer or DP2) will continue with funding from the Common Fund, and K99s will be implemented as in previous years.
However, the NIH Directors Bridge Award (R56) will be suspended in FY10, though ICs will be permitted to use their own appropriated funds for this purpose (i.e., provide limited support to maintain projects just outside the payline).
A separate notice bumps up the FY10 salary cap to $199,700, and another lists the legislative mandates in the FY10 appropriations omnibus bill (e.g., no disseminating false data, no forgetting to acknowledge federal funding, no using federal funds for lobbying, distributing sterile needles, abortions [specified exceptions], human embryo research, promoting legalization of controlled substances, or contracts to contractors who have not paid all their taxes).
Oh, and you get 5 (vs 2) days to correct your electronic applications during the entire Cycle 1 submission period (Jan 25-May 7, 2010).
Remember too that this is the last calendar year to submit any grandfathered A2 applications (no A2s will be accepted after Jan 7, 2011).
ARRA Administrative Supplements
01.4.10 by Michelle Kienholz
When it rains, it pours … in odd and extravagant ways …
Administrative Supplements for Comparative Effectiveness Research Workforce Development
Receipt Date: March 1, 2010
The maximum sum that may be requested is $500,000, as total costs, in FY2010 only. Up to 16 awards may be made, likely by September 2010 and, where possible, shortly after their approval.
This administrative supplement provides the opportunity for eligible grantees to develop, expand, and/or increase CER training, education, and career development programs within existing U.S. NIH-supported grants. Some types of awards from NIH (eg, S10 awards and many R and P awards) are not eligible for administrative supplements. Institutional awards that support the scope of training envisaged in this supplement announcement include the K12, K30, T32, T35, and T90 mechanisms, together with their KL and TL equivalents. Questions about eligibility should be addressed to the program officer and/or the grants management specialist for the award.
To be eligible, the parent grant must be active, and the training, education, and career development activities proposed in the supplement must be accomplished within the current competitive segment. The proposed supplement MUST be within the general scope of the peer-reviewed activities and aims approved within the parent grant, including projects on a no-cost extension.
Some examples of the types of supplements that could be appropriate include, but are not limited to: adding scholar or training slots for CER education and training through short courses, Certificate programs, and advanced degree-awarding programs; creating a course curriculum for early or midcareer researchers to develop or enhance skills in CER; creating an outreach training or course for community based research to develop or enhance skills in CER; and developing a CER training or course to include related fields such as communication and information dissemination science, medical decision-making, and outcomes and evaluation research as long as the proposal’s specific aims supports Federal Coordinating Council-defined CER.
Grantsmanship Downloads Page Added
01.3.10 by Michelle Kienholz
Happy 2010, everyone!
I’ve added a page (see top list of links in the right margin) with downloadable grantsmanship files … some from the NIH and NSF and a couple of writedit originals. One summarizes the shorter application format and enhanced review/scoring procedures, and the other is a big catch-all introduction to NIH terminology, policies, and whatnot for young investigators as well as an overview of early career stage funding mechanisms (fellowships, career development, diversity supplements). I’ll add more as I find them online or refine my own – suggestions welcome (as are critiques of what I’ve put up). Cheers!
AHRQ Embracing Most Enhancements
12.22.09 by Michelle Kienholz
AHRQ Announces Changes to Peer Review Processes, Evaluation Review Criteria, and New Application Forms for Grant Applications
AHRQ will be adopting many, but not all, of the procedures currently being implemented by NIH under the NIH Enhancing Peer Review initiative. This includes implementation of enhancing review criteria for evaluating the scientific and technical merit of research grant applications submitted to AHRQ through the peer review system.
AHRQ will also use the new, restructured versions of the paper PHS 398 and electronic SF 424 application packages, including changes to the Research Plan (i.e., Research Strategy with sections for Significance, Innovation, and Approach), Resources (environment), and Biographical Sketch.
Original new and competing renewal applications that were submitted prior to January 25, 2010 will be permitted two amendments (A1 and A2). For these “grandfathered” applications, AHRQ expects that any A2 will be submitted no later than January 7, 2011, and AHRQ will not accept A2 applications after that date.
BUT …
AHRQ will not be using the new, significantly shorter page limits announced but will instead maintain the page limits it currently uses. However, each AHRQ FOA will specify the page limit requirements for the Research Strategy section of the grant application.
Also, AHRQ will not be using the designation of Early Stage Investigator (ESI) on grant applications.
Best Timing for NIH Applications
12.9.09 by Michelle Kienholz
My standard advice is always to apply when the application is ready – strong, competitive, and not submitted just to get feedback (especially now with just one resubmission). I always give caveats on the various standard receipt dates and review cycles, such as Cycle 1 carrying the highest risk of delayed or deferred funding. Now, NIAID (who else) has a nice table summarizing all this for me.
| New R01 Application Timing | Characteristics and Considerations |
|---|---|
|
Review Cycle 1 |
|
|
Review Cycle 2 |
|
|
Review Cycle 3 |
|
Other great articles in this week’s issue of the NIAID Funding News cover the roles and responsibilities of the Advisory Council , whether securing an R21 helps new investigators receive an R01 later (yes, by a 2:1 ratio), and conducting your own “peer review” prior to submitting an application to the NIH.
