Feedback on NIH Scoring
07.16.10 by Michelle Kienholz
Update: Jeremy Berg has posted similar analyses of Approach and Innovation scores at the Feedback Loop … and now, regression analysis results, too!
My NIGMS Feedback Loop listserv alerted me to Jeremy Berg’s assessment Model Organisms and the Significance of Significance. Not much on model organisms (an interesting comment by Whimple – update: and others now), but Dr. Berg notes that:
To examine how reviewers apply the significance criterion in determining overall impact scores, I analyzed 360 NIGMS R01 applications reviewed during the October 2009 Council round. [he shows a plot, too]
As anticipated, the scores are reasonably strongly correlated, with a Pearson correlation coefficient of 0.63. Similar comparisons with the other peer review criteria revealed correlation coefficients of 0.74 for approach, 0.54 for innovation, 0.49 for investigator and 0.37 for environment.
Hmm. Not too surprising. Research is not likely to have much impact if it is not both significant (meaningful) and well designed/planned. I realized on reading his post that I do indeed tend to discount the scores (and, to some extent, the comments) under the other criteria and focus on the overall impact bullets plus Significance and Approach when reviewing Summary Statements.
I actually like this definition of Overall Impact from Sally Amero’s presentation on peer review at the June 2010 NIH Regional Grant Seminar:
Likelihood for the project to exert a sustained, powerful influence on the research field(s) involved
- Likelihood (i.e., probability) is primarily derived from the investigator(s), approach and environment criteria
- Sustained powerful influence is primarily derived from the significance and innovation criteria
Though I still focus on assessment of Significance and Approach in the review …
I’ll be interested to see if these data change with the just-completed reviews of the first short-format applications submitted during Cycle 1. If anything, I would expect them to become more tightly correlated, which is I’m sure what Toni Scarpa hopes as well. Then again, the Summary Statements from this round that I’ve already read invariably note something to the effect that details are lacking (in approach), so we’ll see.
(and, after ignoring the blogosphere for a few weeks due to travels & grant overload, I just thought to check, and, yes, DrugMonkey covered this as well … but in case there’s anyone here but not there who might be interested in the NIGMS Feedback …)
NIH Guide 2.0 – Enhanced FOAs
06.25.10 by Michelle Kienholz
Yesterday I attended a last-minute session at which NIH folks presented potential new changes to the way FOAs are released, the idea being that shorter, simpler applications deserve shorter, simpler funding announcements.
The question to you all: what data are most important in an FOA (especially in the Part 1. Overview Information) – that is, what would you not want to see disappear from this section (e.g. key dates, number of applications allowed, etc.) … and what aspects of FOAs do you feel need to be changed (e.g., details on eRA registration, page limitations, submission requirements, etc.)?
At the end of the session, one brave volunteer gave an e-mail address to which comments/suggestions could be sent … or you could add them here in response to this post.
And now … off the grid for a few days. Try not to trash the place while I’m gone …
NIH Regional Grants Seminar
06.24.10 by Michelle Kienholz
Writedit is in Portland for the NIH Regional Grants Seminar (& I recommend everyone attend one of these or at least view the online presentations) – limited Web access and will be off the grid for a few days after. Have fun.
But … a few tidbits already. Later this afternoon, I’ll learn about plans to shorten/streamline/”enhance” the writing of FOAs. More on that later.
At one talk, an SRO shared a good rule of thumb for differentiating Impact from Significance: Significance is the hypothetical benefit to science/technology/clinical practice *if* the aims are achieved … Impact is the real-world impact, taking into account why the investigators & environment will really make this cool study work & shift a paradigmm or two.
Also, for resubmissions, SROs really want the reviewers to look at the A1 as a “new” application (reviewers don’t see old application in any case) evaluated based on its own merit – not in relation to how much it improved from the prior submission or whether all the reviewer critiques were met. Not news – but clearly laid out today.
And Sally Rockey (head of OER) confirmed that the NIH is rigorously sniffing out “new” applications that are not new. Rigorously (investing time & personnel needed). Please remember that just changing PAs does not make the application new (changing mechanisms, resubmission after failing at an RFA do qualify as new). She also noted that so far, there has only been a 10% bump in applications submitted. The next big jump will likely be in 2012, when everyone who had ARRA funding and asked for no-cost extensions comes back to the trough for more ….
Collins on NIH Budget, Investigator-Initiated Research, Basic Science, etc.
06.4.10 by Michelle Kienholz
In the published version of his interview with Science, Francis Collins had some rather grim news for NIH-supported investigators, particularly those in the basic sciences:
Q: Regarding the budget and grant funding, you’ve said that you expect very low success rates in 2011, 15%. What’s this going to mean?
F.C.: We don’t know what the budget will end up being. Obviously, the signs are not particularly good that the Congress will do better than the president’s budget [a 3.2% increase]. Some noises might even indicate that they’ll do worse.
We will undoubtedly have to look at draconian things like downward negotiations, which means cutting the budgets of approved grants in order to try to free up dollars to fund more grants. We are trying to protect certain parts of the enterprise. Postdoc training slots, for instance. But I’m sorry, I can’t come up with a magic solution here that is going to reduce all of the pain.
Q: Some researchers are worried that your emphasis on translational research and big goals will mean cutting back on investigator-initiated grants.
F.C.: I don’t think they should be very worried. Everybody’s going to be stressed, so it will be tempting if your grant didn’t get funded to look around for some reason other than the fact that it was a tough year budgetarily. The amount of additional funds that might go into focusing on translation are going to be maybe 1% or thereabouts of the overall NIH effort. That shouldn’t have a very big effect.
Q: Does that mean 1% less for investigator-initiated research?
F.C.: I would argue that if NIH simply said we’re going to keep doing what we’ve been doing all along, we’re not in a very good position then to ask the Administration or the Congress to give us more resources.
The translational goals get a lot of traction with the Congress, with the public. They should. I mean, we’re the National Institutes of Health. We are supposed to be coming up with ways to prevent and treat disease.
I was a bit taken aback by that last answer, but in the complete transcript of his almost hour-long interview with Science staffers Jocelyn Kaiser, Eliot Marshall, and Laura Zahn, Collins’ full sentiments become more clear. The full question was:
so we wanted to talk about the budget and the stimulus cliff which you’ve been talking about on the Hill lately. You’ve said that you expect very low success rates, 15%. What’s this going to mean? Is it going to shut labs down? We’ve heard plenty of people talking about posdocs being out of work. What’s it going to mean if that’s what you get?
Collins first notes that success rates depend on a lot of things, including “the budget you actually have to spend.” And then …
Yesterday’s primary result was a big disappointment, I think to many people. {clearly the interview was held on May 19th, the day after our primary election here in BICO- writedit} We were hoping to see Senator [Arlen] Spector [(D-PA)] reelected because he’s been such a strong champion for medical research, as you all know. And I am personally grieving at that outcome because of what it may mean for medical research in the congressional appropriation process. And because Senator Specter as a human being has been remarkably devoted to making the case for the importance of this kind of work. And he’s a courageous cancer survivor himself.
And the real pisser is what has happened in the aftermath of Sestak’s win … but don’t get me started.
Collins adds
So we don’t know what the budget will end up being. Obviously the signs are not particularly good that Congress will do better than the president’s budget. Some noises even indicate they’ll do worse with all the other pressures upon them from education needs and so on.
The other question we don’t know the answer to is what will be the number of grant submissions in FY’11. I will tell you, there were some concerns that we would see a huge deluge already in the January submissions because even though they would be submitted in January 2010 because of the cycle time, they wouldn’t be funded until early ’11. And many people were wondering if all those Challenge Grant applications, those 20,000 that came in during the stimulus and only 800 of those got funded, are the other 19,200 going to come back? … {but} We didn’t see a particularly big bump.
Well, a 13% increase in applications received since Oct 2009 (even with some ARRA applications still coming in) isn’t nothing …
We will undoubtedly have to look at draconian things like downward negotiations, which means cutting the budgets of approved grants in order to free up dollars to approve more grants.
Undoubtedly.
When asked whether the NIH would be “facing more flat funding”, Collin states the obvious:
Our system is every year is a brand new year and you’re very much at the mercy of what’s happening with the economy and the political system. And we have to roll with that and do the best we can. … It will obligate all of us who are science managers … to look really hard at whether there are less productive areas of sicence that we can’t continue to push forward just because we always have.
And then we get to the full answer to the “Does that mean 1% less for investigator initiated research?”, which is quite a bit longer and a bit more encouraging – for those in basic research – than the printed excerpt:
… When you no doubt listened to the hearings in the House and the Senate, a lot of interest about translation, about the Cures Acceleration Network and whether it’s going to get appropriated and not just authorized. And I think that puts us in a pretty good position to say, we really are at a critical juncture as far as moving basic science forward discoveries into the clinic in ways that we couldn’t have 5 or 6 years ago.
All that being said, I totally want to protect the basic science foundation of everything we do because that is our future. And I would certainly say to any R01 investigator who’s worried about this that the vast majority of the discoveries that are going to matter are going to come from those hypothesis-driven investigator-initiated efforts.
… But I also want to inspire those individuals who may have thought of themselves as lifelong basic scientists to think translationally if the opportunity arises and not to feel that’s off limits. And that’s not for everybody but I can certainly point to a few basic scientists who found that pretty exhilarating. Why do people go into biomedical research? Curiosity, sure, it’s fascinating, it’s intellectually stimulating, it’s not very profitable if you’re in the academic sector. But also because you have a dream that you’re going to do something, that you’re going to discover something that wasn’t known before and you’re going to help somebody. And if we in this effort to emphasize translation are going to give a few more basic scientists a chance to see that part of their dream become a little more real, I don’t think that’s bad as long as we’re not seen as putting down the basic science part of their careers and many others like them. We can’t do that.
However, the funding emphasis will remain on translational and applied research – accelerating cures … on cue, it seems. And the more pure basic science, seeking to serendipitously discover mechanism … the way things work? I worry that these sorts of projects will be lost as “less productive” … harder to explain to Congress.
But then, in moving on to talk about drug discovery, a question comes up about the new regulatory science initiative between NIH and FDA, which will be paid for out of the NIH Common Fund. Eliot Marshall asks, “They [FDA] do have a research program, right?” To which Collins replies:
Very, very limited and they certainly have not had anything like this. And so we – most of this is actually funded by the NIH since their budget is so tight, so we volunteered because it seems like a great idea that will cover most of our costs. We put out an RFA and we got 59 letters of intent. I’ve read all of those and they’re really interesting. In fact, we’re going to be challenged when the review goes through, we may not have enough money to fund the best stuff and have to go scrabbling around to fund more.
Having read the RFA, I’m impressed they got 59 letters of intent (all read by the NIH Director – not something you think about when jotting off these formalities). I wonder how many applications came in and how much more than the budgeted $6M will go toward this rather unusual RFA … and those to follow as part of the larger “regulatory science” initiative.
NIH & NSF Entering a New Line of Business
05.13.10 by Michelle Kienholz
Funding it, that is, as part of the new i6 Challenge.
I couldn’t possibly describe this program better than the full announcement:
The i6 Challenge is a new, multi-agency innovation competition led by the U.S. Department of Commerce (DOC) and its Economic Development Administration (EDA). The DOC and EDA will coordinate this funding opportunity with the NIH, the NSF, and the U.S. Patent and Trademark Office (USPTO) to leverage federal resources and maximize available funding to i6 Challenge winners.
The i6 Challenge is designed to encourage and reward innovative, ground-breaking ideas that will accelerate technology commercialization and new venture formation across the United States, for the ultimate purpose of helping to drive economic growth and job creation.
To accomplish this, the i6 Challenge targets sections of the research-to-deployment continuum that are in need of additional support, in order to strengthen regional innovation ecosystems. Applicants to the i6 Challenge are expected to propose mechanisms to fill in existing gaps in the continuum or leverage existing infrastructure and institutions, such as economic development organizations, academic institutions, or other non-profit organizations, in new and innovative ways to achieve the i6 objectives.
Applicants are also expected to leverage regional strengths, capabilities, and competitive advantages. Furthermore, they are expected to identify a real or persistent problem or an unaddressed opportunity with a sense of urgency, cultivate strong public-private partnerships, provide a credible plan to access resources, demonstrate how the effort will be sustained, and bring together a well-qualified team and partners.
EDA intends to fund implementation grants for technical assistance through its Economic Adjustment Assistance Program under the i6 Challenge.
EDA will make at least 6 awards of up to $1M – one in each of its 6 regions. EDA can only fund proposals in an area that, on the date of application, meets one (or more) of the following economic distress criteria: 1. An unemployment rate that is, for the most recent 24-month period for which data are available, at least one percentage point greater than the national average unemployment rate; 2. Per capita income that is, for the most recent period for which data are available, 80% or less of the national average per capita income; or 3. Has a “Special Need,” as determined by EDA.
Successful Applicants who are NIH SBIR Grantees with an active SBIR grant as of October 2010 are eligible for up to $500K in supplemental awards.
Successful Applicants who are NSF SBIR Grantees with an active SBIR grant as of July 15, 2010 are eligible for up to $100K (individual) to $500K (collective) in supplemental awards.
USPTO will provide customized intellectual property seminars to entrepreneurs and innovators associated with the winning Applicants.
Applicants must demonstrate a Matching Share of at least $500K, which must be available and committed to the project from non-federal sources. EDA will give preference to applications with higher Matching Shares and to applications with higher levels of cash contributions in their Matching Share.
Strongly recommended letters of intent are due June 15th – full applications due July 15th.
Questions? I can’t imagine … but there will be a conference call on Monday, May 17, 2010 at 2:00 p.m. EDT.
Collins FY11 Budget Testimony
04.28.10 by Michelle Kienholz
Most of Francis Collins’ remarks to the House Subcommittee on Labor – HHS – Education Appropriations were of the fluffy, feel-good variety, but he closed with some talking points (references for each are available in the transcript) that you all might want to keep in mind when communicating with your Congressfolk about supporting the NIH appropriation:
It is crucial to keep in mind that investing in NIH not only improves America’s health and strengthens our nation’s biomedical research potential, it empowers the entire U.S. economy. Consider the following statistics:
- A report issued by Families USA calculated that in 2007, every $1 in NIH funding resulted in an additional $2.11 in economic output in the U.S.
- In FY 2007, a typical NIH grant supported the salaries of about 7 high-tech jobs in full or in part.
- The 351,000 jobs resulting from NIH awards paid an average annual wage of more than $52,000 per annum and account for more than $18 billion in wages for FY 2007.
- Long term, NIH funded R&D sparks U.S. economic innovation in the high-technology and high value-added pharmaceutical and biotechnology industries. For example, between 1982 and 2006, one-third of all drugs and nearly 60 percent of promising new molecular entities approved by the FDA cited either an NIH-funded publication or an NIH patent.
- NIH-funded research has contributed to overall gains in average U.S. life expectancy from 1970-2000 that were worth an estimated $95 trillion.
NIH FY09 Success Rates
04.8.10 by Michelle Kienholz
So as the 2009 applicants await word as to whether they will be funded with FY10 dollars, I thought I’d post the lastest success rate data from the NIH.
The big NIH-wide scoreboard shows an overall success rate of 20.6%. This includes all competitive applications (e.g., new, renewals, supplements) for all mechanisms. As a reminder, NIH Success Rates:
include applications that are peer reviewed and either scored or unscored by an Initial Review Group. Success rates are determined by dividing the number of competing applications funded by the sum of the total number of competing applications reviewed and the number of funded carryovers [i.e., applications reviewed and scored but not funded the fiscal year prior]. Applications having one or more amendments in the same fiscal year are only counted once.
Grants funded jointly by 2 or more ICs are counted only by the IC footing the largest chunk of the bill.
On the master file, you can click on your favorite ICs to get their specific success rate stats.
Across the NIH, R21s (all Type 1s) have success rates well below R01s (which include Type 2/3 applications in their success rate). Among the big ICs, NHLBI is at 14.5% for the R21; NCI, 13.7%; NINDS, 12.8%; NIAID 11.8%; NIGMS, 7.9%; and NIDDK (which discourages applicants from using this mechanism except for specific types of work), 4.6%. NIMH is an outlier with an R21 success rate of 20.1%. In general, in fact, NIMH looks to be a nice place to go for money … except for R03s, which have a success rate of 9.6% (go to NIDDK, 58%, or NCI, 30.8%, for this mechanism … though these are probably mainly secondary data analyses awards).
No data on F, K, or T awards here … you need to scroll down the main success rate page and check the appropriate Excel spreadsheet for these data … or better yet, check the NIH Data Book for trends in Career Development Awards and Training Grants & Fellowships.
Payline Complexity Explained & CSR College of Reviewers Updated by NIAID
03.31.10 by Michelle Kienholz
As always, the latest issue of NIAID Funding News is a treasure trove of information and good advice.
First, for the hundreds of you out there wondering why your IC hasn’t set a payline yet, NIAID reports, shockingly, that the trend toward score clustering has increased and explains how score clustering causes jumps in assigned percentiles. An impact score of 20 seems to be the sweet spot thus far: “In the first two review cycles of this fiscal year, approximately 3% of applications reviewed by CSR received a score of 20.” NIAID gives an example in which a score of 20 in one study section might land at the 9th percentile, with a score of 21 in turn at the 11th percentile (payline at the 10th percentile).
NIAID also includes an update on the CSR College of Reviewers (discussed here previously), including the current membership roster. As a reminder, these folks (“editorial board members”) will be asked to provide written reviews only on up to 12 applications a year for 2 years as part of 2-stage reviews (with the second stage being the face-to-face meetings of “editors”).
Lots of other good intel and advice, so be sure to scroll through the entire newsletter and, no matter which IC is “yours”, sign up for delivery to your very own digital mail box.
NIH Plans to Eliminate Error-Correction Window
03.18.10 by Michelle Kienholz
within the next year, along with AHRQ and the CDC, according to a notice seeking input on this proposed change.
The rationale:
Eliminating the error correction window will allow NIH to enforce a fair and consistent submission deadline for all applicants. In addition, eliminating the error correction window will help NIH reduce the time needed to process applications and forward them through the peer review process.
They also note that “applications now tak[e] minutes to process through both systems [Grants.gov and eRA] on average instead of days” and that safeguards remain in place for systems problems that might interfere with on-time submission/error-checking.
You can submit comments on the impact of such a change as well as any preference on timing (in the next 3-6 months, in a year, or later).
Input Sought on NIGMS Training & Career Development Plan
03.17.10 by Michelle Kienholz
As NIGMS Director Jeremy Berg noted on their blog the Feedback Loop, NIGMS would very much appreciate your input as they develop a strategic plan for training and career development.
You can submit comments online until April 21st.
You can participate in person at one of four regional stakeholder meetings:
- Philly: April 29th
- San Francisco: May 12th
- Chicago: May 25th
- Atlanta: June 4th
You can participate in a Web discussion on Friday, June 11th, from 1-4 p.m. Eastern Time.
Please do participate, though. Obama’s proposed FY11 budget includes higher $ amounts for stipends (6% increase!) but a loss of 140 funded slots on institutional awards (T mechanisms) and only 48 additional individual awards (Fs); these numbers are NIH-wide, not just at NIGMS. With stipend levels more attractive but awards/slots increasingly competitive, your input will be more important than ever.
