More on the Impact-Criterion Score Correlation
03.8.11 by Michelle Kienholz
This time by Sally Rockey on Rock Talk.
Jeremy Berg introduced the concept of correlating overall impact score with the individual criterion scores, first using NIGMS and then NIH-wide data.
Based on the 32,546 applications (of 54,727 submitted) that received overall impact scores in FY10, OER played with the numbers a bit more but came up with the same conclusions: Approach and then Significance drive Overall Impact scores.
For applications receiving numerical impact scores (about 60% of the total), we used multiple regression to create a descriptive model to predict impact scores using the applications’ criterion scores, while attempting to control for ten different “institutional” factors (e.g., whether the application was new, a renewal, or a resubmission). In the model, scores for the approach criterion had the largest regression weight, followed by criterion scores for significance, innovation, investigator, and environment. The same pattern of results was observed across multiple rounds of peer review and institute funding decisions.
She also notes, as can be seen in her figure, that scores for Approach showed the widest range, followed by Significance.
So, the work you propose doing better be important … and, more importantly, better be done right.
Surveying Peer Review Enhancements
03.1.11 by Michelle Kienholz
In the midst of grant deadlines, writedit has been staring longingly at the psychiatric hospital up the hill, where a room with a view and a valium drip sounds good about now, but has just enough time for a quick post to distract all of you with freshly assigned impact scores from obsessively searching for any hint of funding success … and those of you with stale impact scores from wondering again when paylines might be known.
The NIGMS Feedback Loop and Rock Talk both have current posts on OER survey data on Enhancing Peer Review. The Feedback Loop pulls out respondent assessment of the value of the individual criterion scores, a topic of recent interest to Director Berg both at the NIGMS and NIH-wide level. Seems less than half of you feel the criterion scores are particularly helpful …
Peer Review Survey
10.19.10 by Michelle Kienholz
The Comparative Assessment of Peer Review (who knew?), an NSF-funded project of the Center for the Study of Interdisciplinarity (who knew?) at the University of North Texas, has an online survey that you are all invited (and encouraged) to complete. The CAPR “examines the peer review process at 6 science agencies worldwide: NSF, NIH, NOAA, NSERC, the EU’s 7th Framework Programme, and the Dutch STW.”
Probably not entirely what you might expect, but still an interesting thought exercise with plenty of opportunity to enter free-text comments and input.
The project is also creating a digital repository for the aforementioned science agencies (the sorts of program & policy documents not easily found in one place) and examining the broader impacts criteria for NSF-funded research (other than their own).
And, speaking of peer review & broader impacts, for those of you familiar with the Rocket Boys story (and even more so for those of you who are not familiar with it!), I think you’ll enjoy this adorable little (3’32″) video from the NIH.
NIH-Wide Data on Impact & Criterion Score Correlations
09.30.10 by Michelle Kienholz
Ask and ye shall receive … NIH-wide data on the correlation between individual review criterion scores and overall impact score, compliments of Jeremy Berg.
Correlation coefficients between the overall impact score and the five criterion scores for 32,608 NIH applications from the Fiscal Year 2010 October, January and May Council rounds
As he notes, the trends across the ICs mirror what he found at NIGMS. The NIH-wide data also include more mechanisms … whereas Jeremy analyzed 654 R01s from one cycle, these data include all RPG, research center, and SBIR/STTR applications over 3 cycles (Oct-Jan-May Councils). Not sure if we’ll get all his other lovely data at the NIH level, but we can dream. In the meantime, thanks so much for your leadership in disseminating the NIGMS and now these NIH-wide data, Dr. Berg.
Research Productivity among NIGMS-funded PIs
09.27.10 by Michelle Kienholz
Update: Jeremy Berg’s analyses of productivity among NIGMS-funded PIs has been covered in Nature News, with some additional commentary and a new composite figure.
NIGMS Director Jeremy Berg continues to anticipate the sorts of questions NIH-funded investigators would like to have answered. In his latest Feedback Loop post, he analyzes data on publications from 2007-2010 linked to NIGMS funding and the impact factor of the journals involved according to annual direct costs.
So what do the data show? Among the 2,938 investigators who held at least one NIGMS R01 or P01 grant in Fiscal Year 2006:
Median number of grant-linked publications, 6 (2007-2010)
Median journal impact factor, 5.5
Median annual direct costs of funding received, $220K
Please note (at the original post) the ranges in each category, though, and that funding totals are by PI vs award. PI productivity peaks at about $700K per year in annual DC. Anyone surprised? Jeremy notes this supports the NIGMS threshold of $750K in defining (and limiting addtional funding to) “well-funded laboratories.”
Anyone have specific sorts of peer review/grant funding/research productivity data they would like to see analyzed?
One hopes OER has recognized the scientific community’s interest in these types of data … perhaps they could cover NIH-wide and IC-specific trends on their RePORT site down the road.
Status of PIs Who Score $ with NIGMS …
09.14.10 by Michelle Kienholz
Jeremy Berg has posted another great display of application outcome data for the 655 NIGMS R01 applications reviewed during the January 2010 Council. This time, he shows application outcome (awarded, not awarded) by overall impact score and percentile and PI status (ESI, new, established).
Separately, he adds a line graph of the cumulative fraction of applications by percentile in four classes: ESI, new, established Type 1 (new application), and established Type 2 (competing renewal). The latter two symbols are a bit tricky to discern on the graph, but the Type 2s are the clear winners, as expected.
And as PP points out in his comment, how did a new investigator with a 3rd percentile/impact score 11 application not get funded? Or the ESI at the 20th percentile with a 19? Hopefully neither of these symbols actually represent more than one unfunded applicant in that status with that score.
Enhancing Summary Statements
08.9.10 by Michelle Kienholz
or touching up the previously enhanced version at least.
The Extramural Nexus from OER this month indicates that “help is on the way” … (that does not involve making Madame Zelda an allowable consultant cost):
NIH will begin requiring reviewers to include a paragraph in their written critiques to explain the factors that informed his or her overall impact score … [to] provide applicants with greater insight into how each reviewer assessed scientific merit of the grant application and determined his/her overall impact score. … You will see the additional information in summary statements for applications reviewed this fall.
Well then, something to look forward to. One wonders if reviewers will be required to remember verbatim the drunken comment made by the applicant at a Gordon conference …
Feedback on NIH Scoring
07.16.10 by Michelle Kienholz
Even updateder update: Jeremy Berg has posted an analysis of application scoring for the October 2010 Council pool (654 R01s) at the Feedback Loop, with similar trends in Approach and Significance.
Update: Jeremy Berg has posted similar analyses of Approach and Innovation scores at the Feedback Loop … and now, regression analysis results, too!
My NIGMS Feedback Loop listserv alerted me to Jeremy Berg’s assessment Model Organisms and the Significance of Significance. Not much on model organisms (an interesting comment by Whimple – update: and others now), but Dr. Berg notes that:
To examine how reviewers apply the significance criterion in determining overall impact scores, I analyzed 360 NIGMS R01 applications reviewed during the October 2009 Council round. [he shows a plot, too]
As anticipated, the scores are reasonably strongly correlated, with a Pearson correlation coefficient of 0.63. Similar comparisons with the other peer review criteria revealed correlation coefficients of 0.74 for approach, 0.54 for innovation, 0.49 for investigator and 0.37 for environment.
Hmm. Not too surprising. Research is not likely to have much impact if it is not both significant (meaningful) and well designed/planned. I realized on reading his post that I do indeed tend to discount the scores (and, to some extent, the comments) under the other criteria and focus on the overall impact bullets plus Significance and Approach when reviewing Summary Statements.
I actually like this definition of Overall Impact from Sally Amero’s presentation on peer review at the June 2010 NIH Regional Grant Seminar:
Likelihood for the project to exert a sustained, powerful influence on the research field(s) involved
- Likelihood (i.e., probability) is primarily derived from the investigator(s), approach and environment criteria
- Sustained powerful influence is primarily derived from the significance and innovation criteria
Though I still focus on assessment of Significance and Approach in the review …
I’ll be interested to see if these data change with the just-completed reviews of the first short-format applications submitted during Cycle 1. If anything, I would expect them to become more tightly correlated, which is I’m sure what Toni Scarpa hopes as well. Then again, the Summary Statements from this round that I’ve already read invariably note something to the effect that details are lacking (in approach), so we’ll see.
(and, after ignoring the blogosphere for a few weeks due to travels & grant overload, I just thought to check, and, yes, DrugMonkey covered this as well … but in case there’s anyone here but not there who might be interested in the NIGMS Feedback …)
NIH Regional Grants Seminar
06.24.10 by Michelle Kienholz
Writedit is in Portland for the NIH Regional Grants Seminar (& I recommend everyone attend one of these or at least view the online presentations) – limited Web access and will be off the grid for a few days after. Have fun.
But … a few tidbits already. Later this afternoon, I’ll learn about plans to shorten/streamline/”enhance” the writing of FOAs. More on that later.
At one talk, an SRO shared a good rule of thumb for differentiating Impact from Significance: Significance is the hypothetical benefit to science/technology/clinical practice *if* the aims are achieved … Impact is the real-world impact, taking into account why the investigators & environment will really make this cool study work & shift a paradigmm or two.
Also, for resubmissions, SROs really want the reviewers to look at the A1 as a “new” application (reviewers don’t see old application in any case) evaluated based on its own merit – not in relation to how much it improved from the prior submission or whether all the reviewer critiques were met. Not news – but clearly laid out today.
And Sally Rockey (head of OER) confirmed that the NIH is rigorously sniffing out “new” applications that are not new. Rigorously (investing time & personnel needed). Please remember that just changing PAs does not make the application new (changing mechanisms, resubmission after failing at an RFA do qualify as new). She also noted that so far, there has only been a 10% bump in applications submitted. The next big jump will likely be in 2012, when everyone who had ARRA funding and asked for no-cost extensions comes back to the trough for more ….
NIH FY09 Success Rates
04.8.10 by Michelle Kienholz
So as the 2009 applicants await word as to whether they will be funded with FY10 dollars, I thought I’d post the lastest success rate data from the NIH.
The big NIH-wide scoreboard shows an overall success rate of 20.6%. This includes all competitive applications (e.g., new, renewals, supplements) for all mechanisms. As a reminder, NIH Success Rates:
include applications that are peer reviewed and either scored or unscored by an Initial Review Group. Success rates are determined by dividing the number of competing applications funded by the sum of the total number of competing applications reviewed and the number of funded carryovers [i.e., applications reviewed and scored but not funded the fiscal year prior]. Applications having one or more amendments in the same fiscal year are only counted once.
Grants funded jointly by 2 or more ICs are counted only by the IC footing the largest chunk of the bill.
On the master file, you can click on your favorite ICs to get their specific success rate stats.
Across the NIH, R21s (all Type 1s) have success rates well below R01s (which include Type 2/3 applications in their success rate). Among the big ICs, NHLBI is at 14.5% for the R21; NCI, 13.7%; NINDS, 12.8%; NIAID 11.8%; NIGMS, 7.9%; and NIDDK (which discourages applicants from using this mechanism except for specific types of work), 4.6%. NIMH is an outlier with an R21 success rate of 20.1%. In general, in fact, NIMH looks to be a nice place to go for money … except for R03s, which have a success rate of 9.6% (go to NIDDK, 58%, or NCI, 30.8%, for this mechanism … though these are probably mainly secondary data analyses awards).
No data on F, K, or T awards here … you need to scroll down the main success rate page and check the appropriate Excel spreadsheet for these data … or better yet, check the NIH Data Book for trends in Career Development Awards and Training Grants & Fellowships.
