A recent study published on the Open Research platform F1000 Research, conducted by Ivan Buljan, David G. Pina (REA), Antonija Mijatović and Ana Marušić, analysed the importance of numerical scores in Marie Skłodowska-Curie Actions (MSCA) projects. The study focused on Innovative Training Networks (ITN) proposals submitted under Horizon 2020 calls. The ITN funding scheme has been renamed MSCA Doctoral Networks (DN) under the Horizon Europe framework programme.
The aim of the study was to assess whether there were differences in reviewer comments’ linguistic characteristics after the numerical scoring was removed by the European Commission in 2020. Comments were compared to those from 2019, when numerical scoring was still present. The findings indicate that the removal of numerical scores did not contribute to meaningful differences in the evaluation procedure of the ITN proposals or its outcome. These results support the finding that the procedure used for the evaluation of MSCA grant proposals is robust and stable.
Method:
The research team used anonymised datasets, without insight into the actual content of the proposal, or the names of applicants or expert evaluators. Personal data protection regulations were therefore not applicable. Data analysed concerned the Individual Evaluation Reports (IER) and Evaluation Summary Reports (ESR) of all ITN proposals evaluated from the 2019 and 2020 calls.
Each report included textual comments referring to the three different evaluation criteria (Excellence, Impact, and Implementation). Scores of IER were only available for the year 2019. The authors collected the data on the proposal status after evaluation (“Main list”, “Reserve list” or “Rejected”), call year (2019 or 2020), research domain, total evaluation scores, as well as numerical scores for each criterion, together with the corresponding comments which separately described proposal strengths and weaknesses.
Linguistic characteristics of experts’ comments were assessed using the Linguistic Inquiry and Word Count software. The latter is a program that counts words related to different psychological states and phenomena, and gives a score that is a proportion of the specific category in the entire text. By proceeding this way, the researchers were able to calculate separately for strengths and weaknesses of each of the criteria assessed.
Findings:
The researchers found that the differences in linguistic characteristics between reports from both calls (2019 and 2020) were small and negligible from a practical viewpoint. This indicates that the removal of numerical scores did not result in meaningful changes in the reports’ comments, assessed by quantitative text analysis. For both calls, the comments were conceived objectively, with weaknesses written with less emotion and more analytically than the proposals’ strengths.
In addition, the authors found that the final status of the proposals (i.e. main-listed or rejected) can be predicted by the linguistic characteristics of the reviewer’s comments, especially the tone related to the identified weaknesses, indicating that weaknesses may be crucial in proposal evaluation.
A look into the future:
The study paves the way for many potential research methods. It focuses on the characteristics of the textual content of reports, but not on traits of the reviewers themselves. For example, it would be interesting to employ a quantitative text analysis, sentiment analysis or analysis of the text tone. This method could serve as a useful tool to determine whether there were any differences in the evaluation performed by distinct reviewers after the removal of individual numerical scores.
Disclaimer: All views expressed in this article are strictly those of the authors and may under no circumstances be regarded as an official position of the European Research Executive Agency or the European Commission.
More about the Marie Skłodowska-Curie Actions
Details
- Publication date
- 16 October 2023
- Author
- European Research Executive Agency