sofs.org.ua › About us › Archive › 2022 › №3(117) › 1.4

1.4

PERFORMANCE EVALUATION IN RESEARCH ORGANIZATIONS: EVOLUTION OF METHODOLOGIES   

I.Yu. YEGOROV 1
https://orcid.org/0000-0002-3829-6383
І.A. ZHUKOVYCH 1
https://orcid.org/0000-0001-5027-5991
1 Center for Evaluation of Activity of Research Institutions and Scientific Support of Regional Development of Ukraine of the NАS of Ukraine

Nauka naukozn. 2022, 3(117): 36–50
https://doi.org/10.15407/sofs2022.03.036

Section: Problems of the development of science and technology potential
Language: Ukrainian
Abstract: The article is devoted to a topical issue of performance evaluation in research organizations. Methodological approaches to the evaluation of research organizations, which results determine the amount of their basic funding, require constant improvement of procedures, being subject of numerous studies. This issue is especially relevant for the National Academy of Sciences (NAS) of Ukraine as the largest R&D center in the research system of Ukraine. Robust and regular performance evaluation of research organizations of the NAS of Ukraine will help increase their efficiency, make distribution of funding more transparent, improve reporting on the use of budget funds, reorganize their network in a rational way.

The article highlights trends in performance evaluation of research organizations of Ukraine, based on foreign experiences and results obtained in the first cycle of the evaluation of research organizations of the NAS of Ukraine (2016–2021), with providing recommendations for further improvements in evaluation methods and procedures. Research methods: analysis, generalization of scientific sources of information, synthesis and comparison. Source base: data on the evaluation of research organizations of the NAS of Ukraine, regulatory framework for the evaluation of research organizations, publications of Ukrainian and foreign specialists.

It is emphasized the presently research organizations of the NAS of Ukraine are evaluated using two official evaluation techniques, which main difference is the method for evaluation: professional assessment of the organization’s performance by qualified experts (the methodology of the NAS Ukraine) and formalized self-assessment of the organizations based on quantitative indicators that can be adjusted in a way by a group of experts (the methodology of the Ukrainian Ministry of Education and Science). The problems of constructing a composite indicator to be used for ranking are discussed, with a review of researches’ positions.

The conclusions are made about the unreliability of metrics for evaluating the performance of research organizations and the need to adopt expert evaluation methods supplemented by quantitative indicators. It is emphasized that evaluation results can be made more robust through inviting as many as possible foreign researchers as experts. This approach is in line with global trends in the evaluation of research organizations.

Keywords: research organization, methodology for evaluating the performance of research organization, National Academy of Sciences of Ukraine, methodology of evaluating the performance of R&D and innovation activities of research organization, higher education establishments, universities, ranking, Chinese Academy of Sciences.

References

  1. Xu, F., & Li, X. (2016). The changing role of metrics in research institute evaluations undertaken by the Chinese Academy of Sciences (CAS). Palgrave Communications, 2(1), 1–6. http://doi.org/10.1057/palcomms.2016.78
  2. Yue, W., Gao, J., & Suo, W.-L. (2020). Efficiency evaluation of S&T resource allocation using anaccurate quantification of the time-lag effect and relation effect: a case study of Chinese research institutes. Research Evaluation, 29(1), 77–86. https://doi.org/10.1093/reseval/rvz027
  3. Ortiz Núñez, R. (2021). Altmetrics: alternative metrics for scientific, technological and innovation evaluation. Academia Letters. Art. 1658. https://doi.org/10.20935/AL1658
  4. Abramo, G., D’Angelo, C.A. (2015). The VQR, Italy’s second national research assessment: Methodological failures and ranking distortions. Journal of the Association for Information Science and Technology, 66 (11), 2202—2214. http://doi.org/10.1002/asi.23323
  5. Franceschet, M. & Costantini, A. (2011) The first Italian research assessment exercise: A bibliometric perspective. Journal of Informetrics, 5(2), 275–291. http://doi.org/10.1016/j.joi.2010.12.002
  6. Rybachuk, V.P. (2013). Methodological problems of evaluating the productivity of scientific activity. Science and Science of Science, 2, 46–52 [in Ukrainian].
  7. Karpenko, A.V., & Budytska, Yu.O. (2017). Theoretical and Methodological Basics of Evaluating Scientific PracticeEffectiveness in Ukraine. Scientific works of Central Ukrainian National Technical University. Economics, 31, 51–57 [in Ukrainian].
  8. Laktionova, T.M., Parasochka, I.F., & Havrylova, N.A. (2021). Bibliometric indicators of publication activities in research units of the institute. Science and Science of Science, 4(114), 74–93. https://doi.org/10.15407/sofs2021.04.074 [in Ukrainian].
  9. Mryglod, O.I., & Nazarovets, S.A. (2019). Scientometrics and management of scientific activities once again about the global and Ukrainian. Visn. Nac. Akad. Nauk Ukr., 9, 81–94. https://doi.org/10.15407/visn2019.09.081 [in Ukrainian].
  10. Yegorov, I.Yu., Zhukovych, I.A., & Artyushenko, V.V. (2019). Interim results and problems of implementation of the methodology for evaluation of the scientific institutions of the NAS of Ukraine. Visn. Nac. Akad. Nauk Ukr., 11, 38–44. https://doi.org/10.15407/visn2019.11.038 [in Ukrainian].
  11. Zhukovych, I.A., & Yegorov, I.Yu. (2020). The Evolution of Methodical Approaches for Evaluation the Efficiency of the Activities of Scientific Research Organizations in the Czech Republic. Statistics of Ukraine, 2–3, 117–126. https://doi.org/10.31767/su.2-3(89-90)2020.02-03.13 [in Ukrainian].
  12. Zhukovych, I.A., & Yegorov, I.Yu. (2021). Evolution of Methodological Approaches to Evaluating the Efficiency of the Activities of Research Organizations in Ukraine. Statistics of Ukraine, 2, 4–15. http://doi.org/10.31767/su.2(93)2021.02.01 [in Ukrainian].
  13. Zhukovych, I.A., & Yegorov, I.Yu. (2019) Organization of the NAS of Ukraine in SCIMAGO institutions rankings (SIR). Science, technologies, innovations, 2(10), 15–22. http://doi.org/10.35668/2520-6524-2019-2-03 [in Ukrainian].
  14. (2016). The best decisions are made on the basis of commonsense and positive experience (interview with academician of the National Academy of Sciences of Ukraine A.G. Zagorodny). Visn. Nac. Akad. Nauk Ukr., 3, 3–11 [in Ukrainian].
  15. Holz, C. (2021). University Rankings: Comparing Apples and Pears. Academia Letters. Art. 290. URL: https://doi.org/10.20935/AL290
  16. Hazelkorn, E., & Mihut, G. (Eds.) (2021). Research Handbook on University Rankings. Theory, Methodology, Influence and Impact. N.Y. and Northhampton: Edward Elgar Publisher. https://doi.org/10.4337/9781788974981
  17. Goodhart, C. (1975) Problems of Monetary Management: The UK experience. Papers in Monetary Economics, 1, 1–20. URL: https://www.econbiz.de/Record/problems-of-monetary-management-the-u-k-experience-goodhart-charles/10002525062 (last accessed: 04.05.2022).
  18. Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548). http://doi.org/10.1038/520429a
  19. Bogdanov, V.L., & Kubalskyi, O.N. (2021). On the concept of development of the National Academy of Sciences of Ukraine for 2021–2025. Visn. Nac. Akad. Nauk Ukr., 12, 38–45 [in Ukrainian].

Full Text (PDF)