Academic Evaluation from the Perspective of Multi-source eterogeneous Data:Connotation, Status and Prospect

Authors

Sanhong DENG, School of information management， Nanjing University， Nanjing 210023
Jie YANG, School of information management， Nanjing University， Nanjing 210023
Hao WANG, School of information management， Nanjing University， Nanjing 210023
Jia SHEN, Jiangsu Institute of Quality and Standardization， Nanjing 210023

Keywords

academic evaluation; scientometrics; multi-source and heterogeneous data

Abstract

[Purpose/significance]Academic evaluation and scientometric methods of synthesizing multisource heterogeneous data have gradually become hot spots in the field. At present， there is no systematic theoretical discussion on academic evaluation from the perspective of multisource heterogeneous data.[Method/process]In view of insufficient utilization of multi-source heterogeneous data， inappropriate research methods， and unclear research objectives， this paper combs the development of academic evaluation from the perspective of multi-source heterogeneous data， and clarifies the connotation and extension of the field， and analyzes the frontiers and hot spots in the field.[Result/conclusion]The results state clearly that from the perspective of multi-source heterogeneous data， academic evaluation manifests the trend of increasing data dimension， data granularity refinement， and evaluation index complexity. The definition of academic evaluation connotation， current situation analysis and future prospects from the perspective of multi-source heterogeneous data are of great significance to clarify domain recognition， identify related concepts， clarify research paths and methodological basis， and build an academic evaluation system with Chinese characteristics.

First Page

42

Last Page

56

Digital Object Identifier (DOI)

10.19809/j.cnki.kjqbyj.2023.04.004

Recommended Citation

DENG, Sanhong; YANG, Jie; WANG, Hao; and SHEN, Jia (2023) "Academic Evaluation from the Perspective of Multi-source eterogeneous Data:Connotation, Status and Prospect," Scientific Information Research: Vol. 5: Iss. 4, Article 4.
DOI: 10.19809/j.cnki.kjqbyj.2023.04.004
Available at: https://eng.kjqbyj.com/journal/vol5/iss4/4

Reference

[1] 中共中央办公厅，国务院办公厅．关于深化项目评审、人才评价、机构评估改革的意见[EB/OL].（2018-07-03）[2019-04-18].https：//www.gov.cn/zhengce/2018-07/03/content_5303251.htm.
[2] 国务院．关于优化科研管理提升科研绩效若干措施的通知[EB/OL].（2018-07-18）[2019-04-18].https：//www.gov.cn/zhengce/content/2018-07/24/content_5308787.htm.
[3] 科技部，教育部，人力资源社会保障部，中科院，工程院．关于开展清理“唯论文、唯职称、唯学历、唯奖项”专项行动的通知[EB/OL].（2018-10-15）[2019-04-15].https：//www.gov.cn/zhengce/zhengceku/2018-12/31/content_5446309.htm.
[4] 中华人民共和国中央人民政府.教育部 科技部印发《关于规范高等学校SCI论文相关指标使用 树立正确评价导向的若干意见》的通知[EB/OL].（2020-02-18）[2020-08-10].https：//www.gov.cn/zhengce/zhengcekn/2020-03/03/content_5486229.htm.
[5] SPIER R.The history of the peer-review process[J].Trends in Biotechnology，2002，20（08）：357-358.
[6] BESANCON L，PEIFFER-SMADJA N，SEGALAS C，et al.Open science saves lives：Lessons from the COVID-19 pandemic[J].Medical Research Methodology，2020，21（01）：117.
[7] PAULSON E J，ALEXANDER J，ARMSTRONG S.Peer review re-viewed：Investigating the juxtaposition of composition students' eye movements and peer-review processes[J].Research in the Teaching of English，2007，41（03）：304-335.
[8] SILER K，LEE K，BERO L.Measuring the effectiveness of scientific gatekeeping[J].Proceedings of the National Academy of Sciences，2015，112（02）：360-365.
[9] WILSON C S.Informetrics[J].Annual Review Of Information Science and Technology，1999，34：107-247.
[10] PRITCHARD A.Statistical bibliography or bibliometrics?[J].Journal of Documentation，1969，25：348-349.
[11] GARFILED E.Citation analysis as a tool in journal evaluation[J].Science，1972，178（4060）：471-479.
[12] GARFIELD E.How can impact factors be improved?[J].British Medical Journal，1996，313（7054）：411-413.
[13] HIRSCH J E.An index to quantify an individual's scientific research output[J].Proceedings of the National Academy of Sciences of the United States of America，2005，102（46）：16569-16572.
[14] VAN N R.Controversial impact factor gets a heavyweight rival[J].Nature，2016，540（7633）：325-326.
[15] ATAYERO A，POPOOLA S I.Citation analytics：data exploration and comparative analyses of CiteScores of open access and subscription-based publications indexed in Scopus （2014-2016）[J].Data in Brief，2018，19：198-213.
[16] BERGER J M，BAKER C M.Bibliometrics：An Overview[J].RGUHS Journal of Pharmaceutical Sciences，2014，4（03）：81-92.
[17] GUNASHEKAR S，WOODING S，GUTHRIE S.How do NIHR peer review panels use bibliometric information to support their decisions?[J].Scientometrics：An International Journal for All Quantitative Aspects of the Science of Science Policy，2017，112（03）：1813-1835.
[18] QUINDOS G.Confusing the confused：thoughts on impact factor，h（irsch） index，Q value，and other cofactors that influence the researcher's happiness[J].Revista Iberoamericana De Micologia，2009，26（02）：97-102.
[19] CONROY G.What's wrong with the H-index，according to its inventor[EB/OL].Nature Index，2020，24.
[20] 王浩斌.学术共同体、学术期刊与学术评价之内在逻辑解读[J].中国社会科学评价，2015（03）：69-81，126-127.
[21] 苏新宁，王东波.学术评价相关问题与思考[J].信息资源管理学报，2018，8（03）：4-11.
[22] 叶继元.人文社会科学评价体系探讨[J].南京大学学报（哲学·人文科学·社会科学），2010，47（01）：97-110，160.
[23] 叶继元.学术“全评价”体系与中国特色哲学社会科学学术评价体系的构建与完善[J].情报资料工作，2021，42（03）：15-22.
[24] 邱均平，容瑀航，汤建民.我国“五计学”的交叉融合发展：进程与趋势[J].图书馆理论与实践，2020（06）：1-6，11.
[25] 邱均平，余波，杨思洛.大数据背景下一门交叉学科的兴起：论数据计量学的构建[J].中国图书馆学报，2021，47（05）：48-58.
[26] 邓三鸿，陈超群，王昊，等.基于组合赋权法的高校网络影响力综合评价：以“双一流”高校为例[J].西南民族大学学报（人文社科版），2019，40（09）：227-235.
[27] 张宝隆，王昊，邓三鸿，等.基于差异性视角的期刊区分性测度及分析研究[J].情报学报，2020，39（03）：284-296.
[28] 闫晓慧，邓三鸿，张艺炜，等.基于BJRC模型的学术论文创新度测量研究[J].信息资源管理学报，2022，12（04）：46-55.
[29] 张琳，孙蓓蓓，黄颖.交叉科学研究：内涵、测度与影响[J].科研管理，2020，41（07）：279-288.
[30] 赵蓉英，刘卓著，张兆阳，等.论学术话语权及其评价[J].图书情报工作，2022，66（11）：14-23.
[31] 王贤文，方志超，王虹茵.连续、动态和复合的单篇论文评价体系构建研究[J].科学学与科学技术管理，2015，36（08）：37-48.
[32] 罗卓然，陆伟，蔡乐，等.学术文本词汇功能识别：在论文新颖性度量上的应用[J].情报学报，2022，41（07）：720-732.
[33] 杨思洛，郑梦雪.完善人文社会科学成果国际影响力评价[N].中国社会科学报，2022-05-31（008）.
[34] PRIEM J，TARABORELLI D，GROTH P，et al.Altmetrics：A manifesto[EB/OL].（2010-10-26）[2019-04-15].http：//altmetrics.org/manifesto.
[35] GU F，WIDEN-WULFF G.Scholarly communication and possible changes in the context of social media：A Finnish case study[J].The Electronic Library，2011，29（06）：762-776.
[36] ROWLANDS I，NICHOLAS D，RUSSELL B，et al.Social media use in the research workflow[J].Learned Publishing，2011，24（03）：183-195.
[37] TENOPIR C，VOLENTINE R，KING D W.Social media and scholarly reading[J].Online Information Review，2013，37（02）：193-216.
[38] ADIE E，ROE W.Altmetric：enriching scholarly content with article-level discussion and metrics[J]. Learned Publishing，2013，26（01）：11-17.
[39] ALHOORI H，FURUTA R.Do altmetrics follow the crowd or does the crowd follow altmetrics?[C]//2014 IEEE/ACM Joint Conference on Digital Libraries，London：2015.
[40] THOMA B，ROLSTON D，LIN M.Global Emergency Medicine Journal Club：Social Media Responses to the March 2014 Annals of Emergency Medicine Journal Club on Targeted Temperature Management[J]. Annals of Emergency Medicine，2014，64（02）：207-212.
[41] BORNMANN L.Scientific Revolution in Scientometrics：the Broadening of Impact from Citation to Societal[J].Theories of Informetrics and Scholarly Communication，2016：347-359.
[42] VAN-NOORDEN R.Online collaboration：Scientists and the social network[J].Nature，2014，512（7513）：126-129.
[43] 苏娜.科学研究的社会影响力评价：研究与实践进展[J].情报学报，2020，39（10）：1114-1119.
[44] GRANDE D，GOLLUST S E，PANY M，et al.Translating Research For Health Policy：Researchers' Perceptions And Use Of Social Media[J].Health Affairs，2014，33（07）：1278-1285.
[45] HARLEY D，ACORD S K，EARL-NOVELL S，et al.Assessing the future landscape of scholarly communication：an exploration of faculty values and needs in seven disciplines[C]//The Center for Studies in Higher Education，Univ of California Press，Berkeley：2010.
[46] ALMIND T C，INGWERSEN P.Informetric analyses on the world wide web：methodological approaches to 'webometrics'[J].Journal of Documentation，1997，53：404-426.
[47] NEWMAN M E J.The structure and function of complex networks[J].Siam Review，2003，45（02）：167-256.
[48] FONTOURA C L，OLIVEIRA O N，TRAVIESO G.et al.Analyzing and modeling real-world phenomena with complex networks：A survey of applications[J].Advances in Physics，2011，60：329-412.
[49] VAN-ECK N J，WALTMAN L.Visualizing Bibliometric Networks[J].Measuring scholarly impact：Methods and practice，2014：285-320.
[50] BERGSTROM C T，WEST J D，WISEMAN M A.The Eigenfactor metrics[J].Neurosci，2008，28（45）：11433-11434.
[51] YE F Y，LEYDESDORFF L.The "Academic Trace" of the Performance Matrix：A Mathematical Synthesis of the h-Index and the Integrated Impact Indicator（I3）[J].Journal of the Association for Information Science and Technology，2014，65（04）：742-750.
[52] HUTCHINS B I，YUAN X，ANDERSION J M，et al.Relative citation ratio （RCR）：A new metric that uses citation rates to measure influence at the article level[J].PLos Biology，2016，14（09）：e1002541.
[53] FUNK R J，OWEN S J.A Dynamic Network Measure of Technological Change[J].Management science：Journal of the Institute of Management Sciences，2017，63（03）：791-817.
[54] WU L，WANG D，EVANS J A.Large teams develop and small teams disrupt science and technology[J].Nature，2019，566（7744）：378.
[55] BU Y，WALTMAN L，HUANG Y.A multi-dimensional framework for characterizing the citation impact of scientific publications[J].Quantitative Science Studies，2021，2（01）：155-183.
[56] LEYDESDORFF L，BORNMANN L.Disruption indices and their calculation using web-of-science data： Indicators of historical developments or evolutionary dynamics?[J/OL].Journal of Informetrics，2021，15（04）：Article 101219.
[57] LATHABAI H H，NANDY A，SINGH V K.X-index：Identifying core competency and thematic research strengths of institutions using an NLP and network based ranking framework[J].Scientometrics，2021，126（12）：9557-9583
[58] SODERLAND S.Learning information extraction rules for semi-structured and free text[J].Machine Language，1999，34：233-272.
[59] 陆伟，李鹏程，张国标，等.学术文本词汇功能识别：基于BERT向量化表示的关键词自动分类研究[J].情报学报，2020，39（12）：1320-1329.
[60] YANG L，ZHANG Z，CAI X，et al.Attention-Based Personalized Encoder-Decoder Model for Local Citation Recommendation[J].COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE，2019：1-7.
[61] 姜艺，黄永，夏义堃，等.学术文本词汇功能识别：在关键词自动抽取中的应用[J].情报学报，2021，40（02）：152-162.
[62] KONDO T，NANBA H.Technical trend analysis by analyzing research papers' titles[C]//Proceedings of the Language and Technology Conference on Human Language Technology，Challenges for Computer Science and Linguistics.Heidelberg：Springer，2009：512-521.
[63] NANBA H，KONDO T，TAKEZAWA T.Automatic creation of a technical trend map from research papers and patents[C]//Proceedings of the 3rd International Workshop on Patent Information Retrieval. New York：ACM Press，2010.
[64] CHEN L，XU S，ZHU L，et al.A deep learning based method for extracting semantic information from patent documents[J].Scientometrics，2020，125（01）：289-312.
[65] DOUCET A，COUSTATY M.Enhancing table of contents extraction by system aggregation[C]//2017 14th IAPR international conference on document analysis and recognition（ICDAR）.IEEE，Kyoto：2017.
[66] SAFDER I，HASSAN S U，ALJOHANI N R.AI cognition in searching for relevant knowledge from scholarly big data，using a multi-layer perceptron and recurrent convolutional neural network model[C]//Companion of the web conference 2018.Lyon：2018.
[67] LI C，XING J，SUN A，et al.Effective document labeling with very few seed words：A topic model approach[C]//Proceedings of the 25th ACM international on conference on information and knowledge management， Indianapolis：2016.
[68] SIEGEL N，LOURIE N.Extracting scientific figures with distantly supervised neural networks[C]//Proceedings of the 18th ACM/IEEE on joint conference on digital libraries，New York：2018.
[69] AL-ZAIDY R A，GILES C L.Automatic knowledge base construction from scholarly documents[C]//Proceedings of the 2017 ACM symposium on document engineering，Beijing：2017.
[70] GARFIELD E.Cam citation indexing be automated[C]//Statistical association methods for mechanized documentation，Symposium proceedings.1964：189-192．
[71] BROOKS T A．Evidence of complex citer motivations[J].Journal of the American society for Information Science，1986，37（01）：34-36．
[72] TEUFEL S，CARLETTA J，MOENS M.An annotation scheme for discourse-level argumentation in research articles[C]//Proceedings of the Ninth Conference on European Chapter of the Association for Computational Linguistics，Bergen：1999.
[73] HUANG H，ZHU D，WANG X.Evaluating scientific impact of publications：combining citation polarity and purpose[EB/OL].Scientometrics，2021.
[74] DING Y，LIU X，GUO C，et al.The distribution of references across texts：some implications for citation analysis[J].Journal of Informetrics，2013，7（03）：583-592.
[75] WAN X，LIU F.Are all literature citations equally important?Automatic citation strength estimation and its applications[J].Journal of the Association for Information Science and Technology，2014，65（09）：1929-1938.
[76] EBRAHIMY S，OSAREH F.Design，validation，and reliability determination a citing conformity instrument at three levels：normative，informational，and identification[J].Scientometrics，2014，99（02）：581-597.
[77] 陆伟，孟睿，刘兴帮.面向引用关系的引文内容标注框架研究[J].中国图书馆学报，2014（06）：93-104.
[78] AN J，KIM N，KAN M Y，et al.Exploring characteristics of highly cited authors according to citation location and content[J].Journal of the Association for Information Science and Technology，2017，68（08）：1975-1988.
[79] ZHAO S X，QIAO L，QIAN X H D I.Man：An analytical framework for solving complex Issues[J]. Science，2021，374（Oct.1 App.TN.6563）：39-41.