A Semantic Understanding Oriented Evaluation of the Intelligent Q&A Service on Chinese Provincial Government Websites

Authors

Fang WANG, 1. Department of Information Resources Management, Business School of Nankai University, Tianjin 300071;2. Center for Network Society Governance, Nankai University, Tianjin 300072
Zhonghan WEI, Information Center of the Ministry of Science and Technology of China, Beijing 100862
Zhixuan LIAN, 1. Department of Information Resources Management, Business School of Nankai University, Tianjin 300071; 2. Center for Network Society Governance, Nankai University, Tianjin 300072
Jia KANG, 1. Department of Information Resources Management, Business School of Nankai University, Tianjin 300071; 2. Center for Network Society Governance, Nankai University, Tianjin 300072

Keywords

Government Website, intelligent customer service, semantic understanding ability, Q&A system, quality evaluation, dialogue robot

Abstract

[Purpose/significance]Intelligent question answering （Q&A） system has become an important facility for websites to provide consulting services. The complexity of government consultation issues poses higher requirements for the semantic understanding ability of intelligent Q&A systems on government websites.[Method/process]This study evaluates the Q&A systems of 30 Chinese provincial government websites from three aspects of problem solving quality,service interaction quality and basic construction quality by using the "Semantic Understanding based Evaluation Indicator System for Intelligent Q&A Service on Government Websites" developed by the Center for Network Society Governance of Nankai University and the supporting test sets.[Result/conclusion]The results show that Shanghai, Zhejiang and Beijing rank in the top three in terms of total scores. The Q&A systems on government websites have significant shortcomings in semantic understanding and scenario based services, with only 30% scoring above the passing line. Some Q&A systems have significant room for improvement in basic functionality and service interaction. Finally, this study proposes a few of suggestions to enhance the semantic understanding ability of government Q&A systems, such as expanding knowledge base, improving the accuracy of problem matching, and increasing humanistic care.

First Page

67

Last Page

84

Digital Object Identifier (DOI)

10.19809/j.cnki.kjqbyj.2023.03.006

Recommended Citation

WANG, Fang; WEI, Zhonghan; LIAN, Zhixuan; and KANG, Jia (2023) "A Semantic Understanding Oriented Evaluation of the Intelligent Q&A Service on Chinese Provincial Government Websites," Scientific Information Research: Vol. 5: Iss. 3, Article 6.
DOI: 10.19809/j.cnki.kjqbyj.2023.03.006
Available at: https://eng.kjqbyj.com/journal/vol5/iss3/6

Reference

[1] 中华人民共和国中央人民政府.国务院办公厅关于印发政府网站发展指引的通知:国办发[2017]47号[A/OL].(2017-06-08)[2021-12-31].www.gov.cn/zhengce/content/2017-06/08/content_5200760.htm.[2] 周斐,李锦芝.基于人工智能的广州市政府智能服务机器人云平台设计及应用[J].数字技术与应用,2018,36(12):120-122.[3] 央广网.商务部联手京东AI打造政务智能问答系统[EB/OL].(2019-07-15)[2022-02-22].https://baijiahao.baidu.com/s?id=1639129781659108953&wfr=spider&for=pc.[4] 亓廷廷.济南市疾病预防控制中心两款智能问答机器人上线[EB/OL].(2020-01-29)[2022-02-22].https://baijiahao.baidu.com/s?id=1657048191273956690&wfr=spider&for=pc.[5] 王友奎,张楠,赵雪娇.政务服务中的智能问答机器人:现状、机理和关键支撑[J].电子政务,2020(02):34-45.[6] Sohu city-Tianjin."2022 Data enabling government governance" evaluation index was released,"Chengdu-Chongqing Twin Cities Economic Circle" had the strong momentum of development[EB/OL].(2022-06-11)[2022-12-21].https://3g.k.sohu.com/t/n611346354?serialId=5e31e1e9b574c173a3daf5a3380db33e&showType=.[7] 肖路. 基于机器阅读理解的政务问答系统研究与实现[D].西安:西安电子科技大学,2021.[8] 姬美琳. 面向政务领域的知识库问答系统研究[D].武汉:中南民族大学,2021.[9] 李兆兆. 基于语义理解的智能问答系统关键技术研究[D].西安:西安邮电大学,2019.[10] 张恺. 基于DEA和多元智能的教育信息管理理论与模型研究[D].天津:天津大学,2010.[11] 郭艳华,周昌乐.自然语言理解研究综述[J].杭州电子工业学院学报,2000(01):58-65.[12] 钟义信. 自然语言理解的全信息方法论[J].北京邮电大学学报,2004,27(04):1-12.[13] NORASETA T,LOWPHANSIRIKUL L,Tuarob S.WabiQA:A Wikipedia-Based Thai Question-Answering System[J].Information Processing & Management,2020,58(01):209-216.[14] STORKS S,GAO Q,CHAI J Y.Commonsense reasoning for natural language understanding:A survey of benchmarks,resources,and approaches[J].arXiv preprint arXiv:1904.01172,2019.[15] 陈涛,冉龙亚,明承瀚.政务服务的人工智能应用研究[J].电子政务,2018(03):22-30.[16] ADAN B G,LEONARDO M G.Question answering systems in e-government:a proposal for the Colombian context[C]//Proceedings of the International Conference on Advances in Information Processing and Communication Technology-IPCT:2014.[17] CHAN H Y,TSAI M H.Question-answering dialogue system for emergency operations[J].International Journal of Disaster Risk Reduction,2019,41:1-13.[18] ANDROUTSOPOULOU A,KARACAPILIDIS N,LOUKIS E,et al.Transforming the communication between citizens and government through AI-guided chatbots[J].Government Information Quarterly,2019,36(02):358-367.[19] 谢炯坤. 面向人机互动的自然语言理解的研究[D].合肥:中国科学技术大学,2015.[20] TURING A M.Computing machinery and intelligence[J].Mind,1950,59(236):433-460.[21] 吴友政,赵军,段湘煜,等.问答式检索技术及评测研究综述[J].中文信息学报,2005(03):1-13.[22] JURCICEK F,KEIZER S,GASIC M,et al.Real user evaluation of spoken dialogue systems using Amazon Mechanical Turk[C]//Proceedings of Interspeech:2011.[23] 李菁,张海松,宋彦.面向非任务型对话系统的人工标注中文数据集[J].中文信息学报,2019,33(03):17-24.[24] ROQUE G,SOUZA R,NASCIMENTO J,et al.Content validation and usability of a chatbot of guidelines for wound dressing[J].International Journal of Medical Informatics,2021,151(Jul):1-6.[25] 李焱冬. 人工智能在电子政务建设中的实例应用初探[J].电子政务,2008(05):87-93.[26] AOKI N.An experimental study of public trust in AI chatbots in the public sector[J].Government Information Quarterly,2020,37(04):1-10.[27] 王友奎,张楠,赵雪娇.政务服务中的智能问答机器人:现状、机理和关键支撑[J].电子政务,2020(02):34-45.[28] 王芳,阴雨轩,刘汪洋,等.我国城市政府运用大数据提升治理效能评价研究[J].图书与情报,2020(02):81-93.[29] 程璇. 我国政府门户网站评价指标体系研究[D].武汉:华中师范大学,2007.[30] 徐林明,李美娟.动态综合评价中的数据预处理方法研究[J].中国管理科学,2020,28(01):162-169.[31] 新华网.“中国政务信息无障碍服务体系”获2017信息社会世界峰会项目大奖[EB/OL].(2017-06-14)[2022-02-22].http://it.people.com.cn/n1/2017/0614/c1009-29338648.html.