摘要
【目的】基于大语言模型实现科技论文语义评价指标的自动量化,支撑科技文献语义评价研究。【方法】从科技论文中抽取与评价指标相关的语步句,设计标准、简化、详细三种不同详尽程度的提示词,横向对比提示词效果,利用少量标注样本微调大语言模型,得到科技论文语义评价指标量化模型。【结果】基于论文文本的语义内容,从“实验条件的苛刻程度”维度量化论文评价指标并开展分析。实验结果表明,基于详细提示词微调的模型取得最佳效果。在训练样本数为100时,Micro-Acc和Fuzzy-Acc分别达到0.72和0.87。【局限】仅选取计算机领域科技论文进行实验,未考察所提方法在不同学科上的效果差异。【结论】基于提示微调大语言模型的指标量化方法具有较高的精确度和可靠性,提高提示词的详尽程度可明显提升量化效果。此外,增加微调阶段的样本数虽可提升总体效果,但不同得分段的提升程度存在差异。
[Objective]This research aims to achieve the automatic quantification of semantic evaluation metrics for scientific papers using large language models,supporting the study of semantic evaluation of scientific literature.[Methods]First,we extracted rhetorical moves related to evaluation metrics from scientific papers with three levels of prompt detail—standard,simplified,and detailed.Then,we compared the effectiveness of these prompts.Third,we fine-tuned a large language model with a small number of annotated samples to develop a model for quantifying semantic evaluation metrics.[Results]Based on the semantic content of the papers,we analyzed the“difficulty of experimental conditions”dimension.The proposed model achieved the best performance.With a training sample size of 100,its Micro-Acc and Fuzzy-Acc reached 0.72 and 0.87,respectively.[Limitations]The experiment only included scientific papers in computer science,and we need to explore the effectiveness of the proposed method across different disciplines was not explored.[Conclusions]The proposed method demonstrates high accuracy and reliability in evaluating scientific papers.Increasing the level of detail in prompts significantly improves the quantification effect.While increasing the number of samples during the fine-tuning stage improves overall performance,the degree of improvement varies across different scoring ranges.
作者
李西雨
钱力
张智雄
Li Xiyu;Qian Li;Zhang Zhixiong(National Science Library,Chinese Academy of Sciences,Beijing 100190,China;Department of Information Resources Management,School of Economics and Management,University of Chinese Academy of Sciences,Beijing 100190,China;Key Laboratory of New Publishing and Knowledge Services for Scholarly Journals,Beijing 100190,China)
出处
《数据分析与知识发现》
EI
CSSCI
CSCD
北大核心
2024年第8期200-212,共13页
Data Analysis and Knowledge Discovery
基金
国家社会科学基金重大项目(项目编号:21&ZD329)的研究成果之一
关键词
科技论文语义评价
指标量化
大语言模型
提示微调
Semantic Evaluation of Scientific Papers
Scoring for Evaluation Metrics
Large Language Models
Prompt Tuning
作者简介
通讯作者:钱力,ORCID:0000-0002-0931-2882,E-mail:qianl@mail.las.ac.cn。
