摘要
在传统的知识蒸馏中,若教师、学生模型的参数规模差距过大,则会出现学生模型无法学习较大教师模型的负面结果。为了获得在不同任务上均拥有较好表现的学生模型,深入研究了现有的模型蒸馏方法、不同教师模型的优缺点,提出了一种新型的来自Transformers的双向编码器表示(Bidrectional Enoceder Respresentations from Transformers,BERT)模型的多教师蒸馏方案,即使用BERT、鲁棒优化的BERT方法(Robustly optimized BERT approach,RoBERTa)、语言理解的广义自回归预训练模型(XLNET)等多个拥有BERT结构的教师模型对其进行蒸馏,同时修改了对教师模型中间层知识表征的蒸馏方案,加入了对Transformer层的蒸馏。该蒸馏方案在通用语言理解评估(General Language Understanding Evaluation,GLUE)中的多个数据集上的实验结果表明,最终蒸馏实验的结果较为理想,可以保留教师模型95.1%的准确率。
In traditional knowledge distillation,if there is a significant difference in the parameter scale between the teacher and student models,there will be a negative impact that the student model cannot learn from the larger teacher model.In order to achieve better performance of student models on different tasks in knowledge distillation for BERT,a new multi-teacher distillation scheme for BERT model is proposed by making intensive research on existing model distillation methods and analysis on the advantages and disadvantages of different teacher models,that is,BERT,RoBERTA and XLNET are used to distill the teacher model with BERT structure.Meanwhile,the distillation scheme for the knowledge representation of the middle layer of the teacher model is modified,and the distillation on Transformer layer is added.Finally,experiments on several datasets in GLUE show that the final distillation results are ideal and can retain 95.1%accuracy of the teacher model.
作者
石佳来
郭卫斌
SHI Jialai;GUO Weibin(School of Information Science and Engineering,East China University of Science and Technology,Shanghai 200237,China)
出处
《华东理工大学学报（自然科学版）》
CAS
CSCD
北大核心
2024年第2期293-300,共8页
Journal of East China University of Science and Technology
基金
几何信息融合的分类学习研究(62076094)。
作者简介
石佳来(1998-),男,浙江台州人,硕士生,主要研究方向为NLU和知识蒸馏。E-mail:1244047234@qq.com;通信联系人:郭卫斌,E-mail:gweibin@ecust.edu.cn。
