摘要
预训练语言模型(pre-trained languages model,PTLM)在自然语言处理(natural language processing,NLP)领域取得了令人瞩目的成功,并由此引发了下游任务从监督学习到预训练-微调范式的转变。在此之后,一系列预训练模型的创新研究涌现出来。本文系统性、全面的回顾了自然语言处理的代表性工作和最新进展,并按照类别系统性的介绍了自然语言处理领域的预训练模型。首先我们简要介绍了预训练模型,以及不同的模型特点和框架。之后,我们介绍并分析了预训练模型的影响和挑战以及下游任务中的应用。最后,我们简要总结并阐述了预训练模型未来的研究方向。
Pre-trained language models have achieved striking success in natural language processing(NLP),leading to a paradigm shift from supervised learning to pre-training followed by fine-tuning.The NLP community has witnessed a surge of research interest in improving pre-trained models.This article presents a comprehensive review of representative work and recent progress in the NLP field and introduces the taxonomy of pre-trained models.We first give a brief introduction of pre-trained models,followed by characteristic methods and frameworks.We then introduce and analyze the impact and challenges of pre-trained models and their downstream applications.Finally,we briefly conclude and address future research directions in this field.
作者
王海峰
李纪为
Hua Wu
Eduard Hovy
Yu Sun
Haifeng Wang;Jiwei Li;Hua Wu;Eduard Hovy;Yu Sun(Baidu Inc.,Beijing 100193,China;College of Computer Science and Technology,Zhejiang University,Hangzhou 310058,China;Language Technologies Institute,Carnegie Mellon University,Pittsburgh,PA 15213,USA)
出处
《Engineering》
SCIE
EI
CAS
CSCD
2023年第6期51-65,M0004,共16页
工程（英文）
作者简介
Corresponding author:Haifeng Wang,E-mail address:wanghaifeng@baidu.com。
