Background Generally speaking. Chinese college graduates in the fifties and sixties took Russian as their second language, and those who graduated in the seventies had no second language to speak of. Now, in the years...Background Generally speaking. Chinese college graduates in the fifties and sixties took Russian as their second language, and those who graduated in the seventies had no second language to speak of. Now, in the years of our Open Door Policy, they find they have to learn some English and learn it quickly. They try to learn from radio and TV and many take English courses of 4 to 6 months, with varying degree of success. Their chief stumbling blocks展开更多
Automatic word-segmentation is widely used in the ambiguity cancellation when processing large-scale real text,but during the process of unknown word detection in Chinese word segmentation,many detected word candidate...Automatic word-segmentation is widely used in the ambiguity cancellation when processing large-scale real text,but during the process of unknown word detection in Chinese word segmentation,many detected word candidates are invalid.These false unknown word candidates deteriorate the overall segmentation accuracy,as it will affect the segmentation accuracy of known words.In this paper,we propose several methods for reducing the difficulties and improving the accuracy of the word-segmentation of written Chinese,such as full segmentation of a sentence,processing the duplicative word,idioms and statistical identification for unknown words.A simulation shows the feasibility of our proposed methods in improving the accuracy of word-segmentation of Chinese.展开更多
随着能源行业的快速发展和技术革新,大量的专业术语和表达方式不断更新,新词不断涌现。然而,传统的新词发现方法通常依赖于词典或规则,且难以高效率地处理和更新大量的专业术语,特别是在快速变化的能源领域。因此,结合能源领域文本数据...随着能源行业的快速发展和技术革新,大量的专业术语和表达方式不断更新,新词不断涌现。然而,传统的新词发现方法通常依赖于词典或规则,且难以高效率地处理和更新大量的专业术语,特别是在快速变化的能源领域。因此,结合能源领域文本数据特性,提出了一种融合N-Gram和多重注意力机制的能源领域新词发现方法(new word discovery method in the energy field combining N-Gram and multiple attention mechanism, ENFM)。该方法首先利用N-Gram模型对能源领域的文本数据进行初步处理,通过统计和分析词频来生成新词候选列表。随后,引入融合多重注意力机制的ERNIE-BiLSTM-CRF模型,以进一步提升新词发现的准确性和效率。与传统的新词发现技术相比,在新词的准确识别和整体效率上均有显著提升,将其于能源领域政策文本数据集,准确率、召回率和F1分别为95.71%、95.56%、95.63%。实验结果表明,该方法能够准确地在能源领域的大量文本数据中识别新词,有效识别出能源领域特有的词汇和表达方式,显著提高了中文分词任务中对能源领域专业术语的识别能力。展开更多
文摘Background Generally speaking. Chinese college graduates in the fifties and sixties took Russian as their second language, and those who graduated in the seventies had no second language to speak of. Now, in the years of our Open Door Policy, they find they have to learn some English and learn it quickly. They try to learn from radio and TV and many take English courses of 4 to 6 months, with varying degree of success. Their chief stumbling blocks
文摘Automatic word-segmentation is widely used in the ambiguity cancellation when processing large-scale real text,but during the process of unknown word detection in Chinese word segmentation,many detected word candidates are invalid.These false unknown word candidates deteriorate the overall segmentation accuracy,as it will affect the segmentation accuracy of known words.In this paper,we propose several methods for reducing the difficulties and improving the accuracy of the word-segmentation of written Chinese,such as full segmentation of a sentence,processing the duplicative word,idioms and statistical identification for unknown words.A simulation shows the feasibility of our proposed methods in improving the accuracy of word-segmentation of Chinese.
文摘随着能源行业的快速发展和技术革新,大量的专业术语和表达方式不断更新,新词不断涌现。然而,传统的新词发现方法通常依赖于词典或规则,且难以高效率地处理和更新大量的专业术语,特别是在快速变化的能源领域。因此,结合能源领域文本数据特性,提出了一种融合N-Gram和多重注意力机制的能源领域新词发现方法(new word discovery method in the energy field combining N-Gram and multiple attention mechanism, ENFM)。该方法首先利用N-Gram模型对能源领域的文本数据进行初步处理,通过统计和分析词频来生成新词候选列表。随后,引入融合多重注意力机制的ERNIE-BiLSTM-CRF模型,以进一步提升新词发现的准确性和效率。与传统的新词发现技术相比,在新词的准确识别和整体效率上均有显著提升,将其于能源领域政策文本数据集,准确率、召回率和F1分别为95.71%、95.56%、95.63%。实验结果表明,该方法能够准确地在能源领域的大量文本数据中识别新词,有效识别出能源领域特有的词汇和表达方式,显著提高了中文分词任务中对能源领域专业术语的识别能力。
