Hits:

Indexed by:期刊论文

Date of Publication:2018-01-01

Journal:CHINESE JOURNAL OF ELECTRONICS

Included Journals:SCIE

Volume:27

Issue:1,1

Page Number:1-8

ISSN No.:1022-4653

Key Words:Low-frequency unknown words; Information entropy; Independence of strings; Modified usage of Accessor variety (AV)

Abstract:As a prerequisite task in Natural language processing (NLP), Chinese word segmentation (CWS), is challenged by unknown words. Aiming to effectively detect Chinese unknown words, especially the low-frequency unknown words in unstructured microblog data, we modify the usage of Accessor variety (AV) to measure the context environments of core fragments and propose a novel variable, the Independence of strings, which is derived from the internal structure of segments. Our approach is unsupervised without using any manual materials. Due to the lack of manual resources of microblog-oriented unknown words extraction, we use sampling approach to assess the effectiveness of our method. Eicperimental results suggest our best system beats the baseline system as well as the state-of-the-art system by a significant improvement in F1-measure and the recall of low -frequency unknown words.