论文类型：期刊论文
发表刊物：CHINESE JOURNAL OF ELECTRONICS
收录刊物：SCIE
卷号：27
期号：1,1
页面范围：1-8
ISSN号：1022-4653
关键字：Low-frequency unknown words; Information entropy; Independence of strings; Modified usage of Accessor variety (AV)
摘要：As a prerequisite task in Natural language processing (NLP), Chinese word segmentation (CWS), is challenged by unknown words. Aiming to effectively detect Chinese unknown words, especially the low-frequency unknown words in unstructured microblog data, we modify the usage of Accessor variety (AV) to measure the context environments of core fragments and propose a novel variable, the Independence of strings, which is derived from the internal structure of segments. Our approach is unsupervised without using any manual materials. Due to the lack of manual resources of microblog-oriented unknown words extraction, we use sampling approach to assess the effectiveness of our method. Eicperimental results suggest our best system beats the baseline system as well as the state-of-the-art system by a significant improvement in F1-measure and the recall of low -frequency unknown words.