点击次数：

论文类型：期刊论文

发表时间：2009-06-01

发表刊物：Journal of Information and Computational Science

收录刊物：EI、Scopus

卷号：6

期号：3

页面范围：1355-1363

ISSN号：15487741

摘要：In order to maintain the high freshness of a search engine's local repository, it is crucial for a web crawler to detect changed web pages and new web pages as many as possible. This paper proposes a novel tag-based filtering method for identifying the index-pages that are valuable in new page discovery. Our method can filter out more index-pages and lots of useless hops can be avoided in new page discovery. The efficiency of obtaining new web pages increases significantly. Empirical studies on tens of thousands of web pages show that more than 80% of new web pages can be discovered using index-pages filtered by our method, which outperforms the traditional approach in cover level. 1548-7741/ Copyright ? 2009 Binary Information Press.