论文类型：期刊论文
发表刊物：Journal of Information and Computational Science
收录刊物：Scopus、EI
卷号：6
期号：3
页面范围：1355-1363
ISSN号：15487741
摘要：In order to maintain the high freshness of a search engine's local repository, it is crucial for a web crawler to detect changed web pages and new web pages as many as possible. This paper proposes a novel tag-based filtering method for identifying the index-pages that are valuable in new page discovery. Our method can filter out more index-pages and lots of useless hops can be avoided in new page discovery. The efficiency of obtaining new web pages increases significantly. Empirical studies on tens of thousands of web pages show that more than 80% of new web pages can be discovered using index-pages filtered by our method, which outperforms the traditional approach in cover level. 1548-7741/ Copyright ? 2009 Binary Information Press.