Hits:

Indexed by:期刊论文

Date of Publication:2009-06-01

Journal:Journal of Information and Computational Science

Included Journals:EI、Scopus

Volume:6

Issue:3

Page Number:1355-1363

ISSN No.:15487741

Abstract:In order to maintain the high freshness of a search engine's local repository, it is crucial for a web crawler to detect changed web pages and new web pages as many as possible. This paper proposes a novel tag-based filtering method for identifying the index-pages that are valuable in new page discovery. Our method can filter out more index-pages and lots of useless hops can be avoided in new page discovery. The efficiency of obtaining new web pages increases significantly. Empirical studies on tens of thousands of web pages show that more than 80% of new web pages can be discovered using index-pages filtered by our method, which outperforms the traditional approach in cover level. 1548-7741/ Copyright ? 2009 Binary Information Press.