A Three Layer Distributed Architecture for Large-Scale Duplicated Web Page Detection
BEN Xinglong
JIA Dawen
YUAN Lin
Abstract：Duplicated web page detection is a necessary step .Currently ,researchers focus on the accuracy and time complexity of duplicated web pages detection algorithms based on the similarity of web page content .A three layer distribu‐ted architecture for large‐scale duplicated web page detection is proposed ,which can detect duplicated web pages efficiently u‐sing the combination of local memory caches ,distributed caches and distributed index .This architecture is especially applica‐ble for those applications which need crawl the web pages repeatedly .The experimental results indicate our proposed archi‐tecture can satisfy the requirement of large scale duplicated web page detection in distribute web crawler application .Moreo‐ver ,this architecture is scalable .
Keywords：duplicated web page detectionweb crawlerdistribute architecture
Publication Date：2015-01-01
Online Publishing Date：2025-08-15(First online date of this platform, not the publication date of the document)
Pages：5( 1751-1755 )