Article

Detecting phrase-level duplication on the world wide web

Authors:

SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 170 - 177

https://doi.org/10.1145/1076034.1076066

Published: 15 August 2005 Publication History

Get Access

Abstract

Two years ago, we conducted a study on the evolution of web pages over time. In the course of that study, we discovered a large number of machine-generated "spam" web pages emanating from a handful of web servers in Germany. These spam web pages were dynamically assembled by stitching together grammatically well-formed German sentences drawn from a large collection of sentences. This discovery motivated us to develop techniques for finding other instances of such "slice and dice" generation of web pages, where pages are automatically generated by stitching together phrases drawn from a limited corpus. We applied these techniques to two data sets, a set of 151 million web pages collected in December 2002 and a set of 96 million web pages collected in June 2004. We found a number of other instances of large-scale phrase-level replication within the two data sets. This paper describes the algorithms we used to discover this type of replication, and highlights the results of our data mining.

References

[1]

Amitay, E., Carmel, D., Darlow, A., Lempel, R., and Soffer, A. The Connectivity Sonar: Detecting Site Functionality by Structural Patterns. In 14th ACM Conference on Hypertext and Hypermedia (Aug. 2003), 38--47.Bharat, K., Chang, B., Henzinger.

Digital Library

Google Scholar

[2]

M., and Ruhl, M. Who Links to Whom: Mining Linkage between Web Sites. In 2001 IEEE International Conference on Data Mining (Nov. 2001), 51--58.

Digital Library

Google Scholar

[3]

Broder, A. Some applications of Rabin's fingerprinting method. In Capocelli, R., De Santis, A., and Vaccaro, U., editors, Sequences II: Methods in Communications, Security, and Computer Science, 143--152, Springer Verlag, 1993.

Google Scholar

[4]

Broder, A., Glassman, S., Manasse, M., and Zweig, G. Syntactic Clustering of the Web. In 6th International World Wide Web Conference (Apr. 1997), 393--404.

Digital Library

Google Scholar

[5]

Broder, A., Kumar, R., Maghoul, F., Raghavan, P., Rajagopalan, S., Stata, R., Tomkins, A., and Wiener, J. Graph Structure in the Web. In 9th International World Wide Web Conference (May 2000), 309--320.

Digital Library

Google Scholar

[6]

Davison, B. Recognizing Nepotistic Links on the Web. In AAAI-2000 Workshop on Artificial Intelligence for Web Search (July 2000).

Google Scholar

[7]

Fetterly, D., Manasse, M., Najork, M., and Wiener, J. A large-scale study of the evolution of web pages. In 12th International World Wide Web Conference (May 2003), 669--678.

Digital Library

Google Scholar

[8]

Fetterly, D., Manasse, M., and Najork, M. On the Evolution of Clusters of Near-Duplicate Web Pages. In 1st Latin American Web Congress (Nov. 2003), 37--45.

Digital Library

Google Scholar

[9]

Fetterly, D., Manasse, M., and Najork, M. Spam, Damn Spam, and Statistics: Using statistical analysis to locate spam web pages. In 7th International Workshop on the Web and Databases (June 2004), 37--45.

Digital Library

Google Scholar

[10]

M. Henzinger, R. Motwani, C. Silverstein. Challenges in Web Search Engines. SIGIR Forum 36(2), 2002.

Digital Library

Google Scholar

[11]

Rabin, M. Fingerprinting by random polynomials. Report TR-15-81, Center for Research in Computing Technology, Harvard University, 1981.

Google Scholar

Cited By

View all

Derbeko PDolev SGudes EUllman J(2022)Efficient and Privacy Preserving Approximation of Distributed Statistical QueriesIEEE Transactions on Big Data10.1109/TBDATA.2021.30525168:5(1399-1413)Online publication date: 1-Oct-2022
https://doi.org/10.1109/TBDATA.2021.3052516
Li Y(2022)Towards Forecasting Internet Financial Frauds based on Advertising2022 8th International Conference on Big Data and Information Analytics (BigDIA)10.1109/BigDIA56350.2022.9874049(5-11)Online publication date: 24-Aug-2022
https://doi.org/10.1109/BigDIA56350.2022.9874049
Shahzad ANawi NRehman MKhan A(2021)An Improved Framework for Content- and Link-Based Web-Spam DetectionComplexity10.1155/2021/66257392021Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1155/2021/6625739
Show More Cited By

Index Terms

Detecting phrase-level duplication on the world wide web

Recommendations

Detecting spam web pages through content analysis
WWW '06: Proceedings of the 15th international conference on World Wide Web

In this paper, we continue our investigations of "web spam": the injection of artificially-created pages into the web in order to influence the results from search engines, to drive traffic to certain pages for fun or profit. This paper considers some ...
Spam, damn spam, and statistics: using statistical analysis to locate spam web pages
WebDB '04: Proceedings of the 7th International Workshop on the Web and Databases: colocated with ACM SIGMOD/PODS 2004

The increasing importance of search engines to commercial web sites has given rise to a phenomenon we call "web spam", that is, web pages that exist only to mislead search engines into (mis)leading users to certain web sites. Web spam is a nuisance to ...
SAAD, a content based Web Spam Analyzer and Detector

Web Spam is one of the main difficulties that crawlers have to overcome and therefore one of the main problems of the WWW. There are several studies about characterising and detecting Web Spam pages. However, none of them deals with all the possible ...

Comments

Information & Contributors

Information

Published In

SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval

August 2005

708 pages

ISBN:1595930345

DOI:10.1145/1076034

General Chairs:
Ricardo Baeza-Yates
University of Chile, Chile
,
Nivio Ziviani
Federal University of Minas Gerais, Brazil
,
Program Chairs:
Gary Marchionini
University of North Carolina, USA
,
Alistair Moffat
University of Melbourne, Australia
,
John Tait
University of Sunderland, UK

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 August 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR05

Sponsor:

SIGIR

SIGIR05: The 28th ACM/SIGIR International Symposium on Information Retrieval 2005

August 15 - 19, 2005

Salvador, Brazil

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

74
Total Citations
View Citations
1,150
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)2

Reflects downloads up to 20 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Derbeko PDolev SGudes EUllman J(2022)Efficient and Privacy Preserving Approximation of Distributed Statistical QueriesIEEE Transactions on Big Data10.1109/TBDATA.2021.30525168:5(1399-1413)Online publication date: 1-Oct-2022
https://doi.org/10.1109/TBDATA.2021.3052516
Li Y(2022)Towards Forecasting Internet Financial Frauds based on Advertising2022 8th International Conference on Big Data and Information Analytics (BigDIA)10.1109/BigDIA56350.2022.9874049(5-11)Online publication date: 24-Aug-2022
https://doi.org/10.1109/BigDIA56350.2022.9874049
Shahzad ANawi NRehman MKhan A(2021)An Improved Framework for Content- and Link-Based Web-Spam DetectionComplexity10.1155/2021/66257392021Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1155/2021/6625739
Zhou XZafarani R(2020)A Survey of Fake NewsACM Computing Surveys10.1145/339504653:5(1-40)Online publication date: 28-Sep-2020
https://dl.acm.org/doi/10.1145/3395046
Kaur PGosain A(2020)GT2FS-SMOTE: An Intelligent Oversampling Approach Based Upon General Type-2 Fuzzy Sets to Detect Web SpamArabian Journal for Science and Engineering10.1007/s13369-020-04995-546:4(3033-3050)Online publication date: 15-Oct-2020
https://doi.org/10.1007/s13369-020-04995-5
Chatzakou DLeontiadis IBlackburn JCristofaro EStringhini GVakali AKourtellis N(2019)Detecting Cyberbullying and Cyberaggression in Social MediaACM Transactions on the Web10.1145/334348413:3(1-51)Online publication date: 14-Oct-2019
https://dl.acm.org/doi/10.1145/3343484
Malkani CAhuja LKumar Khatri S(2018)Statistical Approach for Combating Web Spamming Using Fisher Technique2018 International Conference on Inventive Research in Computing Applications (ICIRCA)10.1109/ICIRCA.2018.8597230(63-66)Online publication date: Jul-2018
https://doi.org/10.1109/ICIRCA.2018.8597230
Molla ABiadgie YSohn K(2017)Detecting Negative Deceptive Opinion from TweetsMobile and Wireless Technologies 201710.1007/978-981-10-5281-1_36(329-339)Online publication date: 17-Jun-2017
https://doi.org/10.1007/978-981-10-5281-1_36
Alsaleh MAlarifi A(2016)Analysis of Web Spam for Non-English Content: Toward More Effective Language-Based ClassifiersPLOS ONE10.1371/journal.pone.016438311:11(e0164383)Online publication date: 17-Nov-2016
https://doi.org/10.1371/journal.pone.0164383
Fdez-Glez JRuano-Ordás DLaza RMéndez JPavón RFdez-Riverola F(2016)WSF2Scientific Programming10.1155/2016/60913852016(1-1)Online publication date: 1-Jan-2016
https://dl.acm.org/doi/10.1155/2016/6091385
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Detecting spam web pages through content analysis

Spam, damn spam, and statistics: using statistical analysis to locate spam web pages

SAAD, a content based Web Spam Analyzer and Detector