New York

London

Shanghai

Technology

Case Studies

Resources

About Us

Contact

Home

Personalized Retargeting

Branding

For industries

For agencies

Gaming

In-app

For the cookieless world

Offer

Careers

Blog

Partners

Learn more

Modern Slavery Act

Our Technologies

Investors

Tax Strategy 2021

Tax Strategy 2022

Diversity & Inclusion Policy

Stay on top of the latest in digital ads. Sign up to our newsletter.

Newsletter

Facebook

LinkedIn

Contact us

Opt Out

Privacy Center

Our technologies

**What is RTB House Crawler?**
- RTB House Crawler is a software powered by Deep Learning that reads web pages and analyzes them to place the most relevant ads.
Why does RTB House visit your site?
To support contextual advertising, we need to analyze the public web content by crawling web pages. This lets us categorize pages and adjust advertisements.

**What kind of data does RTB House collect?**
- We collect meta tags, text sections (some or all, depending on usability) from HTML documents, and language-related headers from responses returned by the server.

**When will the site be indexed?**
- We try to access a given page as soon as it’s published - the time may vary depending on how fast we find out that it exists. Later on, the data might be refreshed, but it’s not a rule - it mostly depends on the site's popularity.

**How can I authorize/exclude the crawler?**
- We follow the rules defined in robots.txt for default user agents, meaning if a given page is not disallowed, it will be indexed.

If you want to block robots on all sites, add the following:
User-agent: *
Disallow: /

If you want to block a particular path with exceptions:  
User-agent: *
Allow: /path/allowed
Disallow: /path

You can also block everything with exceptions:
User-agent: *
Allow: /path/exception2
Allow: /path/exception1
Disallow: /

More information about creating a robots.txt file https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt

**How long does it take to update the robots.txt published by a domain?**
- The crawler checks the robots.txt file once a week for each domain, meaning if the robots.txt is updated, we will sync its status within a week or less.

**More information**
- If you need to know more about the crawler, please email us at crawler@rtbhouse.com.

RTB House Crawler

data created by MaxMind, available at https://www.maxmind.com

GeoLite2

Our products include the following technologies

**Technologies that we use in** our products 

Create truly personalized marketing campaigns powered by Deep Learning technology that cut through the online noise. 100% ready for the cookieless world.

Technology Powered by Deep Learning

 | RTB House

TECHNOLOGIES

Our technologies