Sigtrip-Scanner, our website crawler

Sigtrip-Scanner reads the public pages of hotel websites. This page is for the people who run those sites: how to recognise it, how it behaves, and how to block it, slow it down or let it through a firewall.

What it reads: Public pages only, up to 60 per visit.
How often: At most about once a week per site, on a schedule.
How to control it: robots.txt, with the token Sigtrip-Scanner.

1. What it is and why it visits

Sigtrip helps hotels see how AI assistants, such as ChatGPT, describe and recommend them. To tell a hotel what those assistants can and cannot find on its website, Sigtrip-Scanner reads the hotel's public pages, much as a simple search crawler would.

It visits the websites of hotels that are tracked in Sigtrip. What it reads is used to show the hotel what its site tells AI assistants and what is missing. For hotels that use AI Actions, it is also the source for the pages Sigtrip builds for them; section 3 of our AI Terms covers that.

2. How to recognise it

Every request it makes sends this user agent:

Sigtrip-Scanner/1.0 (+https://sigtrip.com/crawler/)

Its token in robots.txt is Sigtrip-Scanner. It never sends a browser's user agent or disguises itself as anything else.

3. How it behaves

It follows robots.txt. It reads /robots.txt at the start of every visit. If the file has a group for Sigtrip-Scanner, it follows that group; otherwise it follows the group for all crawlers (*).
It reads public pages only. It never logs in, fills in or submits forms, or books anything.
It stays on your site. It starts from your sitemap and your homepage's links, and does not follow links to other domains.
It is gentle. At most two requests at a time per site. It honours Crawl-delay, and it backs off when your site answers 429 or 503.
It reads up to 60 pages per visit.
It visits at most about once a week per site, on a schedule. Now and then it may read a site once more after a change, for example to check that a new link works.
It does not run JavaScript. It reads the HTML your server sends, so content that only appears after scripts run is not seen.

4. How to block it or slow it down

Add a group to your site's robots.txt. Changes apply from its next visit.

Block it from the whole site

User-agent: Sigtrip-Scanner
Disallow: /

Block it from part of the site

User-agent: Sigtrip-Scanner
Disallow: /private/

Slow it down

Crawl-delay is the number of seconds to wait between requests.

User-agent: Sigtrip-Scanner
Crawl-delay: 10

Note: once your robots.txt has a Sigtrip-Scanner group, Sigtrip-Scanner follows that group instead of the * group. Repeat any Disallow lines from the * group that should still apply to it.

5. How to let it through a firewall

Some sites sit behind a firewall or bot protection that blocks Sigtrip-Scanner before it can read anything, including robots.txt. If you want Sigtrip to be able to read your site, allow the user agent Sigtrip-Scanner in your firewall or bot rules. Sigtrip-Scanner has no fixed IP addresses, so allow it by user agent, not by IP.

If you are a hotelier, you can forward this sentence to whoever runs your website or firewall:

Please allow requests with the user agent Sigtrip-Scanner through our firewall and bot protection; it is Sigtrip's crawler, it reads public pages only, and it is described at https://sigtrip.com/crawler/.

6. Other single requests with the same name

A few Sigtrip checks also identify as Sigtrip-Scanner. They are single requests, not a crawl, and they do not read robots.txt:

When a hotel is scanned in Sigtrip, its homepage is fetched once to identify the hotel.
To check that a web address still works, such as a page an AI assistant cited or a link on a page Sigtrip publishes for a hotel, Sigtrip sends one HEAD request, followed by one GET if needed. These can reach any website, not only hotel sites.

7. Contact

Questions about Sigtrip-Scanner, or a problem with how it reads your site? Email info@sigtrip.com and include your site's address.