AI Hotel Ranking · Methodology
How we measure what AI recommends
The AI Hotel Ranking is Sigtrip’s ongoing measure of which hotels AI recommends to travelers in New York. We put the questions travelers actually ask to ChatGPT — with no hotel or brand names in them — through 4 traveler lenses, and repeat each one because the same question rarely gets the same answer twice. This page is the method behind the boards: what we ask, how we count it, what we leave out, and where it can be wrong. Every number here comes straight from the answers we publish.
The questions
We ask the questions travelers actually put to an AI — in their words, not ours, and never naming a hotel or brand. They span the ways people really search for a room.
Keeping brand names out of the questions is the whole point: we want what the engine offers on its own, not what it says when we lead it. Each broad citywide question is asked 8× and each more specific one 6×, so a single lucky or unlucky answer can’t define a hotel’s standing.
Four traveler lenses
The same question lands differently depending on who's asking. We run the whole panel through four traveler lenses, so a hotel AI recommends to families but not to business travelers reads as exactly that.
- General— no trip type mentioned — what a traveler sees by default
- Business— asked as someone on a business trip
- Family— asked as someone planning a family trip
- Leisure— asked as someone traveling for leisure
On the board, the Blended lens combines all four — the fullest picture of who ChatGPT names — and you can switch to any single lens to see how the picture shifts. Only the framing of the ask changes between lenses; the questions themselves stay word-for-word the same.
How a placement is measured
We record which hotels the engine names, and how it ranks them — then read those answers three ways, each stricter than the last.
A hotel’s placement ratefor a board is the share of that board’s eligible answers in which the engine names the hotel. Recommended rate is stricter — how often it actively recommends the hotel, not just mentions it in passing — and top-pick rateis stricter still: how often the hotel is the first or #1 choice. The “Rank by” menu on the board switches between them.
The overall board uses citywide questions (“best hotels in New York”); each borough, neighborhood, and budget board uses that segment’s questions (24 boards in all). A hotel appears on a board because the engine named it there — not because of any tag we assign.
We keep two signals separate and never merge them into one score: how often a hotel is named (the ranking) and what sources the engine cites (the sources board). One is reputation; the other is where that reputation comes from.
Sample size & confidence
We show percentages, not raw counts — but every percentage carries a sample size and a confidence range underneath.
Ask an AI the same question twice and you can get different answers, so a placement rate is an estimate, not a fixed number. We attach a 95% confidence rangeto each one — the band the true rate most likely sits in, given how many answers back it. A narrow band means we’re fairly sure; a wide band means read it as a rough signal.
The citywide board rests on 352 answers; the borough, neighborhood, and budget boards range from 24 to 96 (median 48). Boards with fewer than 12 eligible answers are withheld as too small to publish.
Wilson 95% score intervals on each placement rate (n = the board's eligible answers). Repeats cluster by prompt, so intervals are approximate.
One caution when reading down a board: a hotel named in just one or two answers sits inside that run-to-run noise. Treat the head of each board as the real signal, and low single-digit percentages as “occasionally mentioned,” not a firm rank.
Which hotels are in the running
The ranking can only name hotels it knows exist. Here's how that list is built — and why being on it says nothing about whether a hotel is a Sigtrip customer.
We assembled 673New York hotels from public directory data — no property’s permission needed, and being on the list doesn’t make a hotel a customer. Of those, ChatGPT named 368 at least once across our questions; the rest simply never came up in an answer a traveler would read.
A further 50hotels are held off the public board on purpose — a validation set kept aside to check our own accuracy against answers we didn’t use to build the ranking.
What we exclude
A ranking is only as trustworthy as what it leaves out.
- Blind-validation hold-out hotels (reserved to validate the method — never ranked)
- Segments with fewer than 12 eligible answers (sample too small)
Conflict of interest & corrections
Sigtrip sells AI-visibility tools to hotels, including some that appear in this market. To keep the ranking honest, being a Sigtrip customer never changes where a hotel lands — we don’t add, remove, or reorder any property, so a customer appears on a board only when the engine names it, exactly like everyone else. We record which hotels are customers for transparency, not to promote them.
These figures are a snapshot of what an AI engine said at a point in time, not a judgment of any hotel. If something looks wrong, tell us and we’ll review it against the underlying answers: hello@sigtrip.com.
The AI Hotel Ranking is Sigtrip’s own measurement, produced from our ChatGPT runs. Percentages are placement rates over eligible answers.