Most AI guest messaging tools look alike in a demo. The questions below are the ones that separate them once real guests start writing in. They work whether you're on Guesty, Hostaway or another PMS, and whether you're comparing two vendors or deciding if your PMS's built-in AI is enough.
What it actually handles
1. Does it stop at the reply?
Ask for a walk-through of a message that needs an action. A guest reporting a broken heater is a good one. Does the tool answer and leave the rest to you, or does it create the maintenance task, brief the right person and update the guest when it's fixed? That single example tells you whether you're looking at an auto-reply or an autopilot.
2. Can it work with your third parties?
Cleaners, handymen, co-hosts, owners. Find out who the system can contact and how. If your cleaning team works through WhatsApp or a separate app, ask whether the autopilot can reach them there, and whether it's two-way: can it ask a housekeeper whether an early check-in is possible, wait for the answer and act on it? Ask too whether it can re-plan the day when someone calls in sick. We cover this in AI staff scheduling for short-term rentals.
3. Does it understand every guest?
Guests write in dozens of languages, in slang, in half-sentences typed on a phone. A strong system understands and replies in any language, including regional dialects, and should cope with a message that asks three things at once.
How much control you keep
4. Can you set its policies and tone?
You should be able to decide how strict it is on check-in times, what it says about parties or pets, when it can offer a discount, and how formal it sounds. Ask how you'd change one of those next week, and how long the change takes to apply.
On tone, ask to read whole conversations, not single replies. A reply can be correct and still cost you the guest. Service is getting them the wifi code, hospitality is how it felt to ask. The AI should sound like your team, learned from how your team already writes, not like a polite template.
5. What happens when it isn't sure?
This is the most important question on the list. One wrong promise in a hundred replies costs more than the other 99 save, so escalating correctly matters more than replying. A good autopilot recognises what it shouldn't answer and passes it to your team quickly, with a summary. Ask to see examples of conversations it handed over, and ask what it does with an angry guest or a refund request.
6. Can your team see and override everything?
Every action the AI takes should be visible in your PMS, and anyone on your team should be able to step into a conversation at any point. If the vendor runs its own separate inbox, ask how it keeps that in sync.
How to judge the numbers
7. How do they count their automation rate?
Ask exactly what's counted. Messages your team reviewed and approved shouldn't be, because a person did the work. Scheduled messages your PMS already sent shouldn't be either. Ask whether the figure is averaged across all customers or taken from the best one.
8. Will they show you results on your own conversations?
A demo on invented guests proves very little. Ask the vendor to run on a few of your properties, or on your past conversations, and show you what it would have handled. The best set-up is a shadow week, where the AI prepares every reply and escalation with its reasoning but doesn't send, followed by two weeks live. For larger operators we've written a playbook for trialling on a slice of your portfolio.
Set-up and commitment
9. Who does the set-up?
Some tools hand you a rule builder and a help centre. Others read your listings, saved replies and past conversations, build the playbook for you and ask you to approve it. The second kind gets you live in days, and it tends to sound more like your team because it learnt from your team.
10. Does it get better on its own?
Your properties change, and so do the questions guests ask. Ask how the system learns from your team's corrections, and whether that happens automatically or needs you to update it. On a large portfolio this matters more than almost anything else, because keeping hundreds of properties current by hand isn't realistic. We've written more on what self-learning should mean and how to test it.
Putting it together
One more thing to weigh is depth against breadth. A vendor that integrates with thirty PMSs can be a good fit if you might switch platforms. One built deeply on two can usually write more back to your PMS reliably, and ship new AI capability to you within weeks rather than quarters.
There's no single right answer. A small portfolio with simple properties may be well served by the auto-reply in its PMS. A larger operator, or anyone whose team spends the day on tasks and coordination, will usually get more from a full autopilot. The ten questions above should make the difference obvious within a couple of calls.
For PMS-specific detail, see our guides for Guesty and Hostaway.