AI Performance Report — ROI & Automation Metrics

What it does

The AI Performance Report shows you exactly how much time and money Vanio AI saves your operation. See the breakdown of AI-handled versus human-handled messages, estimated cost savings, action counts by type, per-listing automation rates, and response time comparisons — all with configurable date ranges.

Getting started

Accessing the report

1. Navigate to Settings > AI Assistant from the main sidebar
2. Click the Performance tab
3. The report loads with the current month selected
4. Use the date picker to analyze different periods — the URL updates so you can bookmark specific views

Reading the headline metrics

The top section displays four KPI cards:

• Messages Sent — Total messages your AI actually sent to guests (excludes shadow drafts and internal notes)
• Shadow Drafts — Messages AI prepared but didn't send automatically (requires review in shadow mode)
• Escalated — Messages AI flagged for human attention due to complexity or uncertainty
• Action Needed — Messages waiting for your review or approval

Understanding live vs. shadow mode metrics

The report clearly separates messages your AI actually sent to guests from drafts it prepared for review. This distinction helps you understand your AI's true impact on guest communication versus preparatory work that still requires human oversight.

How it works

AI vs. human messaging timeline

The main chart shows message volume over time with two series:

• Blue line — Messages handled entirely by AI (auto-responded, auto-classified, auto-actioned)
• Orange line — Messages that required human intervention (escalated, manually responded)

Hover over any point to see the exact counts for that day. A widening gap between the lines means your AI is taking on more of the workload over time.

Response time comparison

A highlighted section shows side-by-side response time metrics:

• Vanio AI — Median response time for AI-generated messages (now under 15 seconds with performance optimizations)
• Your team — Median response time for human-handled messages
• Speed advantage — How much faster AI responds compared to human staff (typically 4x faster than previous benchmarks)

This comparison uses median times rather than averages to give you a more realistic picture of typical performance, filtering out extreme outliers that can skew average calculations. Recent infrastructure improvements have significantly reduced response times, with most guest inquiries now answered in seconds rather than minutes.

Report loading speed

The AI Performance Report now loads significantly faster. Whether you're pulling up a 7-day snapshot or a full historical view, results appear much more quickly — and running your report no longer slows things down for other members of your team who happen to be using Vanio at the same time.

Cost savings calculation

The report displays estimated dollar savings based on:

• Number of messages AI handled completely
• Your configured average cost per human response
• Time saved by automated responses versus manual handling
• Improved reliability from eliminated timeout errors

The calculation now accounts for messages actually sent to guests rather than all AI activity, providing more accurate ROI projections. Performance improvements have increased the volume of messages AI can handle automatically, boosting overall savings.

Action breakdown

Below the timeline, a bar chart categorizes every action the AI performed:

• Auto-reply — Guest messages answered without human involvement
• Task creation — Cleaning or maintenance tasks created automatically
• Payment processing — Charges, refunds, or payment links generated
• Reservation updates — Date changes, cancellations, or modifications handled
• Escalation — Cases sent to a human team member for review

This shows you not just that AI is working, but what it is doing. Higher action volumes indicate improved AI performance and reliability.

Per-listing automation rate

A table lists every property with its individual AI automation rate:

• Property name — The listing
• Total messages — Number of guest messages received
• AI-handled — Messages fully handled by AI
• Team-handled — Messages requiring human intervention
• Automation rate — Percentage of messages AI handled independently

Sort by automation rate to find properties where AI struggles — these may need updated knowledge base articles or procedure adjustments. Recent performance improvements should show increased automation rates across all properties.

Daily AI digest email consistency

Your daily AI digest emails now pull from the same data source as the dashboard report. This ensures the metrics in your morning email always match what you see in the performance report — no more confusion about different numbers from different sources.

Key features

• Faster report loading — The report loads significantly faster than before, so you spend less time waiting and more time acting on your data
• No slowdowns for your team — Running a report no longer affects performance for other property managers using Vanio at the same time
• 4x faster response times — Recent optimizations deliver guest responses in under 15 seconds instead of minutes
• Eliminated timeout errors — Improved reliability means more messages handled successfully without system failures
• Accurate message counting — Distinguishes between messages actually sent to guests versus internal drafts and shadow mode preparation
• Response time comparison — Side-by-side AI versus human response speed using median calculations for more accurate comparisons
• ROI quantification — Dollar estimates of AI savings based on your actual message volume and labor costs, now higher due to improved performance
• AI vs. human timeline — Visualize automation growth over time
• Action breakdown — See exactly what types of actions AI performs
• Per-listing rates — Identify properties that need AI tuning with detailed message handling breakdowns
• Configurable date ranges — Analyze any period with URL-persistent date selection
• Consistent metrics — Dashboard and email reports now use identical data sources for reliable tracking
• Export — Download the full report data for stakeholder presentations or owner reports

Tips & best practices

Review monthly and share with stakeholders. The AI Performance Report makes a compelling case for the value of automation. Include it in monthly reports to property owners, emphasizing both the cost savings and dramatically faster response times.

Highlight the speed improvement in client presentations. The side-by-side comparison now shows AI responding in seconds versus humans taking minutes or hours — a 4x improvement over previous performance that's particularly compelling for prospective clients.

Run reports without worrying about timing. You and your team can now pull reports simultaneously without affecting each other's experience. There's no need to coordinate who runs a report when — go ahead and check your numbers whenever it suits you.

Monitor shadow draft volumes. High shadow draft counts may indicate your AI needs more training or confidence adjustments to handle more conversations independently. Performance improvements should reduce shadow drafts over time.

Investigate low-automation properties. If a property has a significantly lower AI automation rate, check its knowledge base and procedures. Missing information is usually the cause, but performance improvements should boost rates across all properties.

Track cost savings trends. Configure your hourly labor cost in Settings > AI Assistant > Economics to get accurate savings projections. Recent performance gains mean higher savings potential — useful for justifying Vanio AI to new clients or expanding your portfolio.

Trust your daily digest emails. The metrics in your morning AI performance emails now match the dashboard exactly, so you can confidently share those numbers with clients or use them in reports.

Compare before/after performance periods. Use date range filtering to compare metrics before and after recent system improvements — you should see measurably faster response times and higher automation rates.

Common questions

Why does the report load so much faster now? Recent improvements to how Vanio retrieves your message data have made the AI Performance Report significantly faster to load. You'll notice the difference especially during busy periods when multiple people on your team are using the platform at the same time.

Why are my response times suddenly much faster? Recent infrastructure optimizations significantly improved AI performance, reducing typical response times from minutes to seconds. This 4x improvement should be reflected in your metrics.

Why don't my email and dashboard numbers match? They do now. Both the daily AI digest email and the dashboard report pull from the same data source, ensuring consistent metrics across all surfaces.

What's the difference between "sent" and "drafted" messages? Sent messages went directly to guests automatically. Drafted messages were prepared by AI but held for your review, typically when running in shadow mode or when AI isn't confident enough to send automatically.

How are response times calculated? The report uses median response times rather than averages because they better represent typical performance and aren't skewed by extreme outliers like messages sent days later.

Why did my cost savings number increase? Performance improvements mean your AI handles more messages automatically and responds much faster, increasing the time and cost savings compared to manual handling.