AI Performance Report — ROI & Automation Metrics

What it does

The AI Performance Report shows you exactly how much time and money Vanio AI saves your operation. See the breakdown of AI-handled versus human-handled messages, estimated cost savings, action counts by type, per-listing automation rates, and response time comparisons — all with configurable date ranges.

Getting started

Accessing the report

1. Navigate to Settings > AI Assistant from the main sidebar
2. Click the Performance tab
3. The report loads with the current month selected
4. Use the date picker to analyze different periods — the URL updates so you can bookmark specific views

Reading the headline metrics

The top section displays four KPI cards:

• Messages Sent — Total messages your AI actually sent to guests (excludes shadow drafts and internal notes)
• Shadow Drafts — Messages AI prepared but didn't send automatically (requires review in shadow mode)
• Escalated — Messages AI flagged for human attention due to complexity or uncertainty
• Action Needed — Messages waiting for your review or approval

Understanding live vs. shadow mode metrics

The report clearly separates messages your AI actually sent to guests from drafts it prepared for review. This distinction helps you understand your AI's true impact on guest communication versus preparatory work that still requires human oversight.

How it works

AI vs. human messaging timeline

The main chart shows message volume over time with two series:

• Blue line — Messages handled entirely by AI (auto-responded, auto-classified, auto-actioned)
• Orange line — Messages that required human intervention (escalated, manually responded)

Hover over any point to see the exact counts for that day. A widening gap between the lines means your AI is taking on more of the workload over time.

Response time comparison

A highlighted section shows side-by-side response time metrics:

• Vanio AI — Median response time for AI-generated messages (now under 15 seconds with performance optimizations)
• Your team — Median response time for human-handled messages
• Speed advantage — How much faster AI responds compared to human staff (typically 4x faster than previous benchmarks)

This comparison uses median times rather than averages to give you a more realistic picture of typical performance, filtering out extreme outliers that can skew average calculations. Recent infrastructure improvements have significantly reduced response times, with most guest inquiries now answered in seconds rather than minutes.

Cost savings calculation

The report displays estimated dollar savings based on:

• Number of messages AI handled completely
• Your configured average cost per human response
• Time saved by automated responses versus manual handling
• Improved reliability from eliminated timeout errors

The calculation now accounts for messages actually sent to guests rather than all AI activity, providing more accurate ROI projections. Performance improvements have increased the volume of messages AI can handle automatically, boosting overall savings.

Action breakdown

Below the timeline, a bar chart categorizes every action the AI performed:

• Auto-reply — Guest messages answered without human involvement
• Task creation — Cleaning or maintenance tasks created automatically
• Payment processing — Charges, refunds, or payment links generated
• Reservation updates — Date changes, cancellations, or modifications handled
• Escalation — Cases sent to a human team member for review

This shows you not just that AI is working, but what it is doing. Higher action volumes indicate improved AI performance and reliability.

Per-listing automation rate

A table lists every property with its individual AI automation rate:

• Property name — The listing
• Total messages — Number of guest messages received
• AI-handled — Messages fully handled by AI
• Team-handled — Messages requiring human intervention
• Automation rate — Percentage of messages AI handled independently

Sort by automation rate to find properties where AI struggles — these may need updated knowledge base articles or procedure adjustments. Recent performance improvements should show increased automation rates across all properties.

Daily AI digest email consistency

Your daily AI digest emails now pull from the same data source as the dashboard report. This ensures the metrics in your morning email always match what you see in the performance report — no more confusion about different numbers from different sources.

Key features

• 4x faster response times — Recent optimizations deliver guest responses in under 15 seconds instead of minutes
• Eliminated timeout errors — Improved reliability means more messages handled successfully without system failures
• Accurate message counting — Distinguishes between messages actually sent to guests versus internal drafts and shadow mode preparation
• Response time comparison — Side-by-side AI versus human response speed using median calculations for more accurate comparisons
• ROI quantification — Dollar estimates of AI savings based on your actual message volume and labor costs, now higher due to improved performance
• AI vs. human timeline — Visualize automation growth over time
• Action breakdown — See exactly what types of actions AI performs
• Per-listing rates — Identify properties that need AI tuning with detailed message handling breakdowns
• Configurable date ranges — Analyze any period with URL-persistent date selection
• Consistent metrics — Dashboard and email reports now use identical data sources for reliable tracking
• Export — Download the full report data for stakeholder presentations or owner reports

Tips & best practices

Review monthly and share with stakeholders. The AI Performance Report makes a compelling case for the value of automation. Include it in monthly reports to property owners, emphasizing both the cost savings and dramatically faster response times.

Highlight the speed improvement in client presentations. The side-by-side comparison now shows AI responding in seconds versus humans taking minutes or hours — a 4x improvement over previous performance that's particularly compelling for prospective clients.

Monitor shadow draft volumes. High shadow draft counts may indicate your AI needs more training or confidence adjustments to handle more conversations independently. Performance improvements should reduce shadow drafts over time.

Investigate low-automation properties. If a property has a significantly lower AI automation rate, check its knowledge base and procedures. Missing information is usually the cause, but performance improvements should boost rates across all properties.

Track cost savings trends. Configure your hourly labor cost in Settings > AI Assistant > Economics to get accurate savings projections. Recent performance gains mean higher savings potential — useful for justifying Vanio AI to new clients or expanding your portfolio.

Trust your daily digest emails. The metrics in your morning AI performance emails now match the dashboard exactly, so you can confidently share those numbers with clients or use them in reports.

Compare before/after performance periods. Use date range filtering to compare metrics before and after recent system improvements — you should see measurably faster response times and higher automation rates.

Common questions

Why are my response times suddenly much faster? Recent infrastructure optimizations significantly improved AI performance, reducing typical response times from minutes to seconds. This 4x improvement should be reflected in your metrics.

Why don't my email and dashboard numbers match? They do now. Both the daily AI digest email and the dashboard report pull from the same data source, ensuring consistent metrics across all surfaces.

What's the difference between "sent" and "drafted" messages? Sent messages went directly to guests automatically. Drafted messages were prepared by AI but held for your review, typically when running in shadow mode or when AI isn't confident enough to send automatically.

How are response times calculated? The report uses median response times rather than averages because they better represent typical performance and aren't skewed by extreme outliers like messages sent days later.

Why did my cost savings number increase? Performance improvements mean your AI handles more messages automatically and responds much faster, increasing the time and cost savings compared to manual handling.

STRICTLY FORBIDDEN — never mention:

• LLM/AI model names (Claude, GPT, OpenAI, Anthropic, Sonnet, etc.)
• API endpoints, webhooks, database tables, schemas, migrations
• Internal service/handler/worker/queue names
• Code architecture, file paths, environment variables
• Stripe Connect internals (say "payment processing")
• Any implementation detail a user would never see in the UI

REQUIRED:

• Write for property managers, not developers
• Step-by-step instructions with UI references ("Go to Settings → ...", "Click ...")
• What happens automatically (explain WHAT, never HOW technically)
• [Screenshot: description] placeholders where visuals would help
• Voice: "you" addressing the user. Warm, clear, confident.

Structure

1. What it does — 2-3 sentence overview
2. Getting started — step-by-step with UI references
3. How it works — what happens automatically, what the user sees
4. Key features — bullet list with brief explanations
5. Tips & best practices
6. Common questions — 2-3 FAQs