Model Quality Dashboard

What it does

The Model Quality dashboard gives you complete transparency into how accurate Vanio's pricing recommendations are for your properties. Every night, Vanio automatically compares our recommended prices against your actual bookings, so you can see exactly how well our pricing engine is performing and whether newer versions are improving your revenue. You can now evaluate pricing accuracy across your entire booking history or focus on recent performance.

Getting started

Step 1: Access the dashboard

1. Log into your Vanio account
2. Go to Settings → Pricing Intelligence
3. Click the Model Quality tab

[Screenshot: Model Quality tab in the main navigation showing accuracy metrics]

Step 2: Choose your evaluation mode You'll see two options for analyzing your pricing performance:

• Recent Performance (default): Analyzes the past 60 days using point-in-time recommendations
• Historical Forecast: Evaluates how your current pricing model would have performed across your entire booking history

Step 3: Review your pricing accuracy The dashboard shows your pricing performance across all your markets, with key metrics displayed at the top:

• Pricing Accuracy: How close our recommendations were to optimal market rates
• Revenue Impact: The difference between recommended prices and what guests actually paid
• Model Performance: Error rates and confidence levels for each market

[Screenshot: Main dashboard view with accuracy metrics and market breakdown]

How it works

Vanio automatically runs quality checks every night at 4 AM UTC. Here's what happens behind the scenes:

Automatic analysis: The system reviews your booking data and compares it against pricing recommendations. You can choose between two evaluation approaches:

Recent Performance mode: Reviews your actual bookings from the past 60 days and compares them against the pricing recommendations that were active when guests made their reservations. This shows you exactly how recommendations performed in real-time conditions.

Historical Forecast mode: Uses your current pricing model to evaluate how it would have performed across your entire booking history (up to 4+ years). This gives you a comprehensive view of your model's effectiveness and helps you understand long-term pricing patterns, especially useful when you have extensive booking history but limited recommendation data.

Market-specific insights: Results are broken down by your different markets (like "Whistler," "Park City," or "Banff") so you can see which areas have the most accurate pricing. Each market's data is now properly tracked and displayed with the correct market-specific pricing models.

Version comparison: When Vanio releases improved pricing models, you'll see side-by-side comparisons showing whether the new version is delivering better accuracy and revenue.

Key features

• Dual evaluation modes: Switch between recent performance analysis and historical forecast evaluation to get different perspectives on pricing accuracy

• Extended historical analysis: Review pricing performance across up to 4 years of booking history to identify long-term trends and seasonal patterns

• Accuracy metrics: See Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE) to understand how close recommendations were to optimal prices

• Revenue impact tracking: View the actual revenue difference between our recommendations and your final booking prices

• Market breakdown: Compare pricing performance across different resort towns and destinations in your portfolio — all markets including Fernie, Amsterdam, and other international destinations now display correctly

• Historical trends: Track how pricing accuracy improves over time with 30-day rolling averages

• Model version comparison: When new pricing engines are released, see A/B comparisons between current and previous versions

• Auto-pricing performance: For properties using automatic price updates, see how much additional revenue was generated compared to manual pricing

• Booking volume context: Understand how many actual bookings each accuracy metric is based on

[Screenshot: Market comparison table showing accuracy metrics for different resort destinations]

Understanding the metrics

Mean Absolute Error (MAE): The average dollar difference between recommended and optimal prices. Lower is better. For example, an MAE of $15 means recommendations were typically within $15 of the ideal price.

Mean Absolute Percentage Error (MAPE): Shows pricing accuracy as a percentage. An MAPE of 8% means recommendations were typically within 8% of optimal pricing.

Lift Percentage: For auto-pricing customers, this shows the revenue improvement from using Vanio's recommendations versus your previous manual pricing.

Evaluated Bookings: The number of actual reservations used to calculate each metric, giving you confidence in the statistical significance.

Evaluation Modes:

• Recent Performance: Uses point-in-time recommendations that were actually available when bookings occurred
• Historical Forecast: Shows how today's pricing model would have performed on historical dates (not limited by when recommendations were generated)

Tips & best practices

Use both evaluation modes: Start with Historical Forecast to understand long-term patterns across your entire booking history, then switch to Recent Performance to see how current recommendations are performing in real-time.

Monitor weekly: Check your Model Quality dashboard weekly to spot any concerning trends in pricing accuracy or significant changes in performance.

Leverage extended history: Use the Historical Forecast mode to analyze seasonal trends and pricing patterns across multiple years, especially helpful for understanding how your pricing model handles different market conditions.

Focus on high-volume markets: Pay closest attention to accuracy metrics for markets where you have the most properties and bookings, as these have the biggest revenue impact.

Compare model versions: When you see a "new model version available" notification, use this dashboard to verify that the update actually improves your pricing accuracy before fully adopting it.

Seasonal awareness: Expect accuracy metrics to fluctuate during shoulder seasons or major events when booking patterns change rapidly. The extended historical view can help you understand these patterns.

Use lift data for decisions: If you're considering switching to automatic pricing, the lift percentage data shows you the potential revenue impact based on your actual booking history.

Check all your markets: Now that market-specific pricing models display properly for all destinations, make sure to review performance across your entire portfolio, including international markets that may have different booking patterns.

Common questions

How often is this data updated? The dashboard refreshes every night with the previous day's analysis. Recent Performance mode shows results based on the most recent 60 days, while Historical Forecast mode can analyze up to 4+ years of booking history.

What's the difference between Recent Performance and Historical Forecast modes? Recent Performance uses the actual recommendations that were available when guests booked (point-in-time accuracy), while Historical Forecast shows how your current pricing model would perform on historical dates. Historical Forecast is especially useful when you have years of booking data but only weeks or months of recommendation history.

Why do some markets show better accuracy than others? Markets with more consistent booking patterns and larger numbers of comparable properties typically show higher accuracy. Newer or smaller markets may have more variable results as the model learns local patterns.

What should I do if accuracy seems to be declining? Accuracy can fluctuate due to seasonal changes, local events, or shifts in market conditions. Use the Historical Forecast mode to see if similar patterns occurred in previous years. If you notice a sustained decline over several weeks, contact support to review your market factors and property settings.

I have properties in markets like Fernie or Amsterdam — will I see data for these? Yes! We recently fixed an issue that was preventing some international and smaller markets from displaying correctly. All your markets now show their proper pricing model data and recommendations in the dashboard.

[Screenshot: Historical accuracy trend chart showing performance over time]