AI Quality Score Review

Important Notice: The AI Quality Score Review feature is temporarily unavailable while we optimize the system. Your property listings continue to work normally, and this feature will return in a future update.

The AI Quality Score Review feature allows your support team to review and validate property quality scores before they're rolled out across your entire listing portfolio. This ensures the accuracy and reliability of AI-powered property assessments at scale.

What it does

Vanio's AI system analyzes your properties across six quality dimensions—cleanliness, design, amenities, professionalism, photo accuracy, and property fit—and assigns scores to help you understand how your listings perform. The Review feature gives your team the ability to spot-check these scores, validate the AI's assessments, and catch any potential issues before they impact your entire property portfolio.

Current Status

We've temporarily disabled the AI Quality Score Review feature to fix a cost issue that was causing unnecessary image analysis requests to run in the background. This was draining account balances without providing any benefit to your property management.

What this means for you:

• Your existing property listings continue to work normally
• No impact on your current booking flow or guest experience
• Previously generated quality scores remain available for reference
• The feature will return once we've optimized the system for better cost efficiency

Getting started (when feature returns)

Accessing the review dashboard

1. Navigate to your admin panel
2. Go to Atlas → DNA Review
3. You'll see a list of recently analyzed properties with their quality scores

[Screenshot: DNA Review dashboard showing a table of properties with thumbnails, titles, locations, quality tiers, and Atlas scores]

Reviewing property scores

1. Browse the property list: Each row shows essential property details including location, quality tier, and overall Atlas score
2. Filter your view: Use the dropdown filters to focus on specific property segments, quality tiers, score ranges, or assessment versions
3. Click on any property to open the detailed review panel
4. Examine the breakdown: Review scores for each of the six quality dimensions with specific highlights and red flags
5. Rate the assessment: Use the thumbs up/down buttons to mark whether you agree with the AI's analysis—this helps train the system
6. Add notes: Include specific feedback about the assessment to help improve future scoring accuracy

[Screenshot: Detailed property review panel showing individual dimension scores, highlights, and identified issues]

How it works

When you request new property analyses, the system automatically:

• Selects a diverse sample of properties from your portfolio
• Analyzes property photos, descriptions, amenities, and host information
• Generates detailed scores across all six quality dimensions
• Provides specific reasoning for each score with highlighted strengths and concerns
• Calculates processing costs so you can track analysis expenses

The review interface shows you exactly what information was analyzed and how the AI arrived at each score, giving you full transparency into the assessment process.

Key features

• Comprehensive scoring: View detailed breakdowns across cleanliness, design, amenities, professionalism, photo accuracy, and property fit

• On-demand rescoring: Click "Rescore Now" to get an updated analysis using the latest assessment criteria—see results in seconds with side-by-side comparison

• Manual feedback system: Rate each assessment with thumbs up/down and add detailed notes to help train the scoring system

• Assessment accuracy tracking: View a confusion matrix showing how well AI recommendations align with your manual ratings

• Customer match insights: See which of your customers' properties were matched as similar comparables and evaluate the accuracy of those connections

• Version tracking: Filter reviews by assessment criteria version to identify improvements or changes in scoring standards

• Smart sampling: Request analysis of additional properties with automatic geographic and price tier distribution to ensure representative samples

• Real-time validation: Compare current scoring criteria against previously analyzed properties to identify any changes in assessment standards

• Cost tracking: Monitor analysis expenses with detailed breakdowns of processing costs per property

• Batch processing: Trigger analysis of up to 50 properties at once for efficient quality assurance workflows

• Historical comparison: Track how scores and assessment criteria evolve over time

[Screenshot: Quality dimension breakdown showing individual scores for cleanliness, design, amenities, etc. with specific feedback]

[Screenshot: Rescore interface showing before/after comparison with highlighted changes]

Tips & best practices

Getting the most accurate reviews

Review regularly: Check new property analyses weekly to catch any scoring inconsistencies early. This helps maintain quality standards across your portfolio.

Use the feedback system: Consistently rate assessments with thumbs up/down and add specific notes about what the AI got right or wrong. This training data directly improves future scoring accuracy.

Test with rescoring: When you notice questionable scores, use the "Rescore Now" feature to see if updated criteria produce different results. This helps identify whether issues are with older assessment versions.

Focus on edge cases: Pay special attention to properties that received unexpected scores—either much higher or lower than you anticipated. These often reveal important insights about the assessment criteria.

Use diverse samples: When requesting new analyses, ensure your sample includes properties from different markets, price points, and property types to validate scoring consistency.

Managing your review workflow

Set up a review schedule: Establish regular review sessions with your team to evaluate recent property assessments and discuss any concerning patterns.

Track the confusion matrix: Monitor the accuracy statistics to see how well AI recommendations align with your team's manual ratings over time.

Review customer matches: Check the "Matched by customers" panel to validate whether properties are being correctly identified as similar comparables.

Track score trends: Monitor whether property scores are trending higher or lower over time, which could indicate changes in your portfolio quality or assessment criteria.

Document findings: Keep notes on any scoring issues you discover to help improve future property assessments.

Common questions

When will the AI Quality Score Review feature be available again?

We're working to optimize the system for better cost efficiency and will restore the feature in a future update. Your property listings continue to work normally in the meantime, and any previously generated quality scores remain available for reference.

How often should we review property scores?

We recommend reviewing new property analyses at least weekly, especially when you're first implementing quality scoring across your portfolio. Once you're confident in the consistency, you can reduce the frequency to bi-weekly or monthly reviews.

What should we do if we find inaccurate scores?

If you notice scores that don't align with your expectations, first try the "Rescore Now" feature to see if updated criteria produce better results. Always rate the assessment with thumbs down and add specific notes about what was incorrect—this feedback directly improves the system. You can also check whether the assessment criteria have changed since the property was analyzed, as the system will flag when scores were generated using older criteria.

How does the manual feedback affect future scoring?

Your thumbs up/down ratings and detailed notes create training data that helps improve the AI's assessment accuracy over time. The confusion matrix shows you how well the system's recommendations align with your manual ratings, giving you insight into improvement trends.

Can we analyze specific properties on demand?

Currently, the system analyzes properties in batches to ensure efficient processing and cost management. You can trigger analysis of up to 50 additional properties at once, and the system will automatically select a diverse sample from your unanalyzed listings. However, you can instantly rescore any individual property that's already been analyzed to see how it performs with the latest assessment criteria.