Analyze AI Performance
Analyze AI Performance #
Once your AI Agent is live and handling customer conversations, it's crucial to monitor its performance, identify improvement opportunities, and track key metrics. Support Unicorn provides comprehensive analytics to help you understand how your AI Agent is performing and where to focus your optimization efforts.
Conversation History #
View and search all conversations your AI Agent has handled.
Accessing Conversations #
- Navigate to your AI Agent's page
- Click the "Chats" tab
- See a chronological list of all conversations
Conversation List View #
Each conversation shows:
Status Indicators:
- Active: Ongoing conversation
- Waiting on Customer: Last message from agent/AI
- Waiting on Agent: Customer waiting for response
- Closed: Conversation ended
Key Information:
- Customer name or identifier
- Channel type (Live Chat, SMS, WhatsApp)
- First message preview
- Message count (customer vs agent)
- Duration
- Created/updated timestamps
- AI handoff status (if escalated)
Filtering and Search #
Find specific conversations using filters:
By Status:
- Active conversations
- Closed conversations
- Escalated to humans
- AI-handled only
By Channel:
- Live Chat
- SMS
- WhatsApp
- All channels
By Date Range:
- Today
- Last 7 days
- Last 30 days
- Custom date range
By Customer:
- Search by email
- Search by phone number
- Search by name
- Search by customer ID
By Content:
- Keyword search in messages
- Search AI responses
- Search customer queries
Conversation Details #
Click any conversation to see complete details:
Message Thread:
- Full conversation history
- Timestamps for each message
- Sender identification (customer, AI, or human agent)
- Message content and formatting
- Rich media (images, attachments)
AI Performance Data:
For each AI response:
- Confidence Score: How confident the AI was (0-1)
- Response Time: How long it took to generate
- Source Attribution: Which knowledge base chunks were used
- Relevance Scores: How relevant each source was
- Token Usage: Prompt and completion tokens
Conversation Metadata:
- Session ID
- User agent (browser/device)
- IP address (if available)
- Custom attributes
- Tags and labels
Handoff Tracking:
If conversation was escalated:
- When escalation occurred
- Why it was escalated (confidence, sentiment, explicit request)
- Which human agent took over
- How long until human responded
- Resolution details
Conversation Actions #
From the conversation details page:
Close Conversation:
Mark resolved and close the conversation thread.
Reopen Conversation:
Reactivate a closed conversation if customer returns.
Disable/Enable AI:
Toggle AI responses on/off for this specific conversation.
Export Conversation:
Download full conversation history (JSON or PDF).
Add Tags:
Categorize for better organization and analytics.
Key Metrics Dashboard #
Track your AI Agent's performance with real-time metrics.
Overview Metrics #
Total Conversations:
- Total handled by AI
- Comparison to previous period
- Trend over time (line chart)
AI Automation Rate:
- Percentage of conversations handled entirely by AI
- No human intervention required
- Target: 60-80% for most use cases
Average Response Time:
- How fast AI responds to customer messages
- Typically 2-5 seconds
- Impacts customer satisfaction
Customer Satisfaction:
- Based on post-conversation ratings
- Scale of 1-5 stars
- CSAT (Customer Satisfaction Score)
AI Performance Metrics #
Confidence Score Distribution:
- Average confidence score across all responses
- Distribution chart (how many at each confidence level)
- Trend over time
High Confidence Rate:
- Percentage of responses with confidence > 0.7
- Indicates knowledge base coverage
- Target: 70%+ for well-trained agents
Escalation Rate:
- Percentage of conversations escalated to humans
- Why escalations occurred
- Target: 20-40% depending on AI mode
Knowledge Base Utilization:
- Which sources are used most frequently
- Which sources are never retrieved
- Gaps in knowledge base
Customer Engagement Metrics #
Message Volume:
- Total customer messages received
- Messages per conversation (avg)
- Peak hours and days
Conversation Length:
- Average number of message exchanges
- Duration of conversations
- Drop-off points
Return Customers:
- Customers with multiple conversations
- Frequency of repeat contacts
- Trends (increasing or decreasing)
Channel Distribution:
- Breakdown by channel (Live Chat, SMS, WhatsApp)
- Most popular channels
- Conversion rates per channel
Agent Efficiency Metrics #
First Contact Resolution (FCR):
- Conversations resolved without escalation
- Critical metric for AI success
- Target: 60%+ for common questions
Time to Resolution:
- How long from first message to resolution
- AI-only vs escalated conversations
- Benchmarks by topic/category
Credits Consumed:
- Total AI messages sent
- Credits used (1 per message default)
- Cost per conversation
- Trend over time
Handoff Analytics #
Understand when and why AI escalates to human agents.
Handoff Metrics #
Total Handoffs:
- Number of conversations escalated
- Percentage of all conversations
- Trend over time
Handoff Reasons:
Breakdown by escalation trigger:
- Low Confidence: AI uncertain about answer (X%)
- Customer Request: Explicitly asked for human (X%)
- Sentiment: Negative emotion detected (X%)
- Topic-Based: Predefined topic requiring human (X%)
- Timeout: No resolution after N exchanges (X%)
Time to Handoff:
- How long into conversation before escalation
- Average message count before handoff
- Patterns (immediate vs after attempts)
Human Response Time:
- How long until human agent responds after handoff
- SLA compliance
- By time of day/day of week
Handoff Success Rate #
Successful Handoffs:
- Human agent responded and resolved
- Customer satisfaction after handoff
- Resolution time
Failed Handoffs:
- No human response
- Customer abandoned before human reply
- Handoff to wrong team/person
Improving Handoff Flow #
Use handoff analytics to:
Reduce Unnecessary Escalations:
- Identify low-confidence topics → add to knowledge base
- Refine escalation thresholds
- Train AI with actual customer questions
Faster Human Response:
- Optimize routing rules
- Staff appropriately during peak times
- Use urgent notifications for high-priority handoffs
Better Context Transfer:
- Ensure humans see full conversation history
- Include customer metadata
- Provide AI's uncertainty reason
Content Performance #
Understand which knowledge base sources are working and which aren't.
Source Analytics #
Most Retrieved Sources:
- Documents or Q&As retrieved most frequently
- Indicates valuable content
- Ensure these are accurate and up-to-date
Underutilized Sources:
- Content that's never or rarely retrieved
- May be irrelevant, duplicate, or poorly formatted
- Consider removing or improving
Highest Confidence Sources:
- Which sources lead to high-confidence responses
- Model for creating new content
- Indicates good structure and clarity
Source Gaps:
- Common questions with no matching sources
- Low relevance scores across all sources
- Topics missing from knowledge base
Question Analysis #
Most Frequent Questions:
- What customers ask most often
- Ensure comprehensive answers in knowledge base
- Create dedicated Q&A pairs for top questions
Unanswered Questions:
- Questions AI couldn't answer confidently
- Explicitly said "I don't know"
- Priority for knowledge base expansion
Low Confidence Questions:
- Questions answered but with low confidence
- May need better content or clarification
- Review and improve
Question Categories:
Auto-categorization of questions by topic:
- Product information
- Billing and pricing
- Technical support
- Shipping and delivery
- Returns and refunds
- Account management
Content Recommendations #
Based on analytics, Support Unicorn suggests:
Add Content For:
- Top unanswered questions
- Frequent low-confidence responses
- Emerging topics from recent conversations
Update Content:
- Outdated information (detected from context)
- Sources with consistently low relevance
- Conflicting information across sources
Remove Content:
- Never-retrieved sources
- Duplicate content
- Irrelevant or outdated material
Customer Feedback #
Collect and analyze customer feedback on AI interactions.
Post-Conversation Surveys #
Automatic Survey Trigger:
After conversation closes, prompt customer:
- "How was your experience?" (1-5 stars)
- "Did the AI help solve your issue?" (Yes/No)
- "Any feedback?" (Optional text)
Survey Results:
- Overall satisfaction score
- Distribution of ratings
- Trends over time
- Correlation with AI confidence scores
Feedback Categories:
Analyze qualitative feedback:
- Positive: "Quick and helpful"
- Negative: "Couldn't understand my question"
- Suggestions: "Add information about X"
Thumbs Up/Down #
Allow customers to rate individual AI responses:
👍 Helpful responses:
- Mark in analytics
- Identify strong knowledge base areas
- Model for future responses
👎 Not helpful responses:
- Flag for review
- Identify gaps or inaccuracies
- Priority for improvement
Implicit Feedback #
Track behavioral signals:
Conversation Abandonment:
- Customer stops responding mid-conversation
- May indicate frustration or problem solved elsewhere
- Review abandoned conversation patterns
Repeat Questions:
- Customer rephrases same question multiple times
- AI response wasn't helpful or clear
- Knowledge base improvement needed
Escalation Requests:
- "Can I talk to a person?"
- Indicates AI couldn't meet need
- Review what led to request
Reports and Exports #
Generate custom reports and export data for deeper analysis.
Pre-Built Reports #
Daily Summary:
- Conversations handled
- AI automation rate
- Escalations
- Customer satisfaction
- Sent via email at 9 AM daily
Weekly Performance Report:
- Week-over-week comparisons
- Trending metrics
- Top questions
- Recommendations
- Sent via email every Monday
Monthly Business Review:
- Comprehensive monthly analysis
- ROI calculations (time saved, costs)
- Strategic recommendations
- Exported as PDF
Custom Reports #
Build your own reports with:
Metrics Selection:
Choose which metrics to include
- Conversation volume
- AI performance
- Customer satisfaction
- Channel breakdown
- etc.
Date Range:
- Custom start and end dates
- Compare two periods
- Year-over-year comparisons
Filters:
- Specific channels
- Customer segments
- AI agents (if multiple)
- Tags or labels
Visualization:
- Charts and graphs
- Tables
- Summary statistics
Schedule:
- One-time
- Daily, Weekly, Monthly
- Email recipients
Data Export #
Export raw data for external analysis:
Formats:
- CSV (spreadsheet)
- JSON (programmatic access)
- PDF (presentation-ready)
What Can Be Exported:
- Conversation histories
- Message-level data
- Metrics time-series
- Customer feedback
- Handoff logs
Use Cases:
- Import into business intelligence tools
- Share with stakeholders
- Compliance and record-keeping
- Custom analysis in Excel/Python
Continuous Improvement Workflow #
Use analytics to systematically improve your AI Agent.
Weekly Review Process #
1. Review Top Metrics (5 min)
- Automation rate
- Customer satisfaction
- Escalation rate
- Any concerning trends?
2. Identify Top Issues (10 min)
- Most common unanswered questions
- Lowest-confidence topics
- Highest escalation triggers
- Negative customer feedback
3. Prioritize Improvements (5 min)
- Quick wins (add 1-2 Q&As)
- Medium effort (update document)
- Large projects (new knowledge source)
4. Implement (throughout week)
- Add/update knowledge base content
- Adjust system prompts
- Refine escalation rules
- Test changes
5. Measure Impact (next review)
- Did metrics improve?
- Were specific issues resolved?
- Any new issues introduced?
Quarterly Deep Dive #
Every quarter, conduct comprehensive analysis:
Performance Review:
- Compare to previous quarter
- Benchmark against goals
- Identify major wins and misses
Knowledge Base Audit:
- Review all sources for accuracy
- Remove outdated content
- Consolidate duplicates
- Add missing information
Customer Journey Analysis:
- Map common conversation flows
- Identify friction points
- Optimize handoff experiences
ROI Calculation:
- Time saved by AI automation
- Cost per conversation
- Support team capacity increase
- Customer satisfaction improvement
Strategic Planning:
- Set goals for next quarter
- Plan major knowledge base projects
- Explore new channels or features
- Team training needs
Benchmarks and Goals #
Industry Benchmarks #
AI Automation Rate:
- Excellent: 70%+
- Good: 50-70%
- Needs Work: <50%
Customer Satisfaction (CSAT):
- Excellent: 4.5+/5
- Good: 4.0-4.5/5
- Needs Work: <4.0/5
First Contact Resolution:
- Excellent: 70%+
- Good: 50-70%
- Needs Work: <50%
Average Confidence Score:
- Excellent: 0.80+
- Good: 0.65-0.80
- Needs Work: <0.65
Setting Your Goals #
Consider your context when setting targets:
Industry:
- B2B SaaS: Higher complexity, lower automation
- E-commerce: High volume, higher automation potential
- Healthcare: Lower automation (safety/compliance)
Team Size:
- Small team (1-5): Target 60%+ automation
- Medium team (6-20): Target 50%+ automation
- Large team (20+): Target 40%+ automation (higher complexity)
AI Mode:
- Auto Response: Aim for 70%+ automation
- Hybrid: Aim for 90%+ suggestion acceptance
- Fallback: Aim for <10% fallback activation
FAQ #
How often should I review analytics? #
Daily: Quick glance at key metrics (5 min)
Weekly: Detailed review and minor adjustments (30 min)
Monthly: Comprehensive analysis and reporting (2 hours)
Quarterly: Deep dive and strategic planning (half day)
What's a good AI automation rate? #
Depends on your industry and complexity. For most businesses, 60-70% is excellent. Don't aim for 100% - some conversations genuinely need human judgment.
Why is my escalation rate so high? #
Common reasons:
- Knowledge base doesn't cover common questions
- Confidence threshold set too high (too conservative)
- Customers explicitly requesting humans (trust building needed)
- System prompt doesn't match customer language
How do I know which content to add? #
Use the "Unanswered Questions" report to see what customers are asking that the AI can't answer. Start with the most frequent questions first.
Can I track ROI of my AI Agent? #
Yes! Calculate:
- Time saved: Conversations automated × avg handling time × hourly rate
- Cost per conversation: Credits used / total conversations
- Capacity increase: % of team time freed up
- Customer satisfaction: Compare before/after AI deployment
What if customer satisfaction decreases after adding AI? #
This is uncommon but can happen. Check:
- Are responses accurate and helpful?
- Is AI tone appropriate for your brand?
- Are escalations handled smoothly?
- Are customers aware they're talking to AI?
- Consider starting in Hybrid mode for more control
How do I compare performance across multiple AI agents? #
If you have multiple agents (e.g., support vs sales), the dashboard shows comparative metrics side-by-side. You can also filter reports by specific agent.