todos.md

Support Ticket Test Data Initiative

Overview

Create high-quality test data for support tickets, focusing on 'new' status tickets to test our LLM Agent's RAG capabilities.

Requirements

• 30 'new' status tickets
• Each ticket has 1 customer message, 0 agent responses
• Questions mostly align with existing documentation
• Some questions intentionally unanswerable
• Data matches TSV format

Current Documentation Sources

• Articles (articles.tsv)

  • Solar Panel Maintenance Guide
  • Understanding Solar Tax Credits
  • Troubleshooting Common Issues
  • Battery Backup Systems

• Documents (markdown files)

  • Installation Guide
  • Maintenance Schedule
  • System Specifications
  • Warranty Terms

Implementation Plan

1. Document Analysis

• Review all documentation sources
• Create list of key topics covered
• Identify gaps in documentation for unanswerable queries
• Map common customer question types to docs

2. Question Generation

• Create 25 questions based on documentation
  • Installation questions (5)
  • Maintenance questions (5)
  • Warranty questions (5)
  • Technical specifications (5)
  • Financial/billing questions (5)
• Create 5 unanswerable questions
  • Edge cases
  • Out of scope queries
  • Questions about undocumented features

3. Data Preparation

• Format all tickets in TSV schema
• Ensure proper status flags
• Validate message counts
• Check data integrity

4. Quality Assurance

• Verify question distribution
• Check documentation coverage
• Validate TSV format
• Test data loading

Success Criteria

1. 30 new tickets created
2. Each ticket has exactly 1 customer message
3. Questions cover full range of documentation
4. Include realistic unanswerable queries
5. Data matches existing TSV format
6. Questions are realistic and varied

Notes

• Focus on common customer pain points
• Include both simple and complex queries
• Maintain realistic language and tone
• Ensure questions reflect actual customer behavior

Summary of Created Test Data

1. Installation Questions (5)

   • Roof assessment
   • Installation timeline
   • Safety requirements
   • Documentation
   • System commissioning

2. Maintenance Questions (5)

   • Cleaning schedule
   • Performance monitoring
   • Winter preparation
   • Annual inspection
   • System analysis

3. Warranty Questions (5)

   • Warranty transfer
   • Performance claims
   • Extended options
   • Component replacement
   • Maintenance requirements

4. Technical Questions (5)

   • Panel efficiency
   • Inverter compatibility
   • Temperature impact
   • Monitoring capabilities
   • Wind resistance

5. Financial Questions (5)

   • Tax credits
   • Production estimates
   • System sizing
   • State rebates
   • Performance guarantees

6. Unanswerable Questions (5)

   • Smart home integration
   • International warranty
   • Custom panel design
   • Mobile app features
   • Cryptocurrency mining