Continuous Alignment Testing (CAT) Framework

A framework for implementing Continuous Alignment Testing (CAT) for LLM-based applications.

Overview

CAT provides the infrastructure to:

• Run and track automated tests against an AI application.
• Store and analyze AI test results over iterations.
• Monitor reliability changes in an AI application as prompts, models, code, and data evolve.
• Integrate AI tests into the CI pipeline.
• Build the AI application as an API service.
• Ensure AI applications are reliable while maintaining creativity.
• Iterate on prompts and code while measuring improvements.

Example Apps

• team_recommender - Team Recommender

Documentation

Github Pages: https://thisisartium.github.io/continuous-alignment-testing/

Wiki

https://github.com/thisisartium/continuous-alignment-testing/wiki