Red Teaming python-framework for testing chatbots and LLM systems

Install 🚀

pip install llamator==2.3.1

Documentation 📚

Documentation Link: https://romiconez.github.io/llamator

Guides ▶️

• LLM Red Teaming with LLAMATOR
• LLAMATOR: Quick Start Guide

Examples 💡

• 📄 RAG bot testing via REST API
• 🧙‍♂️ Gandalf web bot testing via Selenium
• 💬 Telegram bot testing via Telethon
• 📱 WhatsApp bot testing via Selenium
• 🔗 LangChain client testing with custom attack
• 🐋 Running LLAMATOR in Docker

Supported Clients 🛠️

• 🌐 All LangChain clients
• 🧠 OpenAI-like API
• ⚙️ Custom Class (Telegram, WhatsApp, Selenium, etc.)

Unique Features 🌟

• ️🗡 Support for custom attacks from the user
• 👜 Large selection of attacks on RAG / Agent / Prompt in English and Russian
• 🛡 Custom configuration of chat clients
• 📊 History of attack requests and responses in Excel and CSV format
• 📄 Test report document in DOCX format

OWASP Classification 🔒

• 💉 LLM01: Prompt Injection and Jailbreaks
• 🕵 LLM07: System Prompt Leakage
• 🎭 LLM09: Misinformation

Community 🌍

• 📣 Telegram Channel — AI Security Lab
• 💬 Telegram Chat — LLAMATOR | AI Red Team Community

Supported by 🚀

• AI Security Lab ITMO
• Raft Security
• AI Talent Hub

License 📜

© Roman Neronov, Timur Nizamov, Nikita Ivanov

This project is licensed under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license. See the LICENSE file for details.