v0.3.0

What's Changed

• Add new standardized annotated pairs data format by @timokau in #18
• Add OpenRouter Integration and Dev Container Enhancements by @timokau in #17

Full Changelog: v0.2.1...v0.3.0

v0.2.1

What's Changed

• Update old configs to new config structure by @rdnfn in #14
• Add pairrm baseline training and eval code by @timokau in #9
• Improve functional annotator support by @rdnfn in #16
• Update plotting code to work with updated output format (including functional annotators)

Full Changelog: v0.2.0...v0.2.1

v0.2.0

What's Changed

• Add support for function-based annotators by @rdnfn in #13. Note that this version changes the structure of the annotator part of the experiments configuration (cfg.annotator). Using old configurations will throw errors with messages explaining how to adapt configurations to the new structure. In short, AlpacaEval specific configurations moved to cfg.annotator.alpaca_eval.

Full Changelog: v0.1.3...v0.2.0

v0.1.3

What's Changed

• Add experiment configs by @rdnfn in #8
• Add support for non-AlpacaEval annotators by @rdnfn in #5
• Add package release actions by @rdnfn in #10, #11
• Update versioning setup by @rdnfn in #12

Full Changelog: v0.1.2...v0.1.3

v0.1.2

Small update with the following main changes:

• Fix issues with caching of votes
• Update documentation to include description of vote caching method

v0.1.1

Small release with some minor fixes

• Fix #2: Enable use of built-in annotator configs by default to avoid forcing experiment to be run in cloned ICAI repo
• Pin AlpacaEval version to 0.6.5 or lower to avoid issues with analysis code
• Add more comprehensive tests via GitHub actions (including CLI test)

v0.1.0

First official release.