v2.6.0

Summary of changes

Data Preprocessor Updates:

• Renaming of Data handlers to make it self explanatory:
  apply_dataset_formatting renamed to add_tokenizer_eos_token
apply_custom_data_formatting_jinja_template renamed to apply_custom_jinja_template
• Added support for Jinja template rendering in the dataset.
• Introduced a data handler for the EPT use case.
• Added support for renaming and retaining dataset columns in Data handlers.
• Introduced an EOS token flag to disable its use in data handlers.

Documentation Additions:

• Added documentation on running EPT with our library.

Dependency Updates:

• Updated the upper limit of TRL from v0.11 to v0.14 and the lower limit from v0.9.3to v0.13.
• Updated the upper limit of Transformers to v4.48.1 and the lower limit from v4.45 to v4.46.

Additional Changes:

• Setting Legacy=True in AutoTokenizer to suppress warnings.
• Bug fixed in Aim tracker where server-based tracking was not being picked up.

Full list of Changes

• chore: bump python version by @dushyantbehl in #449
• fix: Hardcode Legacy behavior to True to resolve warning. by @Luka-D in #446
• feat: Add support for jinja based template rendering of the dataset by @Abhishek-TAMU in #438
• fix: bug in aim tracker where the server based tracking was not picked by @dushyantbehl in #454
• fix: remove duplicate main_process_port entry by @prince14322 in #444
• fix: space missing from data_formatter_template causing mismatch with response_template by @HarikrishnanBalagopal in #455
• fix: Using Sandboxed Environment in handler rendering Jinja template by @Abhishek-TAMU in #456
• docs: Add documentation on how to do EPT runs with our library. by @dushyantbehl in #461
• feat: Rename data handlers and add a new one for EPT scenarios by @dushyantbehl in #460
• fix:Retrieval of tokenizer_kwargs in data handler tokenize_and_apply_input_masking by @Abhishek-TAMU in #465
• feat: Add support for renaming and retaining columns in data preprocessor by @dushyantbehl in #466
• chore(deps): upgrade trl and transformers by @willmj in #448
• feat: adding eos token to be made a flag so we don't force it on every handler by @dushyantbehl in #467
• chore:Merge set of changes for v2.6.0 by @Abhishek-TAMU in #472

New Contributors

• @prince14322 made their first contribution in #444

Full Changelog: v2.5.0...v2.6.0