feat: Support for add special tokens via cli args

[YashasviChaurasia]

Description of the change

Adds support for add_special_tokens to tokenizer's vocabulary via cli args
List of Special Tokens can be passed as follows:

Related issue number

Partially Solves #470 , Support for Reserved Special Tokens is not supported yet

How to verify the PR

Run a small sample training job with --add_special_tokens flag

python tuning/sft_trainer.py    --include_tokens_per_second="true" \
    --learning_rate="1e-05" \
    --logging_steps="1" \
    --logging_strategy="steps" \
    --max_steps="100" \
    --model_name_or_path="/your/model" \
    --output_dir="/output/directory" \
    --per_device_train_batch_size="1" \
    --save_steps="50" \
    --save_strategy="steps" \
    --add_special_tokens "<|assistant|>" "<|user|>" \
    --use_flash_attn=false \
    --training_data_path="/dataset"

once the training is completed load the trained model and check if the special token is tokenized properly as a single token/ or if it is part of the tokenizer's vocabulary

Was the PR tested

Debugging was done to ensure the updated tokens are reflected in the Tokenizer Vocab.

• I have added >=1 unit test(s) for every new method I have added.
• I have ensured all unit tests pass

[github-actions]

Thanks for making a pull request! 😃
One of the maintainers will review and advise on the next steps.

[YashasviChaurasia]

@kmehant

[Ssukriti]

surely there must be some unit tests we can add for this ? @Abhishek-TAMU @willmj can you guide if you see feasible unit tests?

[Ssukriti]

maybe after this ongoing refactor is done https://github.com/foundation-model-stack/fms-hf-tuning/pull/475/files , unit tests can be added to this to at least ensure all special tokens passed as args are present in special_tokens_dict

then other set of unit tests can ensure whatever is in special tokens dict is added to tokenizer

[Abhishek-TAMU]

unit tests can be added to this to at least ensure all special tokens passed as args are present in special_tokens_dict

Yes, in this refactor PR, this could be taken care of.

then other set of unit tests can ensure whatever is in special tokens dict is added to tokenizer

For this, unit test case could be added now where reference could be taken from this unit test case where sft_trainer.train() is called and after _validate_training/_test_run_inference, saved model in tempdir could be loaded with Autotokenizer and could be then checked for the special tokens.

[kmehant]

Also fix this @YashasviChaurasia

tuning/sft_trainer.py:302:8: W1203: Use lazy % or .format() formatting in logging functions (logging-fstring-interpolation)