Fixed logging issue in rloo_trainer.py that where generate_completions() is not reflective of actual model generations

[swkarlekar]

What does this PR do?

Issue:

there is a bug in the generations that get logged as model checkpoints. Specifically, rloo_trainer.py's generate_completions() is not reflective of actual model generations, which seems to not be the desired behavior. In particular the temperature in generate_completions is set to 0.01.

   def generate_completions(self, sampling: bool = False):
        args = self.args
        processing_class = self.processing_class
        generation_config = **GenerationConfig(
            max_new_tokens=self.args.response_length,
            temperature=(0.01 + 1e-7),
            top_k=0.0,
            top_p=1.0,
            do_sample=True,
        )**
...

trl/trl/trainer/rloo_trainer.py

Line 575 in 4659ad9

temperature=(0.01 + 1e-7),

The generation_completions()'s temperature is not the same as that used in train():

...
        generation_config = GenerationConfig(
            max_new_tokens=args.response_length,
            temperature=(args.temperature + 1e-7),
            top_k=0.0,
            top_p=1.0,
            do_sample=True,
        )
...

trl/trl/trainer/rloo_trainer.py

Line 261 in 4659ad9

temperature=(args.temperature + 1e-7),

This results in logging of model checkpoints that are different from what gets used in the actual RLOO process.

Fix:

Adjusted the arguments in the generation_config in generate_completions() to use self.args.temperature instead of the hardcoded value 0.01.

Fixes # (issue)

#2678

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.