logging issue: generation_config in rloo_trainer.py's generate_completions() is not reflective of actual model generations

[swkarlekar]

Reproduction

Hi! I think there is a bug in the generations that get logged as model checkpoints. Specifically, rloo_trainer.py's generate_completions() is not reflective of actual model generations, which seems to not be the desired behavior. In particular the temperature in generate_completions is set to 0.01.

   def generate_completions(self, sampling: bool = False):
        args = self.args
        processing_class = self.processing_class
        generation_config = **GenerationConfig(
            max_new_tokens=self.args.response_length,
            temperature=(0.01 + 1e-7),
            top_k=0.0,
            top_p=1.0,
            do_sample=True,
        )**
...

The generation_completions()'s temperature is not the same as that used in train():

...
        generation_config = GenerationConfig(
            max_new_tokens=args.response_length,
            temperature=(args.temperature + 1e-7),
            top_k=0.0,
            top_p=1.0,
            do_sample=True,
        )
...

This results in logging of model checkpoints that are different from what gets used in the actual RLOO process.

I would be open to doing a small pull request to fix this!

System Info

Examined files in main

trl/trl/trainer/rloo_trainer.py

Line 575 in 4659ad9

temperature=(0.01 + 1e-7),

trl/trl/trainer/rloo_trainer.py

Line 261 in 4659ad9

temperature=(args.temperature + 1e-7),

Checklist

• I have checked that my issue isn't already filed (see open issues)
• I have included my system information
• Any code provided is minimal, complete, and reproducible (more on MREs)
• Any code provided is properly formatted in code blocks, (no screenshot, more on code blocks)
• Any traceback provided is complete