Add llava model for 🤗 Transformers

[lewtun]

This PR adds the modelling code needed to evaluate llava models in the transformers format: https://huggingface.co/collections/llava-hf/llava-15-65f762d5b6941db5c2ba07e0

Example command to run:

accelerate launch --num_processes=8 -m lmms_eval --model llava_hf   --model_args pretrained="llava-hf/llava-1.5-7b-hf"   --tasks mme --batch_size 1 --output_path ./logs/ --log_samples

I will share some benchmark numbers shortly, but the code can be reviewed in any case :)

[lewtun]

I think using chat templates is the best way enable flexibility across fine-tuned models so that one doesn't have to manually implement the template each time (like is currently done for the llava-next models)

[kcz358]

A duplicate line here

[kcz358]

Looks quite good for me, I think adding the device_map=auto option and loglikelihood support then should be able to merge

[jzhang38]

Should we prepend vicuna 1.1 system prompt? @kcz358

[kcz358]

Yeah, I think so. But that should be determined by the pass in conv template? So that it is much more flexible

[lewtun]

Good idea, I'll add a chat_template arg so users can set / override the template in the tokenizer if desired

[lewtun]

Looks quite good for me, I think adding the device_map=auto option and loglikelihood support then should be able to merge

I'm working on loglikelihood support - is there a way to test it works as expected? If you can point me to a benchmark or command to run, that would be very helpful!

[kcz358]

Looks quite good for me, I think adding the device_map=auto option and loglikelihood support then should be able to merge

I'm working on loglikelihood support - is there a way to test it works as expected? If you can point me to a benchmark or command to run, that would be very helpful!

Hi, I think you can try seedbench_ppl which is a multiple_choice output type and appending the options one by one to the context to calculate the loglikelihood.

Or you can use this yaml file which is revised on llava_in_the_wild

dataset_path: lmms-lab/llava-bench-coco
dataset_kwargs:
  token: True
task: "llava_in_the_wild_ppl"
test_split: train
output_type: loglikelihood
doc_to_visual: !function utils.llava_doc_to_visual
doc_to_text: !function utils.llava_doc_to_text
doc_to_target: "gpt_answer"
metric_list:
  - metric: perplexity
    higher_is_better: true
metadata:
  version: 0.0
model_specific_prompt_kwargs:
  default:
    pre_prompt: ""
    post_prompt: ""

This will test the model's perplexity on a generation task

[lewtun]

Hello @kcz358 @jzhang38 I've now tidied up the code and pushed support for:

• Chat templates that the user can specify at runtime. Note that I opted for the Jinja templates we use in transformers so that users don't have to install the GitHub Llava repo
• A proper implementation of the Vicuna chat template with system prompt in the Jinja format
• Loglikelihood benchmarks
• Flash attention

I also ran the 7B model over several benchmarks to compare against the original llava implementation. In some we have good agreement, while in others there is some significant difference. One possible reason is that the image processing differs across implementations (see here) and/or some slight differences in how the inputs are formatted.

Spreadsheet: https://docs.google.com/spreadsheets/d/1CbV-SOSVNl1S60Ns8B0-DhHBH5k5zPAm9M6XcpwFG5w/edit?usp=sharing

Do you have some ideas about why e.g. mme can be so different, given that other benchmarks like mmbench and mmmu are quite similar?

For the loglikelihood benchmarks, here's the chat template that is being applied (inspired by the llava code):

A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image> <image> <image> <image> <image> <image> <image> <image>
Please identify the sequence of actions in this video and record them in sequence. Answer :  ASSISTANT:  scoop sugar, pour milk, carry milk, reach cup, carry cup, reach cup</s>

Please let me know if this is not correct, e.g. should the EOS token be omitted?

Edit: I double checked the prompt template for the llava implementation of loglikelihood and spotted a bug in llava.py. Fixed in 7c7b969

[lewtun]

I think this explains the main diff on seedbench_ppl compared to llava_hf (running eval now to compare). Basically the problem was that contexts is a string for batch_size=1 and thus the prompt was just the first letter of the prompt

[lewtun]

OK this didn't have much impact after all: seedbench_ppl went from 0.168 -> 0.112.

What is odd is that the numbers reported in the paper are much higher ~0.6 which suggests there is also an issue in the original llava.py implementation as well:

For reference, this is the command I am running:

accelerate launch --num_processes=8 -m lmms_eval --model llava   --model_args pretrained=liuhaotian/llava-v1.5-7b   --tasks seedbench_ppl --batch_size 1 --output_path ./logs/ --log_samples

Perhaps this is something that can be dealt with in a follow up PR?

[kcz358]

Hmmm, I also notice this huge result difference when I write the seedbench ppl. The original llava use generation instead of ppl to generate answer and I achieve similar result using the generation version. So I feel this is just whether you use generation or perplexity. May I ask in your seedbench ppl logs, are the answer being matched correctly?

[kcz358]

Wow your work is amazing @lewtun ! Currently all seem quite good to me and thank you very much for spotting out the loglikelihood issue for us.

For mme disagreement, have you checked that the prompt are exactly the same for hf version and the llava version?

Also, I think just like you mentioned, different image processing implementation will also affect the score. Based on some of the tests in our development, this could cause some significant shift to the score. I checked the eval scripts of the llava and seems like the image processing implementation llava 1.5 used on mme is pad the image to a square.

https://github.com/haotian-liu/LLaVA/blob/4e2277a060da264c4f21b364c867cc622c945874/llava/mm_utils.py#L152-L163

Another factor that may affect the final score is the torch version you use. We provide a reproduce environment here that can exactly reproduce mme score on llava. Whether you use flash attn or not may also affect the score a bit but not too much and can be ignored

[lewtun]

For mme disagreement, have you checked that the prompt are exactly the same for hf version and the llava version?

Yes, I've checked they are exactly the same which suggests image processing is the culprit.

Another factor that may affect the final score is the torch version you use. We provide a reproduce environment here that can exactly reproduce mme score on llava. Whether you use flash attn or not may also affect the score a bit but not too much and can be ignored

Thanks, I am using torch==2.1.2 which produces an MME score of 1513.673 for llava which is compatible with the paper. I know there are plans to enable the same padding logic for llava_hf models, so perhaps we can merge this as-is and revisit MME at a future date?

[kcz358]

Thanks, I am using torch==2.1.2 which produces an MME score of 1513.673 for llava which is compatible with the paper. I know there are plans to enable the same padding logic for llava_hf models, so perhaps we can merge this as-is and revisit MME at a future date?

Yeah I think this is okay for now since for most of the benchmark the scores are similar

[lewtun]

Yeah I think this is okay for now since for most of the benchmark the scores are similar

Great! Any chance we could merge this soon? We are working on VLM integration in trl and would like to point the community to lmms-eval for the release :)

[kcz358]

Should this be moved into the model_utils here? Can create a folder inside the utils named llava and put this inside

[lewtun]

Good idea, will move it there!

[lewtun]

Moved to the llava_hf.py file for now (to keep things simple) in 6d08fe8

[kcz358]

Great! Any chance we could merge this soon? We are working on VLM integration in trl and would like to point the community to lmms-eval for the release :)

Hi @Luodian, most of the parts of this PR LGTM for me. Do you think we can merge it now or wait until next release? You might also wanna review the changes and see whether there are things that need to change.

[Luodian]

Great! Any chance we could merge this soon? We are working on VLM integration in trl and would like to point the community to lmms-eval for the release :)

Hi @Luodian, most of the parts of this PR LGTM for me. Do you think we can merge it now or wait until next release? You might also wanna review the changes and see whether there are things that need to change.

Hi I think it can be merged directly, but let me see the changes and after checking I will merge it~