Unable to reproduce llava evaluation

[jacob-hansen]

To Reproduce (on a linux machine with 1xA100 80GB):

1. I first create a new environment of python 3.10 with anaconda:

conda create --name lmms python=3.10
conda activate lmms

2. Install lmms-eval from source (when in lmms-eval repo)

pip install --no-deps -U -e .

3. Install llava requirements

pip install -r llava_repr_requirements.txt
This Fails due to a conflict in transformer requirement:

  ERROR: Cannot install -r llava_repr_requirements.txt (line 1) and transformers>=4.36.2 because these package versions have conflicting dependencies.
  
  The conflict is caused by:
      The user requested transformers>=4.36.2
      llava 1.1.3 depends on transformers==4.31.0

  To fix this you could try to:
  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip attempt to solve the dependency conflict

  ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

4. I then tried manually installed transformers>=4.36.2 and retried install after removing it from llava_repr_requirements.txt

pip install transformers
pip install -r llava_repr_requirements.txt

5. Run main command

accelerate launch --num_processes=1 -m lmms_eval --model llava --model_args pretrained="liuhaotian/llava-v1.5-7b" --tasks mme --batch_size 1 --log_samples --log_samples_sufix reproduce --output_path ./logs/

Received error:
__main__.py: error: unrecognized arguments: --log_samples_sufix reproduce

Removed unknown commands (log_samples_sufix)

accelerate launch --num_processes=1 -m lmms_eval --model llava --model_args pretrained="liuhaotian/llava-v1.5-7b" --tasks mme --batch_size 1 --log_samples --output_path ./logs/

Received ValueError:
ValueError: Attempted to load model 'llava', but no model for this name found! Supported model names: qwen_vl, gpt4V, instructblip, minicpm_v

I've tried from many fresh installations. I'm am up to date with the main branch of lmms-eval (cloned it few hours ago). What should I do?

[jacob-hansen]

Interesting, I observe different behavior for the log_samples_suffix reproduce (does not work) and log_samples_suffix llava_v1.5_mme_mmbenchen (I got to work)

In order to get it functioning, I also had to introduce two changes to the lmms-eval > lmms_eval > models > llava.py

1. Comment out "image_sizes=gen_kwargs["image_sizes"]," from generation args
2. Skip padding tokens (-200)
   cont = cont.masked_fill(cont == -200, 1)
   result = self.tokenizer.batch_decode(cont, skip_special_tokens=True)
   text_outputs = [i.replace("", "").replace("", "").strip() for i in result]
3. Hard Code device and device_map
   self._device = torch.device("cuda:0")
   self.device_map = "cuda:0"

Would anyone be able to clarify the difference between reproduce and llava_v1.5_mme_mmbenchen? What benchmarks are supported by llava_v1.5_mme_mmbenchen?

Additionally, has anyone ever run into the errors I mentioned? Wondering why I was unable to run the code as is.

[justlovebarbecue]

I almost met the same series of error as @jacob-hansen. Hope can have a clean version for llava soon.

[kcz358]

Hi, @jacob-hansen , @justlovebarbecue , thank you for spotting out the issue.

It seems like current llava_requirement.txt does have some issue. Here are my way to make it correct under current version.

---

Solution

First cd into lmms-eval

pip install --no-deps -U -e .

Then cd into the LLaVA repo from https://github.com/haotian-liu/LLaVA?tab=readme-ov-file and do the same thing

pip install --no-deps -U -e .

This will build llava and lmms_eval without installing any dependency.

Then instead of using the current llava_requirement.txt, please use this

accelerate==0.21.0
datasets==2.16.1
evaluate==0.4.1
hf_transfer==0.1.6
Jinja2==3.1.3
numpy==1.26.4
openai==1.13.3
packaging==23.2
pandas==2.2.1
Pillow==10.2.0
protobuf==4.25.3
pycocoevalcap==1.2
pycocotools==2.0.7
pytablewriter==1.2.0
pytest==8.0.2
python_Levenshtein==0.25.0
pytz==2024.1
PyYAML==6.0.1
PyYAML==6.0.1
Requests==2.31.0
sacrebleu==2.4.0
scikit_learn==1.2.2
sentencepiece==0.1.99
setuptools==68.2.2
sglang==0.1.12
shortuuid==1.0.12
sqlitedict==2.1.0
tenacity==8.2.3
torch==2.0.1
tokenizers==0.15.2
tqdm==4.66.2
transformers==4.37.2

Save this into llava_requirement.txt and

pip install -r llava_requirement.txt

The correct environment will be installed by this requirement file. Noted that this file is generated py pipreqs and possibly there are some redundant packages but it does work.

Then you can run

accelerate launch --num_processes=8 --main_process_port 12345 -m lmms_eval --model llava --model_args pretrained="liuhaotian/llava-v1.5-7b,use_flash_attention_2=False" --tasks mme --batch_size 1 --log_samples --log_samples_suffix llava_v1.5_mme --output_path ./logs/;

Make sure use_flash_attention_2 is set to False in model_args.

Results

---

Additional Note

If you want to use flash attention, you can install it by

pip install flash-attn --no-build-isolation --no-cache-dir

But noted that if you use flash-attn 2, there will be a slight different in the result. I got
"mme_cognition_score": 355.7142857142857,
"mme_percetion_score": 1509.9907963185276,

[kcz358]

Also, @jacob-hansen , there is one typo for the log_samples_suffix in our example script, should be log_samples_suffix instead of log_samples_sufix.

[jacob-hansen]

When following your new protocol, I observe no differences than my work arounds for the installation.

By specifying use_flash_attention_2=False, I no longer had to comment that part out from my code.

But I still observe a gpu and required the following changes:

self.device_map = torch.device(device) # (line 74)

This might be specific to my system though

[kcz358]

@jacob-hansen , you might also need to add device_map=auto or device_map=cuda when you use single device in the model args

[Luodian]

This change has been merged into:

#24