Evaluation with multiple GPUs

[lucasjinreal]

Currently using multiple GPU to eval, the model will load into serveral devices, caused inference error.

How to force them eval one task in parallel on multiple device?

[kcz358]

Hi @lucasjinreal , you can refer to #12 for more details

[lucasjinreal]

@kcz358 Hi, the issue linked seems are runing one model with tensor parallel on 2 GPU.

But my request is doing evaluation same time on 8GPUs, my model is small and I want all GPUs runing at the same time, then gathering the results togather, does it support for now? How to enable that?

[kcz358]

This seems like you should use accelerate launch --num_processes=8

[lucasjinreal]

Am using the main.py directly, since I need my customized llava, does there any other way didn't need to involve with accelerate?

[kcz358]

If you does not want to use accelerate, you might also want to try torchrun. However, this is not tested and you might want figure out how to make it work.

[lucasjinreal]

Got it!

However, I met another problem, when eval mmbench, I got Nan result , while on cmmmu are normal:

num_fewshot: None,
batch_size: 1
|      Tasks       |Version|Filter|n-shot|  Metric  |Value|   |Stderr|
|------------------|-------|------|-----:|----------|-----|---|------|
|mmbench           |N/A    |none  |     0|submission|N/A  |±  |N/A   |
| - mmbench_cn_cc  |Yaml   |none  |     0|submission|N/A  |±  |N/A   |
| - mmbench_cn_dev |Yaml   |none  |     0|submission|N/A  |±  |N/A   |
| - mmbench_cn_test|Yaml   |none  |     0|submission|N/A  |±  |N/A   |
| - mmbench_en_dev |Yaml   |none  |     0|submission|N/A  |±  |N/A   |
| - mmbench_en_test|Yaml   |none  |     0|submission|N/A  |±  |N/A   |

|Groups |Version|Filter|n-shot|  Metric  |Value|   |Stderr|
|-------|-------|------|-----:|----------|-----|---|------|
|mmbench|N/A    |none  |     0|submission|N/A  |±  |N/A   |

any help?

[kcz358]

Hi @lucasjinreal , you can see the metric is submission, which means you need to submit the files to their website by yourself to get the score

You can find the submission file in your log file

[lucasjinreal]

For mmbench, I saw the result file, it has prediciton and gt, why it can not calculate a score offline?

[Luodian]

For mmbench, I saw the result file, it has prediciton and gt, why it can not calculate a score offline?

MMBench is designed for submission only, you need to submit the result file to their official site.

[lucasjinreal]

No, for dev it's certainly not.

I am not refering to test.

[Luodian]

No, for dev it's certainly not.

I am not refering to test.

oh yes you're right. we will plan the update to enable report mmbench-dev score

[lucasjinreal]

thanks, hoping for it

[hxhcreate]

Run into the same problem when evaluation with multigpu, looking forward to any assistance, thanks

RROR Error Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument weight in method wrapper_CUDA__cudnn_convolution) in generating
05-10 21:54:48 [lmms_eval/models/llava.py:407] ERROR Error Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:3! (when checking argument for argument weight in method wrapper_CUDA__cudnn_convolution) in generating