LIME update

kangreen0210 · Oct 14, 2024 · 3af8707 · 3af8707
1 parent 397601d
commit 3af8707
Show file tree

Hide file tree

Showing 59 changed files with 23 additions and 1,790 deletions.
diff --git a/README.md b/README.md
@@ -3,7 +3,7 @@
 
 # LIME: LESS IS MORE FOR MLLM EVALUATION
 ## Annoucement
-- [2024-10.01]  📰 We have released both the [LIME](https://huggingface.co/datasets/LIME-ICLR) dataset and the  data duration pipeline! 
+- [2024-10]  📰 We have released both the [LIME](https://huggingface.co/LIME-ICLR) dataset and the  data duration pipeline! 
 - [2024-09]  🍋 We have open-sourced the evaluation data and corresponding evaluation code for `LIME`. The data duration pipeline for LIME will be open-sourced within two weeks.
 
 ## Introduction
@@ -22,7 +22,7 @@ cd lmms-eval
 pip install -e .
 ```
 ### 2. download dataset from huggingface
-download all datasets from [here](https://huggingface.co/datasets/LIME-ICLR)
+download all datasets from [here](https://huggingface.co/LIME-ICLR)
 
 ### 3.run evaluation
 #### For MLLMs evaluation:

diff --git a/lmms_eval/tasks/ai2d_all_VD/ai2d_all.yaml b/lmms_eval/tasks/ai2d_all_VD/ai2d_all.yaml
diff --git a/lmms_eval/tasks/ai2d_all_VD/upload_ai2d.py b/lmms_eval/tasks/ai2d_all_VD/upload_ai2d.py
diff --git a/lmms_eval/tasks/ai2d_all_VD/utils.py b/lmms_eval/tasks/ai2d_all_VD/utils.py
diff --git a/lmms_eval/tasks/ai2d_suit_VD/ai2d_suit.yaml b/lmms_eval/tasks/ai2d_suit_VD/ai2d_suit.yaml
@@ -1,8 +1,4 @@
-<<<<<<< HEAD
-dataset_path: ./vlms-bench-data-selected-VD/ai2d
-=======
-dataset_path: /ML-A100/team/mm/zk/lmms-eval/vlms-bench-data-selected-VD/ai2d
->>>>>>> 865c7069caf994108f2fb1c2648cb346c8741a4e
+dataset_path: LIME-ICLR/ai2d
 task: "ai2d_suit_vd"
 dataset_kwargs:
   token: True

diff --git a/lmms_eval/tasks/ai2d_suit_VD/utils.py b/lmms_eval/tasks/ai2d_suit_VD/utils.py
@@ -30,8 +30,7 @@ def ai2d_doc_to_text(doc, model_specific_prompt_kwargs=None):
 
 
 def ai2d_doc_to_visual(doc):
-    image=Image.open(doc["image"])
-    return [image.convert("RGB")]
+    return [doc["image"].convert("RGB")]
 
 
 def ai2d_doc_to_target(doc, model_specific_target_kwargs):

diff --git a/lmms_eval/tasks/chartqa_all_vd/chartqa.yaml b/lmms_eval/tasks/chartqa_all_vd/chartqa.yaml