Skip to content

Commit

Permalink
vqav2 (EvolvingLMMs-Lab#25)
Browse files Browse the repository at this point in the history 
* Update tqdm progress bar position

* Merge commit '82108537ee4e3d54d6378fb7faa78199e00a3e8b'

* Squashed commit of the following:

commit 03edad8
Author: Zhang Peiyuan <a1286225768@gmail.com>
Date:   Thu Jan 25 17:08:25 2024 +0800

    add model specific prompt and gen kwargs in sqa (EvolvingLMMs-Lab#19)

    * add mmme

    * black

    * add model specific prompt and gen kwargs

    * black

    * add yaml config to supprot multi-model eval

    * print table at the end

    * refactor multi model code

commit 8210853
Author: kcz358 <92624596+kcz358@users.noreply.github.com>
Date:   Thu Jan 25 09:47:31 2024 +0800

    [Dataset] Add flickr30k (EvolvingLMMs-Lab#18)

    * Add flickr30k support

    * Black lint

    * Align prompt with NoCaps

commit 158c42d
Author: Li Bo <drluodian@gmail.com>
Date:   Wed Jan 24 22:10:14 2024 +0800

    [Datasets] modify NoCaps data path and prompts (EvolvingLMMs-Lab#17)

    * Merge commit '76c213db0f1495c1ececf0b58678f87cc6144e3c'

    * Update dataset paths and improve user prompts

commit 76c213d
Author: Li Bo <drluodian@gmail.com>
Date:   Wed Jan 24 19:51:34 2024 +0800

    Add output path file naming convention (EvolvingLMMs-Lab#16)

    Update datetime format in get_datetime_str() function

* remove useless output file

* Update dataset path in vqav2.yaml

* Squashed commit of the following:

commit 122f420e8450d70eeee97d0e33d30772f781358d
Author: kcz358 <92624596+kcz358@users.noreply.github.com>
Date:   Fri Jan 26 09:56:45 2024 +0800

    Black lint

commit 363c608444b6df57d51e53f2adb8d8cbfeda0852
Author: kcz358 <92624596+kcz358@users.noreply.github.com>
Date:   Fri Jan 26 09:55:47 2024 +0800

    Solve doc_iterator_for_counting crashing issue

commit 87d7ee9a776438ad39d4f275f6dc589433f30931
Author: kcz358 <92624596+kcz358@users.noreply.github.com>
Date:   Fri Jan 26 09:55:13 2024 +0800

    Exclude train in refcoco/+/g config

commit d5571773a0d095a288be47883ffdf53f07a077ee
Merge: 6a1ae69 697a438
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 17:17:13 2024 +0000

    Merge branch 'dev/bli_add_datasets' of https://github.com/EvolvingLMMs-Lab/lmms-eval into dev/bli_add_datasets

commit e8710b2ab15bf0bfdb30a60fbd18dcec404bd2ae
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 17:17:06 2024 +0000

    Fix file path and raise error if config file does not exist

commit 3f3f7481a933a368b5f1b8f267f8003ea0ef82f4
Author: kcz358 <92624596+kcz358@users.noreply.github.com>
Date:   Fri Jan 26 00:47:24 2024 +0800

    Fix tasks issue for nocaps, refcoco/+/g

commit c0dcef296c7076d5ad9992c1afef229e447b9851
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 10:09:43 2024 +0000

    Remove unused files and update task configuration

commit 59e99665b53fb0d1b95f59ae7c2bfffdb1f6d93b
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 09:43:56 2024 +0000

    Add submission file for coco, flickr30k, nocaps, and textcaps tasks

commit 68364a32bd346514262a957e89899a4c1c057bf9
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 09:32:54 2024 +0000

    Refactor get_task_dict function to handle nested groups

commit 18f51856fef5fd1b62b1189068c9837fb67195e3
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 09:13:46 2024 +0000

    Fix bug in login functionality

    Refactor code for better performance

    Add new feature for user authentication

    Update UI layout for improved user experience

    Fix typo in variable name

    Optimize database queries for faster response time

    Add error handling for edge cases

    Update dependencies to latest versions

    Remove unused code

    Improve code readability and maintainability

commit 927d9e80856e6b14fd81ddac273ba2468dddc076
Author: kcz358 <92624596+kcz358@users.noreply.github.com>
Date:   Thu Jan 25 17:07:20 2024 +0800

    Fix cli itself can not run with config file

commit 589b3a2a124c241441b3180a67bab57412bbe5ef
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 09:09:04 2024 +0000

    Squashed commit of the following:

    commit 03edad8
    Author: Zhang Peiyuan <a1286225768@gmail.com>
    Date:   Thu Jan 25 17:08:25 2024 +0800

        add model specific prompt and gen kwargs in sqa (EvolvingLMMs-Lab#19)

        * add mmme

        * black

        * add model specific prompt and gen kwargs

        * black

        * add yaml config to supprot multi-model eval

        * print table at the end

        * refactor multi model code

commit 4c9c7db3ba26973d130a111506b4d5d77ab00c95
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 09:02:57 2024 +0000

    Squashed commit of the following:

    commit a9bdc9b952df662cd7156ccc63af31ae0a83d2ff
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 11:59:12 2024 +0800

        refactor multi model code

    commit 7313f07606ec94f555d50d4523adcb2c1714922e
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 11:51:16 2024 +0800

        print table at the end

    commit 69f7f0be0eaa855c6c46e7c748a7ac69a04606e8
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 11:20:59 2024 +0800

        add yaml config to supprot multi-model eval

    commit 9173602a072c669f3348a58b715c77cfef4f0fbf
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 10:39:42 2024 +0800

        black

    commit 9386d0011c4d6ed7190373d0951d903c7548ccb5
    Merge: 7e8b57d 1d3fdd4
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 10:37:57 2024 +0800

        resolve conflicts in sqa

    commit 77079bc826943e187247863d5473237de05b3cf2
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 10:36:46 2024 +0800

        add model specific prompt and gen kwargs

    commit 1a284e6a412da3cc503297f33417dad19dd59aee
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Wed Jan 24 13:56:51 2024 +0800

        black

    commit d1c04e8c8e509a375c117020b3c241cc736f9365
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Wed Jan 24 13:55:43 2024 +0800

        add mmme

commit 4f7ad7f56de08cddd2b3af64635d0d3a2c37ddb7
Merge: 22c3adf 1d3fdd4
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 08:43:15 2024 +0000

    Merge commit '82108537ee4e3d54d6378fb7faa78199e00a3e8b' into dev/bli_add_datasets

commit e7cd3c23c345d1ed54e9085ac0cf28006489c434
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 08:38:52 2024 +0000

    Squashed commit of the following:

    commit a9bdc9b952df662cd7156ccc63af31ae0a83d2ff
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 11:59:12 2024 +0800

        refactor multi model code

    commit 7313f07606ec94f555d50d4523adcb2c1714922e
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 11:51:16 2024 +0800

        print table at the end

    commit 69f7f0be0eaa855c6c46e7c748a7ac69a04606e8
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 11:20:59 2024 +0800

        add yaml config to supprot multi-model eval

    commit 9173602a072c669f3348a58b715c77cfef4f0fbf
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 10:39:42 2024 +0800

        black

    commit 9386d0011c4d6ed7190373d0951d903c7548ccb5
    Merge: 7e8b57d 1d3fdd4
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 10:37:57 2024 +0800

        resolve conflicts in sqa

    commit 77079bc826943e187247863d5473237de05b3cf2
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 10:36:46 2024 +0800

        add model specific prompt and gen kwargs

    commit 8210853
    Author: kcz358 <92624596+kcz358@users.noreply.github.com>
    Date:   Thu Jan 25 09:47:31 2024 +0800

        [Dataset] Add flickr30k (EvolvingLMMs-Lab#18)

        * Add flickr30k support

        * Black lint

        * Align prompt with NoCaps

    commit 158c42d
    Author: Li Bo <drluodian@gmail.com>
    Date:   Wed Jan 24 22:10:14 2024 +0800

        [Datasets] modify NoCaps data path and prompts (EvolvingLMMs-Lab#17)

        * Merge commit '76c213db0f1495c1ececf0b58678f87cc6144e3c'

        * Update dataset paths and improve user prompts

    commit 1a284e6a412da3cc503297f33417dad19dd59aee
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Wed Jan 24 13:56:51 2024 +0800

        black

    commit d1c04e8c8e509a375c117020b3c241cc736f9365
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Wed Jan 24 13:55:43 2024 +0800

        add mmme

commit 33b1143ba4f86461cc37c5b4f86c3a20523768e5
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 08:38:11 2024 +0000

    Add coco_val and coco_test tasks to coco.yaml

commit b163d1076b6df227914352cb7a23e5cbc282c683
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 04:58:28 2024 +0000

    Update dataset_path in flickr30k.yaml

commit 44df7c8fa06090b48a47c5c87f988d2e14c663f9
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 02:12:25 2024 +0000

    Merge commit '82108537ee4e3d54d6378fb7faa78199e00a3e8b'

commit 96626ddda0df111ef5498f294e58ed01b51bdbbd
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 02:10:18 2024 +0000

    Add submission folder and update file paths for storing prediction results

commit 8210853
Author: kcz358 <92624596+kcz358@users.noreply.github.com>
Date:   Thu Jan 25 09:47:31 2024 +0800

    [Dataset] Add flickr30k (EvolvingLMMs-Lab#18)

    * Add flickr30k support

    * Black lint

    * Align prompt with NoCaps

commit 158c42d
Author: Li Bo <drluodian@gmail.com>
Date:   Wed Jan 24 22:10:14 2024 +0800

    [Datasets] modify NoCaps data path and prompts (EvolvingLMMs-Lab#17)

    * Merge commit '76c213db0f1495c1ececf0b58678f87cc6144e3c'

    * Update dataset paths and improve user prompts

commit 739dc3f823ab434707c23160c1bf51712ccbdc43
Merge: c6370bf 51f2eaa
Author: Li Bo <drluodian@gmail.com>
Date:   Wed Jan 24 22:10:07 2024 +0800

    Merge branch 'main' into dev/bli_add_datasets

commit 7ae42b4e7f1895429ffaa4ffb2d57f6aab2a470c
Author: Bo Li <drluodian@gmail.com>
Date:   Wed Jan 24 14:08:06 2024 +0000

    Update dataset paths and improve user prompts

commit 55d8c58b5446e432de7d397eb251028608d08edd
Author: Bo Li <drluodian@gmail.com>
Date:   Wed Jan 24 11:52:33 2024 +0000

    Merge commit '76c213db0f1495c1ececf0b58678f87cc6144e3c'

commit 76c213d
Author: Li Bo <drluodian@gmail.com>
Date:   Wed Jan 24 19:51:34 2024 +0800

    Add output path file naming convention (EvolvingLMMs-Lab#16)

    Update datetime format in get_datetime_str() function

* Fix bug in login functionality

* create vqav2_val

* Update vqav2_test.yaml

* Update vqav2_test.yaml

* Update vqav2_val.yaml

---------

Co-authored-by: Li Bo <drluodian@gmail.com>
  • Loading branch information
@pufanyi @Luodian
pufanyi and Luodian authored Jan 27, 2024
1 parent 1dc5155 commit 47252f5
Show file tree
Hide file tree
Showing 6 changed files with 322 additions and 4 deletions.
2 changes: 1 addition & 1 deletion lmms_eval/api/task.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -348,7 +348,7 @@ def build_all_requests(self, limit=None, rank=None, world_size=None) -> None:
doc_id_iterator = utils.create_iterator([i for i in range(len(docs))], rank, world_size, limit)
doc_id_iterator, doc_id_iterator_counting = itertools.tee(doc_id_iterator)
total_docs = sum(1 for _ in doc_id_iterator_counting)
pbar = tqdm(total=total_docs, desc="Building context")
pbar = tqdm(total=total_docs, desc=f"Building context {rank}", position=rank)
for doc_id in doc_id_iterator:
# sample fewshot context #TODO: need to offset doc_id by rank now!
fewshot_ctx = self.fewshot_context(doc_id, 0 if self.config.num_fewshot is None else self.config.num_fewshot, self.config.training_split if self.has_training_docs() else split)
Expand Down
22 changes: 22 additions & 0 deletions lmms_eval/tasks/docvqa/docvqa.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
task: docvqa
dataset_path: lmms-lab/DocVQA
test_split: test
output_type: generate_until
doc_to_visual: !function utils.vqav2_doc_to_visual
doc_to_text: !function utils.vqav2_doc_to_text
doc_to_target: "answer"
generation_kwargs:
until:
- "ASSISTANT:"
metric_list:
- metric: exact_match
aggregation: mean
higher_is_better: true
ignore_case: true
ignore_punctuation: true
- metric: submission
aggregation: !function utils.vqav2_aggreate_submissions
higher_is_better: true
metadata:
- version: 0.0
process_results: !function utils.vqav2_process_results
2 changes: 1 addition & 1 deletion lmms_eval/tasks/vqav2/utils.py → lmms_eval/tasks/vqav2_test/utils.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,7 @@ def vqav2_doc_to_text(doc):
def vqav2_aggreate_submissions(results):
now_date_time = datetime.datetime.now().strftime("%Y-%m-%d-%H-%M-%S")
os.makedirs("./submissions", exist_ok=True)
submission_file_name = f"./submissions/vqav2-submission-{now_date_time}.json"
submission_file_name = f"./submissions/vqav2-test-submission-{now_date_time}.json"
path = os.path.abspath(submission_file_name)
with open(path, "w") as f:
json.dump(results, f)
Expand Down
4 changes: 2 additions & 2 deletions lmms_eval/tasks/vqav2/vqav2.yaml → lmms_eval/tasks/vqav2_test/vqav2_test.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
task: "vqav2"
task: "vqav2_test"
dataset_path: lmms-lab/VQAv2
dataset_kwargs:
token: True
test_split: testdev
test_split: test
output_type: generate_until
doc_to_visual: !function utils.vqav2_doc_to_visual
doc_to_text: !function utils.vqav2_doc_to_text
Expand Down
272 changes: 272 additions & 0 deletions lmms_eval/tasks/vqav2_val/utils.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,272 @@
import re
import os
import json
import datetime
import statistics


def vqav2_doc_to_visual(doc):
return [doc["image"].convert("RGB")]


class EvalAIAnswerProcessor:
"""
Processes an answer similar to Eval AI
copied from
https://github.com/facebookresearch/mmf/blob/c46b3b3391275b4181567db80943473a89ab98ab/pythia/tasks/processors.py#L897
"""

CONTRACTIONS = {
"aint": "ain't",
"arent": "aren't",
"cant": "can't",
"couldve": "could've",
"couldnt": "couldn't",
"couldn'tve": "couldn't've",
"couldnt've": "couldn't've",
"didnt": "didn't",
"doesnt": "doesn't",
"dont": "don't",
"hadnt": "hadn't",
"hadnt've": "hadn't've",
"hadn'tve": "hadn't've",
"hasnt": "hasn't",
"havent": "haven't",
"hed": "he'd",
"hed've": "he'd've",
"he'dve": "he'd've",
"hes": "he's",
"howd": "how'd",
"howll": "how'll",
"hows": "how's",
"Id've": "I'd've",
"I'dve": "I'd've",
"Im": "I'm",
"Ive": "I've",
"isnt": "isn't",
"itd": "it'd",
"itd've": "it'd've",
"it'dve": "it'd've",
"itll": "it'll",
"let's": "let's",
"maam": "ma'am",
"mightnt": "mightn't",
"mightnt've": "mightn't've",
"mightn'tve": "mightn't've",
"mightve": "might've",
"mustnt": "mustn't",
"mustve": "must've",
"neednt": "needn't",
"notve": "not've",
"oclock": "o'clock",
"oughtnt": "oughtn't",
"ow's'at": "'ow's'at",
"'ows'at": "'ow's'at",
"'ow'sat": "'ow's'at",
"shant": "shan't",
"shed've": "she'd've",
"she'dve": "she'd've",
"she's": "she's",
"shouldve": "should've",
"shouldnt": "shouldn't",
"shouldnt've": "shouldn't've",
"shouldn'tve": "shouldn't've",
"somebody'd": "somebodyd",
"somebodyd've": "somebody'd've",
"somebody'dve": "somebody'd've",
"somebodyll": "somebody'll",
"somebodys": "somebody's",
"someoned": "someone'd",
"someoned've": "someone'd've",
"someone'dve": "someone'd've",
"someonell": "someone'll",
"someones": "someone's",
"somethingd": "something'd",
"somethingd've": "something'd've",
"something'dve": "something'd've",
"somethingll": "something'll",
"thats": "that's",
"thered": "there'd",
"thered've": "there'd've",
"there'dve": "there'd've",
"therere": "there're",
"theres": "there's",
"theyd": "they'd",
"theyd've": "they'd've",
"they'dve": "they'd've",
"theyll": "they'll",
"theyre": "they're",
"theyve": "they've",
"twas": "'twas",
"wasnt": "wasn't",
"wed've": "we'd've",
"we'dve": "we'd've",
"weve": "we've",
"werent": "weren't",
"whatll": "what'll",
"whatre": "what're",
"whats": "what's",
"whatve": "what've",
"whens": "when's",
"whered": "where'd",
"wheres": "where's",
"whereve": "where've",
"whod": "who'd",
"whod've": "who'd've",
"who'dve": "who'd've",
"wholl": "who'll",
"whos": "who's",
"whove": "who've",
"whyll": "why'll",
"whyre": "why're",
"whys": "why's",
"wont": "won't",
"wouldve": "would've",
"wouldnt": "wouldn't",
"wouldnt've": "wouldn't've",
"wouldn'tve": "wouldn't've",
"yall": "y'all",
"yall'll": "y'all'll",
"y'allll": "y'all'll",
"yall'd've": "y'all'd've",
"y'alld've": "y'all'd've",
"y'all'dve": "y'all'd've",
"youd": "you'd",
"youd've": "you'd've",
"you'dve": "you'd've",
"youll": "you'll",
"youre": "you're",
"youve": "you've",
}

NUMBER_MAP = {
"none": "0",
"zero": "0",
"one": "1",
"two": "2",
"three": "3",
"four": "4",
"five": "5",
"six": "6",
"seven": "7",
"eight": "8",
"nine": "9",
"ten": "10",
}
ARTICLES = ["a", "an", "the"]
PERIOD_STRIP = re.compile(r"(?!<=\d)(\.)(?!\d)")
COMMA_STRIP = re.compile(r"(?<=\d)(\,)+(?=\d)")
PUNCTUATIONS = [
";",
r"/",
"[",
"]",
'"',
"{",
"}",
"(",
")",
"=",
"+",
"\\",
"_",
"-",
">",
"<",
"@",
"`",
",",
"?",
"!",
]

def __init__(self, *args, **kwargs):
pass

def word_tokenize(self, word):
word = word.lower()
word = word.replace(",", "").replace("?", "").replace("'s", " 's")
return word.strip()

def process_punctuation(self, in_text):
out_text = in_text
for p in self.PUNCTUATIONS:
if (p + " " in in_text or " " + p in in_text) or (re.search(self.COMMA_STRIP, in_text) is not None):
out_text = out_text.replace(p, "")
else:
out_text = out_text.replace(p, " ")
out_text = self.PERIOD_STRIP.sub("", out_text, re.UNICODE)
return out_text

def process_digit_article(self, in_text):
out_text = []
temp_text = in_text.lower().split()
for word in temp_text:
word = self.NUMBER_MAP.setdefault(word, word)
if word not in self.ARTICLES:
out_text.append(word)
else:
pass
for word_id, word in enumerate(out_text):
if word in self.CONTRACTIONS:
out_text[word_id] = self.CONTRACTIONS[word]
out_text = " ".join(out_text)
return out_text

def __call__(self, item):
item = self.word_tokenize(item)
item = item.replace("\n", " ").replace("\t", " ").strip()
item = self.process_punctuation(item)
item = self.process_digit_article(item)
return item


def vqav2_process_results(doc, result):
eval_ai_processor = EvalAIAnswerProcessor()
assert len(result) == 1, f"The result should be a list of length 1, but got {len(result)}."
resAns = eval_ai_processor(result[0])
accuracy = 0

if "answers" in doc and doc["answers"] is not None:
for ansDic in doc["answers"]:
ansDic["answer"] = ansDic["answer"].replace("\n", " ")
ansDic["answer"] = ansDic["answer"].replace("\t", " ")
ansDic["answer"] = ansDic["answer"].strip()
gtAcc = []
gtAnswers = [ans["answer"] for ans in doc["answers"]]

if len(set(gtAnswers)) > 1:
for ansDic in doc["answers"]:
ansDic["answer"] = eval_ai_processor.process_punctuation(ansDic["answer"])
ansDic["answer"] = eval_ai_processor.process_digit_article(ansDic["answer"])
resAns = eval_ai_processor.process_punctuation(resAns)
resAns = eval_ai_processor.process_digit_article(resAns)

for gtAnsDatum in doc["answers"]:
otherGTAns = [item for item in doc["answers"] if item != gtAnsDatum]
matchingAns = [item for item in otherGTAns if item["answer"] == resAns]
acc = min(1, float(len(matchingAns)) / 3)
gtAcc.append(acc)
accuracy = statistics.mean(gtAcc)

return {
"exact_match": accuracy,
"submission": {
"question_id": doc["question_id"],
"answer": resAns,
},
}


def vqav2_doc_to_text(doc):
return f"{doc['question']}\nAnswer the question using a single word or phrase."


def vqav2_aggreate_submissions(results):
now_date_time = datetime.datetime.now().strftime("%Y-%m-%d-%H-%M-%S")
submission_file_name = f"vqav2-val-submission-{now_date_time}.json"
path = os.path.abspath(submission_file_name)
with open(path, "w") as f:
json.dump(results, f)
print(f"Submission file saved to {path}")
return 0
24 changes: 24 additions & 0 deletions lmms_eval/tasks/vqav2_val/vqav2_val.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
task: "vqav2_val"
dataset_path: lmms-lab/VQAv2
dataset_kwargs:
token: True
test_split: validation
output_type: generate_until
doc_to_visual: !function utils.vqav2_doc_to_visual
doc_to_text: !function utils.vqav2_doc_to_text
doc_to_target: "answer"
generation_kwargs:
until:
- "ASSISTANT:"
metric_list:
- metric: exact_match
aggregation: mean
higher_is_better: true
ignore_case: true
ignore_punctuation: true
- metric: submission
aggregation: !function utils.vqav2_aggreate_submissions
higher_is_better: true
metadata:
- version: 0.0
process_results: !function utils.vqav2_process_results

0 comments on commit 47252f5

Please sign in to comment.