Skip to content

Commit

Permalink
vqav2 (EvolvingLMMs-Lab#25)
Browse files Browse the repository at this point in the history 
* Update tqdm progress bar position

* Merge commit '52ee4a18dad22b2399a4248d2aa9204dbfe88624'

* Squashed commit of the following:

commit f7a7db5
Author: Zhang Peiyuan <a1286225768@gmail.com>
Date:   Thu Jan 25 17:08:25 2024 +0800

    add model specific prompt and gen kwargs in sqa (EvolvingLMMs-Lab#19)

    * add mmme

    * black

    * add model specific prompt and gen kwargs

    * black

    * add yaml config to supprot multi-model eval

    * print table at the end

    * refactor multi model code

commit 52ee4a1
Author: kcz358 <92624596+kcz358@users.noreply.github.com>
Date:   Thu Jan 25 09:47:31 2024 +0800

    [Dataset] Add flickr30k (EvolvingLMMs-Lab#18)

    * Add flickr30k support

    * Black lint

    * Align prompt with NoCaps

commit 04303b0
Author: Li Bo <drluodian@gmail.com>
Date:   Wed Jan 24 22:10:14 2024 +0800

    [Datasets] modify NoCaps data path and prompts (EvolvingLMMs-Lab#17)

    * Merge commit 'bee5794a597d8a87794b4bcd9b57a1553efad857'

    * Update dataset paths and improve user prompts

commit bee5794
Author: Li Bo <drluodian@gmail.com>
Date:   Wed Jan 24 19:51:34 2024 +0800

    Add output path file naming convention (EvolvingLMMs-Lab#16)

    Update datetime format in get_datetime_str() function

* remove useless output file

* Update dataset path in vqav2.yaml

* Squashed commit of the following:

commit 75bb7043ea5a533ab6351fc0f5ab055e86106423
Author: kcz358 <92624596+kcz358@users.noreply.github.com>
Date:   Fri Jan 26 09:56:45 2024 +0800

    Black lint

commit 6635a8aa34cfbd3c7a4afb6fcd214a7283ce01cb
Author: kcz358 <92624596+kcz358@users.noreply.github.com>
Date:   Fri Jan 26 09:55:47 2024 +0800

    Solve doc_iterator_for_counting crashing issue

commit 080f42b88ea8acacd527b8d67b84ba1d7d135b03
Author: kcz358 <92624596+kcz358@users.noreply.github.com>
Date:   Fri Jan 26 09:55:13 2024 +0800

    Exclude train in refcoco/+/g config

commit 4da84069c08c95e49e8ab0e64a1e103ff7ac8730
Merge: 6a1ae69 697a438
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 17:17:13 2024 +0000

    Merge branch 'dev/bli_add_datasets' of https://github.com/EvolvingLMMs-Lab/lmms-eval into dev/bli_add_datasets

commit 6a1ae69923d79ae32a001edac38206b605274ec3
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 17:17:06 2024 +0000

    Fix file path and raise error if config file does not exist

commit 697a4387827ceeec3e393237dd1baa217c714c88
Author: kcz358 <92624596+kcz358@users.noreply.github.com>
Date:   Fri Jan 26 00:47:24 2024 +0800

    Fix tasks issue for nocaps, refcoco/+/g

commit 47e40437126d39a5f062c9a33b4de426c1a29804
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 10:09:43 2024 +0000

    Remove unused files and update task configuration

commit 9976eb8e9ed03c8613725fdbd822ef5d8cf70e47
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 09:43:56 2024 +0000

    Add submission file for coco, flickr30k, nocaps, and textcaps tasks

commit 95f97a69faa6129676e89eee14960fcfe2076b7c
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 09:32:54 2024 +0000

    Refactor get_task_dict function to handle nested groups

commit 3b79ee842b2488714baf92ab34528ef77989d392
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 09:13:46 2024 +0000

    Fix bug in login functionality

    Refactor code for better performance

    Add new feature for user authentication

    Update UI layout for improved user experience

    Fix typo in variable name

    Optimize database queries for faster response time

    Add error handling for edge cases

    Update dependencies to latest versions

    Remove unused code

    Improve code readability and maintainability

commit f5c353f2ce93a2d96add4312b695b57432f68cbb
Author: kcz358 <92624596+kcz358@users.noreply.github.com>
Date:   Thu Jan 25 17:07:20 2024 +0800

    Fix cli itself can not run with config file

commit 9a68fec37be74cfe8d4a73390bc83edee147ae24
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 09:09:04 2024 +0000

    Squashed commit of the following:

    commit f7a7db5
    Author: Zhang Peiyuan <a1286225768@gmail.com>
    Date:   Thu Jan 25 17:08:25 2024 +0800

        add model specific prompt and gen kwargs in sqa (EvolvingLMMs-Lab#19)

        * add mmme

        * black

        * add model specific prompt and gen kwargs

        * black

        * add yaml config to supprot multi-model eval

        * print table at the end

        * refactor multi model code

commit 93f847c5851fd246716367935d6b807b17d53949
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 09:02:57 2024 +0000

    Squashed commit of the following:

    commit 4d48d0c9b88e62dfebe05ec909b7f1851e9cd75d
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 11:59:12 2024 +0800

        refactor multi model code

    commit 4a4b7bec200c72332b61a0c277cd8f8a34e4f721
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 11:51:16 2024 +0800

        print table at the end

    commit 63739fc6fa0a462d807ae81de0db0173102de584
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 11:20:59 2024 +0800

        add yaml config to supprot multi-model eval

    commit edcc752f97ea3845cefad56624e5d2855066f680
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 10:39:42 2024 +0800

        black

    commit 41f4b63d3a6e83babe92bac32a7432a8ef740bb5
    Merge: 7e8b57d 1d3fdd4
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 10:37:57 2024 +0800

        resolve conflicts in sqa

    commit 7e8b57d3bcc21d2a049d3abbc8a8201631641db4
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 10:36:46 2024 +0800

        add model specific prompt and gen kwargs

    commit 7d5058337d3de3cd4f0e85368e3dd463f34e703c
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Wed Jan 24 13:56:51 2024 +0800

        black

    commit 73918654650daa0dad965d1b786d53e7c3585010
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Wed Jan 24 13:55:43 2024 +0800

        add mmme

commit fa4ad4404e26d8924f55208746dbb9143b464011
Merge: 22c3adf 1d3fdd4
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 08:43:15 2024 +0000

    Merge commit '52ee4a18dad22b2399a4248d2aa9204dbfe88624' into dev/bli_add_datasets

commit 22c3adfd0645acc23b6d7c06b487f4ffd47666c4
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 08:38:52 2024 +0000

    Squashed commit of the following:

    commit 4d48d0c9b88e62dfebe05ec909b7f1851e9cd75d
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 11:59:12 2024 +0800

        refactor multi model code

    commit 4a4b7bec200c72332b61a0c277cd8f8a34e4f721
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 11:51:16 2024 +0800

        print table at the end

    commit 63739fc6fa0a462d807ae81de0db0173102de584
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 11:20:59 2024 +0800

        add yaml config to supprot multi-model eval

    commit edcc752f97ea3845cefad56624e5d2855066f680
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 10:39:42 2024 +0800

        black

    commit 41f4b63d3a6e83babe92bac32a7432a8ef740bb5
    Merge: 7e8b57d 1d3fdd4
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 10:37:57 2024 +0800

        resolve conflicts in sqa

    commit 7e8b57d3bcc21d2a049d3abbc8a8201631641db4
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Thu Jan 25 10:36:46 2024 +0800

        add model specific prompt and gen kwargs

    commit 52ee4a1
    Author: kcz358 <92624596+kcz358@users.noreply.github.com>
    Date:   Thu Jan 25 09:47:31 2024 +0800

        [Dataset] Add flickr30k (EvolvingLMMs-Lab#18)

        * Add flickr30k support

        * Black lint

        * Align prompt with NoCaps

    commit 04303b0
    Author: Li Bo <drluodian@gmail.com>
    Date:   Wed Jan 24 22:10:14 2024 +0800

        [Datasets] modify NoCaps data path and prompts (EvolvingLMMs-Lab#17)

        * Merge commit 'bee5794a597d8a87794b4bcd9b57a1553efad857'

        * Update dataset paths and improve user prompts

    commit 7d5058337d3de3cd4f0e85368e3dd463f34e703c
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Wed Jan 24 13:56:51 2024 +0800

        black

    commit 73918654650daa0dad965d1b786d53e7c3585010
    Author: jzhang38 <a1286225768@gmail.com>
    Date:   Wed Jan 24 13:55:43 2024 +0800

        add mmme

commit 4c712336b6f7438e717a865910bb241e413a4688
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 08:38:11 2024 +0000

    Add coco_val and coco_test tasks to coco.yaml

commit b5547126c855927fd4dc8384211e4aceee40870f
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 04:58:28 2024 +0000

    Update dataset_path in flickr30k.yaml

commit f786f61e2559f082072f21aa9030e2080ddaf809
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 02:12:25 2024 +0000

    Merge commit '52ee4a18dad22b2399a4248d2aa9204dbfe88624'

commit 796a011000e0df90f66f8e80cb34dc2318ae9ac8
Author: Bo Li <drluodian@gmail.com>
Date:   Thu Jan 25 02:10:18 2024 +0000

    Add submission folder and update file paths for storing prediction results

commit 52ee4a1
Author: kcz358 <92624596+kcz358@users.noreply.github.com>
Date:   Thu Jan 25 09:47:31 2024 +0800

    [Dataset] Add flickr30k (EvolvingLMMs-Lab#18)

    * Add flickr30k support

    * Black lint

    * Align prompt with NoCaps

commit 04303b0
Author: Li Bo <drluodian@gmail.com>
Date:   Wed Jan 24 22:10:14 2024 +0800

    [Datasets] modify NoCaps data path and prompts (EvolvingLMMs-Lab#17)

    * Merge commit 'bee5794a597d8a87794b4bcd9b57a1553efad857'

    * Update dataset paths and improve user prompts

commit 118744c63eb2d9724571d85fbbd85fcc9ad05b59
Merge: c6370bf 51f2eaa
Author: Li Bo <drluodian@gmail.com>
Date:   Wed Jan 24 22:10:07 2024 +0800

    Merge branch 'main' into dev/bli_add_datasets

commit c6370bff65903681f00cf3d07111d8e15a57b619
Author: Bo Li <drluodian@gmail.com>
Date:   Wed Jan 24 14:08:06 2024 +0000

    Update dataset paths and improve user prompts

commit 810daf458fa94cb3ec2b4a6cc5ecb1e656a24002
Author: Bo Li <drluodian@gmail.com>
Date:   Wed Jan 24 11:52:33 2024 +0000

    Merge commit 'bee5794a597d8a87794b4bcd9b57a1553efad857'

commit bee5794
Author: Li Bo <drluodian@gmail.com>
Date:   Wed Jan 24 19:51:34 2024 +0800

    Add output path file naming convention (EvolvingLMMs-Lab#16)

    Update datetime format in get_datetime_str() function

* Fix bug in login functionality

* create vqav2_val

* Update vqav2_test.yaml

* Update vqav2_test.yaml

* Update vqav2_val.yaml

---------

Co-authored-by: Li Bo <drluodian@gmail.com>
  • Loading branch information
@pufanyi @Luodian
pufanyi and Luodian authored Jan 27, 2024
1 parent be874f2 commit 54a1cbd
Show file tree
Hide file tree
Showing 6 changed files with 322 additions and 4 deletions.
2 changes: 1 addition & 1 deletion lmms_eval/api/task.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -348,7 +348,7 @@ def build_all_requests(self, limit=None, rank=None, world_size=None) -> None:
doc_id_iterator = utils.create_iterator([i for i in range(len(docs))], rank, world_size, limit)
doc_id_iterator, doc_id_iterator_counting = itertools.tee(doc_id_iterator)
total_docs = sum(1 for _ in doc_id_iterator_counting)
pbar = tqdm(total=total_docs, desc="Building context")
pbar = tqdm(total=total_docs, desc=f"Building context {rank}", position=rank)
for doc_id in doc_id_iterator:
# sample fewshot context #TODO: need to offset doc_id by rank now!
fewshot_ctx = self.fewshot_context(doc_id, 0 if self.config.num_fewshot is None else self.config.num_fewshot, self.config.training_split if self.has_training_docs() else split)
Expand Down
22 changes: 22 additions & 0 deletions lmms_eval/tasks/docvqa/docvqa.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
task: docvqa
dataset_path: lmms-lab/DocVQA
test_split: test
output_type: generate_until
doc_to_visual: !function utils.vqav2_doc_to_visual
doc_to_text: !function utils.vqav2_doc_to_text
doc_to_target: "answer"
generation_kwargs:
until:
- "ASSISTANT:"
metric_list:
- metric: exact_match
aggregation: mean
higher_is_better: true
ignore_case: true
ignore_punctuation: true
- metric: submission
aggregation: !function utils.vqav2_aggreate_submissions
higher_is_better: true
metadata:
- version: 0.0
process_results: !function utils.vqav2_process_results
2 changes: 1 addition & 1 deletion lmms_eval/tasks/vqav2/utils.py → lmms_eval/tasks/vqav2_test/utils.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,7 @@ def vqav2_doc_to_text(doc):
def vqav2_aggreate_submissions(results):
now_date_time = datetime.datetime.now().strftime("%Y-%m-%d-%H-%M-%S")
os.makedirs("./submissions", exist_ok=True)
submission_file_name = f"./submissions/vqav2-submission-{now_date_time}.json"
submission_file_name = f"./submissions/vqav2-test-submission-{now_date_time}.json"
path = os.path.abspath(submission_file_name)
with open(path, "w") as f:
json.dump(results, f)
Expand Down
4 changes: 2 additions & 2 deletions lmms_eval/tasks/vqav2/vqav2.yaml → lmms_eval/tasks/vqav2_test/vqav2_test.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
task: "vqav2"
task: "vqav2_test"
dataset_path: lmms-lab/VQAv2
dataset_kwargs:
token: True
test_split: testdev
test_split: test
output_type: generate_until
doc_to_visual: !function utils.vqav2_doc_to_visual
doc_to_text: !function utils.vqav2_doc_to_text
Expand Down
272 changes: 272 additions & 0 deletions lmms_eval/tasks/vqav2_val/utils.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,272 @@
import re
import os
import json
import datetime
import statistics


def vqav2_doc_to_visual(doc):
return [doc["image"].convert("RGB")]


class EvalAIAnswerProcessor:
"""
Processes an answer similar to Eval AI
copied from
https://github.com/facebookresearch/mmf/blob/c46b3b3391275b4181567db80943473a89ab98ab/pythia/tasks/processors.py#L897
"""

CONTRACTIONS = {
"aint": "ain't",
"arent": "aren't",
"cant": "can't",
"couldve": "could've",
"couldnt": "couldn't",
"couldn'tve": "couldn't've",
"couldnt've": "couldn't've",
"didnt": "didn't",
"doesnt": "doesn't",
"dont": "don't",
"hadnt": "hadn't",
"hadnt've": "hadn't've",
"hadn'tve": "hadn't've",
"hasnt": "hasn't",
"havent": "haven't",
"hed": "he'd",
"hed've": "he'd've",
"he'dve": "he'd've",
"hes": "he's",
"howd": "how'd",
"howll": "how'll",
"hows": "how's",
"Id've": "I'd've",
"I'dve": "I'd've",
"Im": "I'm",
"Ive": "I've",
"isnt": "isn't",
"itd": "it'd",
"itd've": "it'd've",
"it'dve": "it'd've",
"itll": "it'll",
"let's": "let's",
"maam": "ma'am",
"mightnt": "mightn't",
"mightnt've": "mightn't've",
"mightn'tve": "mightn't've",
"mightve": "might've",
"mustnt": "mustn't",
"mustve": "must've",
"neednt": "needn't",
"notve": "not've",
"oclock": "o'clock",
"oughtnt": "oughtn't",
"ow's'at": "'ow's'at",
"'ows'at": "'ow's'at",
"'ow'sat": "'ow's'at",
"shant": "shan't",
"shed've": "she'd've",
"she'dve": "she'd've",
"she's": "she's",
"shouldve": "should've",
"shouldnt": "shouldn't",
"shouldnt've": "shouldn't've",
"shouldn'tve": "shouldn't've",
"somebody'd": "somebodyd",
"somebodyd've": "somebody'd've",
"somebody'dve": "somebody'd've",
"somebodyll": "somebody'll",
"somebodys": "somebody's",
"someoned": "someone'd",
"someoned've": "someone'd've",
"someone'dve": "someone'd've",
"someonell": "someone'll",
"someones": "someone's",
"somethingd": "something'd",
"somethingd've": "something'd've",
"something'dve": "something'd've",
"somethingll": "something'll",
"thats": "that's",
"thered": "there'd",
"thered've": "there'd've",
"there'dve": "there'd've",
"therere": "there're",
"theres": "there's",
"theyd": "they'd",
"theyd've": "they'd've",
"they'dve": "they'd've",
"theyll": "they'll",
"theyre": "they're",
"theyve": "they've",
"twas": "'twas",
"wasnt": "wasn't",
"wed've": "we'd've",
"we'dve": "we'd've",
"weve": "we've",
"werent": "weren't",
"whatll": "what'll",
"whatre": "what're",
"whats": "what's",
"whatve": "what've",
"whens": "when's",
"whered": "where'd",
"wheres": "where's",
"whereve": "where've",
"whod": "who'd",
"whod've": "who'd've",
"who'dve": "who'd've",
"wholl": "who'll",
"whos": "who's",
"whove": "who've",
"whyll": "why'll",
"whyre": "why're",
"whys": "why's",
"wont": "won't",
"wouldve": "would've",
"wouldnt": "wouldn't",
"wouldnt've": "wouldn't've",
"wouldn'tve": "wouldn't've",
"yall": "y'all",
"yall'll": "y'all'll",
"y'allll": "y'all'll",
"yall'd've": "y'all'd've",
"y'alld've": "y'all'd've",
"y'all'dve": "y'all'd've",
"youd": "you'd",
"youd've": "you'd've",
"you'dve": "you'd've",
"youll": "you'll",
"youre": "you're",
"youve": "you've",
}

NUMBER_MAP = {
"none": "0",
"zero": "0",
"one": "1",
"two": "2",
"three": "3",
"four": "4",
"five": "5",
"six": "6",
"seven": "7",
"eight": "8",
"nine": "9",
"ten": "10",
}
ARTICLES = ["a", "an", "the"]
PERIOD_STRIP = re.compile(r"(?!<=\d)(\.)(?!\d)")
COMMA_STRIP = re.compile(r"(?<=\d)(\,)+(?=\d)")
PUNCTUATIONS = [
";",
r"/",
"[",
"]",
'"',
"{",
"}",
"(",
")",
"=",
"+",
"\\",
"_",
"-",
">",
"<",
"@",
"`",
",",
"?",
"!",
]

def __init__(self, *args, **kwargs):
pass

def word_tokenize(self, word):
word = word.lower()
word = word.replace(",", "").replace("?", "").replace("'s", " 's")
return word.strip()

def process_punctuation(self, in_text):
out_text = in_text
for p in self.PUNCTUATIONS:
if (p + " " in in_text or " " + p in in_text) or (re.search(self.COMMA_STRIP, in_text) is not None):
out_text = out_text.replace(p, "")
else:
out_text = out_text.replace(p, " ")
out_text = self.PERIOD_STRIP.sub("", out_text, re.UNICODE)
return out_text

def process_digit_article(self, in_text):
out_text = []
temp_text = in_text.lower().split()
for word in temp_text:
word = self.NUMBER_MAP.setdefault(word, word)
if word not in self.ARTICLES:
out_text.append(word)
else:
pass
for word_id, word in enumerate(out_text):
if word in self.CONTRACTIONS:
out_text[word_id] = self.CONTRACTIONS[word]
out_text = " ".join(out_text)
return out_text

def __call__(self, item):
item = self.word_tokenize(item)
item = item.replace("\n", " ").replace("\t", " ").strip()
item = self.process_punctuation(item)
item = self.process_digit_article(item)
return item


def vqav2_process_results(doc, result):
eval_ai_processor = EvalAIAnswerProcessor()
assert len(result) == 1, f"The result should be a list of length 1, but got {len(result)}."
resAns = eval_ai_processor(result[0])
accuracy = 0

if "answers" in doc and doc["answers"] is not None:
for ansDic in doc["answers"]:
ansDic["answer"] = ansDic["answer"].replace("\n", " ")
ansDic["answer"] = ansDic["answer"].replace("\t", " ")
ansDic["answer"] = ansDic["answer"].strip()
gtAcc = []
gtAnswers = [ans["answer"] for ans in doc["answers"]]

if len(set(gtAnswers)) > 1:
for ansDic in doc["answers"]:
ansDic["answer"] = eval_ai_processor.process_punctuation(ansDic["answer"])
ansDic["answer"] = eval_ai_processor.process_digit_article(ansDic["answer"])
resAns = eval_ai_processor.process_punctuation(resAns)
resAns = eval_ai_processor.process_digit_article(resAns)

for gtAnsDatum in doc["answers"]:
otherGTAns = [item for item in doc["answers"] if item != gtAnsDatum]
matchingAns = [item for item in otherGTAns if item["answer"] == resAns]
acc = min(1, float(len(matchingAns)) / 3)
gtAcc.append(acc)
accuracy = statistics.mean(gtAcc)

return {
"exact_match": accuracy,
"submission": {
"question_id": doc["question_id"],
"answer": resAns,
},
}


def vqav2_doc_to_text(doc):
return f"{doc['question']}\nAnswer the question using a single word or phrase."


def vqav2_aggreate_submissions(results):
now_date_time = datetime.datetime.now().strftime("%Y-%m-%d-%H-%M-%S")
submission_file_name = f"vqav2-val-submission-{now_date_time}.json"
path = os.path.abspath(submission_file_name)
with open(path, "w") as f:
json.dump(results, f)
print(f"Submission file saved to {path}")
return 0
24 changes: 24 additions & 0 deletions lmms_eval/tasks/vqav2_val/vqav2_val.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
task: "vqav2_val"
dataset_path: lmms-lab/VQAv2
dataset_kwargs:
token: True
test_split: validation
output_type: generate_until
doc_to_visual: !function utils.vqav2_doc_to_visual
doc_to_text: !function utils.vqav2_doc_to_text
doc_to_target: "answer"
generation_kwargs:
until:
- "ASSISTANT:"
metric_list:
- metric: exact_match
aggregation: mean
higher_is_better: true
ignore_case: true
ignore_punctuation: true
- metric: submission
aggregation: !function utils.vqav2_aggreate_submissions
higher_is_better: true
metadata:
- version: 0.0
process_results: !function utils.vqav2_process_results

0 comments on commit 54a1cbd

Please sign in to comment.