Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataset/update climate fever #1873

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Dataset/update climate fever #1873

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d7b6831
Updated ClimateFEVER dataset with new version
mina-parham Jan 25, 2025
42634ae
Adds Fill in the empty metadata.
mina-parham Jan 25, 2025
5511f49
Updates the date tuple
mina-parham Jan 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion mteb/models/gme_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
from __future__ import annotations

import logging
from functools import partial

from mteb.model_meta import ModelMeta

Expand Down
8 changes: 1 addition & 7 deletions mteb/models/lens_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,6 @@
from __future__ import annotations

from functools import partial

import torch

from mteb.encoder_interface import PromptType
from mteb.model_meta import ModelMeta, sentence_transformers_loader
from mteb.models.instruct_wrapper import instruct_wrapper
from mteb.model_meta import ModelMeta

from .bge_models import bge_full_data

Expand Down
36 changes: 36 additions & 0 deletions mteb/tasks/Retrieval/eng/ClimateFEVERRetrieval.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,3 +72,39 @@ class ClimateFEVERHardNegatives(AbsTaskRetrieval):
primaryClass={cs.CL}
}""",
)


class ClimateFEVERv2(AbsTaskRetrieval):
Copy link
Contributor

@KennethEnevoldsen KennethEnevoldsen Feb 2, 2025 •
edited by isaac-chung
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You will need to add supeseded_by to ClimateFEVER

If we want to consistently name tasks we should probably call this

Suggested change
class ClimateFEVERv2(AbsTaskRetrieval):
class ClimateFEVERRetrievalv2(AbsTaskRetrieval):

The same with the name

metadata = TaskMetadata(
name="ClimateFEVER.v2",
description="CLIMATE-FEVER is a dataset adopting the FEVER methodology that consists of 1,535 real-world claims regarding climate-change. ",
reference="https://www.sustainablefinance.uzh.ch/en/research/climate-fever.html",
dataset={
"path": "Mina76/climate-fever",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would love to move this over to the mteb org to ensure that it doesn't get taken down (I have sent you an invite to the org).

Not to say that you would do it, but it has happened sometimes (often people just cleaning up the datasets)

"revision": "e438c9586767800aeb10dbe8a245c41dbea4e5f4",
},
type="Retrieval",
category="s2p",
modalities=["text"],
eval_splits=["test"],
eval_langs=["eng-Latn"],
main_score="ndcg_at_10",
date=("2020-12-11", "2020-12-11"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The date should refer to when the source data was written. E.g. articles from the period 2014-2018.

domains=["Academic"],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
domains=["Academic"],
domains=["Academic", "Written"],

What is the source data of climate fever? Research articles? (would be great to update the description to make this clearer

task_subtypes=["Question answering"],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't it Claim Verification?

license="cc-by-sa-4.0",
annotations_creators="human-annotated",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the metadata is not filled out in the old one - could you move this up there as well?

dialect=[],
sample_creation="found",
bibtex_citation="""@misc{diggelmann2021climatefever,
title={CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims},
author={Thomas Diggelmann and Jordan Boyd-Graber and Jannis Bulian and Massimiliano Ciaramita and Markus Leippold},
year={2021},
eprint={2012.00614},
archivePrefix={arXiv},
primaryClass={cs.CL}
}""",
prompt={
"query": "Given a claim about climate change, retrieve documents that support or refute the claim"
},
)
Loading