diff --git a/docs/tasks.md b/docs/tasks.md index 481718bb94..c2ffc7bcbf 100644 --- a/docs/tasks.md +++ b/docs/tasks.md @@ -14,9 +14,9 @@ The following tables gives you an overview of the tasks in MTEB. | [AlloProfClusteringS2S](https://huggingface.co/datasets/lyon-nlp/alloprof) | {'fra'} | Clustering | s2s | | | | | [AlloprofReranking](https://huggingface.co/datasets/antoinelb7/alloprof) | {'fra'} | Reranking | s2s | | | | | [AlloprofRetrieval](https://huggingface.co/datasets/antoinelb7/alloprof) | {'fra'} | Retrieval | s2p | | | | -| [AmazonCounterfactualClassification](https://arxiv.org/abs/2104.06893) | {'deu', 'jpn', 'eng'} | Classification | s2s | | {'validation': 335, 'test': 670} | {'validation': 109.2, 'test': 106.1} | +| [AmazonCounterfactualClassification](https://arxiv.org/abs/2104.06893) | {'eng', 'deu', 'jpn'} | Classification | s2s | | {'validation': 335, 'test': 670} | {'validation': 109.2, 'test': 106.1} | | [AmazonPolarityClassification](https://huggingface.co/datasets/amazon_polarity) | {'eng'} | Classification | s2s | | {'test': 400000} | {'test': 431.4} | -| [AmazonReviewsClassification](https://arxiv.org/abs/2010.02573) | {'spa', 'jpn', 'cmn', 'eng', 'deu', 'fra'} | Classification | s2s | | {'validation': 30000, 'test': 30000} | {'validation': 159.2, 'test': 160.4} | +| [AmazonReviewsClassification](https://arxiv.org/abs/2010.02573) | {'cmn', 'fra', 'deu', 'jpn', 'eng', 'spa'} | Classification | s2s | | {'validation': 30000, 'test': 30000} | {'validation': 159.2, 'test': 160.4} | | [AngryTweetsClassification](https://aclanthology.org/2021.nodalida-main.53/) | {'dan'} | Classification | s2s | | {'test': 1050} | {'test': 156.1} | | [ArguAna](http://argumentation.bplaced.net/arguana/data) | {'eng'} | Retrieval | s2p | | | | | [ArguAna-PL](https://huggingface.co/datasets/clarin-knext/arguana-pl) | {'pol'} | Retrieval | s2p | | | | @@ -27,7 +27,7 @@ The following tables gives you an overview of the tasks in MTEB. | [BIOSSES](https://tabilab.cmpe.boun.edu.tr/BIOSSES/DataSet.html) | {'eng'} | STS | s2s | | | | | [BQ](https://aclanthology.org/2021.emnlp-main.357) | {'cmn'} | STS | s2s | | | | | [BSARDRetrieval](https://huggingface.co/datasets/maastrichtlawtech/bsard) | {'fra'} | Retrieval | s2p | | | | -| [BUCC](https://comparable.limsi.fr/bucc2018/bucc2018-task.html) | {'cmn', 'eng', 'deu', 'fra', 'rus'} | BitextMining | s2s | | {'test': 641684} | {'test': 101.3} | +| [BUCC](https://comparable.limsi.fr/bucc2018/bucc2018-task.html) | {'cmn', 'fra', 'deu', 'rus', 'eng'} | BitextMining | s2s | | {'test': 641684} | {'test': 101.3} | | [BambaraSentimentClassification](https://arxiv.org/abs/2009.08712) | {'mlt'} | Classification | s2s | [Reviews] | {'test': 673} | {'test': 29.4} | | [Banking77Classification](https://arxiv.org/abs/2003.04807) | {'eng'} | Classification | s2s | | {'test': 3080} | {'test': 54.2} | | [BengaliHateSpeechClassification](https://huggingface.co/datasets/bn_hate_speech) (Karim et al., 2020) | {'ben'} | Classification | s2s | [News] | {'train': 3418} | {'train': 103.42} | @@ -61,7 +61,7 @@ The following tables gives you an overview of the tasks in MTEB. | [ClimateFEVER](https://www.sustainablefinance.uzh.ch/en/research/climate-fever.html) | {'eng'} | Retrieval | s2p | | | | | [CmedqaRetrieval](https://aclanthology.org/2022.emnlp-main.357.pdf) | {'cmn'} | Retrieval | s2p | | | | | [Cmnli](https://huggingface.co/datasets/clue/viewer/cmnli) | {'cmn'} | PairClassification | s2s | | | | -| [CodeSearchNetRetrieval](https://huggingface.co/datasets/code_search_net/viewer) (Husain et al., 2019) | {'go', 'php', 'python', 'javascript', 'java', 'ruby'} | Retrieval | p2p | [Programming] | {'test': 1000} | {'test': 1196.4609} | +| [CodeSearchNetRetrieval](https://huggingface.co/datasets/code_search_net/viewer) (Husain et al., 2019) | {'go', 'java', 'ruby', 'python', 'javascript', 'php'} | Retrieval | p2p | [Programming] | {'test': 1000} | {'test': 1196.4609} | | [Core17InstructionRetrieval](https://arxiv.org/abs/2403.15246) (Orion Weller, 2024) | {'eng'} | InstructionRetrieval | s2p | [News] | {'eng': 39470} | {'eng': 2747.2883966244726} | | [CovidRetrieval](https://arxiv.org/abs/2203.03367) | {'cmn'} | Retrieval | s2p | | | | | [CroatianSentimentClassification](https://arxiv.org/abs/2009.08712) | {'hrv'} | Classification | s2s | [Reviews] | {'validation': 214, 'test': 437} | {'validation': 166.9, 'test': 151.4} | @@ -72,7 +72,7 @@ The following tables gives you an overview of the tasks in MTEB. | [DalajClassification](https://spraakbanken.gu.se/en/resources/superlim) | {'dan'} | Classification | s2s | | {'test': 444} | {'test': 243.8} | | [DanFEVER](https://aclanthology.org/2021.nodalida-main.47/) | {'dan'} | Retrieval | p2p | [Encyclopaedic, Non-fiction] | {'train': 8897} | {'train': 124.84} | | [DanishPoliticalCommentsClassification](https://huggingface.co/datasets/danish_political_comments) | {'dan'} | Classification | s2s | | {'train': 9010} | {'train': 69.9} | -| [DiaBlaBitextMining](https://inria.hal.science/hal-03021633) | {'fra', 'eng'} | BitextMining | s2s | | | | +| [DiaBlaBitextMining](https://inria.hal.science/hal-03021633) | {'eng', 'fra'} | BitextMining | s2s | | | | | [DuRetrieval](https://aclanthology.org/2022.emnlp-main.357.pdf) | {'cmn'} | Retrieval | s2p | | | | | [DutchBookReviewSentimentClassification](https://github.com/benjaminvdb/DBRD) (Benjamin et al., 2019) | {'nld'} | Classification | s2s | [Reviews] | {'test': 2224} | {'test': 1443.0} | | [EcomRetrieval](https://arxiv.org/abs/2203.03367) | {'cmn'} | Retrieval | s2p | | | | @@ -87,7 +87,7 @@ The following tables gives you an overview of the tasks in MTEB. | [FiQA2018](https://sites.google.com/view/fiqa/) | {'eng'} | Retrieval | s2p | | | | | [FilipinoHateSpeechClassification](https://pcj.csp.org.ph/index.php/pcj/issue/download/29/PCJ%20V14%20N1%20pp1-14%202019) (Neil Vicente Cabasag et al., 2019) | {'fil'} | Classification | s2s | [Social] | {'validation': 2048, 'test': 2048} | {'validation': 88.1, 'test': 87.4} | | [FinParaSTS](https://huggingface.co/datasets/TurkuNLP/turku_paraphrase_corpus) | {'fin'} | STS | s2s | [News, Subtitles] | {'test': 1000, 'validation': 1000} | {'test': 59.0, 'validation': 58.8} | -| [FloresBitextMining](https://huggingface.co/datasets/facebook/flores) | {'afr', 'ces', 'bel', 'zul', 'ita', 'slv', 'snd', 'kor', 'fur', 'tzm', 'bjn', 'ory', 'hin', 'war', 'bos', 'zho', 'spa', 'mai', 'bod', 'tir', 'azj', 'yor', 'lmo', 'vec', 'kam', 'knc', 'tha', 'bak', 'som', 'umb', 'tpi', 'ron', 'nso', 'grn', 'min', 'bam', 'uig', 'sna', 'tso', 'fin', 'bul', 'lua', 'fra', 'kat', 'mag', 'lin', 'pes', 'azb', 'kaz', 'pap', 'kin', 'tgk', 'guj', 'ast', 'sat', 'kas', 'ars', 'mlt', 'isl', 'kmr', 'fao', 'kea', 'luo', 'nob', 'twi', 'ban', 'mni', 'lij', 'mya', 'lim', 'jav', 'quy', 'cat', 'mri', 'gaz', 'slk', 'ibo', 'apc', 'jpn', 'fij', 'zsm', 'sag', 'lao', 'tur', 'tat', 'ltz', 'sin', 'ajp', 'bho', 'szl', 'run', 'kbp', 'glg', 'tam', 'acq', 'sun', 'ckb', 'kir', 'ell', 'ben', 'mkd', 'kan', 'cjk', 'scn', 'pag', 'por', 'khk', 'lit', 'ilo', 'arb', 'lug', 'aeb', 'tel', 'ary', 'deu', 'yue', 'amh', 'hne', 'aka', 'gla', 'awa', 'nus', 'dik', 'nno', 'hrv', 'hun', 'pbt', 'ssw', 'fuv', 'npi', 'asm', 'ltg', 'lvs', 'tum', 'rus', 'crh', 'eus', 'nld', 'khm', 'kon', 'ewe', 'heb', 'ace', 'sot', 'fon', 'mar', 'srp', 'hau', 'san', 'swh', 'arz', 'eng', 'smo', 'bug', 'ukr', 'tuk', 'lus', 'pan', 'hye', 'mos', 'dyu', 'als', 'wol', 'bem', 'mal', 'kmb', 'swe', 'xho', 'gle', 'uzn', 'plt', 'vie', 'oci', 'shn', 'taq', 'nya', 'ydd', 'epo', 'urd', 'ind', 'ceb', 'ayr', 'kab', 'pol', 'dzo', 'acm', 'srd', 'cym', 'kik', 'prs', 'tsn', 'kac', 'dan', 'est', 'hat', 'tgl'} | BitextMining | s2s | | {'dev': 997, 'devtest': 1012} | | +| [FloresBitextMining](https://huggingface.co/datasets/facebook/flores) | {'srp', 'yue', 'ltz', 'pap', 'ita', 'deu', 'ary', 'dzo', 'quy', 'lit', 'acm', 'scn', 'bem', 'kan', 'npi', 'fuv', 'mri', 'bjn', 'azb', 'apc', 'kon', 'swh', 'aeb', 'mar', 'sin', 'nus', 'lua', 'mos', 'ars', 'luo', 'dan', 'plt', 'tsn', 'uzn', 'eng', 'tum', 'cym', 'mal', 'fur', 'tha', 'cat', 'ydd', 'kik', 'dyu', 'mai', 'aka', 'eus', 'oci', 'wol', 'mlt', 'zho', 'run', 'kas', 'pan', 'pag', 'ckb', 'nld', 'ell', 'kea', 'hun', 'nno', 'lao', 'hrv', 'tel', 'urd', 'uig', 'ltg', 'kat', 'lus', 'tzm', 'fin', 'mag', 'bug', 'slk', 'lij', 'arz', 'kab', 'lmo', 'als', 'zsm', 'guj', 'som', 'tur', 'knc', 'ayr', 'ilo', 'szl', 'yor', 'fao', 'lug', 'bos', 'san', 'gaz', 'war', 'awa', 'epo', 'khm', 'ces', 'fra', 'lvs', 'kor', 'afr', 'hne', 'lim', 'shn', 'ast', 'gle', 'snd', 'prs', 'kmr', 'tuk', 'gla', 'ssw', 'est', 'dik', 'heb', 'ind', 'ajp', 'vec', 'bod', 'fon', 'por', 'bak', 'smo', 'spa', 'arb', 'jav', 'mya', 'pol', 'ceb', 'twi', 'tir', 'grn', 'tpi', 'xho', 'slv', 'sag', 'tso', 'sna', 'hin', 'sun', 'mni', 'tat', 'sat', 'hau', 'kin', 'hat', 'ben', 'kmb', 'asm', 'sot', 'ory', 'kam', 'amh', 'cjk', 'ewe', 'jpn', 'azj', 'khk', 'kbp', 'bho', 'hye', 'kac', 'zul', 'nso', 'min', 'rus', 'taq', 'kaz', 'lin', 'crh', 'pes', 'nya', 'bul', 'isl', 'glg', 'srd', 'nob', 'ban', 'bam', 'acq', 'ron', 'ibo', 'umb', 'vie', 'mkd', 'ukr', 'tgk', 'tgl', 'kir', 'fij', 'tam', 'pbt', 'bel', 'swe', 'ace'} | BitextMining | s2s | | {'dev': 997, 'devtest': 1012} | | | [FloresClusteringS2S](https://huggingface.co/datasets/facebook/flores) | {'spa'} | Clustering | s2s | | | | | [GerDaLIR](https://github.com/lavis-nlp/GerDaLIR) | {'deu'} | Retrieval | s2p | | | | | [GerDaLIRSmall](https://github.com/lavis-nlp/GerDaLIR) | {'deu'} | Retrieval | p2p | [Legal] | | | @@ -105,12 +105,12 @@ The following tables gives you an overview of the tasks in MTEB. | [HotpotQA-PL](https://hotpotqa.github.io/) | {'pol'} | Retrieval | s2p | | | | | [HunSum2AbstractiveRetrieval](https://arxiv.org/abs/2404.03555) (Botond Barta, 2024) | {'hun'} | Retrieval | s2p | [News] | {'test': 1998} | {'test': 2462.2177177177177} | | [IFlyTek](https://www.cluebenchmarks.com/introduce.html) | {'cmn'} | Classification | s2s | | | | -| [IN22ConvBitextMining](https://huggingface.co/datasets/ai4bharat/IN22-Conv) (Jay Gala, 2023) | {'san', 'tam', 'guj', 'snd', 'eng', 'sat', 'brx', 'kas', 'ory', 'ben', 'kan', 'hin', 'pan', 'mai', 'tel', 'mal', 'doi', 'mni', 'asm', 'npi', 'urd', 'gom', 'mar'} | BitextMining | s2s | [Social, Spoken, Fiction] | {'conv': 1503} | {'conv': 54.3} | -| [IN22GenBitextMining](https://huggingface.co/datasets/ai4bharat/IN22-Gen) (Jay Gala, 2023) | {'san', 'tam', 'guj', 'snd', 'eng', 'sat', 'brx', 'kas', 'ory', 'ben', 'kan', 'hin', 'pan', 'mai', 'tel', 'mal', 'doi', 'mni', 'asm', 'npi', 'urd', 'gom', 'mar'} | BitextMining | s2s | [Web, Legal, Government, News, Religious, Non-fiction] | {'gen': 1024} | {'gen': 156.7} | +| [IN22ConvBitextMining](https://huggingface.co/datasets/ai4bharat/IN22-Conv) (Jay Gala, 2023) | {'kas', 'pan', 'snd', 'kan', 'npi', 'brx', 'tel', 'urd', 'mar', 'hin', 'eng', 'guj', 'mni', 'gom', 'sat', 'mal', 'doi', 'ben', 'asm', 'ory', 'tam', 'san', 'mai'} | BitextMining | s2s | [Social, Spoken, Fiction] | {'conv': 1503} | {'conv': 54.3} | +| [IN22GenBitextMining](https://huggingface.co/datasets/ai4bharat/IN22-Gen) (Jay Gala, 2023) | {'kas', 'pan', 'snd', 'kan', 'npi', 'brx', 'tel', 'urd', 'mar', 'hin', 'eng', 'guj', 'mni', 'gom', 'sat', 'mal', 'doi', 'ben', 'asm', 'ory', 'tam', 'san', 'mai'} | BitextMining | s2s | [Web, Legal, Government, News, Religious, Non-fiction] | {'gen': 1024} | {'gen': 156.7} | | [ImdbClassification](http://www.aclweb.org/anthology/P11-1015) | {'eng'} | Classification | p2p | | {'test': 25000} | {'test': 1293.8} | -| [IndicCrosslingualSTS](https://huggingface.co/datasets/jaygala24/indic_sts) (Ramesh et al., 2022) | {'tam', 'guj', 'asm', 'eng', 'tel', 'mal', 'ory', 'ben', 'kan', 'mar', 'urd', 'hin', 'pan'} | STS | s2s | [News, Non-fiction, Web, Spoken, Government] | {'test': 10020} | {'test': 76.22} | -| [IndicLangClassification](https://arxiv.org/abs/2305.15814) | {'san', 'tam', 'guj', 'snd', 'sat', 'brx', 'kas', 'ory', 'ben', 'kan', 'hin', 'pan', 'mai', 'tel', 'mal', 'doi', 'mni', 'asm', 'npi', 'urd', 'gom', 'mar'} | Classification | s2s | [Web, Non-fiction] | {'test': 30418} | {'test': 106.5} | -| [IndicSentimentClassification](https://arxiv.org/abs/2212.05409) (Sumanth Doddapaneni, 2022) | {'tam', 'guj', 'asm', 'tel', 'brx', 'mal', 'ory', 'ben', 'kan', 'mar', 'urd', 'hin', 'pan'} | Classification | s2s | [Reviews] | {'test': 1000} | {'test': 137.6} | +| [IndicCrosslingualSTS](https://huggingface.co/datasets/jaygala24/indic_sts) (Ramesh et al., 2022) | {'kan', 'mar', 'ben', 'asm', 'ory', 'tel', 'urd', 'tam', 'pan', 'hin', 'eng', 'guj', 'mal'} | STS | s2s | [News, Non-fiction, Web, Spoken, Government] | {'test': 10020} | {'test': 76.22} | +| [IndicLangClassification](https://arxiv.org/abs/2305.15814) | {'kas', 'pan', 'snd', 'kan', 'npi', 'brx', 'tel', 'urd', 'mar', 'hin', 'guj', 'mni', 'gom', 'sat', 'mal', 'doi', 'ben', 'asm', 'ory', 'tam', 'san', 'mai'} | Classification | s2s | [Web, Non-fiction] | {'test': 30418} | {'test': 106.5} | +| [IndicSentimentClassification](https://arxiv.org/abs/2212.05409) (Sumanth Doddapaneni, 2022) | {'kan', 'mar', 'ben', 'brx', 'asm', 'ory', 'tel', 'tam', 'pan', 'hin', 'urd', 'guj', 'mal'} | Classification | s2s | [Reviews] | {'test': 1000} | {'test': 137.6} | | [IndonesianIdClickbaitClassification](http://www.sciencedirect.com/science/article/pii/S2352340920311252) | {'ind'} | Classification | s2s | [News] | {'train': 2048} | {'train': 64.28} | | [IsiZuluNewsClassification](https://huggingface.co/datasets/dsfsi/za-isizulu-siswati-news) (Madodonga et al., 2023) | {'zul'} | Classification | s2s | [News] | {'train': 752} | {'train': 43.1} | | [ItaHateClassification](https://aclanthology.org/2022.woah-1.15/) | {'ita'} | Classification | s2s | [Constructed] | {'test': 1845} | {'test': 50.4} | @@ -145,48 +145,48 @@ The following tables gives you an overview of the tasks in MTEB. | [MSMARCO](https://microsoft.github.io/msmarco/) | {'eng'} | Retrieval | s2p | | | | | [MSMARCO-PL](https://microsoft.github.io/msmarco/) | {'pol'} | Retrieval | s2p | | | | | [MSMARCOv2](https://microsoft.github.io/msmarco/TREC-Deep-Learning.html) | {'eng'} | Retrieval | s2p | | | | -| [MTOPDomainClassification](https://arxiv.org/pdf/2008.09335.pdf) | {'spa', 'tha', 'eng', 'deu', 'fra', 'hin'} | Classification | s2s | | {'validation': 2235, 'test': 4386} | {'validation': 36.5, 'test': 36.8} | -| [MTOPIntentClassification](https://arxiv.org/pdf/2008.09335.pdf) | {'spa', 'tha', 'eng', 'deu', 'fra', 'hin'} | Classification | s2s | | {'validation': 2235, 'test': 4386} | {'validation': 36.5, 'test': 36.8} | +| [MTOPDomainClassification](https://arxiv.org/pdf/2008.09335.pdf) | {'tha', 'fra', 'deu', 'hin', 'eng', 'spa'} | Classification | s2s | | {'validation': 2235, 'test': 4386} | {'validation': 36.5, 'test': 36.8} | +| [MTOPIntentClassification](https://arxiv.org/pdf/2008.09335.pdf) | {'tha', 'fra', 'deu', 'hin', 'eng', 'spa'} | Classification | s2s | | {'validation': 2235, 'test': 4386} | {'validation': 36.5, 'test': 36.8} | | [MacedonianTweetSentimentClassification](https://aclanthology.org/R15-1034/) | {'mkd'} | Classification | s2s | [Social] | {'test': 1139} | {'test': 67.6} | | [MalteseSentimentClassification](https://arxiv.org/abs/2009.08712) | {'mlt'} | Classification | s2s | [Reviews] | {'validation': 85, 'test': 171} | {'validation': 119.7, 'test': 132.4} | -| [MasakhaNEWSClassification](https://arxiv.org/abs/2304.09972) | {'ibo', 'run', 'sna', 'som', 'lug', 'tir', 'eng', 'fra', 'orm', 'yor', 'amh', 'lin', 'pcm', 'hau', 'swa', 'xho'} | Classification | s2s | | {'test': 422} | {'test': 5116.6} | -| [MasakhaNEWSClusteringP2P](https://huggingface.co/datasets/masakhane/masakhanews) | {'ibo', 'run', 'sna', 'som', 'lug', 'tir', 'eng', 'fra', 'orm', 'yor', 'amh', 'lin', 'pcm', 'hau', 'swa', 'xho'} | Clustering | p2p | | | | -| [MasakhaNEWSClusteringS2S](https://huggingface.co/datasets/masakhane/masakhanews) | {'ibo', 'run', 'sna', 'som', 'lug', 'tir', 'eng', 'fra', 'orm', 'yor', 'amh', 'lin', 'pcm', 'hau', 'swa', 'xho'} | Clustering | s2s | | | | -| [MassiveIntentClassification](https://arxiv.org/abs/2204.08582#:~:text=MASSIVE%20contains%201M%20realistic%2C%20parallel,diverse%20languages%20from%2029%20genera.) | {'afr', 'ita', 'slv', 'tam', 'kor', 'eng', 'ben', 'ell', 'kan', 'isl', 'hin', 'por', 'aze', 'hye', 'msa', 'spa', 'nob', 'fas', 'sqi', 'tel', 'deu', 'mal', 'amh', 'swe', 'swa', 'mya', 'tha', 'ara', 'lav', 'vie', 'jav', 'hun', 'mon', 'ron', 'urd', 'ind', 'rus', 'cmo', 'nld', 'khm', 'pol', 'jpn', 'fin', 'heb', 'cym', 'fra', 'kat', 'dan', 'tur', 'tgl'} | Classification | s2s | | {'validation': 2033, 'test': 2974} | {'validation': 34.8, 'test': 34.6} | -| [MassiveScenarioClassification](https://arxiv.org/abs/2204.08582#:~:text=MASSIVE%20contains%201M%20realistic%2C%20parallel,diverse%20languages%20from%2029%20genera.) | {'afr', 'ita', 'slv', 'tam', 'kor', 'eng', 'ben', 'ell', 'kan', 'isl', 'hin', 'por', 'aze', 'hye', 'msa', 'spa', 'nob', 'fas', 'sqi', 'tel', 'deu', 'mal', 'amh', 'swe', 'swa', 'mya', 'tha', 'ara', 'lav', 'vie', 'jav', 'hun', 'mon', 'ron', 'urd', 'ind', 'rus', 'cmo', 'nld', 'khm', 'pol', 'jpn', 'fin', 'heb', 'cym', 'fra', 'kat', 'dan', 'tur', 'tgl'} | Classification | s2s | | {'validation': 2033, 'test': 2974} | {'validation': 34.8, 'test': 34.6} | +| [MasakhaNEWSClassification](https://arxiv.org/abs/2304.09972) | {'ibo', 'lin', 'fra', 'orm', 'swa', 'tir', 'som', 'xho', 'amh', 'run', 'lug', 'sna', 'eng', 'yor', 'pcm', 'hau'} | Classification | s2s | | {'test': 422} | {'test': 5116.6} | +| [MasakhaNEWSClusteringP2P](https://huggingface.co/datasets/masakhane/masakhanews) | {'ibo', 'lin', 'fra', 'orm', 'swa', 'tir', 'som', 'xho', 'amh', 'run', 'lug', 'sna', 'eng', 'yor', 'pcm', 'hau'} | Clustering | p2p | | | | +| [MasakhaNEWSClusteringS2S](https://huggingface.co/datasets/masakhane/masakhanews) | {'ibo', 'lin', 'fra', 'orm', 'swa', 'tir', 'som', 'xho', 'amh', 'run', 'lug', 'sna', 'eng', 'yor', 'pcm', 'hau'} | Clustering | s2s | | | | +| [MassiveIntentClassification](https://arxiv.org/abs/2204.08582#:~:text=MASSIVE%20contains%201M%20realistic%2C%20parallel,diverse%20languages%20from%2029%20genera.) | {'lav', 'khm', 'hye', 'fra', 'swa', 'deu', 'fas', 'ita', 'kor', 'afr', 'cmo', 'kan', 'mon', 'msa', 'ell', 'nld', 'heb', 'hun', 'ind', 'tel', 'swe', 'urd', 'sqi', 'por', 'spa', 'kat', 'isl', 'fin', 'aze', 'jav', 'mya', 'pol', 'dan', 'nob', 'slv', 'hin', 'eng', 'cym', 'mal', 'ron', 'tha', 'tur', 'vie', 'tgl', 'ben', 'ara', 'tam', 'amh', 'rus', 'jpn'} | Classification | s2s | | {'validation': 2033, 'test': 2974} | {'validation': 34.8, 'test': 34.6} | +| [MassiveScenarioClassification](https://arxiv.org/abs/2204.08582#:~:text=MASSIVE%20contains%201M%20realistic%2C%20parallel,diverse%20languages%20from%2029%20genera.) | {'lav', 'khm', 'hye', 'fra', 'swa', 'deu', 'fas', 'ita', 'kor', 'afr', 'cmo', 'kan', 'mon', 'msa', 'ell', 'nld', 'heb', 'hun', 'ind', 'tel', 'swe', 'urd', 'sqi', 'por', 'spa', 'kat', 'isl', 'fin', 'aze', 'jav', 'mya', 'pol', 'dan', 'nob', 'slv', 'hin', 'eng', 'cym', 'mal', 'ron', 'tha', 'tur', 'vie', 'tgl', 'ben', 'ara', 'tam', 'amh', 'rus', 'jpn'} | Classification | s2s | | {'validation': 2033, 'test': 2974} | {'validation': 34.8, 'test': 34.6} | | [MedicalQARetrieval](https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-019-3119-4) (Asma et al., 2019) | {'eng'} | Retrieval | s2s | [Medical] | {'test': 2048} | {'test': 1205.9619140625} | | [MedicalRetrieval](https://arxiv.org/abs/2203.03367) | {'cmn'} | Retrieval | s2p | | | | | [MedrxivClusteringP2P](https://api.medrxiv.org/) | {'eng'} | Clustering | p2p | | {'test': 375000} | {'test': 1981.2} | | [MedrxivClusteringS2S](https://api.medrxiv.org/) | {'eng'} | Clustering | s2s | | {'test': 375000} | {'test': 114.7} | | [MindSmallReranking](https://msnews.github.io/assets/doc/ACL2020_MIND.pdf) | {'eng'} | Reranking | s2s | | {'test': 107968} | {'test': 70.9} | -| MintakaRetrieval | {'spa', 'jpn', 'ita', 'ara', 'deu', 'fra', 'hin', 'por'} | Retrieval | s2p | | | | +| MintakaRetrieval | {'ara', 'fra', 'deu', 'ita', 'hin', 'jpn', 'por', 'spa'} | Retrieval | s2p | | | | | [MovieReviewSentimentClassification](https://github.com/TheophileBlard/french-sentiment-analysis-with-bert) (Théophile Blard, 2020) | {'fra'} | Classification | s2s | [Reviews] | {'validation': 1024, 'test': 1024} | {'validation': 550.3, 'test': 558.1} | -| [MultiLongDocRetrieval](https://arxiv.org/abs/2402.03216) (Jianlv Chen, 2024) | {'spa', 'jpn', 'ita', 'tha', 'ara', 'kor', 'cmn', 'eng', 'deu', 'fra', 'rus', 'hin', 'por'} | Retrieval | s2p | | | | +| [MultiLongDocRetrieval](https://arxiv.org/abs/2402.03216) (Jianlv Chen, 2024) | {'tha', 'cmn', 'ara', 'fra', 'deu', 'ita', 'rus', 'jpn', 'kor', 'hin', 'eng', 'por', 'spa'} | Retrieval | s2p | | | | | [MultilingualSentiment](https://github.com/tyqiangz/multilingual-sentiment-datasets) | {'cmn'} | Classification | s2s | | | | | [NFCorpus](https://www.cl.uni-heidelberg.de/statnlpgroup/nfcorpus/) | {'eng'} | Retrieval | s2p | | | | | [NFCorpus-PL](https://www.cl.uni-heidelberg.de/statnlpgroup/nfcorpus/) | {'pol'} | Retrieval | s2p | | | | | [NQ](https://ai.google.com/research/NaturalQuestions/) | {'eng'} | Retrieval | s2p | | | | | [NQ-PL](https://ai.google.com/research/NaturalQuestions/) | {'pol'} | Retrieval | s2p | | | | -| [NTREXBitextMining](https://huggingface.co/datasets/xianf/NTREX) | {'spa', 'jpn', 'ita', 'tha', 'ara', 'vie', 'kor', 'eng', 'deu', 'fra', 'ind', 'rus', 'tur', 'hin', 'por', 'zho'} | BitextMining | s2s | [News] | {'train': 1997} | {'train': 120.0} | +| [NTREXBitextMining](https://huggingface.co/datasets/xianf/NTREX) | {'vie', 'tha', 'tur', 'ara', 'zho', 'fra', 'deu', 'ita', 'rus', 'ind', 'jpn', 'kor', 'hin', 'eng', 'por', 'spa'} | BitextMining | s2s | [News] | {'train': 1997} | {'train': 120.0} | | [NarrativeQARetrieval](https://metatext.io/datasets/narrativeqa) | {'eng'} | Retrieval | s2p | | | | | [NepaliNewsClassification](https://github.com/goru001/nlp-for-nepali) | {'nep'} | Classification | s2s | [News] | {'train': 5975, 'test': 1495} | {'train': 196.61, 'test': 196.017} | -| [NeuCLIR2022Retrieval](https://neuclir.github.io/) (Lawrie et al., 2023) | {'fas', 'rus', 'zho'} | Retrieval | s2p | [News] | {'fas': 2232130, 'zho': 3179323, 'rus': 4627657} | {'fas': 3500.5143969099317, 'zho': 2543.1140667919617, 'rus': 3214.755239654659} | -| [NeuCLIR2023Retrieval](https://neuclir.github.io/) (Dawn Lawrie, 2024) | {'fas', 'rus', 'zho'} | Retrieval | s2p | [News] | {'fas': 2232092, 'zho': 3179285, 'rus': 4627619} | {'fas': 3579.508213937439, 'zho': 2704.44834488453, 'rus': 3466.8192213553616} | +| [NeuCLIR2022Retrieval](https://neuclir.github.io/) (Lawrie et al., 2023) | {'zho', 'fas', 'rus'} | Retrieval | s2p | [News] | {'fas': 2232130, 'zho': 3179323, 'rus': 4627657} | {'fas': 3500.5143969099317, 'zho': 2543.1140667919617, 'rus': 3214.755239654659} | +| [NeuCLIR2023Retrieval](https://neuclir.github.io/) (Dawn Lawrie, 2024) | {'zho', 'fas', 'rus'} | Retrieval | s2p | [News] | {'fas': 2232092, 'zho': 3179285, 'rus': 4627619} | {'fas': 3579.508213937439, 'zho': 2704.44834488453, 'rus': 3466.8192213553616} | | [News21InstructionRetrieval](https://arxiv.org/abs/2403.15246) (Orion Weller, 2024) | {'eng'} | InstructionRetrieval | s2p | [News] | {'eng': 60258} | {'eng': 2331.381203215969} | | [NewsClassification](https://arxiv.org/abs/1509.01626) | {'eng'} | Classification | s2s | [News] | {'test': 7600} | {'test': 235.29} | | [NoRecClassification](https://aclanthology.org/L18-1661/) | {'nob'} | Classification | s2s | | {'test': 2050} | {'test': 82.0} | | [NorQuadRetrieval](https://aclanthology.org/2023.nodalida-1.17/) | {'nob'} | Retrieval | p2p | [Encyclopaedic, Non-fiction] | {'test': 2602} | {'test': 502.19} | -| [NordicLangClassification](https://aclanthology.org/2021.vardial-1.8/) | {'nob', 'nno', 'dan', 'isl', 'swe', 'fao'} | Classification | s2s | | {'test': 3000} | {'test': 78.2} | -| [NorwegianCourtsBitextMining](https://opus.nlpl.eu/ELRC-Courts_Norway-v1.php) | {'nob', 'nno'} | BitextMining | s2s | [Spoken, Legal] | {'test': 456} | {'test': 82.11} | +| [NordicLangClassification](https://aclanthology.org/2021.vardial-1.8/) | {'isl', 'dan', 'nob', 'nno', 'fao', 'swe'} | Classification | s2s | | {'test': 3000} | {'test': 78.2} | +| [NorwegianCourtsBitextMining](https://opus.nlpl.eu/ELRC-Courts_Norway-v1.php) | {'nno', 'nob'} | BitextMining | s2s | [Spoken, Legal] | {'test': 456} | {'test': 82.11} | | [NorwegianCourtsBitextMining](https://opus.nlpl.eu/index.php) | {'nob', 'nno'} | BitextMining | s2s | | {'test': 2050} | {'test': 1884.0} | | [NorwegianParliamentClassification](https://huggingface.co/datasets/NbAiLab/norwegian_parliament) | {'nob'} | Classification | s2s | | {'test': 1200, 'validation': 1200} | {'test': 1884.0, 'validation': 1911.0} | | [Ocnli](https://arxiv.org/abs/2010.05444) | {'cmn'} | PairClassification | s2s | | | | | [OnlineShopping](https://aclanthology.org/2023.nodalida-1.20/) | {'cmn'} | Classification | s2s | | | | -| [OpusparcusPC](https://gem-benchmark.com/data_cards/opusparcus) | {'fin', 'eng', 'deu', 'fra', 'rus', 'swe'} | PairClassification | s2s | | | | +| [OpusparcusPC](https://gem-benchmark.com/data_cards/opusparcus) | {'fin', 'fra', 'deu', 'rus', 'eng', 'swe'} | PairClassification | s2s | | | | | [PAC](https://arxiv.org/pdf/2211.13112.pdf) | {'pol'} | Classification | p2p | | {'test': 3453} | {'test': 185.3} | | [PAWSX](https://aclanthology.org/2021.emnlp-main.357) | {'cmn'} | STS | s2s | | | | | [PSC](http://www.lrec-conf.org/proceedings/lrec2014/pdf/1211_Paper.pdf) | {'pol'} | PairClassification | s2s | | | | -| [PawsX](https://arxiv.org/abs/1908.11828) | {'spa', 'jpn', 'kor', 'cmn', 'eng', 'deu', 'fra'} | PairClassification | s2s | | | | +| [PawsX](https://arxiv.org/abs/1908.11828) | {'cmn', 'fra', 'deu', 'jpn', 'kor', 'eng', 'spa'} | PairClassification | s2s | | | | | [PersianFoodSentimentClassification](https://hooshvare.github.io/docs/datasets/sa) (Mehrdad Farahani et al., 2020) | {'fas'} | Classification | s2s | [Reviews] | {'validation': 2048, 'test': 2048} | {'validation': 90.37, 'test': 90.58} | | [PolEmo2.0-IN](https://aclanthology.org/K19-1092.pdf) | {'pol'} | Classification | s2s | | | | | [PolEmo2.0-OUT](https://aclanthology.org/K19-1092.pdf) | {'pol'} | Classification | s2s | | {'test': 722} | {'test': 756.2} | @@ -216,11 +216,11 @@ The following tables gives you an overview of the tasks in MTEB. | [STS14](https://www.aclweb.org/anthology/S14-1002) | {'eng'} | STS | s2s | | | | | [STS15](https://www.aclweb.org/anthology/S15-2010) | {'eng'} | STS | s2s | | | | | [STS16](https://www.aclweb.org/anthology/S16-1001) | {'eng'} | STS | s2s | | | | -| [STS17](http://alt.qcri.org/semeval2016/task1/) | {'spa', 'ita', 'ara', 'kor', 'eng', 'deu', 'fra', 'tur', 'nld'} | STS | s2s | | {'test': 500} | {'test': 43.3} | -| [STS22](https://competitions.codalab.org/competitions/33835) | {'spa', 'ita', 'ara', 'cmn', 'eng', 'deu', 'fra', 'tur', 'rus', 'pol'} | STS | p2p | | {'test': 8060} | {'train': 1992.8} | +| [STS17](http://alt.qcri.org/semeval2016/task1/) | {'tur', 'nld', 'ara', 'fra', 'deu', 'ita', 'kor', 'eng', 'spa'} | STS | s2s | | {'test': 500} | {'test': 43.3} | +| [STS22](https://competitions.codalab.org/competitions/33835) | {'tur', 'cmn', 'pol', 'ara', 'fra', 'deu', 'ita', 'rus', 'eng', 'spa'} | STS | p2p | | {'test': 8060} | {'train': 1992.8} | | [STSB](https://aclanthology.org/2021.emnlp-main.357) | {'cmn'} | STS | s2s | | | | | [STSBenchmark](https://github.com/PhilipMay/stsb-multi-mt/) | {'eng'} | STS | s2s | | | | -| [STSBenchmarkMultilingualSTS](https://github.com/PhilipMay/stsb-multi-mt/) | {'spa', 'ita', 'cmn', 'eng', 'deu', 'fra', 'rus', 'por', 'nld', 'pol'} | STS | s2s | | | | +| [STSBenchmarkMultilingualSTS](https://github.com/PhilipMay/stsb-multi-mt/) | {'nld', 'cmn', 'pol', 'fra', 'deu', 'ita', 'rus', 'eng', 'por', 'spa'} | STS | s2s | | | | | [STSES](https://huggingface.co/datasets/PlanTL-GOB-ES/sts-es) | {'spa'} | STS | s2s | | | | | [ScalaDaClassification](https://aclanthology.org/2023.nodalida-1.20/) | {'dan'} | Classification | s2s | | {'test': 1024} | {'test': 109.4} | | [ScalaNbClassification](https://aclanthology.org/2023.nodalida-1.20/) | {'nob'} | Classification | s2s | | {'test': 1024} | {'test': 98.4} | @@ -253,7 +253,7 @@ The following tables gives you an overview of the tasks in MTEB. | [TRECCOVID](https://ir.nist.gov/covidSubmit/index.html) | {'eng'} | Retrieval | s2p | | | | | [TRECCOVID-PL](https://ir.nist.gov/covidSubmit/index.html) | {'pol'} | Retrieval | s2p | | | | | [TV2Nordretrieval](https://huggingface.co/datasets/alexandrainst/nordjylland-news-summarization) | {'dan'} | Retrieval | p2p | [News, Non-fiction] | {'test': 4096} | {'test': 784.11} | -| [Tatoeba](https://github.com/facebookresearch/LASER/tree/main/data/tatoeba/v1) | {'afr', 'arq', 'bel', 'ces', 'ita', 'slv', 'kor', 'max', 'fry', 'war', 'hin', 'bos', 'pms', 'spa', 'uzb', 'swg', 'ang', 'tha', 'cmn', 'ron', 'kzj', 'uig', 'fin', 'bul', 'fra', 'kat', 'pes', 'kaz', 'ast', 'tzl', 'hsb', 'isl', 'fao', 'aze', 'dsb', 'nob', 'bre', 'jav', 'mon', 'ile', 'nov', 'cbk', 'cat', 'slk', 'jpn', 'zsm', 'tur', 'tat', 'ber', 'orv', 'glg', 'tam', 'ben', 'gsw', 'ell', 'mkd', 'por', 'lit', 'tel', 'yid', 'deu', 'yue', 'amh', 'gla', 'awa', 'nds', 'nno', 'ara', 'hrv', 'hun', 'lvs', 'pam', 'rus', 'cha', 'eus', 'nld', 'kur', 'khm', 'heb', 'mar', 'srp', 'lfn', 'wuu', 'swh', 'arz', 'eng', 'ukr', 'tuk', 'hye', 'lat', 'dtp', 'csb', 'sqi', 'ina', 'mal', 'swe', 'xho', 'mhr', 'gle', 'cor', 'vie', 'oci', 'epo', 'urd', 'ceb', 'ind', 'kab', 'pol', 'cym', 'dan', 'est', 'ido', 'tgl'} | BitextMining | s2s | | {'test': 2000} | {'test': 39.4} | +| [Tatoeba](https://github.com/facebookresearch/LASER/tree/main/data/tatoeba/v1) | {'srp', 'yue', 'pam', 'csb', 'ido', 'deu', 'ita', 'orv', 'lit', 'swh', 'mar', 'yid', 'nds', 'dan', 'eng', 'cym', 'mal', 'tha', 'kur', 'cat', 'wuu', 'cha', 'max', 'eus', 'oci', 'mhr', 'gsw', 'mon', 'nld', 'ell', 'hun', 'nno', 'hrv', 'tel', 'urd', 'uig', 'ang', 'kat', 'fin', 'aze', 'slk', 'arz', 'nov', 'kab', 'tzl', 'ina', 'zsm', 'tur', 'cor', 'lat', 'ile', 'fao', 'bos', 'swg', 'war', 'awa', 'epo', 'khm', 'ces', 'fra', 'pms', 'lvs', 'kor', 'afr', 'dsb', 'ast', 'gle', 'tuk', 'bel', 'gla', 'est', 'cmn', 'heb', 'ind', 'por', 'spa', 'jav', 'ceb', 'pol', 'slv', 'xho', 'hin', 'tat', 'ben', 'ara', 'amh', 'jpn', 'cbk', 'hye', 'kaz', 'bre', 'pes', 'kzj', 'lfn', 'bul', 'sqi', 'fry', 'uzb', 'isl', 'glg', 'nob', 'ron', 'vie', 'dtp', 'arq', 'ukr', 'tgl', 'mkd', 'ber', 'hsb', 'tam', 'rus', 'swe'} | BitextMining | s2s | | {'test': 2000} | {'test': 39.4} | | [TenKGnadClusteringP2P](https://tblock.github.io/10kGNAD/) | {'deu'} | Clustering | p2p | | {'test': 45914} | {'test': 2641.03} | | [TenKGnadClusteringS2S](https://tblock.github.io/10kGNAD/) | {'deu'} | Clustering | s2s | | {'test': 45914} | {'test': 50.96} | | [ThuNewsClusteringP2P](http://thuctc.thunlp.org/) | {'cmn'} | Clustering | p2p | | | | @@ -272,13 +272,13 @@ The following tables gives you an overview of the tasks in MTEB. | [UyghurSentimentClassification](https://arxiv.org/abs/2009.08712) | {'uig'} | Classification | s2s | [Reviews] | {'test': 841} | {'test': 245.2} | | [VGClustering](https://huggingface.co/datasets/navjordj/VG_summarization) (Navjord et al., 2023) | {'nob'} | Clustering | p2p | [News, Non-fiction] | {'test': 2048} | {'test': 1009.65} | | [VideoRetrieval](https://arxiv.org/abs/2203.03367) | {'cmn'} | Retrieval | s2p | | | | -| [VieMedEVBitextMining](https://aclanthology.org/2015.iwslt-evaluation.11/) (Nhu Vo, 2024) | {'vie', 'eng'} | BitextMining | s2s | [Medical] | {'test': 2048} | {'test': 139.23} | +| [VieMedEVBitextMining](https://aclanthology.org/2015.iwslt-evaluation.11/) (Nhu Vo, 2024) | {'eng', 'vie'} | BitextMining | s2s | [Medical] | {'test': 2048} | {'test': 139.23} | | [VieQuADRetrieval](https://aclanthology.org/2020.coling-main.233.pdf) | {'vie'} | Retrieval | s2p | [Encyclopaedic, Non-fiction] | {'validation': 2048} | {'validation': 790.24} | | [VieStudentFeedbackClassification](https://ieeexplore.ieee.org/document/8573337) (Nguyen et al., 2018) | {'vie'} | Classification | s2s | [Reviews] | {'test': 2048} | {'test': 14.22} | | [WRIMEClassification](https://aclanthology.org/2021.naacl-main.169/) | {'jpn'} | Classification | s2s | [Social] | {'test': 2048} | {'test': 47.78} | | [Waimai](https://aclanthology.org/2023.nodalida-1.20/) | {'cmn'} | Classification | s2s | | | | | [WikiCitiesClustering](https://huggingface.co/datasets/wikipedia) | {'eng'} | Clustering | p2p | | | | -| XMarket | {'deu', 'spa', 'eng'} | Retrieval | s2p | | | | -| [XPQARetrieval](https://arxiv.org/abs/2305.09249) | {'spa', 'jpn', 'ita', 'ara', 'tam', 'kor', 'cmn', 'deu', 'fra', 'hin', 'por', 'pol'} | Retrieval | s2p | | | | +| XMarket | {'eng', 'deu', 'spa'} | Retrieval | s2p | | | | +| [XPQARetrieval](https://arxiv.org/abs/2305.09249) | {'cmn', 'pol', 'ara', 'fra', 'tam', 'deu', 'ita', 'kor', 'hin', 'jpn', 'por', 'spa'} | Retrieval | s2p | | | | | [YueOpenriceReviewClassification](https://github.com/Christainx/Dataset_Cantonese_Openrice) (Xiang et al., 2019) | {'yue'} | Classification | s2s | [Reviews] | {'test': 6161} | {'test': 173.0} | \ No newline at end of file