Integrate JMTEB

[Muennighoff]

Would be great to have the datasets in JMTEB (https://github.com/sbintuitions/JMTEB) integrated into MTEB for those which aren't yet already, so we can also add a Japanese leaderboard sometime 😊 cc @lsz05 @ryokan0123 @masaya-ohagi & anyone who may be interested

[awinml]

I would like to help integrate JMTEB. The following datasets from JMTEB are already present in MTEB:

• AmazonCounterfactualClassification
• AmazonReviewClassification
• MassiveIntentClassification
• MassiveScenarioClassification
• JSTS
• PawsX PairClassification

The following ones would need to be added:

• Clustering: (fix: Add JMTEB Clustering datasets #768)

  • Livedoor News
  • MewsC-16-ja

• STS: (fix: Add JSICK dataset #769)

  • JSICK

• Retrieval: (fix: Add JaGovFaqs and NLPJournal datasets #808)

  • NLP Journal title-abs
  • NLP Journal title-intro
  • NLP Journal abs-intro
  • JaGovFaqs-22k
  • JAQKET
  • Mr.TyDi-ja

• Reranking:

  • ESCI

[AlexeyVatolin]

@Muennighoff, I can take over the integration of ESCI. I have already looked at the contents of this dataset. In the original dataset https://huggingface.co/datasets/tasksource/esci there are three languages: English, Spanish and Japanese. Do we want to add all three languages or just Japanese?

[Muennighoff]

@Muennighoff, I can take over the integration of ESCI. I have already looked at the contents of this dataset. In the original dataset https://huggingface.co/datasets/tasksource/esci there are three languages: English, Spanish and Japanese. Do we want to add all three languages or just Japanese?

I think if we can add all 3 that'd be even better! 🙌

[KennethEnevoldsen]

@awinml do you still want to finish up this issue or should I mark is as "up for graps"?

[awinml]

@KennethEnevoldsen, I don't have the bandwidth at the moment to add the remaining three datasets. Feel free to mark it as open for contributions. I’m happy to help review any PRs that come in!

[KennethEnevoldsen]

I suspected as much - thanks for the help so far though