Improve belebele retrieval task

[akshita-sukhlecha]

This enhances the Belebele retrieval task - to make it cross-lingual and add all the language scripts. This will support 122 language variants (including 115 distinct languages and their scripts).

Checklist

• I have tested that the dataset runs with the mteb package.
• I have run the following models on the task (adding the results to the pr). These can be run using the mteb -m {model_name} -t {task_name} command.
  • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
  • intfloat/multilingual-e5-small
• I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
• If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
• I have filled out the metadata object in the dataset file (find documentation on it here).
• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.

[KennethEnevoldsen]

I don't have too much experience with cross-lingual tasks (@imenelydiaker do you have time for this?)

But I would def. say that you should:

1. Create a new version of the task instead of replacing it and then add the superseeded_by to the old dataset (you can find an example in the arxivClustering tasks)
2. Compare runtime of old dataset and new dataset (how much additional time is added)
3. compare scores of old and new dataset (does it make a difference in model ranking)
4. see comment on potentially not creating a full grid of comparisons

[KennethEnevoldsen]

All possible pairs might be unreasonable as some pairs are in practice never seen. I believe we have another dataset (Bible I believe), which created a meaningful subset.

[akshita-sukhlecha]

Thanks for reviewing ! I have pushed the requested changes.

1. Create a new version (superseeded) task - Done 


2. & 3. Compare runtime & scores - Runtime and score remain the same when providing a language to the old-task and providing a pair with the same source and target language to the new task.



3. Do not create a full grid of comparisons - Done

[imenelydiaker]

Hello, thanks for this work! Some questions:

• How do you match a question in language XX with a passage in language YY? The order of samples on HF is not the same from one language to another (I manually checked Arabic and French and found that the first sample of each language is not the same). I can't see any IDs in the dataset to identify them.
• Why not update the old task instead of creating a new one? We'd have something like MLQA.

[KennethEnevoldsen]

Why not update the old task instead of creating a new one? We'd have somethig like MLQA.

Ahh right, I was suspecting that the dataset changed due to the changes in the dataset transform. I see that it does not. We can keep it as one.

[akshita-sukhlecha]

@imenelydiaker

1. How do you match a question in language XX with a passage in language YY? - Using the link field. A link corresponds to a particular passage.

2. Sure, I'll make the changes to update the old task itself.

[akshita-sukhlecha]

@imenelydiaker I have one question on how to represent the language-pairs - usage of hyphen vs underscore between language and script code.
Should it be "acm-Arab-eng-Latn" or "acm_Arab-eng_Latn" ?

• For languages, "eng-Latn" (hyphen) has been used in the repo.
• For language pairs in Bitext-mining multilingual tasks, "acm_Arab-eng_Latn" (underscore) has been used

[imenelydiaker]

• For language pairs in Bitext-mining multilingual tasks, "acm_Arab-eng_Latn" (underscore) has been used

I'd advise to use the same standard as FloresBitextMining task. So the second option "acm_Arab-eng_Latn".

[imenelydiaker]

LGTM! Thank you! Ready for merging?

[akshita-sukhlecha]

@imenelydiaker Yes, please go ahead and merge it. (It showing that some tests have failed, but they are related to different tasks. Don't know what to do in this case.)

[imenelydiaker]

@imenelydiaker Yes, please go ahead and merge it. (It showing that some tests have failed, but they are related to different tasks. Don't know what to do in this case.)

Hey @akshita-sukhlecha can you please update your branch?

@isaac-chung @KennethEnevoldsen do you know why teh CI is failing? Should we merge either way?