fix: Add datasets in CodeRAG-Bench

[hepengfe]

This PR address #1151
It has a blocker as the dataset cannot be downloaded as reported here code-rag-bench/code-rag-bench#5

Update on 1/3/2025: the dataset server has recovered.

Checklist

• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.

Adding datasets checklist

Reason for dataset addition: ...

• I have run the following models on the task (adding the results to the pr). These can be run using the mteb -m {model_name} -t {task_name} command.
  • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
  • intfloat/multilingual-e5-small
• I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
• If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
• I have filled out the metadata object in the dataset file (find documentation on it here).
• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.

[hepengfe]

@KennethEnevoldsen @isaac-chung Hi, can I get the approval for the workflow.

[isaac-chung]

Thanks! Good first try. Added a few suggestions. And since the dataset_transform and split_by_first_newline are mostly repeated, let's put these classes in the same file. That way the functions can be written once and reused.

[hepengfe]

Thanks! Good first try. Added a few suggestions. And since the dataset_transform and split_by_first_newline are mostly repeated, let's put these classes in the same file. That way the functions can be written once and reused.

I noticed that mteb/tasks/Retrieval/code/CodeSearchNetCCRetrieval.py and mteb/tasks/Retrieval/code/COIRCodeSearchNetRetrieval.py have the same function _load_code_search_code_retrieval. Plus, any suggestion on where to write those helper functions? Do I just create a new file such as mteb/task_helper_function.py and write functions there?

[isaac-chung]

I noticed that mteb/tasks/Retrieval/code/CodeSearchNetCCRetrieval.py and mteb/tasks/Retrieval/code/COIRCodeSearchNetRetrieval.py have the same function _load_code_search_code_retrieval. Plus, any suggestion on where to write those helper functions? Do I just create a new file such as mteb/task_helper_function.py and write functions there?

That can be one option. I'd prefer to limit this PR to CodeRAG-bench and avoid any refactoring other files.

[hepengfe]

I also noticed that the evaluation score for nauc_ndcg_at_*_std is negative. Is it expected? If not, any pointer on the solution for the error?

[isaac-chung]

I also noticed that the evaluation score for nauc_ndcg_at_*_std is negative. Is it expected? If not, any pointer on the solution for the error?

I've not paid attention to that before. Maybe @Samoed or @orionw might? Could be related to these being reranking tasks?

[Samoed]

This is common on many tasks. E.g. voyage results on MIRACL

[isaac-chung]

Nice. I see two main items from docs before we merge:

1. Add dataset metrics: "Add metadata to the task (run task.calculate_metadata_metrics())"
2. Add a benchmark entry in https://github.com/embeddings-benchmark/mteb/blob/main/mteb/benchmarks/benchmarks.py to reference these datasets.

That will complete the PR. Let us know if you have any questions. Thanks again for iterating!

[hepengfe]

Any way to run task.calculate_metadata_metrics() in a low-memory setting? I am trying it out but stackoverflow-posts is too large (2.65GB) and it results in OOM even for 128 GB RAM. @Samoed @isaac-chung

[isaac-chung]

@hepengfe hmm I don't have a good suggestion. In light of that, I'd say adding a benchmark entry (more important) and whatever dataset metrics you already have should complete the PR. That way we can look at the descriptive stats issue separately.

[Samoed]

Can you submit results of a model on this benchmark to results repository?

[isaac-chung]

@hepengfe thanks for your contributions! This PR is good to go.

[Samoed]

I've looked on stackoverflow-posts and it has 9x6gb jsonl files. I can't image how long this would take to run