Add model spice

[iamgroot42]

Adding a model checklist

• I have filled out the ModelMeta object to the extent possible
• I have ensured that my model can be loaded using
  • mteb.get_model(model_name, revision) and
  • mteb.get_model_meta(model_name, revision)
• I have tested the implementation works on a representative set of tasks.

[KennethEnevoldsen]

Implementation looks fine. Though some of the metadata on training data + code is missing. Can you please fill these out.

Generally it seems like the models doesn't have a lot of documentation surrounding how it was trained. Is that intentional or is there an upcoming paper?

[iamgroot42]

Hi @KennethEnevoldsen - the training data is intentionally private. The training recipe is private for now but will be released later with an upcoming paper, with some details on the general mix of data.

[KennethEnevoldsen]

This is perfectly fine then you simply need to mark it training_data=False and training_code=False.

I will also need confirmation that your model is zero-shot on MTEB? (i.e. haven't trained on the training/dev/test data)

[iamgroot42]

public_training_code and public_training_data are both already set to None- do you want me to add the fields training_data and training_code in addition to these? I don't see them described anywhere in the fields forModelMeta

I am a bit confused about the zero-shot comment: are models not supposed to use the training data from MTEB datasets?

[KennethEnevoldsen]

public_training_code and public_training_data are both already set to None

oh, right, we changed it to a link instead of a boolean, that is my bad. None is just fine.

I am a bit confused about the zero-shot comment: are models not supposed to use the training data from MTEB datasets?

It was never clarified in the original paper, and you are welcome to do so, but if you do, we will need to mark the model as non-zero-shot. (will be filtered out by default in the updated version of the leaderboard, which will be released soon)

As shown by voyage-3-exp (which is not recommended to use), they perform notably better when trained on the train set without the model generally performing better (when applied to an unseen task). So, if we care about performance on an unseen task, then we care about zero-shot generalization.

This is to ensure fairness of comparison in the leaderboard (and that model performance on MTEB can be converted into meaningful model selection).

Feel free to ask if there is anything that is unclear

[iamgroot42]

I see- thanks for the clarification! The data distribution does have some sources that might appear similar to the MTEB data, and might have some samples from the training set, but these exact datasets were not used. I think for now you can mark the model as non-zero-shot.

[x-tabdeveloping]

We have discussed this case with the team. Based on your previous submission of identical scores to voyage-3-exp for your own model, we will not accept your submission, unless you provide a detailed description of the training procedure and data.