Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Added Task selection and aggregation #1140

Merged
merged 49 commits into from
Aug 15, 2024
Merged

fix: Added Task selection and aggregation #1140

merged 49 commits into from
Aug 15, 2024

Conversation

KennethEnevoldsen
Copy link
Contributor

@KennethEnevoldsen KennethEnevoldsen commented Aug 8, 2024 •
edited
Loading

This code present work required for performing task selection and developing benchmarks within mteb.

The jupyter notebook provides and example of how this could work using Danish, but once we agree that this is a reasonable approach I can redo the workflow for:

  • mteb lite
  • mteb multilingual
  • mteb eu languages
  • mteb african languages
  • mteb some domain, probably legal

Checklist

  • Run tests locally to make sure nothing is broken using make test.
  • Run the formatter to format the code using make lint.

KennethEnevoldsen and others added 30 commits June 7, 2024 10:48
@KennethEnevoldsen @vaibhavad
task metric selection initial commit (#891)
3287c44 
Co-authored-by: vaibhavad <vaibhavadlakha95@gmail.com>
@KennethEnevoldsen
Add changes to PR
76b85fb
@KennethEnevoldsen
save
f9845ca
@KennethEnevoldsen
save
fa4c81f
@KennethEnevoldsen
Added significant rank
c6f8703
@KennethEnevoldsen
clean up
47a4ce7
@KennethEnevoldsen
save attemtps
b655227
@KennethEnevoldsen
add scores plot
0ee55bb
@KennethEnevoldsen
update scores plot
ddd7455
@KennethEnevoldsen
latent models
d5d9d14
@KennethEnevoldsen
Added utility function for downloading models results
0b41918
@KennethEnevoldsen
Added result tables
ef14ae3
@KennethEnevoldsen
Merge branch 'main' of https://github.com/embeddings-benchmark/mteb i…
f4b8e42 
…nto results_utils
@KennethEnevoldsen
Added bench construction
888958d
@KennethEnevoldsen
format
6cd508e
@KennethEnevoldsen
Merge branch 'main' of https://github.com/embeddings-benchmark/mteb i…
3f4dda3 
…nto added-benchmark-construction
@KennethEnevoldsen
Added correction citations
6aa8933
@KennethEnevoldsen
format
09d3892
@KennethEnevoldsen
Merge branch 'main' into bitext-analysis
edef69c
@KennethEnevoldsen
Merge branch 'main' of https://github.com/embeddings-benchmark/mteb i…
f6f60a8 
…nto added-benchmark-construction
@KennethEnevoldsen
Merge branch 'bitext-analysis' into added-benchmark-construction
c538935
@KennethEnevoldsen
sav
205e2f4
@KennethEnevoldsen
sav
e777e3b
@KennethEnevoldsen
sav
8922c28
@KennethEnevoldsen
merge
1ddb351
@KennethEnevoldsen
Merge branch 'task_selection' into add_euro_analysis
906298b
@KennethEnevoldsen
added task selection for EU languages
fc04784
@KennethEnevoldsen
remove token
865c9cb
@KennethEnevoldsen
Merge branch 'main' of https://github.com/embeddings-benchmark/mteb i…
b263319 
…nto add_euro_analysis
@KennethEnevoldsen
clean up branch
0b8372c
@KennethEnevoldsen
clean up branch
ff172f5
@KennethEnevoldsen
merge
8cdbbc5
@KennethEnevoldsen
Merge branch 'add-benchmarks' of https://github.com/embeddings-benchm…
04fdbd7 
…ark/mteb into add-benchmarks
@KennethEnevoldsen
minor
36fee6e
@KennethEnevoldsen
minor
208fd77
@KennethEnevoldsen
fix metadata
f93fa43
@KennethEnevoldsen
fix: Move benchmarks from script to benchmarks.py
b91d3f8 
- A few of the benchmarks weren't implemented in the benchmark list. The best choice seem to move them benchmarks and remove the script (there were already a few issues). To ensure that they are maintained with the rest.
- Removed trivial (e.g. running one task) or outdated examples. Removing them ensure that we have less to maintain.
- moved mteb specific script into a mteb-folder.
@KennethEnevoldsen
format
ed9685c
@KennethEnevoldsen
fix: Convert arbitrary imports to absolute imports.
fd0f039 
This avoids loading in classes as modules (see PR ...)
@KennethEnevoldsen
Added example
8aa5c39
@KennethEnevoldsen
Merge branch 'fix--abstract-imports' of https://github.com/embeddings…
32bd2d8 
…-benchmark/mteb into add-benchmarks
@KennethEnevoldsen
Final correction to notebook
5e7a5af
@KennethEnevoldsen
Merge branch 'main' of https://github.com/embeddings-benchmark/mteb i…
1a63a5c 
…nto add-benchmarks
@KennethEnevoldsen
format
a7b9b7c
@KennethEnevoldsen
reorder scripts
19adbc8
@KennethEnevoldsen
remove additional files
078b3c3
@KennethEnevoldsen KennethEnevoldsen changed the title wip: Adding benchmark fix: Added Task selection and aggregation Aug 14, 2024
isaac-chung
isaac-chung reviewed Aug 14, 2024
View reviewed changes
Copy link
Collaborator

@isaac-chung isaac-chung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Just a few suggestions.

mteb/load_results/load_results.py Show resolved Hide resolved
mteb/task_aggregation.py Outdated Show resolved Hide resolved
mteb/task_selection.py Outdated Show resolved Hide resolved
scripts/task_selection/task_selection_example.ipynb Outdated Show resolved Hide resolved
@KennethEnevoldsen KennethEnevoldsen enabled auto-merge (squash) August 15, 2024 09:12
@KennethEnevoldsen KennethEnevoldsen merged commit 57f4fb2 into main Aug 15, 2024
9 checks passed
@KennethEnevoldsen KennethEnevoldsen deleted the add-benchmarks branch August 15, 2024 09:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@isaac-chung isaac-chung isaac-chung left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@KennethEnevoldsen @isaac-chung