fix: Compare Cluster and ClusterFast scores and speedup

[isaac-chung]

1) What

Part of #835 .

• Runs english benchmark's clustering (except Arxiv since it's hierarchical) for both Clustering and ClusteringFast on e5-small, e5-base, e5-large, and a few more models.

python scripts/run_mteb_english_cluster.py

• Calculates the spearman's correlation on the v_measures from both task types
• sum the times to run both benchmarks, divide cluster by clusterfast to obtain speedup multiplier.

python scripts/mteb_english_cluster_spearman.py

2) Hardware

These were run on a single A10.

3) Scores on ClusteringFast + Speedup

These are the Spearman's correlation scores and speedups from Clustering to ClusteringFast. The old notes are hidden within each expand.

4% n_samples for every task except for RedditClustering and StackExchangeClustering with 32768)

In addition, p-values for each model is reported.

Task	Spearman	Significant Spearman	Speedup
BiorxivClusteringP2P	0.9505	0.9390	31.50x
BiorxivClusteringS2S	0.9890	0.9679	14.31x
MedrxivClusteringP2P	0.9615	0.8200	21.48x
MedrxivClusteringS2S	0.9560	0.9510	8.39x
RedditClustering	0.9670	0.9790	11.72x
RedditClusteringP2P	0.9670	0.7370	22.77x
StackExchangeClustering	0.9121	0.9486	9.55x
StackExchangeClusteringP2P	0.9670	0.9497	20.20x
TwentyNewsgroupsClustering	1.0000	0.9832	5.02x
Average	0.9634	0.9195	16.11x

Old notes comparing across tasks.

a) Current Implementation

Model	Spearman	Speedup
BiorxivClusteringP2P	0.5	34.46x
BiorxivClusteringS2S	1.0	20.11x
MedrxivClusteringP2P	0.5	17.30x
MedrxivClusteringS2S	1.0	10.30x
RedditClustering	1.0	23.15x
RedditClusteringP2P	1.0	27.32x
StackExchangeClustering	1.0	18.71x
StackExchangeClusteringP2P	0.5	4.50x
TwentyNewsgroupsClustering	1.0	11.44x

b) New Commit: b8a5be4

Model	Spearman	Speedup
BiorxivClusteringP2P	0.5	3.68x
BiorxivClusteringS2S	1.0	3.46x
MedrxivClusteringP2P	1.0	1.83x
MedrxivClusteringS2S	1.0	1.80x
RedditClustering	1.0	17.38x
RedditClusteringP2P	1.0	21.55x
StackExchangeClustering	0.5	14.43x
StackExchangeClusteringP2P	0.5	3.68x
TwentyNewsgroupsClustering	1.0	2.61x

c) New Commit: 6295dcc (swap max_documents_per_cluster and max_documents_to_embed values)

Model	Spearman	Speedup
BiorxivClusteringP2P	0.5	34.38x
BiorxivClusteringS2S	1.0	14.12x
MedrxivClusteringP2P	1.0	17.09x
MedrxivClusteringS2S	1.0	8.63x
RedditClustering	1.0	75.32x
RedditClusteringP2P	1.0	156.63x
StackExchangeClustering	1.0	46.77x
StackExchangeClusteringP2P	1.0	29.39x
TwentyNewsgroupsClustering	1.0	9.68x

d) New Commit: 3a0d9c4 (c + increasing max_documents_per_cluster to 65536)

Model	Spearman	Speedup
BiorxivClusteringP2P	0.5	31.38x
BiorxivClusteringS2S	1.0	10.44x
MedrxivClusteringP2P	0.5	15.58x
MedrxivClusteringS2S	1.0	5.54x
RedditClustering	1.0	50.16x
RedditClusteringP2P	1.0	132.17x
StackExchangeClustering	1.0	32.46x
StackExchangeClusteringP2P	1.0	25.43x
TwentyNewsgroupsClustering	1.0	5.97x

Old notes comparing across models.

a) Current Implementation

Model	Spearman	Speedup
e5-small	0.7833	14.56x
e5-base	0.8333	15.80x
e5-large	0.8000	16.93x

b) New Commit: b8a5be4

Model	Spearman	Speedup
e5-small	0.8500	5.76x
e5-base	0.8833	5.97x
e5-large	0.8000	6.17x

c) New Commit: 6295dcc (swap max_documents_per_cluster and max_documents_to_embed values)

Model	Spearman	Speedup
e5-small	0.8833	32.05x
e5-base	0.9000	40.41x
e5-large	0.9000	52.77x

d) New Commit: 3a0d9c4 (c + increasing max_documents_per_cluster to 65536)

Model	Spearman	Speedup
e5-small	0.8833	22.21x
e5-base	0.9000	32.11x
e5-large	0.9500	47.11x

Checklist

• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.

[KennethEnevoldsen]

This looks great @isaac-chung! A few surprising results here.

I might be afraid that the new method (resampling many documents) produces results closer to the original (which is nice), but what we are really interested in is that they produce the same model ordering and with the same (or better) power to differentiate between similar models.

Though resampling many times does seem to be quite cheap.

[isaac-chung]

Looking ahead to what's needed to close/merge this PR, the following should be completed first:

• Move plotting scripts and plots to MMTEB scripts
• Run a few more models and commit results (fix: Compare Cluster and ClusterFast scores and speedup #892 (comment))

Anything else I'm missing?

[KennethEnevoldsen]

I think that it good

[isaac-chung]

Added plotting scripts and plots here. Will remove them from this PR, then we should be good to merge.

[isaac-chung]

@KennethEnevoldsen just wanted a sanity ✅ before we merge, thanks!

[KennethEnevoldsen]

A few minor comments otherwise looks good - once these are in feel free to merge. Thanks for taking the time on this one!