lums/tmp/teo/kmeans

[lums658]

This PR manually reapplies the various commits in the diverged descendants of teo/kmeans that led to such a disaster in PR #130:

• Started with d085f66 (?)
• Updated C++ kmeans and kmeans++ was completed in commit e5e3690
• Merged main (37c9c9b?) into e5e3690
• Merged 0f73373 into e5e3690
• Rename e5e3690 to be lums/tmp/teo/kmeans

Updates since d085f66:

• Added code in kmeans to reassign centroids that have small (including empty) partitions. There is now a member variable reassign_ratio_ of the kmeans_index class that will reassign partitions that have size smaller than reassign_ratio_ times the size of the largest partition. The default value of reassign_ratio_ is 0.025 (the ratio of largest to smallest partition will be bounded by a factor of 40).
  • For each centroid c_i to be reassigned, we choose the vector in the training set with the i'th largest score (i.e., the vector farthest away from its own centroid).
  • This helps quite a bit in the case of random initialization. It does not help as much in the case of kmeans++ initialization because the centroids are already well-distributed.
  • Also tried reassignment with random vector, but that did not work as well as reassigning most distance vector.
• Added code in kmeans to check for convergence. At each iteration it measures how much the centroids have changed since the previous iteration and compares that to the total "potential" of all the centroids. If the delta is less than a tolerance times the total, the kmeans iteration breaks.
• Created numerous unit tests for various components of the library used by kmeans_index
• Updated fixed_min_heap to take a comparison function objects (to be able to create fixed max heaps, e.g.)
• Created qv_partition_with_scores to provide a scores vector to be used in reassignment.

There are a few todos:

• Parallelize the loop over all thetraining_set vectors in kmeans (Partitions are, well, partitioned among threads. Every thread loops over the entire training_set but only processes vectors from its designated set of partitions.)
• Implement greedy kmeans++ algorithm, which uses multiple candidate centroids at each iteration and chooses the one that most decreases the potential. This approach is what scikit_learn uses and is reported in the literature to be better than plain kmeans++.

[teo-tsirpanis]

Random initialization:
sklearn score: 6.619185447692871
tiledb score: 5.118894100189209
K-means++:
sklearn score: 3.9469563961029053
tiledb score: 4.516618251800537

Random initialization:
sklearn score: 6.619184494018555
tiledb score: 5.32197380065918
K-means++:
sklearn score: 3.9469563961029053
tiledb score: 4.792552471160889

Random initialization:
sklearn score: 6.619185447692871
tiledb score: 5.255888938903809
K-means++:
sklearn score: 3.9469563961029053
tiledb score: 4.764191150665283