Results of SISAP 2024 Indexing challenge

[maumueller]

Dear @Neiko2002 @cole-foster @elgerpus @danielbenedi6 @eth42 @ProchazkaDavid

We are glad to share a summary of the SISAP 2024 indexing challenge results. In particular, the ranking per subset per track; more detailed results will be listed in the overview and discussed during the challenge track of the SISAP 2024 conference. Please recall the task descriptions from https://sisap-challenges.github.io/tasks/.

We prepared docker Linux images using the submitted solutions and ran it on a 28-core (56 threads) Intel(R) Xeon(R) CPU E5-2690 V4 workstation with 512GiB of RAM (enforcing the CPU/memory limits through docker). The evaluation was run on a private query set taken from an unseen batch of the LAION2B dataset. While for the public query set, we removed near-duplicate objects/queries, (considering as near-duplicate any object inside a radius of 0.15 of an already selected query point); for the private query set, we remove near-duplicates within a radius of 0.2, which made the queries more diverse. You will see this reflected in your results. We will make this private query set available soon so that you can reproduce the results.

Thank you for your participation,
Eric (@sadit), Martin (@maumueller), and Vladimir (@VladimirMic)

Overall ranking

Individual tasks

The individual plots showing the performance/quality trade-off that was achieved by the proposed implementations can be seen in the following plots:

You can find the detailed result files that these plots are based on in the following Github repository: https://github.com/sisap-challenges/challenge2024/

[ProchazkaDavid]

Hello @maumueller,

Thanks for listing the results of the indexing challenge. I have three comments/questions.

First, the query time of our solution (LMI) seems to be more than an order of magnitude higher than that obtained on our hardware. It appears that the problem may be that our implementation did not have access to multiple cores, which should speed up the search process and was one of our engineering efforts for this year's challenge. Is there any way to verify that this was the case and that our solution did or did not have access to single/multiple CPU cores? In terms of build time, the values are in the same order of magnitude.

Second, there seems to be an incorrect size of the private query set, as https://sisap-challenges.github.io/2024/tasks/ indicates that there should be 100K queries, not 10K.

Finally, based on the provided plot for Task 3, the HIOB solution should also have a rank in the overall ranking. It exceeded the threshold of 0.4 recall, as indicated by the "second" mark in the plot (and as listed in https://github.com/sisap-challenges/challenge2024/blob/main/sisap24-task3.csv). Or am I missing something? On the other hand, BL-SearchGraph did not achieve the minimum 0.4 recall (see https://github.com/sisap-challenges/challenge2024/blob/main/sisap24-task3.csv) as per the requirements of this task, and yet it is ranked.

Thanks for your answers.

David

[maumueller]

Thanks for your comments @ProchazkaDavid. I agree that there seem to be some irregularities. @sadit and I will have a look into it and get back to you as soon as we collected the data.

[sadit]

Hi @ProchazkaDavid,

Thank you for checking the results with such detail.

On the first comment, we forked your repository at https://github.com/sisap-challenges/sisap24-lmi so you can check any change we made to be able to run. In particular, you can check the docker-instructions.txt, which are some notes used for running. As you may observe, it is incomplete, but they are the example for the rest of the commands. We ran Task 1 and 3 with --cpus 54, and Task 3 was memory limited with --memory 64G; on the other hand, Task 2 used --cpus 8 and --memory 32G. In particular, while running the evaluation, I observed the time costs and thought the same way you mentioned. So, I checked with htop; there were many working threads, but each had a low usage percentage. It is also true that there are many stages in the running process, so I am not sure what part I observed at that moment. I guess you are working with newer/better hardware with CPU features not present in our testing hardware or perhaps the precise version of a library; since you are using neural networks, this could have a significant impact; but I don't know what is happening. Please let me know if I can do something to help with this.

On the second point, thank you for highlighting that issue. There is an internal historical reason behind that mistake, but we use 10K on public and private sets; I will update the site.

Thirdly, we will check the HIOB issue and update the overview.

Finally, after the numerical rounding, the BL-SearchGraph achieved 0.40; we will remove the ranking and put an asterisk with the note about the rounding.

Thank you,
Eric

[jozefCer]

Hi @sadit ,
It looks like inside the container, the environment variable OMP_NUM_THREAD was set to a default value equal to the actual number of CPU cores of the underlying HW. Since this value did not match the limit enforced by Docker, which is set by the 'cpus' flag, it caused the implementation to spawn more threads than it could effectively use. See the updated version with the fixed values:
Coda-Research-Group/LearnedMetricIndex@a09bcbf

Best regards,
Jozef Čerňanský

[sadit]

Dear @Neiko2002 @cole-foster @elgerpus @danielbenedi6 @eth42 @ProchazkaDavid

You can find the private query set and its gold standard on the SISAP Indexing Challenge site
https://sisap-challenges.github.io/2024/datasets/, so you can test your system solutions.

Please note that you may need to download using the "Save as" option and accept some security advice from the Chrome browser since these links are under the plain HTTP protocol.

Best,
Eric (@sadit), Martin (@maumueller), and Vladimir (@VladimirMic)