Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run queries twice to test for idempotence #159

Merged
merged 4 commits into from
Dec 12, 2023

Conversation

jparismorgan
Copy link
Contributor

@jparismorgan jparismorgan commented Dec 11, 2023

What

Adds tests for the idempotence of query() to address https://app.shortcut.com/tiledb-inc/story/32367/add-tests-for-query-idempotence. There is not much of a speed difference from repeating .query() twice, though I'm very open to other suggestions on how to test this. Alternatives include:

  1. New separate idempotence tests in apis/python/test/test_index.py, i.e. test_flat_index_idempotence and test_ivf_flat_index_idempotence
  2. New separate idempotence tests in apis/python/test/test_ingestion.py, i.e. test_flat_index_idempotence and test_ivf_flat_index_idempotence
  3. New separate idempotence tests in apis/python/test/test_query.py, i.e. test_flat_index_idempotence and test_ivf_flat_index_idempotence
  4. Move this logic into apis/python/test/test_ingestion.py and run the loop on the query() calls in there

Testing

Before:

(TileDB-Vector-Search) ~/repo/TileDB-Vector-Search/apis/python pytest test/test_index.py -s                                                  jparismorgan/query-idempotence 
============================================================================ test session starts ============================================================================
platform darwin -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /Users/parismorgan/repo/TileDB-Vector-Search/apis/python
collected 2 items                                                                                                                                                           

test/test_index.py ..

============================================================================= 2 passed in 2.89s =============================================================================

After:

(TileDB-Vector-Search) ~/repo/TileDB-Vector-Search/apis/python pytest test/test_index.py -s                                                  jparismorgan/query-idempotence 
============================================================================ test session starts ============================================================================
platform darwin -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /Users/parismorgan/repo/TileDB-Vector-Search/apis/python
collected 2 items                                                                                                                                                           

test/test_index.py ..

============================================================================= 2 passed in 3.07s =============================================================================

@jparismorgan jparismorgan marked this pull request as ready for review December 11, 2023 11:29
assert {ind.MAX_UINT64} == set(result_i[0])
for _ in range(2):
result_d, result_i = index.query(np.array([[2, 2, 2]], dtype=np.float32), k=3)
assert {ind.MAX_UINT64} == set(result_i[0])
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets add a util function that runs x queries and asserts results to avoid duplication and make the code more readable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, updated:

(TileDB-Vector-Search) ~/repo/TileDB-Vector-Search/apis/python pytest test/test_index.py -s
================================================= test session starts =================================================
platform darwin -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /Users/parismorgan/repo/TileDB-Vector-Search/apis/python
collected 2 items                                                                                                     

test/test_index.py ..

================================================== 2 passed in 3.33s ==================================================

@jparismorgan jparismorgan merged commit 386304e into main Dec 12, 2023
4 checks passed
@jparismorgan jparismorgan deleted the jparismorgan/query-idempotence branch December 12, 2023 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants