Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: rename blocks_preload_pool_size to preload_pool_size #17

Closed
wants to merge 1 commit into from

Conversation

bokobza
Copy link
Contributor

@bokobza bokobza commented Dec 12, 2024

The blocks_preload_pool_size has a default value of 200 and is used in the code to grab a list of blocks.
A variable with almost the same name preload_pool_size is one of the LakeConfig object parameters. This one can be set but isn't used anywhere.
I believe the intent was for this to be 1 variable that the caller can set when creating the LakeConfig object.
I renamed blocks_preload_pool_size to preload_pool_size rather than the other way around to avoid any backward incompatibility.

@frol
Copy link
Collaborator

frol commented Dec 12, 2024

@bokobza What are you talking about? The variable is clearly used and you introduced a breaking change with this PR. If you believe otherwise, comment here and re-open the PR with more details on what you want to achieve

@frol frol closed this Dec 12, 2024
@bokobza
Copy link
Contributor Author

bokobza commented Dec 12, 2024

The preload_pool_size isn't used from what I can see. It's set in the contructor of LakeConfig but then isn't used.
What I want to achieve is to set the size of blocks requested in the list_objects_v2 method.
With the code as it is, this number will always be 200, regardless of how I set the preload_pool_size in LakeConfig.

@frolvanya
Copy link
Owner

frolvanya commented Dec 12, 2024

@bokobza

  1. It's used in list_blocks method:
    block_heights_prefixes = await s3_fetchers.list_blocks(
    s3_client,
    config.s3_bucket_name,
    start_from_block_height,
    config.blocks_preload_pool_size * 2,
    )

    async def list_blocks(
    s3_client,
    s3_bucket_name: str,
    start_from_block_height: near_primitives.BlockHeight,
    number_of_blocks_requested: int,
    ) -> list[near_primitives.BlockHeight]:

    and for the asyncio queue configuration:
    maxsize=config.blocks_preload_pool_size
  2. I don't understand how changing the name of the variable will fix your problem. I'm not going to merge a PR and update pypi package version for a variable renaming (especially if it's going to introduce a breaking change), unless there's a very good reason to do so

@bokobza
Copy link
Contributor Author

bokobza commented Dec 12, 2024

It's the blocks_preload_pool_size that is used, not the preload_pool_size .
You have 2 variables here. One is set in the constructor and not used (preload_pool_size) and one is set as property in LakeConfig and not changeable (blocks_preload_pool_size).
I believe it should be 1 variable. Did you intend to have 2 variables to hold the number of blocks to get in list blocks?

Again, at the moment a user cannot change the number of blocks they want to get in the list blocks method.
I'm not just renaming a variable, I'm renaming the variable so that it works.

image

@frolvanya
Copy link
Owner

Okay, now I understood the problem, but still let's avoid a breaking change, we can rename a local variable, instead of an attribute in a configuration. Thanks for the heads up!

@frolvanya
Copy link
Owner

@bokobza I opened a new PR, but actually it will make more sense to update this one. So if you want, update this PR and I'll close #18

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants