Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for layered package cache #1003

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

kelvinou01
Copy link
Contributor

@kelvinou01 kelvinou01 commented Dec 23, 2024

Description

As per #43, we want to implement layered package caches for Rattler.

Stuff to note

  • I tried to make this not a break, but alas. Specifically, i made PackageCacheError a #[non_exhaustive] enum, and modified some variants.

Old PR for ref

@kelvinou01 kelvinou01 changed the title Add support for layered package cache feat: add support for layered package cache Dec 23, 2024
@wolfv
Copy link
Contributor

wolfv commented Dec 23, 2024

Hi @kelvinou01 - I also started hacking on this a bit but it looks like you are already much further progressed :)

Here was my attempt: #998

@kelvinou01 kelvinou01 mentioned this pull request Dec 23, 2024
@gzm55
Copy link

gzm55 commented Feb 7, 2025

@kelvinou01 is this pr ready?

@gzm55
Copy link

gzm55 commented Feb 15, 2025

it seems the pr does not contains the changes of the config key.

@kelvinou01
Copy link
Contributor Author

kelvinou01 commented Mar 5, 2025

@kelvinou01 is this pr ready?

@gzm55 not yet, is there an urgent need for it?

@gzm55
Copy link

gzm55 commented Mar 5, 2025

no urgent need. but expect this feature for multi-user environment

@kelvinou01 kelvinou01 marked this pull request as ready for review March 8, 2025 11:43
@kelvinou01
Copy link
Contributor Author

kelvinou01 commented Mar 8, 2025

@wolfv hey, could you help me review this?

Also, I did try implementing storing the capabilities cache like #998. However, I thought this PR was large enough already, so the capabilities cache can be introduced in a subsequent PR.

@gzm55
Copy link

gzm55 commented Mar 9, 2025

when select a read or write cache, can we prefer the one from which we can make a hard link?

@kelvinou01
Copy link
Contributor Author

@gzm55 I see how that can be useful! Do you think this feature is necessary to introduce with this PR, or can we do it in a subsequent one as i mentioned above?

@gzm55
Copy link

gzm55 commented Mar 9, 2025

@gzm55 I see how that can be useful! Do you think this feature is necessary to introduce with this PR, or can we do it in a subsequent one as i mentioned above?

I think a subsequent pr for the feature mentioned above is ok.

Comment on lines +170 to +172
fn(PathBuf) -> _,
Pin<Box<dyn Future<Output = Result<(), _>> + Send>>,
std::io::Error,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes unfortunately, as the compiler has to infer the types somehow if we're passing in None for fetch and reporter.

One way around this would be to assign default types to F, Fut, E during the declaration of validate_package_common<F, Fut, E>, but that feature is being phased out of Rust.

}

/// Validate the package, and throw if invalid.
pub async fn validate_or_throw(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesnt really "throw" anything. Maybe call try_validate?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, thanks!


/// Constructs a new [`PackageCache`] located at the specified paths.
/// Read-only layers are queried first.
/// Within read-only layers, the ordering is defined in this constructor. Ditto for writable layers.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont fully understand this comment and I also dont see how it relates to the code? To me it makes most sense to pass in the paths in the order the user wants the layers to be ordered.

I would assume you look through the layers in the order they are passed in, and only write to the first writable layer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like I didn't make the comment clear. What I meant was, the querying order is defined by

  • Primary key: read-only before write-only
  • Secondary key: order it was passed into the constructor

But now that I think about it, querying all layers in the order passed into the constructor makes more sense.

///
/// The permissions are checked at the time of the function call.
pub fn split_layers(&self) -> (Vec<&PackageCacheLayer>, Vec<&PackageCacheLayer>) {
let readonly_layers = self
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

@kelvinou01 kelvinou01 Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the tip ^^ added this to the code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants