Include environment variables in interpreter info caching #11601
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
konstin
added
the
enhancement
zanieb
approved these changes
Feb 18, 2025
zanieb
reviewed
Feb 18, 2025
crates/uv-python/src/interpreter.rs
Outdated
Comment on lines
740
to
748
| Self { | ||
| pythonhome: env::var_os("PYTHONHOME"), | ||
| pythonpath: env::var_os("PYTHONPATH"), | ||
| pythonsafepath: env::var_os("PYTHONSAFEPATH"), | ||
| pythonplatlibdir: env::var_os("PYTHONPLATLIBDIR"), | ||
| pythonnousersite: env::var_os("PYTHONNOUSERSITE"), | ||
| pythonuserbase: env::var_os("PYTHONUSERBASE"), | ||
| appdata: env::var_os("APPDATA"), | ||
| home: env::var_os("HOME"), |
There was a problem hiding this comment.
Technically, we should add all these to the static EnvVars declarations so we can document what we read these for. Could you do that please?
We want to use `sys.path` for package discovery (#2500, #9849). For that, we need to know the correct value of `sys.path`. `sys.path` is a runtime-changeable value, which gets influenced from a lot of different sources: Environment variables, CLI arguments, `.pth` files with scripting, `sys.path.append()` at runtime, etc. We cannot capture them all accurately, especially since it's possible to change `sys.path` mid-execution. Instead, we do a best effort attempt at matching the user's expectation. A common way to influence `sys.path` that is not using venvs is setting `PYTHONPATH`. To support this we're capturing `PYTHONPATH` as part of the cache invalidation, i.e. we refresh the interpreter metadata if it changed. For completeness, we're also capturing other environment variables documented as influencing `sys.path` or other fields in the interpreter info. This PR does not include reading registry values for `sys.path` additions on Windows as documented in https://docs.python.org/3.11/using/windows.html#finding-modules. It notably also does not include parsing of python CLI arguments, we only consider their environment variable versions for package installation and listing. We could try parsing CLI flags in `uv run python`, but we'd still miss them when Python is launched indirectly through a script, and it's more consistent to only consider uv's own arguments and environment variables, similar to uv's behavior in other places.
konstin
force-pushed
the
konsti/cache-python-interpreter-by-env-vars
branch
from
02a8350 to
07be9ce
Compare
zanieb
pushed a commit
that referenced
this pull request
Feb 19, 2025
…1622) Revert #11601 for now We run Python interpreter discovery with `-I` (#2500) which means these environments variables are ignored when determining `sys.path`. Unless we decide to remove the `-I` flag from the `sys.path` query, we shouldn't release these changes to interpreter discovery caching.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We want to use
sys.pathfor package discovery (#2500, #9849). For that, we need to know the correct value ofsys.path.sys.pathis a runtime-changeable value, which gets influenced from a lot of different sources: Environment variables, CLI arguments,.pthfiles with scripting,sys.path.append()at runtime, a distributor patching Python, etc. We cannot capture them all accurately, especially since it's possible to changesys.pathmid-execution. Instead, we do a best effort attempt at matching the user's expectation.The assumption is that package installation generally happens in venv site-packages, system/user site-packages (including pypy shipping packages with std), and
PYTHONPATH. Specifically, we reusePYTHONPATHas dedicated way for users to tell uv to include specific directories in package discovery.A common way to influence
sys.paththat is not using venvs is settingPYTHONPATH. To support this we're capturingPYTHONPATHas part of the cache invalidation, i.e. we refresh the interpreter metadata if it changed. For completeness, we're also capturing other environment variables documented as influencingsys.pathor other fields in the interpreter info.This PR does not include reading registry values for
sys.pathadditions on Windows as documented in https://docs.python.org/3.11/using/windows.html#finding-modules. It notably also does not include parsing of python CLI arguments, we only consider their environment variable versions for package installation and listing. We could try parsing CLI flags inuv run python, but we'd still miss them when Python is launched indirectly through a script, and it's more consistent to only consider uv's own arguments and environment variables, similar to uv's behavior in other places.