-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix duplicate packages with multiple conflicting extras declared #11513
Open
BurntSushi
wants to merge
3
commits into
main
Choose a base branch
from
ag/fix-11479
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+3,349
−33
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The place to look in this snapshot is the `name = "e3nn"` dependency. Its dependencies on `sympy` and `torch` consist of multiple versions with overlapping conflict markers. They are getting incorrectly simplified to `true`.
The particular example I honed in on here was the `e3nn -> sympy 1.13.1` and `e3nn -> sympy 1.13.3` dependency edges. In particular, while the former correctly has a conflict marker, the latter's conflict marker was getting simplified to `true`. This makes the edges trivially overlapping, and results in both of them getting installed simultaneously. (A similar problem happens for the `e3nn -> torch` dependency edges.) Why does this happen? Well, conflict marker simplification works by detecting which extras are known to be enabled (and disabled) for each node in the graph. This ends up being expressed as a set of sets, where each inner set contains items corresponding to "extras is included" or "extra is excluded." The logic then is if _all_ of these sets are satisfied by the conflict marker on the dependency edge, then this conflict marker can be simplified by assuming all of the inclusions/exclusions to be true. In this particular case, we run into an issue where the set of assumptions discovered for `e3nn` is: {test[sevennet]}, {}, {~test[m3gnet], ~test[alignn], test[all]} And the corresponding conflict marker for `e3nn -> sympy 1.13.1` is: extra == 'extra-4-test-all' or extra == 'extra-4-test-chgnet' or (extra != 'extra-4-test-alignn' and extra != 'extra-4-test-m3gnet') And the conflict marker for `e3nn -> sympy 1.13.3` is: extra == 'extra-4-test-alignn' or extra == 'extra-4-test-m3gnet' Evaluating each of the sets above for `sympy 1.13.1`'s conflict marker results in them all being true. Simplifying in turn results in the marker being true. For `sympy 1.13.3`, not all of the sets are satisfied, so this marker is not simplified. I think the fundamental problem here is that our inferences aren't quite rich enough to make these logical leaps. In particular, the conflict marker for `e3nn -> sympy 1.13.3` is not satisfied by _any_ of our sets. One might therefore conclude that this dependency edge is impossible. But! The `test[sevennet]` set doesn't actually rule out `test[m3gnet]` from being included, for example, because there is no conflict. So it is actually possible for this marker to evaluate to true. And I think this reveals the problem: for the `e3nn -> sympy 1.13.1` conflict marker, the inferences don't capture the fact that `test[sevennet]` _might_ have `test[m3gnet]` enabled, and that would in turn result in the conflict marker evaluating to `false`. This directly implies that our simplification here is inappropriate. It would be nice to revisit how we build our inferences here so that they are richer and enable us to make correct logical leaps. For now, we fix this particular bug with a bit of a cop-out: we skip conflict marker simplification when there are ambiguous dependency edges. Fixes #11479
…marker simplification This is fallout from skipping simplification when two or more edges with the same package name exist.
019683d
to
67dbf9b
Compare
@@ -93,6 +93,14 @@ impl ResolutionGraphNode { | |||
} | |||
} | |||
} | |||
|
|||
#[allow(dead_code)] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Necessary?
I'm sort of just confirming my understanding, but if we simplified the conflict marker for each set independently, and then checked if each simplification resulted in the same outcome, could we use that? Or, what if for each set
Would that be sound? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implements a somewhat of a cop-out fix for #11479, where the lock
file produced was missing some conflict markers. This in turn could lead
to multiple versions of the same package being installed into the same
environment.
(What follows is one of the commit messages that gets into the weeds
about the specific problem here.)
The particular example I honed in on here was the
e3nn -> sympy 1.13.1
and
e3nn -> sympy 1.13.3
dependency edges. In particular, while theformer correctly has a conflict marker, the latter's conflict marker was
getting simplified to
true
. This makes the edges triviallyoverlapping, and results in both of them getting installed
simultaneously. (A similar problem happens for the
e3nn -> torch
dependency edges.)
Why does this happen? Well, conflict marker simplification works by
detecting which extras are known to be enabled (and disabled) for each
node in the graph. This ends up being expressed as a set of sets, where
each inner set contains items corresponding to "extras is included" or
"extra is excluded."
The logic then is if all of these sets are satisfied by the conflict
marker on the dependency edge, then this conflict marker can be
simplified by assuming all of the inclusions/exclusions to be true.
In this particular case, we run into an issue where the set of
assumptions discovered for
e3nn
is:And the corresponding conflict marker for
e3nn -> sympy 1.13.1
is:And the conflict marker for
e3nn -> sympy 1.13.3
is:Evaluating each of the sets above for
sympy 1.13.1
's conflictmarker results in them all being true. Simplifying in turn results in
the marker being true. For
sympy 1.13.3
, not all of the sets aresatisfied, so this marker is not simplified.
I think the fundamental problem here is that our inferences aren't quite
rich enough to make these logical leaps. In particular, the conflict
marker for
e3nn -> sympy 1.13.3
is not satisfied by any of our sets.One might therefore conclude that this dependency edge is impossible.
But! The
test[sevennet]
set doesn't actually rule outtest[m3gnet]
from being included, for example, because there is no conflict. So it is
actually possible for this marker to evaluate to true.
And I think this reveals the problem: for the
e3nn -> sympy 1.13.1
conflict marker, the inferences don't capture the fact that
test[sevennet]
might havetest[m3gnet]
enabled, and that would inturn result in the conflict marker evaluating to
false
. This directlyimplies that our simplification here is inappropriate.
It would be nice to revisit how we build our inferences here so that
they are richer and enable us to make correct logical leaps. For now, we
fix this particular bug with a bit of a cop-out: we skip conflict marker
simplification when there are ambiguous dependency edges.
Fixes #11479