Packages depending on ez-conf-lib can't find the ez-conf-lib script

[gridbugs]

For example, building apronext.1.0.4 with dune package management, dune produces the error:

File "dune.lock/conf-gmp-paths.pkg", line 5, characters 2-4:
5 |   sh
      ^^
Error: Logs for package conf-gmp-paths
/usr/bin/sh: 0: cannot open /home/s/apronext.1.0.4/_build/.sandbox/0b313d221691802236609a779e591e93/_private/default/.pkg/ez-conf-lib/target/lib/ez-conf-lib/ez-conf-lib: No such file

It's a little suspicious that the path it's using to run ez-conf-lib is inside the ez-conf-lib's build sandbox, as that path won't necessarily exist later in the build when a dependent package is being built. This could be a similar issue to what we see with ocamlbuild and ocamlfind which we fixed by forking those projects into an overlay repo. Possibly that is necessary here too.

An unusual feature of ez-conf-lib is that when it's installed with opam the executable ends up in ~/.opam/default/lib/ez-conf-lib/ez-conf-lib. It's not in the bin directory but instead the package's lib directory. Need to double check that this is being added to the PATH used by dune to find executables scoped within packages.

[gridbugs]

The problem appears to be that variables set in a package's generated <name>.config file are always treated by dune as strings, which means that paths are not translated out of dune's sandbox. My plan to solve this is to have dune attempt to detect when a variable contains a path inside the project's _build directory and treat such values as paths rather than strings, which will allow dune to translate sandboxed paths out of the sandbox so they can be referred to by other packages.

[rgrinberg]

Did you consider just making this package relocatable? We've also have some support for non-relocatable packages like the compiler. Perhaps we could generalize that? Both sound a lot more appealing than post-processing the config file.

[gridbugs]

I'll see what that would involve. We'd need to make the generated ez-conf-lib.config file contain the correct path to the ez-conf-lib executable outside of the sandbox so other packages can find the executable.

[gridbugs]

I've played around with modifying ez-conf-lib to make it portable on this branch. Here's some more background on the problem and how I've attempted to solve it.

Clients of ez-conf-lib invoke the script like:

build: [
  "sh"
  ez-conf-lib:exe
  "gmp"
  "gmp.h"
  "test-gmp.c"
  "--package-name"
  "conf-gmp-paths"
  "--"
  "/usr/local" {os != "macos" & os != "win32"}
  "/opt/homebrew" {os = "macos"}
  "/opt/local" {os = "macos"}
]

ez-conf-lib:exe is a variable from the ez-conf-lib.config generated during ez-conf-libs's install. The reason ez-conf-lib is not portable is that the generated ez-conf-lib.config file contains the absolute path to the ez-conf-lib shell script inside the temporary build sandbox where the package is installed. This causes clients to attempt to invoke the script from its build sandbox rather than its final destination.

The ideal solution would be to remove the exe variable altogether and modify all clients (there are only 2 revdeps in opam) to locate the script in a different way (e.g. something like ez-conf-lib:lib/ez-conf-lib). This would respect ez-conf-lib's intentions that the ez-conf-lib script be installed to lib rather than bin (as the script is not supposed to be accessible from the command line) and fix the relocatibility issue. The problem with this approach is it would require forking and maintaining patched versions of more packages.

Alternatively (and what I currently do on my branch) we attempt to fix the problem by just patching ez-conf-lib and not any of its revdeps. If we change ez-conf-lib to install its script into the bin directory rather than lib and set the value of the ez-conf-lib:exe variable to simply ez-conf-lib (rather than the full path to the script inside lib) then clients will continue to work unmodified. The drawback is that this change probably can't be upstreamed, since presumably the author of ez-conf-lib chose to install it into libexec rather than bin for a good reason.

So in the short term we should probably use the latter solution and add a patched version of ez-conf-lib to our overlay repo, and then submit patches upstream to ez-conf-lib and its 2 clients to implement the former solution in the long run.

[Leonidas-from-XIV]

I wrote some of my thoughts in the pull request you opened. The most general way is to post-process the .config files on installation and then pre-process the files upon usage but this adds a lot of complexity to make .config files relocatable.

Given @nberth is the author of both ez-conf-lib as well as both the revdeps that support it maybe we can discuss some other solution to get this working? I am fairly sure that we could implement some acceptable way to make ez-conf-lib relocatable upstream and that would benefit both Dune and OPAM (e.g. when relocatable lands, opam switches become mostly relocatable thus they will also have the same issues as we run into at the moment so might as well solve the issue early).

@nberth, what do you think about this?

[nberth]

Hi all. Reading all the discussions now I think I understand what you are trying to achieve. I would be totally fine to make ez-conf-lib relocatable.
I hacked that a long time ago (at least, that's how it feels after working on lots of other projects), and I'm not sure to grasp all the details of dune's internals: could you explain why, if I understand correctly, using ez-conf-lib:lib/ez-conf-lib instead of using ez-conf-lib:exe would solve the issue (would it?).

[nberth]

Let me add that I would be very uncomfortable adding the script to the user PATH as it really is quite a bunch of hacks.

[Leonidas-from-XIV]

@nberth Thanks for reaching out! Let me give you a quick rundown of the issue.

The general issue is that whenever an absolute path is saved into some of the files it means that that path has to exist in the future, when people want to refer to it. The way Dune works is that it builds artifacts in a sandbox location (which gets recorded in the .config file), moves the file to the install location (and the cache) and deletes the sandbox. However when using the artifacts, they might still refer to the old sandbox location and thus fail.

(One could of course not sandbox, but one of the ideas behind package management in Dune is that artifacts can be cached, thus a different project would be able to restore ez-conf-lib from the cache in a different location and have it just work, cutting down installation times immensely)

The way we solve that usually is by avoiding to store the absolute path, instead making it relative to.. something. So /home/leonidas-from-xiv/projects/myproject/_build/.sandbox/ex-conf-lib/bin becomes $PREFIX/ex-conf-lib/bin. When using the path the consumer has to translate it back to an actual path - but the .config file itself stays relocatable. This is how we patched ocamlfind to find its configuration.

could you explain why, if I understand correctly, using ez-conf-lib:lib/ez-conf-lib instead of using ez-conf-lib:exe would solve the issue (would it?).

The way @gridbugs explained to me here: It sidesteps the issue because it doesn't refer to the binary via absolute path, but finds the binary in the (Dune-created) PATH instead. Thus the recorded path is never used.

Of course if other entries in the .config file use absolute paths to the sandbox location, that is not enough and that would still break but it allows to install the 2 existing revdeps.

[nberth]

Hum. I think I remember a bit more what I did. When opam builds ez-conf-lib it uses a scripts gen-config that essentially creates an OPAM variable ez-conf-lib:exe that is an absolute path to the script executable. I think that was only a simple trick to make the package easier to use by its revdeps. Using ez-conf-lib:lib/ez-conf-lib instead of the variable is more redundant and less explicit from a user point of view, but, well… that does not make so many users. If you're saying your mechanism transparently deals with ez-conf-lib:lib, that could be the way to go.