Making use of workspaces that need unregistered packages

[MarkNahabedian]

Pluto Project Manager is a great new feature, but, alas, is not of much use to me since most of my notebooks make use of at least one of my own packages. None of my Julia packages are mature enough to register yet and some are personal mods to other Julia packages whose authors have not accepted my changes.

This notebook

https://github.com/MarkNahabedian/DesignWithSVG/blob/master/floor_drain/floor_drain.jl

has six calls to Pkg.add. It takes about 2 minutes to build the workspace each time the notebook is opened.

If I were only working from my laptop I could have a local directory host that workspace, but this does not work if I want to access the notebook on a mobile device through Binder.

This repository

https://github.com/MarkNahabedian/WebBasedWorkspace

is an experiment to use a web based workspace. I currently feel it is too cumbersome to use though since that package itself must be added before it can be used.

When I was first learning pluto Notebook, all of the useful UI components were invisible to me since my poor eyesight requires that I use a dark theme. The controls are all still invisible,but I've at least learned where a few of them are. I investigated what it would take to fix this and got as far as determining that the problem is actuatlly in the icon library that Pluto is using to draw these buttons.

Since I couldn't figure out how to add a new cell, I created PlutoTool

https://github.com/MarkNahabedian/PlutoTool.jl

as a sort of swiss army knife command line utility for manipulating a Pluto notebook off-line. BTW: the code in Pluto's notebook.jl was easy to understand and leverage for this.

With the introduction of the Pluto Package Manager, I added a subcommand to PlutoTool to inject PROJECT and MANIFEST cells into a notebook file. I was hoping I could embed the workspace into a notebook before pushing it to GitHub. I've not yet grasped all of the subtlety to this though and Pluto seems to overwrite the workspace that PlutoTool embedded.

I think that the most ideal solution would be if Julia added a method to Pkg.activate that took a URI as argument. Maybe I should put in a request for that.

Another alternative is to provide a PlutoUI function that can activate a workspace given a URI.

What do people feel is a good approach to address using unregistered packages from a Pluto notebook without huge setup times?

Thanks.

[disberd]

If you have packages that are unregistered but are still hosted on either github or gitlab, you can set up a local registry and Pluto Pkg Manager also looks into custom registries for fetching packages.
I also have a very similar case where I am using Pluto with custom packages and I did set up my private registry using this package https://github.com/GunnarFarneback/LocalRegistry.jl.

Set up is fairly straightforward and then you can have the convenience of the package manager. Only problem is that if you need to share this notebook with someone that does not have your same registry added as custom registry, things break.

[MarkNahabedian]

I want my notebooks to be accessible on Binder so I can still work with them from my phone.

Is there a way to tell Pluto where to find my registry that doesn't also turn off it's package manager?

[disberd]

@MarkNahabedian It seems that you can actually add manually non-registered packages from the julia repl using Pluto.activate_notebook_environment(notebook_path) as explained in https://github.com/fonsp/Pluto.jl/wiki/%F0%9F%8E%81-Package-management#advanced-edit-the-notebook-environment.

You can use this to add unregistered packages using github repos. Didn't test if it works on binder but I don't see why it shouldn't

[MarkNahabedian]

Thanks. I was initially hopeful when I read your response, but it doesn't work with URIs:

my_workspace_uri = "https://raw.githubusercontent.com/MarkNahabedian/DesignWithSVG/master/floor_drain"
"https://raw.githubusercontent.com/MarkNahabedian/DesignWithSVG/master/floor_drain"
Pluto.activate_notebook_environment(my_workspace_uri)
ERROR: SystemError: opening file "https://raw.githubusercontent.com/MarkNahabedian/DesignWithSVG/master/floor_drain": Invalid argument
Stacktrace:
  [1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
    @ Base .\error.jl:168
  [2] #systemerror#62
    @ .\error.jl:167 [inlined]
  [3] systemerror
    @ .\error.jl:167 [inlined]
  [4] open(fname::String; lock::Bool, read::Nothing, write::Nothing, create::Nothing, truncate::Nothing, append::Nothing)
    @ Base .\iostream.jl:293
  [5] open
    @ .\iostream.jl:282 [inlined]
  [6] open(f::Base.var"#326#327"{String}, args::String; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Base .\io.jl:328
  [7] open
    @ .\io.jl:328 [inlined]
  [8] read
    @ .\io.jl:434 [inlined]
  [9] readwrite
    @ ~\.julia\packages\Pluto\8cQn7\src\notebook\PathHelpers.jl:130 [inlined]
 [10] load_notebook(path::String; disable_writing_notebook_files::Bool)
    @ Pluto ~\.julia\packages\Pluto\8cQn7\src\notebook\Notebook.jl:297
Pluto.activate_notebook_environment(URIs.URI(my_workspace_uri))
ERROR: MethodError: no method matching activate_notebook_environment(::URI)
Closest candidates are:
  activate_notebook_environment(!Matched::String) at C:\Users\Mark Nahabedian\.julia\packages\Pluto\8cQn7\src\packages\PkgUtils.jl:109
Stacktrace:
 [1] top-level scope
   @ none:1

[disberd]

I don't understand what you wanted to achieve here..
You should provide a notebook file path to the activate_notebook_environment function, so some local file that is a .jl file.
You are trying to use the raw github url for a folder (which is anyway broken as a link).

What you should do for achieving what I believe you want, is put the local file of the floor_drain.jl file that you have in your clone of the repo, modify the environment to add whichever package you want, and after push your modified notebook to your github repo.

Edit:
I had a look at the notebook file in your repo but you should remove these lines https://github.com/MarkNahabedian/DesignWithSVG/blob/d6b286180c1e9503c40935fffb0eba474094f045/floor_drain/floor_drain.jl#L23-L29
As having Pkg.add inside the notebook disables the built in Pkg.manager.

Remove those lines from the file, activate the notebook environment and re-add manually the unregistered packages. Then just have using statements in your notebook and test it locally before pushing the modified file to github

[MarkNahabedian]

I created my own registry of my repositories. The process imposes a number or requirements that I was not initially willing to accept, among them, that the registry have its own repository. My original intent was to have this registry only be used when developing my design notebooks, so I wanted to just have it be part of my ShaperOriginDesignLib repository. Once I gave up on that intention and accepted that my regsitry had to be its own repository, I made slow progress.

My registry is at https://github.com/MarkNahabedian/NahaJuliaRegistry.git. The README there details my efforts and observations. I got to use the excellent JuliaWeb/GitHub.jl package to access the GitHub REST API. This allowed me to automate identifying my Julia repositories, and pulling clean clones of the most recent tagged versions of each. For some reason LocalPackage requires local clean clones rather than going to the web. I guess it would either have to only work with GitHub or implement an abstraction barrier for other web based version control services.

It turns out that building a registry was not necessary to achieve my goal of making my Pluto notebooks work on Binder though.

Now that I understand what it does, Pluto.activate_notebook_environment, suggested by @disberd, appears to be the solution.

using Pkg
using Pluto
Pluto.activate_notebook_environment("c:/Users/Mark Nahabedian/Documents/svg-design/floor_drain/floor_drain.jl")
Pkg.add(PackageSpec(; url="https://github.com/MarkNahabedian/ShaperOriginDesignLib.git",
                      name="ShaperOriginDesignLib"))
Pkg.add(PackageSpec(; name="NativeSVG",
                      url="https://github.com/MarkNahabedian/NativeSVG.jl.git"))
Pkg.add(["Unitful", "UnitfulUS", "PlutoUI", "DisplayAs"])
Pkg.status()
      Status `C:\Users\Mark Nahabedian\AppData\Local\Temp\jl_n5nktx\floor_drain.jl\Project.toml`
  [0b91fe84] DisplayAs v0.1.2
  [5f95e7b2] NativeSVG v0.1.0-naha `https://github.com/MarkNahabedian/NativeSVG.jl.git#master`
  [7f904dfe] PlutoUI v0.7.27
  [919415d1] ShaperOriginDesignLib v0.2.0 `https://github.com/MarkNahabedian/ShaperOriginDesignLib.git#master`
  [1986cc42] Unitful v1.10.0
  [7dc9378f] UnitfulUS v0.2.0

I'm not quite there yet though.

My notebook runs fine in a local Pluto, but if I run it on
Binder

I get the error

ArgumentError: Package NativeSVG not found in current path:

- Run `import Pkg; Pkg.add("NativeSVG")` to install the NativeSVG package.

require(::Module, ::Symbol)@loading.jl:893

Why can't it find NativeSVG?
It's right there in the manifest.

[MarkNahabedian]

I think we're misunderstanding each other.

As I said in my original post:

If I were only working from my laptop I could have a local directory host that workspace, but this does not work if I want to access the notebook on a mobile device through Binder.

I have a github repository that contains a Julia workspace. I want several different notebooks to be able to use that workspace, from Binder, with minimal code addition to each notebook and minimal time overhead in running the notebook.

Part of the problem is that Julia's Pkg.activate seems to require a local directory, presumably so that the Pkg API can modify it. One could imagine it allowing activation of a workspace from a URI and throwing an error if one tried to modify it.

One could have ones notebook read the workspace from Github and add each package, but that takes a lot of runtime to recompute the manifest and requires additional code, which must be added and used.

One could have one's notebook create a tempdir and copy the Project.toml and Manifest.toml files there and then activate that tempdir like

begin
	using Pkg
	Pkg.add("HTTP")
	using HTTP

	workspace = mktempdir()
	from_package = "https://raw.githubusercontent.com/MarkNahabedian/PanelCutting.jl/master/"

	for f in ["Project.toml", "Manifest.toml"]
		from = joinpath(from_package, f)
		to = joinpath(workspace, f)
		response = HTTP.request("GET", from)
		@assert response.status == 200
		open(to, "w") do f
			write(f, String(response.body))
		end
	end
	
	#Pkg.activate(workspace)
	workspace
end

but I've not yet managed to make that work in a Pluto notebook. It complains that I need to Pkg.add VectorLogging, even though it is present in both the Project.toml andManifest.toml files in the tempdir.

As near as I can tell, activate_notebook_environment requires a local file path.

Tiratde: During college, and for a decade after, I got to use the MIT Lisp Machine and its descendants. The lisp machine was an individual workstation that didn't necessarily have enough disk space for a local file system. The system primitives for manipulating files could do so on any FTP or NFS server that was reachable over the network. That was in the 1980s and I've not seen that capability in any environment since.

[MarkNahabedian]

I'm adding a comment to this discussion rather than opening an issue because the discussion provides context into why I'm trying to do what I'm trying to do: have a Pluto notebook use a pre-generated workspace that is stored on a web site to reduce notebook startup time.

This notebook copied the Project.tomland Manifest.toml files from
https://raw.githubusercontent.com/MarkNahabedian/PanelCutting.jl/master/workspace_for_binder
to a local temporary directory and activates that directory.

When run in Pluto, that notebook gets the error

ArgumentError: Package PanelCutting [c1372bff-64f3-4c29-a5fb-167619e966f3] is required but does not seem to be installed:

inthe second cell, but when run within Julia gets no such error:

c:\Users\Mark Nahabedian\.julia\dev>julia
julia
include("PanelCutting/src/panels_notebook.jl")
    Updating git-repo `https://github.com/MarkNahabedian/NahaJuliaLib.jl`
    Updating registry at `C:\Users\Mark Nahabedian\.julia\registries\General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
  No Changes to `C:\Users\Mark Nahabedian\.julia\environments\v1.6\Project.toml`
  No Changes to `C:\Users\Mark Nahabedian\.julia\environments\v1.6\Manifest.toml`
Precompiling project...
  ✓ PanelCutting
1 dependency successfully precompiled in 20 seconds (227 already precompiled)
   Resolving package versions...
  No Changes to `C:\Users\Mark Nahabedian\.julia\environments\v1.6\Project.toml`
  No Changes to `C:\Users\Mark Nahabedian\.julia\environments\v1.6\Manifest.toml`
Precompiling project...
  ✓ PanelCutting
1 dependency successfully precompiled in 18 seconds (227 already precompiled)
  Activating environment at `C:\Users\MARKNA~1\AppData\Local\Temp\jl_sGR9T5\Project.toml`
VectorLogger(LogEntry[LogEntry(Info, "dot exit status", PanelCutting, :generic_dot, :PanelCutting_061d5675, "C:\\Users\\Mark Nahabedian\\.julia\\packages\\PanelCutting\\2u0g6\\src\\generic_dot.jl", 45, Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:status, :cmd), Tuple{Int64, Cmd}}}(:status => 0, :cmd => `dot -Tsvg '-oc:/Users/Mark Nahabedian/.julia/dev/PanelCutting/src/two-panels-0.svg'`)), LogEntry(Info, "c:/Users/Mark Nahabedian/.julia/dev/PanelCutting/src/two-panels-0.svg", Main, :panels_notebook, :Main_5d71d5d7, "c:\\Users\\Mark Nahabedian\\.julia\\dev\\PanelCutting\\src\\panels_notebook.jl", 550, Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}()), LogEntry(Info, "dot exit status", PanelCutting, :generic_dot, :PanelCutting_061d5675, "C:\\Users\\Mark Nahabedian\\.julia\\packages\\PanelCutting\\2u0g6\\src\\generic_dot.jl", 45, Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:status, :cmd), Tuple{Int64, Cmd}}}(:status => 0, :cmd => `dot -Tsvg '-oc:/Users/Mark Nahabedian/.julia/dev/PanelCutting/src/two-panels-1.svg'`)), LogEntry(Info, "c:/Users/Mark Nahabedian/.julia/dev/PanelCutting/src/two-panels-1.svg", Main, :panels_notebook, :Main_008c4aa5, "c:\\Users\\Mark Nahabedian\\.julia\\dev\\PanelCutting\\src\\panels_notebook.jl", 552, Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}()), LogEntry(Info, "dot exit status", PanelCutting, :generic_dot, :PanelCutting_061d5675, "C:\\Users\\Mark Nahabedian\\.julia\\packages\\PanelCutting\\2u0g6\\src\\generic_dot.jl", 45, Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:status, :cmd), Tuple{Int64, Cmd}}}(:status => 0, :cmd => `dot -Tsvg '-oc:/Users/Mark Nahabedian/.julia/dev/PanelCutting/src/two-panels-2.svg'`)), LogEntry(Info, "c:/Users/Mark Nahabedian/.julia/dev/PanelCutting/src/two-panels-2.svg", Main, :panels_notebook, :Main_bf4050c8, "c:\\Users\\Mark Nahabedian\\.julia\\dev\\PanelCutting\\src\\panels_notebook.jl", 554, Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}()), LogEntry(Info, "dot exit status", PanelCutting, :generic_dot, :PanelCutting_061d5675, "C:\\Users\\Mark Nahabedian\\.julia\\packages\\PanelCutting\\2u0g6\\src\\generic_dot.jl", 45, Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:status, :cmd), Tuple{Int64, Cmd}}}(:status => 0, :cmd => `dot -Tsvg '-oc:/Users/Mark Nahabedian/.julia/dev/PanelCutting/src/two-panels-3.svg'`)), LogEntry(Info, "c:/Users/Mark Nahabedian/.julia/dev/PanelCutting/src/two-panels-3.svg", Main, :panels_notebook, :Main_8d18900f, "c:\\Users\\Mark Nahabedian\\.julia\\dev\\PanelCutting\\src\\panels_notebook.jl", 556, Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}())])
:done
:done
exit()

I don't understand why these behaviors are not consistent.

Is this a bug?

Thanks.