deal with reworked Doom initialization

[marienz]

doomemacs/doomemacs@8cafbe4 makes some changes to Doom's startup process that breaks how Unstraightened hooks into that process. I'm not sure yet how to deal with this.

Previously, Doom's early-init.el would load doom-start.el, which would load $DOOMDIR/init.el early. It would do so relative to doom-user-dir, which we control (by setting DOOMDIR to a store path): we use this to load Unstraightened's init.el (which loads the user's real init.el). It then loads the profile's init.29.elc, relative to doom-profile-dir, which we also control (either because we set doom-profile-data-dir in the profile loader and Doom initialized doom-profile-dir relative to that, or because we set it in init.el).

The changes in that commit include (assuming I read everything correctly):

• Doom now loads the profile init file using (doom-profile-init-file doom-profile), which always returns a path relative to doom-data-dir. We can't override doom-data-dir, it's used to store all kinds of mutable data. We were already overriding (using advice) doom-profile-init-file, however...
• Doom now loads $DOOMDIR/init.el from the profile init file (doom-profile-generate includes (doom-load (file-name-concat doom-user-dir "init.el") t) as one of the first things in the profile init file). That means our doom-profile-init-file advice is installed too late to fix the problem (catch-22).

Short of "patch Doom", I think that leaves me with the profile loader as the only point to gain control early enough. Possible approaches:

• Set up advice from there
• Point doom-local-dir into the store and then undo that from init.el

But I'm going to hold off for a week or so: the new code contains some intriguing references to a doom-profile struct (which presumably exists in Doom 3.0), and the commit message mentions one more Doom 3.0 backport is incoming. I'm expecting that next backport to further break Unstraightened (because it'll change doom! blocks, which is fairly likely to break my dumping of the user's init.el), but it just might also include more ways of overriding profile-related paths from the profile loader. It might turn out to be easier to update directly to that state than do this in stages (and if not, I can still update in stages, just delayed).

[marienz]

doomemacs/doomemacs@b3aa41f will require further changes on my part, but those changes will be unrelated to the initialization ones. So it's starting to look like a piecemeal migration will be easier after all.

I'll give it a bit more time for things to settle before tackling that.

[marienz]

I'll have to deal with this soonish (next couple of days if possible): CI (which has been failing for a bit because of NixOS/nixpkgs#353702, I was waiting for that change to make it into nixpkgs-unstable...) is now failing because the use-package repo moved. Doom just fixed that with doomemacs/doomemacs@ef33cb3, and although I could pin to the same revision in our fetch-overrides.nix I don't want to go down that path...

[marienz]

Progress:

• 84d9ec1 pulls in all of Doom's init refactoring, and seems to work locally.
• Further updates trigger problems in org-noter (nixpkgs applies a patch that does not apply, looks like I'll need to refresh the patch) and emacsql (looks like nixpkgs tries to build a custom sqlite binary upstream dropped). Probably not too difficult to fix...

[marienz]

ff9ea90 brings us back up-to-date (assuming it passes tests, which it should as it passed them locally).

I'll push that to main after https://nixpk.gs/pr-tracker.html?pr=356505 reaches the nixos-24.11 channel (CI only covers nixpkgs-unstable, been meaning to fix that...).

[zarthross]

I'm using 07aa2a8 and it appears to work for me.
I was having issues with org-roam after a nix update.

Thanks for the work on this!

[marienz]

Thanks for testing!

Glad to hear org-roam works now (I knew it was broken, wasn't sure it was entirely fixed). Unfortunate timing there, with those emacsql changes around the same time as the Doom initialization changes...

Still waiting for that emacsql fix to reach nixos-24.11 (I don't understand nixpkgs build/release well enough to predict how long that'll take...).

[marienz]

Good news: the emacsql fix reached nixos-24.11, so I pushed the changes to main.

Bad news: doing so revealed org-roam v1 (aka (org +roam) is still broken if your Emacs supports native-comp (which it normally does, but I turn that off for nix flake check for performance reasons). org-roam v2 (aka (org +roam2)) does build. This breaks CI (and I assume breaks users if they have (org +roam) and a normal Emacs).

@zarthross maybe you can help me understand if asking people to upgrade to org-roam v2 is reasonable? Doom has described org-roam v1 as "basically EOL" since 2021, and although it does mention v2 is a full rewrite with a need to migrate your data, I'm having trouble quickly working out how different v2 is to use once you're past that. Is this a "you have to retrain your brain and fingers to use the new thing" situation or more of a "people might want to wait for the new thing to stabilize first" (which it hopefully has in the past 3 years)?

(I'll see if I can fix org-roam's build, but with upstream dead and emacsql changing underneath it I'm a little worried this won't be sustainable)

[marienz]

I think org-roam v1 just hasn't worked since around 2023-04-03, when the emacs-sqlite3 package was dropped from MELPA (and as a result from nixpkgs). At first glance it is not possible to use org-roam-v1 without that (its org-roam-db.el hard-depends on it).

So I think all that changed is that the build error became fatal, and "just don't buid org-roam v1" is reasonable (...although implementing that is unfortunately going to be a bit clunky, as (org +roam +roamv2) picks v1, not v2, so my "all flags" build breaks).

(...I wonder if this is outright broken for normal Doom users as well? If it is, and I can get org-roam-v1 dropped there, that'd simply things. Should test that once I have things papered over...)

[marienz]

...nope, Doom pulls a recipe for emacsql-sqlite3 from emacsattic. So it probably does still work for them.

For now I'll keep it disabled unless someone speaks up. It looks fixable but not trivially: I can use that emacsql-sqlite3 recipe, but the way my elisp-packages.nix works doesn't let me force that dependency onto org-roam just if it's v1 (and I don't want to force the v1 dependencies onto v2 users if nobody's really using v1 anymore).

[zarthross]

@zarthross maybe you can help me understand if asking people to upgrade to org-roam v2 is reasonable? Doom has described org-roam v1 as "basically EOL" since 2021, and although it does mention v2 is a full rewrite with a need to migrate your data, I'm having trouble quickly working out how different v2 is to use once you're past that. Is this a "you have to retrain your brain and fingers to use the new thing" situation or more of a "people might want to wait for the new thing to stabilize first" (which it hopefully has in the past 3 years)?

Sorry, I'm a recent convert into org-roam, started directly onto v2 just a few weeks ago. But from what I've read its possible to migrate, but could take an hour or two with manual work to do so. If someone would rather stay on old, outdated and deprecated versions, I would say they can submit a PR here to fix it... but even then its likely a waste of time, since it appears to be slated for removal from Doom itself https://github.com/orgs/doomemacs/projects/5/views/1?filterQuery=-status%3ARejected%2CDone+roam

[marienz]

Thanks for checking! That doesn't sound worth spending time on, so I'll close this off.