Rustdoc html postprocessing

[SpecificProtagonist]

Objective

#17758 added tags indicating core Bevy traits to doc pages of types. This was implemented by adding JavaScript that modifies the DOM.
#17821 expands this to sort types in sections in module doc pages. Because no information about which item implements which trait is available in the module page, it employs a two step process where the source code is modified to invisibly embed this data. Besides being more complex, this leaves the repository in a dirty state when publishing.

Reimplement #17758 by postprocessing the html output instead. As a side benefit this works with JS disabled and allows using the same source of truth for trait tags and trait sections.

#17821 can then be reimplemented on top of this PR.

Solution

• Implement a wrapper around rustdoc and use it on docs.rs & the docs ci workflow
• Keep getting the information about which types implement which trait from the html (instead of switching to reading rustdoc's json output), as the mapping from item to html file isn't available in an easily digestible format
• Gracefully degrade when the structure of rustdoc's html output changes in an incompatible way, so at worst it will be the same as if we didn't do this customization.

Testing

Run

cargo build --package rustdoc-wrapper --release
cargo doc --config "build.rustdoc = \"target/release/rustdoc-wrapper\"" --workspace

and check out the doc. The tags should appear the same as on main.

[JeanMertz]

I love this.

I would also love for this to extend to the search feature (perhaps adding a fourth column). There seem to be many opportunities to improve the docs once this post-processing binary lands.

I also wonder if there's anything worth upstreaming (or at least suggesting) into rustdoc itself?

[JeanMertz]

One more thing we could do:

I know when we introduced required components, highlighting them in the docs was somewhat of a pain. I know we have it now, but it's not optimally visible. Perhaps we can (in a follow-up PR) make those more visible on component pages?

edit: I take this back. I assume this was iterated on since I last checked, or I misremembered, because they are pretty visible in the dev docs:

[JeanMertz]

I guess I'm using this as a dumping ground for suggestions, but here's another one that has bothered me often:

Whenever I search for a type, the first result (as seen in the above screenshot) that shows up is usually in the prelude. I would rather that one not be first, or completely filtered out/replaced, so that I go directly to the "source" module of the type, instead of any re-export location.

[GuillaumeGomez]

So if I understand correctly what you're trying to do here, you want for each type to list which components it requires, right? If so, instead of generating it like this, why not generating a markdown file you include like this:

#[cfg_attr(doc, doc = include!(concat!(DOC_DIR, "ITEM_NAME-traits.md"))]

The only thing you need then is to generate this file for all your items. You can pretty easily write a small rustc extension to do it for you and update these files before generating docs.

However:

1. This is not fully automatic, you need to add the #[cfg_attr()] attribute wherever needed for a type
2. You need to rerun the command to update the doc files

Luckily it can be mitigated with (proc-)macros that will add the doc file (if needed) for you, preventing doc generation to work if the file is empty or was not created.

EDIT: Forgot to mention but rustdoc HTML is not stable and we're tweaking it quite a lot lately to reduce pages size (among other things). So really you should try any other approach rather than any trying to manipulate/extract information from rustdoc pages using JS.

[SpecificProtagonist]

I would also love for this to extend to the search feature (perhaps adding a fourth column).

Whenever I search for a type, the first result (as seen in the above screenshot) that shows up is usually in the prelude. I would rather that one not be first, or completely filtered out/replaced, so that I go directly to the "source" module of the type, instead of any re-export location.

Those do sound useful, and I've been annoyed by the reexports often enough myself. I'd be hesitant to mess with search though as that means messing with rustdoc's JavaScript.

I could see a new doc attribute to affect priority of an item in the search – or maybe just adjusting the search heuristic – being useful to other crates too, but you'd have to ask rustdoc's maintainers (hi GuillaumeGomez!).

edit: I take this back. I assume this was iterated on since I last checked, or I misremembered, because they are pretty visible in the dev docs:
image

Yeah, they used to be on the impl Component section before I moved them to the main section.

[SpecificProtagonist]

Thanks for your feedback!

So if I understand correctly what you're trying to do here, you want for each type to list which components it requires, right?

Ah no, required components is a separate thing already solved by a doc-attribute generating proc macro :)

Bevy is a dependency injection framework where individual crates expose types that implement one (or rarely multiple) of a set of common traits. When learning how to use a module, the first thing you want to know what kind of item each of its datatypes is – that is, whether it is a Plugin, a SystemSet, a Resource etc., which determines where it can be used – and only then whether its a struct, enum or union, what it's fields/variants are, which intrinsic methods it has and what other traits like Clone it implements.

Without any changes, the first thing to do when looking at an item's doc page is to either guess its kind based on context or to scroll the table of contents down to the "Trait Implementations" section and scan it for these traits. When looking at a module doc page, finding this information is even more important to understanding how the module works, yet it isn't present at all, even though information about whether a type is a struct, enum or union is presented.

To alleviate this issue, we (well, JMS55) have come up with two changes:

• On the page of an individual type, display a label below the heading. This is already merged, so you can check out dev-docs.bevyengine.org to see how that looks.

• On the page of a module, instead of grouping datatypes into struct/enum/union, they're grouped into Component/Resource/Plugin/…

  Example

  

Forgot to mention but rustdoc HTML is not stable

Yep! We are aware that rustdoc's html output is unstable and not intended to ever be stabilized. Luckily it's ok if this breaks when rustdoc's output changes (and there won't be any complaints from me when that happens), because this is a feature that, while useful, nobody relies on. Additionally, when rustdoc changes, this doesn't affect the documentation on docs.rs until a new crate version is published.

So really you should try any other approach rather than any trying to manipulate/extract information from rustdoc pages using JS.

To be fair, the interface for rustc drivers which you suggested is also unstable, and we can't just pin a version either because that would introduce a maximum supported rust version, and Bevy's minimum supported rust version is "latest stable".

[mockersf]

I don't think this is a good approach, and one we will want to maintain.

I would prefer to solve #17821 by including markdown files as suggested by @GuillaumeGomez

[SpecificProtagonist]

I would prefer to solve #17821 by including markdown files

If we decide not to do #17821, I think this PR should still be merged, as it moves the trait tags from being added via JS to being part of the static HTML without making maintenance harder/being less stable.

If we do decide to implement #17821 by adding an additional listing, we'll still need a way to find out which traits a type implements. If we keep this PR, we can use that information for the trait tags too so that the only thing about the HTML structure that we rely on is that the heading is a <h1> tag (which would be an improvement over main).

[alice-i-cecile]

I think this is too intrusive / unstable to build on top of docs.rs. I like the core concept, but I think this is likely to be a maintenance headache.