Host --output-format json files in addition to generated HTML

[deeprobin]

I need this to implement my own "doc"-frontend for me into a multi-page-application.

[syphar]

@deeprobin what kind of information would you need?

docs.rs is just hosting some metadata and is serving the HTML files generated by rustdoc,

[deeprobin]

Oh okay. Then it would be better if I open a issue in the rustdoc repository? I thought this was the right repository for this.

[Nemo157]

Rustdoc has an in-progress JSON backend: rust-lang/rust#76578

There has been some very brief discussion of docs.rs generating and hosting these files along with the html. But at the moment I think it's quite buggy (e.g. rust-lang/rust#80664 is probably going to fail a large proportion of builds) and the format is unstable. I think it would make sense to wait a little longer till it has stabilized a bit more before we add it here (I know of at least a couple of users that are already building alternative renderers based on the json, which will hopefully drive it towards stabilization).

[DottieDot]

Any progress/updates on this?

[syphar]

There are still open ICEs when generating the JSON (see this current discussion on zulip, rust-lang/rust#93518 is one of the fixes.

When this is solved I believe we have to partially finish #795 so we have the additional spare capacity for running the additional builds.

Of course on top of that the actual code change needs to be done.

[jyn514]

I don't think we should start hosting this until the JSON backend is stabilized. The output format is likely to change several more times before stabilization.

[syphar]

yeah, also valid point, I only had in mind that we could find more ICEs when running it for docs, but the format changing could be annoying for users of the JSON

[syphar]

• we're planning on building the JSON with the same docs.rs metadata (= the same feature set) as the HTML docs. ( this seems to be OK for cargo-semver-checks too @obi1kenobi )
• the download will be a single JSON download via CDN, probably gz-compressed.

there are some missing features before we deem the format stable enough: ( coming from @jyn514 )

• github.com/rustdoc-json: Improve Item::attrs to not use debug representation. rust#137645
• https://hackmd.io/@aDot/HkY9b6l-xg (cross crate item Ids) probably in multiple issues / prs ([rustdoc-json] paths is inconsistent and questionably useful rust#93522, [rustdoc-json] Partially remove paths and introduce external_index rust#103085, Rustdoc-Json: some item IDs are missing in paths field. rust#101687, [rustdoc-json] absolute position of item is hard to get rust#93524)

[syphar]

next set of notes coming from the all-hands / rustweek: we can actually start building & hosting rustdoc json.

about the implementation:

• memory usage of the JSON build is higher than the HTML build. We might hit limits for crates that reach that limit.
• Files would be compressed on S3, potentially with zstd
• download API is a redirect to static.docs.rs for download, similar to the archive download.
• format_version could be part of the filename / API.
• question: can we keep hosting old format versions? and provide via API?

Normal rebuilds for the latest version of each crate would then also build & upload the rustdoc json. So after a couple of months of this being live, all latest versions have rustdoc json.

There are some pending additions that will come later that will make it easier to link between json docs easier (notably: extend the information we provide in external_crates)

edit: don't forget, we have a separate JSON output per target to store.

[syphar]

implementation notes from first codebase read:

• rustdoc will clear the /target/doc/ directory before the HTML or JSON builds. So we have to pull out the JSON output before we run the HTML build
• we have to store & support multiple targets.
• due to the potentially higher memory usage of rustdoc json, the JSON build might fail when HTML doesn't. We want HTML to be published even after a failed JSON build. Do we need to store separate logs for crate authors to see this somewhere? Or is our internal sentry / log good enough, similar to coverage-generate?
• any API would probably redirect to default target if no target is given.

structure on S3 bucket is currently:

• rustdoc: /rustdoc/{krate}/{version}.zip
• sources: /sources/{krate}/{version}.zip
• build logs: /build-logs/{build_id}/{target}.txt

and legacy locations before archive-storage

• rustdoc: /rustdoc/{krate}/{version}/
• sources: /sources/{krate}/{version}/

after each build we store the archive, and run delete_prefix on the legacy locations.

For rustdoc JSON we preferably want a folder structure where we can later s3 ls and return the available format-versions?

This means the best structure is probably a separate root: /rustdoc-json/{krate}/{version}/{target}/{format_version}.json.zst, perhaps duplicating krate & version & target on the filename for a nicer download. s3 ls could use all files from the target folder.

That also means every request to docs.rs for a rustdoc JSON file needs an s3 ls call. The only way to prevent that, so we can statically determine the path, would be if we remove the format version from the path, or additionally maintain a file without the format version that we overwrite each rebuild.

[syphar]

@obi1kenobi following up on your request to be able to fetch old format versions, if we have them.

With my approach from above you would be able to run parallel requests to:

https://static.docs.rs/rustdoc-json/{krate}/{version}/{target}/{format_version}.json.zst

using your list of supported format versions. You'll get either the file or a 404 then, the 404s also cached when I updated the cloudfront config then.

Would that suffice for your use-case?

( perhaps I add the other parameters to the filename additionally to make it self consistent in your download directory)