doctests

[epage]

We talked about doctests being an area for improvement

[weihanglo]

rustdoc is responsible for both compiling doc examples and executing them as tests. Cargo current has no way to control the behavior. There are a handful of issues around this:

• Doctests are not cached and always recompile
  • Running doc tests are slow due to being always being recompiled cargo#2944
• Doctests generate a binary per test, causing a lot of extra linking
  • Rustdoc should run all doctests in one binary rust#75341
  • cargo test renders device unresponsive due to MSVC linker RAM usage cargo#12916
• Doctests are treated separately and often forgot to test
  • cargo test --all-targets does not run doc tests cargo#6669
• Compiling and running doc examples require different set of shared libs
  • cargo test --doc not load same shared libraries as cargo test --lib cargo#8531
• Configurations in Cargo are not respected
  • target.runner — RUSTC, RUSTC_WRAPPER are not respected for doctests cargo#12532
  • codegen options — Doctests do not set profile codegen options cargo#6570
• Doctests don't work on bin's, non-pub items
  • Doctests don't work in bin targets, non-public items rust#50784

Bonus

• Warnings/clippy on doctests
  • Oh rust doctest lints, where art þou? (Add a way to run clippy on doctests) rust#56232

I propose to the team to looking into the possibility for rustdoc to delegate test executions to Cargo.

[epage]

If we can put as many doctests into a single binary as possible, that would also improve compilation times

We also want to make it so its as natural to get coverage for doctests as other tests.

I propose to the team to looking into the possibility for rustdoc to delegate test executions to Cargo.

That is the direction I would want to go as well.

[weihanglo]

Yeah we can push it further.

rustdoc, as a doc comments analyzer, provides source information via a flag. Cargo invokes rustdoc with the flag, so it knows where the source is and how to compile and test it.

[GuillaumeGomez]

I'm very fine with this approach as it would remove code from rustdoc. Problem is that the code examples originally were written to allow crate writers to add code examples for their users, meaning they needed to import the crate as if it was a dependency. I think this approach is still good, but should not be the only one (for example we can't run doctest in binary crates, which is bad). If cargo can solve both issues, that'd be awesome. :)

[epage]

Crazy idea: could we change the compilation of #[doc] to generate #[test] fns for each code fence.

• If the test lib builds against the regular lib, then the import names still work
• Otherwise, the doctests would be usable on private items, having access to everything where they are located.

Challenges

• The compiler knowing enough about doctests to do this
• The existence of compile tests (maybe an unstable feature to assert on compilation state?)

[GuillaumeGomez]

Unfortunately, it would require some new attributes for it to work as you can add a lot of different checks on doctests such as a specific edition or even to ensure that the compilation fails. More information here.

[BurntSushi]

Would it be possible to resolve how the doctest should be built with a new attribute? Maybe something like internal to opt into "compile this test as a normal unit test that uses crate internal APIs."

[epage]

Unfortunately, it would require some new attributes for it to work as you can add a lot of different checks on doctests such as a specific edition or even to ensure that the compilation fails

Yeah, I was aware of and called out compile_fail. Being able to set the edition is an odd one. Where can I read up more on the use case for that? As I don't personally have a use case for it, my automatic reaction is "can this be deprecated?" but I recognize there are likely cases I'm not aware of.

Likely switching to this style of doctests would require an opt-in with it being the default in a new edition, so we could get away with changing things.

[epage]

btw rust-lang/rust#123974 is a nice improvement to performance.

[GuillaumeGomez]

It's not finished yet. A few things need to be added:

• only enable it starting 2024 edition
• remove chunking to run all tests at once (and improve even further the performance improvement) and potentially fix libtest being blocked

Once done, I plan to work on binary doctests: I want to replace them with what we do for #[test] so people will be able to finally test their own code.

[epage]

only enable it starting 2024 edition

Sounds like this is because the tests were building and running as separate processes and now they are being combined, causing changes in behavior through any interactions with globals?

Once done, I plan to work on binary doctests: I want to replace them with what we do for #[test] so people will be able to finally test their own code.

Could you clarify what you mean here?

Overall, I'm excited by these changes. The changes in performance will hopefully mean people will stop avoiding doctests. That said, I still feel like we need to do something to make these less special. The more we align with how things are normally built and run, the easier it is to take part in a more unified CLI, reporting, and coverage tracking.

[GuillaumeGomez]

Sounds like this is because the tests were building and running as separate processes and now they are being combined, causing changes in behavior through any interactions with globals?

Exactly! Should have precised. That's also why we'll need to add a new code attribute (named standalone or something along the line).

Once done, I plan to work on binary doctests: I want to replace them with what we do for #[test] so people will be able to finally test their own code.

Could you clarify what you mean here?

Absolutely: if your crate is a binary, then I plan to apply the same transformation to doctests as we do for #[test] tests (as described here). It's kinda what I already to merge all compatible doctests, but this time I would do it so they are generated in the crate source code directly, allowing users to test their internal code (which they can't currently as doctests are currently using the crate as dependency).

Don't hesitate if my explanations are not clear enough.

[epage]

Absolutely: if your crate is a binary, then I plan to apply the same transformation to doctests as we do for #[test] tests (as described here). It's kinda what I already to merge all compatible doctests, but this time I would do it so they are generated in the crate source code directly, allowing users to test their internal code (which they can't currently as doctests are currently using the crate as dependency).

Is this a variant of the idea mentioned earlier of generating #[test]s for binaries or something different? What is the path looking like for this?

[GuillaumeGomez]

Is this a variant of the idea mentioned earlier of generating #[test]s for binaries or something different?

Not that I know.

What is the path looking like for this?

Me doing it. First I want to finish the first PR (the speed up of doctests by merging them).

[epage]

FYI @GuillaumeGomez posted a status update on bins: rust-lang/rust#50784 (comment)

[epage]

For overriding file/line, lang is interested, it just needs someone to champion and workout corner cases, particularly if we also support column. We need to harvest Internals for feedback and respond.

e.g. https://internals.rust-lang.org/t/pre-rfc-enable-setting-of-source-file-line-column/19156

One idea: punt on column and code-generate to the right column

[epage]

@GuillaumeGomez in your update at rust-lang/rust#50784 (comment) you mention either having rustc translate doc-comments into #[test]s or generating the code for it. The former aligns with some of my ideas above for un-specializing doctests but only for tests for unit_test attributes. The latter doesn't immediately help the un-specializing. iirc you are starting your experiments without involving rustc (if nothing else, its much easier to prototype in only a single code base).

The part I don't remember is how much we talked about rustdoc generating, compiling, and running this code or only generating it and having cargo compile and run the code. The latter would help us un-specialize doctests.

[epage]

Doesn't have to be in the first release but an interesting spin off from that is if rustdoc emits json messages for these generated files, some of those could include that they are compile_fail and cargo could do the compile-fail part.

[weihanglo]

• Tracking issue for rustdoc --output-format=doctest command-line flag rust#134529

The --output-format=doctest can be leveraged for the experiment.

[epage]

Wasn't sure before but I just double checked, cargo doc is able to reuse the output of cargo check for dependencies that aren't being documented which means we can implement this without having to do two build passes for every dependency.

[epage]

I posted some feedback on --output-format=doctest at rust-lang/rust#134529 (comment)