diff --git a/src/SUMMARY.md b/src/SUMMARY.md index a7b76233d..449a7d0c3 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -101,6 +101,8 @@ - [Rustdoc internals](./rustdoc-internals.md) - [Search](./rustdoc-internals/search.md) - [The `rustdoc` test suite](./rustdoc-internals/rustdoc-test-suite.md) + - [The `rustdoc-gui` test suite](./rustdoc-internals/rustdoc-gui-test-suite.md) + - [The `rustdoc-json` test suite](./rustdoc-internals/rustdoc-json-test-suite.md) - [Autodiff internals](./autodiff/internals.md) - [Installation](./autodiff/installation.md) - [How to debug](./autodiff/debugging.md) diff --git a/src/rustdoc-internals.md b/src/rustdoc-internals.md index bc91c62d8..0234d4a92 100644 --- a/src/rustdoc-internals.md +++ b/src/rustdoc-internals.md @@ -270,35 +270,6 @@ in `test.rs` is the function `make_test`, which is where hand-written Some extra reading about `make_test` can be found [here](https://quietmisdreavus.net/code/2018/02/23/how-the-doctests-get-made/). -## Dotting i's And Crossing t's - -So that's `rustdoc`'s code in a nutshell, but there's more things in the -compiler that deal with it. Since we have the full `compiletest` suite at hand, -there's a set of tests in `tests/rustdoc` that make sure the final `HTML` is -what we expect in various situations. These tests also use a supplementary -script, `src/etc/htmldocck.py`, that allows it to look through the final `HTML` -using `XPath` notation to get a precise look at the output. The full -description of all the commands available to `rustdoc` tests (e.g. [`@has`] and -[`@matches`]) is in [`htmldocck.py`]. - -To use multiple crates in a `rustdoc` test, add `//@ aux-build:filename.rs` -to the top of the test file. `filename.rs` should be placed in an `auxiliary` -directory relative to the test file with the comment. If you need to build -docs for the auxiliary file, use `//@ build-aux-docs`. - -In addition, there are separate tests for the search index and `rustdoc`'s -ability to query it. The files in `tests/rustdoc-js` each contain a -different search query and the expected results, broken out by search tab. -These files are processed by a script in `src/tools/rustdoc-js` and the `Node.js` -runtime. These tests don't have as thorough of a writeup, but a broad example -that features results in all tabs can be found in `basic.js`. The basic idea is -that you match a given `QUERY` with a set of `EXPECTED` results, complete with -the full item path of each item. - -[`@has`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py#L39 -[`@matches`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py#L44 -[`htmldocck.py`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py - ## Testing Locally Some features of the generated `HTML` documentation might require local diff --git a/src/rustdoc-internals/rustdoc-gui-test-suite.md b/src/rustdoc-internals/rustdoc-gui-test-suite.md new file mode 100644 index 000000000..e155f960e --- /dev/null +++ b/src/rustdoc-internals/rustdoc-gui-test-suite.md @@ -0,0 +1,14 @@ +# The `rustdoc-gui` test suite + +> **FIXME**: This section is a stub. Please help us flesh it out! + +This page is about the test suite named `rustdoc-gui` used to test the "GUI" of `rustdoc` (i.e., the HTML/JS/CSS as rendered in a browser). +For other rustdoc-specific test suites, see [Rustdoc test suites]. + +These use a NodeJS-based tool called [`browser-UI-test`] that uses [puppeteer] to run tests in a headless browser and check rendering and interactivity. For information on how to write this form of test, see [`tests/rustdoc-gui/README.md`][rustdoc-gui-readme] as well as [the description of the `.goml` format][goml-script] + +[Rustdoc test suites]: ../tests/compiletest.md#rustdoc-test-suites +[`browser-UI-test`]: https://github.com/GuillaumeGomez/browser-UI-test/ +[puppeteer]: https://pptr.dev/ +[rustdoc-gui-readme]: https://github.com/rust-lang/rust/blob/master/tests/rustdoc-gui/README.md +[goml-script]: https://github.com/GuillaumeGomez/browser-UI-test/blob/master/goml-script.md diff --git a/src/rustdoc-internals/rustdoc-json-test-suite.md b/src/rustdoc-internals/rustdoc-json-test-suite.md new file mode 100644 index 000000000..e08f77095 --- /dev/null +++ b/src/rustdoc-internals/rustdoc-json-test-suite.md @@ -0,0 +1,3 @@ +# The `rustdoc-json` test suite + +> **FIXME**: This section is a stub. It will be populated by [PR #2422](https://github.com/rust-lang/rustc-dev-guide/pull/2422/). diff --git a/src/rustdoc-internals/rustdoc-test-suite.md b/src/rustdoc-internals/rustdoc-test-suite.md index bad7ac19d..5d8727dc5 100644 --- a/src/rustdoc-internals/rustdoc-test-suite.md +++ b/src/rustdoc-internals/rustdoc-test-suite.md @@ -1,112 +1,190 @@ # The `rustdoc` test suite -This page is specifically about the test suite named `rustdoc`. -For other test suites used for testing rustdoc, see [Rustdoc tests](../rustdoc.md#tests). +This page is about the test suite named `rustdoc` used to test the HTML output of `rustdoc`. +For other rustdoc-specific test suites, see [Rustdoc test suites]. -The `rustdoc` test suite is specifically used to test the HTML output of rustdoc. +Each test file in this test suite is simply a Rust source file `file.rs` sprinkled with +so-called *directives* located inside normal Rust code comments. +These come in two flavors: *Compiletest* and *HtmlDocCk*. -This is achieved by means of `htmldocck.py`, a custom checker script that leverages [XPath]. +To learn more about the former, read [Compiletest directives]. +For the latter, continue reading. -[XPath]: https://en.wikipedia.org/wiki/XPath +Internally, [`compiletest`] invokes the supplementary checker script [`htmldocck.py`]. -## Directives -Directives to htmldocck are similar to those given to `compiletest` in that they take the form of `//@` comments. +[Rustdoc test suites]: ../tests/compiletest.md#rustdoc-test-suites +[`compiletest`]: ../tests/compiletest.md +[`htmldocck.py`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py -In addition to the directives listed here, -`rustdoc` tests also support most -[compiletest directives](../tests/directives.html). +## HtmlDocCk Directives -All `PATH`s in directives are relative to the rustdoc output directory (`build/TARGET/test/rustdoc/TESTNAME`), -so it is conventional to use a `#![crate_name = "foo"]` attribute to avoid -having to write a long crate name multiple times. -To avoid repetition, `-` can be used in any `PATH` argument to re-use the previous `PATH` argument. +Directives to HtmlDocCk are assertions that place constraints on the generated HTML. +They look similar to those given to `compiletest` in that they take the form of `//@` comments +but ultimately, they are completey distinct and processed by different programs. -All arguments take the form of quoted strings -(both single and double quotes are supported), -with the exception of `COUNT` and the special `-` form of `PATH`. +[XPath] is used to query parts of the HTML document tree. + +**Introductory example**: + +```rust,ignore (illustrative) +//@ has file/type.Alias.html +//@ has - '//*[@class="rust item-decl"]//code' 'type Alias = Option;' +pub type Alias = Option; +``` + +Here, we check that documentation generated for crate `file` contains a page for the +public type alias `Alias` where the code block that is found at the top contains the +expected rendering of the item. The `//*[@class="rust item-decl"]//code` is an XPath +expression. -Directives are assertions that place constraints on the generated HTML. +Conventionally, you place these directives directly above the thing they are meant to test. +Technically speaking however, they don't need to be as HtmlDocCk only looks for the directives. -All directives (except `files`) can be negated by putting a `!` in front of their name. +All directives take a `PATH` argument. +To avoid repetition, `-` can be passed to it to re-use the previous `PATH` argument. +Since the path contains the name of the crate, it is conventional to add a +`#![crate_name = "foo"]` attribute to the crate root to shorten the resulting path. + +All arguments take the form of shell-style (single or double) quoted strings, +with the exception of `COUNT` and the special `-` form of `PATH`. + +All directives (except `files`) can be *negated* by putting a `!` in front of their name. +Before you add negated directives, please read about [their caveats](#caveats). Similar to shell commands, directives can extend across multiple lines if their last char is `\`. In this case, the start of the next line should be `//`, with no `@`. -For example, `//@ !has 'foo/struct.Bar.html'` checks that crate `foo` does not have a page for a struct named `Bar` in the crate root. +Use the special string `{{channel}}` in XPaths, `PATTERN` arguments and [snapshot files](#snapshot) +if you'd like to refer to the URL `https://doc.rust-lang.org/CHANNEL` where `CHANNEL` refers to the +current release channel (e.g, `stable` or `nightly`). + +Listed below are all possible directives: + +[XPath]: https://en.wikipedia.org/wiki/XPath ### `has` -Usage 1: `//@ has PATH` -Usage 2: `//@ has PATH XPATH PATTERN` +> Usage 1: `//@ has PATH` -In the first form, `has` checks that a given file exists. +Check that the file given by `PATH` exists. -In the second form, `has` is an alias for `matches`, -except `PATTERN` is a whitespace-normalized[^1] string instead of a regex. +> Usage 2: `//@ has PATH XPATH PATTERN` -### `matches` +Checks that the text of each element / attribute / text selected by `XPATH` in the +whitespace-normalized[^1] file given by `PATH` matches the +(also whitespace-normalized) string `PATTERN`. + +**Tip**: If you'd like to avoid whitespace normalization and/or if you'd like to match with a regex, +use `matches` instead. -Usage: `//@ matches PATH XPATH PATTERN` +### `hasraw` -Checks that the text of each element selected by `XPATH` in `PATH` matches the python-flavored regex `PATTERN`. +> Usage: `//@ hasraw PATH PATTERN` -### `matchesraw` +Checks that the contents of the whitespace-normalized[^1] file given by `PATH` +matches the (also whitespace-normalized) string `PATTERN`. -Usage: `//@ matchesraw PATH PATTERN` +**Tip**: If you'd like to avoid whitespace normalization and / or if you'd like to match with a +regex, use `matchesraw` instead. -Checks that the contents of the file `PATH` matches the regex `PATTERN`. +### `matches` -### `hasraw` +> Usage: `//@ matches PATH XPATH PATTERN` -Usage: `//@ hasraw PATH PATTERN` +Checks that the text of each element / attribute / text selected by `XPATH` in the +file given by `PATH` matches the Python-flavored[^2] regex `PATTERN`. -Same as `matchesraw`, except `PATTERN` is a whitespace-normalized[^1] string instead of a regex. +### `matchesraw` + +> Usage: `//@ matchesraw PATH PATTERN` + +Checks that the contents of the file given by `PATH` matches the +Python-flavored[^2] regex `PATTERN`. ### `count` -Usage: `//@ count PATH XPATH COUNT` +> Usage: `//@ count PATH XPATH COUNT` -Checks that there are exactly `COUNT` matches for `XPATH` within the file `PATH`. +Checks that there are exactly `COUNT` matches for `XPATH` within the file given by `PATH`. ### `snapshot` -Usage: `//@ snapshot NAME PATH XPATH` +> Usage: `//@ snapshot NAME PATH XPATH` -Creates a snapshot test named NAME. -A snapshot test captures a subtree of the DOM, at the location -determined by the XPath, and compares it to a pre-recorded value -in a file. The file's name is the test's name with the `.rs` extension -replaced with `.NAME.html`, where NAME is the snapshot's name. +Checks that the element / text selected by `XPATH` in the file given by `PATH` matches the +pre-recorded subtree or text (the "snapshot") in file `FILE_STEM.NAME.html` where `FILE_STEM` +is the file stem of the test file. -htmldocck supports the `--bless` option to accept the current subtree -as expected, saving it to the file determined by the snapshot's name. -compiletest's `--bless` flag is forwarded to htmldocck. +Pass the `--bless` option to `compiletest` to accept the current subtree/text as expected. +This will overwrite the aforementioned file (or create it if it doesn't exist). It will +automatically normalize the channel-dependent URL `https://doc.rust-lang.org/CHANNEL` to +the special string `{{channel}}`. ### `has-dir` -Usage: `//@ has-dir PATH` +> Usage: `//@ has-dir PATH` -Checks for the existence of directory `PATH`. +Checks for the existence of the directory given by `PATH`. ### `files` -Usage: `//@ files PATH ENTRIES` +> Usage: `//@ files PATH ENTRIES` + +Checks that the directory given by `PATH` contains exactly `ENTRIES`. +`ENTRIES` is a Python-like list of strings inside a quoted string. + +**Example**: `//@ files "foo/bar" '["index.html", "sidebar-items.js"]'` + +[^1]: Whitespace normalization means that all spans of consecutive whitespace are replaced with a single space. +[^2]: They are Unicode aware (flag `UNICODE` is set), match case-sensitively and in single-line mode. + +## Compiletest Directives (Brief) + +As mentioned in the introduction, you also have access to [compiletest directives]. +Most importantly, they allow you to register auxiliary crates and +to pass flags to the `rustdoc` binary under test. +It's *strongly recommended* to read that chapter if you don't know anything about them yet. + +Here are some details that are relevant to this test suite specifically: -Checks that the directory `PATH` contains exactly `ENTRIES`. -`ENTRIES` is a python list of strings inside a quoted string, -as if it were to be parsed by `eval`. -(note that the list is actually parsed by `shlex.split`, -so it cannot contain arbitrary python expressions). +* While you can use both `//@ compile-flags` and `//@ doc-flags` to pass flags to `rustdoc`, + prefer to user the latter to show intent. The former is meant for `rustc`. +* Add `//@ build-aux-docs` to the test file that has auxiliary crates to not only compile the + auxiliaries with `rustc` but to also document them with `rustdoc` -Example: `//@ files "foo/bar" '["index.html", "sidebar-items.js"]'` +## Caveats -[^1]: Whitespace normalization means that all spans of consecutive whitespace are replaced with a single space. The files themselves are also whitespace-normalized. +Testing for the absence of an element or a piece of text is quite fragile and not very future proof. + +It's not unusual that the *shape* of the generated HTML document tree changes from time to time. +This includes for example renamings of CSS classes. + +Whenever that happens, *positive* checks will either continue to match the intended element / +attribute / text if their XPath expression is general / loose enough and thus test the correct thing +or they won't in which case they would fail forcing the author of the change tolook at them. + +Compare that to *negative* checks (e.g., `//@ !has PATH XPATH PATTERN`) which won't fail if their +XPath expression "no longer" matches. The author who changed "the shape" thus won't get notified and +as a result someone else can unintentionally reintroduce `PATTERN` into the generated docs without +the original negative check failing. + +**Note**: Please avoid the use of *negated* checks! + +**Tip**: If you can't avoid it, please **always** pair it with an analogous positive check in the +immediate vicinity, so people changing "the shape" have a chance to notice and to update the +negated check! ## Limitations -`htmldocck.py` uses the xpath implementation from the standard library. + +HtmlDocCk uses the XPath implementation from the Python standard library. This leads to several limitations: + * All `XPATH` arguments must start with `//` due to a flaw in the implementation. * Many XPath features (functions, axies, etc.) are not supported. * Only well-formed HTML can be parsed (hopefully rustdoc doesn't output mismatched tags). +Furthmore, compiletest [revisions] are not supported. + +[revisions]: ../tests/compiletest.md#revisions +[compiletest directives]: ../tests/directives.md diff --git a/src/rustdoc.md b/src/rustdoc.md index de70ba638..52ae48c37 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -67,43 +67,29 @@ does is call the `main()` that's in this crate's `lib.rs`, though.) ## Code structure -* All paths in this section are relative to `src/librustdoc` in the rust-lang/rust repository. +All paths in this section are relative to `src/librustdoc/` in the rust-lang/rust repository. + * Most of the HTML printing code is in `html/format.rs` and `html/render/mod.rs`. - It's in a bunch of `fmt::Display` implementations and supplementary - functions. -* The types that got `Display` impls above are defined in `clean/mod.rs`, right - next to the custom `Clean` trait used to process them out of the rustc HIR. + It's in a bunch of functions returning `impl std::fmt::Display`. +* The data types that get rendered by the functions mentioned above are defined in `clean/types.rs`. + The functions responsible for creating them from the `HIR` and the `rustc_middle::ty` IR + live in `clean/mod.rs`. * The bits specific to using rustdoc as a test harness are in `doctest.rs`. * The Markdown renderer is loaded up in `html/markdown.rs`, including functions for extracting doctests from a given block of Markdown. * Frontend CSS and JavaScript are stored in `html/static/`. + * Re. JavaScript, type annotations are written using [TypeScript-flavored JSDoc] +comments and an external `.d.ts` file. + This way, the code itself remains plain, valid JavaScript. + We only use `tsc` as a linter. -## Tests +[TypeScript-flavored JSDoc]: https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html -* Tests on search engine and index are located in `tests/rustdoc-js` and `tests/rustdoc-js-std`. - The format is specified - [in the search guide](rustdoc-internals/search.md#testing-the-search-engine). -* Tests on the "UI" of rustdoc (the terminal output it produces when run) are in - `tests/rustdoc-ui` -* Tests on the "GUI" of rustdoc (the HTML, JS, and CSS as rendered in a browser) - are in `tests/rustdoc-gui`. These use a [NodeJS tool called - browser-UI-test](https://github.com/GuillaumeGomez/browser-UI-test/) that uses - puppeteer to run tests in a headless browser and check rendering and - interactivity. For information on how to write this form of test, - see [`tests/rustdoc-gui/README.md`][rustdoc-gui-readme] - as well as [the description of the `.goml` format][goml-script] -* Tests on the structure of rustdoc HTML output are located in `tests/rustdoc`, - where they're handled by the test runner of bootstrap and - the supplementary script `src/etc/htmldocck.py`. - [These tests have several extra directives available to them](./rustdoc-internals/rustdoc-test-suite.md). -* Additionally, JavaScript type annotations are written using [TypeScript-flavored JSDoc] - comments and an external d.ts file. The code itself is plain, valid JavaScript; we only - use tsc as a linter. +## Tests -[TypeScript-flavored JSDoc]: https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html -[rustdoc-gui-readme]: https://github.com/rust-lang/rust/blob/master/tests/rustdoc-gui/README.md -[goml-script]: https://github.com/GuillaumeGomez/browser-UI-test/blob/master/goml-script.md +`rustdoc`'s integration tests are split across several test suites. +See [Rustdoc tests suites](tests/compiletest.md#rustdoc-test-suites) for more details. ## Constraints diff --git a/src/tests/compiletest.md b/src/tests/compiletest.md index ee06ca3b6..20dd16c81 100644 --- a/src/tests/compiletest.md +++ b/src/tests/compiletest.md @@ -56,6 +56,9 @@ incremental compilation. The various suites are defined in The following test suites are available, with links for more information: +[`tests`]: https://github.com/rust-lang/rust/blob/master/tests +[`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs + ### Compiler-specific test suites | Test suite | Purpose | @@ -71,6 +74,7 @@ The following test suites are available, with links for more information: | [`mir-opt`](#mir-opt-tests) | Check MIR generation and optimizations | | [`coverage`](#coverage-tests) | Check coverage instrumentation | | [`coverage-run-rustdoc`](#coverage-tests) | `coverage` tests that also run instrumented doctests | +| [`crashes`](#crashes-tests) | Check that the compiler ICEs/panics/crashes on certain inputs to catch accidental fixes | ### General purpose test suite @@ -78,19 +82,23 @@ The following test suites are available, with links for more information: ### Rustdoc test suites -See [Rustdoc tests](../rustdoc.md#tests) for more details. - -| Test suite | Purpose | -|------------------|--------------------------------------------------------------------------| -| `rustdoc` | Check `rustdoc` generated files contain the expected documentation | -| `rustdoc-gui` | Check `rustdoc`'s GUI using a web browser | -| `rustdoc-js` | Check `rustdoc` search is working as expected | -| `rustdoc-js-std` | Check rustdoc search is working as expected specifically on the std docs | -| `rustdoc-json` | Check JSON output of `rustdoc` | -| `rustdoc-ui` | Check terminal output of `rustdoc` | - -[`tests`]: https://github.com/rust-lang/rust/blob/master/tests -[`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs +| Test suite | Purpose | +|--------------------------------------|--------------------------------------------------------------------------| +| [`rustdoc`][rustdoc-html-tests] | Check HTML output of `rustdoc` | +| [`rustdoc-gui`][rustdoc-gui-tests] | Check `rustdoc`'s GUI using a web browser | +| [`rustdoc-js`][rustdoc-js-tests] | Check `rustdoc`'s search engine and index | +| [`rustdoc-js-std`][rustdoc-js-tests] | Check `rustdoc`'s search engine and index on the std library docs | +| [`rustdoc-json`][rustdoc-json-tests] | Check JSON output of `rustdoc` | +| `rustdoc-ui` | Check terminal output of `rustdoc` ([see also](ui.md)) | + +Some rustdoc-specific tests can also be found in `ui/rustdoc/`. +These check rustdoc-related or -specific lints that (also) run as part of `rustc`, not (only) `rustdoc`. +Run-make tests pertaining to rustdoc are typically named `run-make/rustdoc-*/`. + +[rustdoc-html-tests]: ../rustdoc-internals/rustdoc-test-suite.md +[rustdoc-gui-tests]: ../rustdoc-internals/rustdoc-gui-test-suite.md +[rustdoc-js-tests]: ../rustdoc-internals/search.md#testing-the-search-engine +[rustdoc-json-tests]: ../rustdoc-internals/rustdoc-json-test-suite.md ### Pretty-printer tests diff --git a/src/tests/directives.md b/src/tests/directives.md index 2dff21ed6..f73a2811d 100644 --- a/src/tests/directives.md +++ b/src/tests/directives.md @@ -261,7 +261,7 @@ Consider writing the test as a proper incremental test instead. | Directive | Explanation | Supported test suites | Possible values | |-------------|--------------------------------------------------------------|------------------------------------------|---------------------------| -| `doc-flags` | Flags passed to `rustdoc` when building the test or aux file | `rustdoc`, `rustdoc-js`, `rustdoc-json` | Any valid `rustdoc` flags | +| `doc-flags` | Flags passed to `rustdoc` when building the test or aux file | `rustdoc`, `rustdoc-js`, `rustdoc-json` | Any valid `rustdoc` flags | +#### Test-suite-specific directives + +The test suites [`rustdoc`][rustdoc-html-tests], [`rustdoc-js`/`rustdoc-js-std`][rustdoc-js-tests] +and [`rustdoc-json`][rustdoc-json-tests] each feature an additional set of directives whose basic +syntax resembles the one of compiletest directives but which are ultimately read and checked by +separate tools. For more information, please read their respective chapters as linked above. + +[rustdoc-html-tests]: ../rustdoc-internals/rustdoc-test-suite.md +[rustdoc-js-tests]: ../rustdoc-internals/search.html#testing-the-search-engine +[rustdoc-json-tests]: ../rustdoc-internals/rustdoc-json-test-suite.md + ### Pretty printing See [Pretty-printer](compiletest.md#pretty-printer-tests).