Extract code and output; script formatting and updating them

.cargo/config

@@ -0,0 +1,3 @@
+[cargo-new]
+name = "Your Name"
+email = "[email protected]"

ADMIN_TASKS.md

@@ -0,0 +1,105 @@
+# Administrative Tasks
+
+This documentation is for Carol and Steve and anyone else managing the repo to remember how to do
+occasional maintenance tasks.
+
+## Update the `rustc` version
+
+- Change the version number in `.travis.yml`
+- Change the version number in `rust-toolchain`, which should change the version you're using
+  locally with `rustup`
+- Change the version number in `src/title-page.md`
+- Run `./tools/update-rustc.sh` (see its commented code for details on what it does)
+- Inspect the changes (by looking at the files changed according to git) and their effects (by
+  looking at the files in `tmp/book-before` and `tmp/book-after`) and commit them if they look good
+- Grep for `manual-regeneration` and follow the instructions in those places to update output that
+  cannot be generated by a script
+
+## Release a new version of the listings
+
+We now make `.tar` files of complete projects containing every listing available [as GitHub
+Releases](https://github.com/rust-lang/book/releases). To create a new release artifact, for
+example if there have been code changes due to edits or due to updating Rust and `rustfmt`, do the
+following:
+
+- Create a git tag for the release and push it to GitHub, or create a new tag by going to the
+  GitHub UI, [drafting a new release](https://github.com/rust-lang/book/releases/new), and entering
+  a new tag instead of selecting an existing tag
+- Run `cargo run --bin release_listings`, which will generate `tmp/listings.tar.gz`
+- Upload `tmp/listings.tar.gz` in the GitHub UI for the draft release
+- Publish the release
+
+## Add a new listing
+
+To facilitate the scripts that run `rustfmt` on all the listings, update the output when the
+compiler is updated, and produce release artifacts containing full projects for the listings, any
+listing beyond the most trivial should be extracted into a file. To do that:
+
+- Find where the new listing should go in the `listings` directory.
+  - There is one subdirectory for each chapter
+  - Numbered listings should use `listing-[chapter num]-[listing num]` for their directory names.
+  - Listings without a number should start with `no-listing-` followed by a number that indicates
+    its position in the chapter relative to the other listings without numbers in the chapter, then
+    a short description that someone could read to find the code they're looking for.
+  - Listings used only for displaying the output of the code (for example, when we say "if we had
+    written x instead of y, we would get this compiler error:" but we don't actually show code x)
+    should be named with `output-only-` followed by a number that indicates its position in the
+    chapter relative to the other listings used only for output, then a short description that
+    authors or contributors could read to find the code they're looking for.
+  - **Remember to adjust surrounding listing numbers as appropriate!**
+- Create a full Cargo project in that directory, either by using `cargo new` or copying another
+  listing as a starting point.
+- Add the code and any surrounding code needed to create a full working example.
+- If you only want to show part of the code in the file, use anchor comments (`// ANCHOR: some_tag`
+  and `// ANCHOR_END: some_tag`) to mark the parts of the file you want to show.
+- For Rust code, use the `{{#rustdoc_include [fileame:some_tag]}}` directive within the code blocks
+  in the text. The `rustdoc_include` directive gives the code that doesn't get displayed to
+  `rustdoc` for `mdbook test` purposes.
+- For anything else, use the `{{#include [filename:some_tag]}}` directive.
+- If you want to display the output of a command in the text as well, create an `output.txt` file
+  in the listing's directory as follows:
+  - Run the command, like `cargo run` or `cargo test`, and copy all of the output.
+  - Create a new `output.txt` file with the first line `$ [the command you ran]`.
+  - Paste the output you just copied.
+  - Run `./tools/update-rustc.sh`, which should perform some normalization on the compiler output.
+  - Include the output in the text with the `{{#include [filename]}}` directive.
+  - Add and commit output.txt.
+- If you want to display output but for some reason it can't be generated by a script (say, because
+  of user input or external events like making a web request), keep the output inline but make a
+  comment that contains `manual-regeneration` and instructions for manually updating the inline
+  output.
+
+## See the effect of some change on the rendered book
+
+To check, say, updating `mdbook` or changing the way files get included:
+
+- Generate a built book before the change you want to test by running `mdbook build -d
+  tmp/book-before`
+- Apply the changes you want to test and run `mdbook build -d tmp/book-after`
+- Run `./tools/megadiff.sh`
+- Files remaining in `tmp/book-before` and `tmp/book-after` have differences you can manually
+  inspect with your favorite diff viewing mechanism
+
+## Produce new markdown files for No Starch
+
+- Run `./tools/nostarch.sh`
+- Spot check the files that script created in the `nostarch` directory
+- Check them into git if you're starting a round of edits
+
+## Produce markdown from docx for diffing
+
+- TODO Carol to document this next time she does it
+
+## Generate Graphviz dot
+
+We're using [Graphviz](http://graphviz.org/) for some of the diagrams in the
+book. The source for those files live in the `dot` directory. To turn a `dot`
+file, for example, `dot/trpl04-01.dot` into an `svg`, run:
+
+```bash
+$ dot dot/trpl04-01.dot -Tsvg > src/img/trpl04-01.svg
+```
+
+In the generated SVG, remove the width and the height attributes from the `svg`
+element and set the `viewBox` attribute to `0.00 0.00 1000.00 1000.00` or other
+values that don't cut off the image.

Cargo.toml

@@ -21,6 +21,14 @@ path = "tools/src/bin/lfp.rs"
 name = "link2print"
 path = "tools/src/bin/link2print.rs"
 
+[[bin]]
+name = "release_listings"
+path = "tools/src/bin/release_listings.rs"
+
+[[bin]]
+name = "remove_hidden_lines"
+path = "tools/src/bin/remove_hidden_lines.rs"
+
 [[bin]]
 name = "remove_links"
 path = "tools/src/bin/remove_links.rs"
@@ -35,3 +43,5 @@ docopt = "0.6.82"
 rustc-serialize = "0.3.19"
 regex = "0.1.73"
 lazy_static = "0.2.1"
+flate2 = "1.0"
+tar = "0.4.16"

README.md

@@ -83,20 +83,6 @@ before we merge any in, but feel free to start!
 [Translations]: https://github.com/rust-lang/book/issues?q=is%3Aopen+is%3Aissue+label%3ATranslations
 [mdbook support]: https://github.com/rust-lang-nursery/mdBook/issues/5
 
-## Graphviz dot
-
-We're using [Graphviz](http://graphviz.org/) for some of the diagrams in the
-book. The source for those files live in the `dot` directory. To turn a `dot`
-file, for example, `dot/trpl04-01.dot` into an `svg`, run:
-
-```bash
-$ dot dot/trpl04-01.dot -Tsvg > src/img/trpl04-01.svg
-```
-
-In the generated SVG, remove the width and the height attributes from the `svg`
-element and set the `viewBox` attribute to `0.00 0.00 1000.00 1000.00` or other
-values that don't cut off the image.
-
 ## Spellchecking
 
 To scan source files for spelling errors, you can use the `spellcheck.sh`

tools/nostarch.sh

@@ -6,19 +6,22 @@ cargo build --release
 
 mkdir -p tmp
 rm -rf tmp/*.md
+rm -rf tmp/markdown
 
-# Get all the Markdown files in the src dir,
-ls src/${1:-""}*.md | \
-# except for `SUMMARY.md`.
-grep -v SUMMARY.md | \
+# Render the book as Markdown to include all the code listings
+MDBOOK_OUTPUT__MARKDOWN=1 mdbook build -d tmp
+
+# Get all the Markdown files
+ls tmp/markdown/${1:-""}*.md | \
 # Extract just the filename so we can reuse it easily.
 xargs -n 1 basename | \
 # Remove all links followed by `<!-- ignore -->``, then
 # Change all remaining links from Markdown to italicized inline text.
 while IFS= read -r filename; do
-  < "src/$filename" ./target/release/remove_links \
+  < "tmp/markdown/$filename" ./target/release/remove_links \
     | ./target/release/link2print \
-    | ./target/release/remove_markup > "tmp/$filename"
+    | ./target/release/remove_markup \
+    | ./target/release/remove_hidden_lines > "tmp/$filename"
 done
 # Concatenate the files into the `nostarch` dir.
 ./target/release/concat_chapters tmp nostarch