Automation metabug

[brson]

Unprioritized list of automation tasks.

• bundling mingw Package all required components of the MinGW toolchain #11782
• fully test windows Run full test suite on windows bots #16457
• rev1 mingw-w64 64-bit toolchain Upgrade win64 bots to rev1 mingw-w64 toolchain #17394
• rust/cargo install Figure out combined Rust+Cargo installation #16456
• release channel Add release channel automation #16677
• out of sync nightlies Nightly version mismatch #14431
• feature staging Add the ability to completely disable experimental features in release channels #16678
• valgrind Run valgrind on a bot #13217
• testing 32 bit on distcheck 32-bit nightlies are not being tested #17344
• 32-bit cargo on CentOS
• OSX auto builds are segfaulting (no issue)
• high-5 Create bot to assign new PR's to reviewers #16674
• merging win32/64 bots Merge the win32/win64 bots #17354
• cloudfare sync is weird Outdated nightlies on Travis-CI #16649
• msys2 win32 Build with msys2 on the win32 bots #17347
• lexer tests Move checking of tests with little x-platform risk to supplemental auto bot, turn on lexer tests #15880
• benchmark bot Add a dedicated benchmark slave #12458
• lldb tests Enable lldb tests on the mac bots #16112
• test more OS X need to test with more OSX versions #17269
• OSX is segfaulting with nightlies nightly-mac bot is regularly segfaulting #16489
• more android slaves Add more android slaves #16795
• increase builds/slave bump max builds to 3 on bots #17395
• doc/master -> doc/nightly Rename the doc/master S3 folder to doc/nightly #17398
• license packaging Install README.md, COPYRIGHT, LICENSE-MIT, LICENSE-APACHE #17690
• doc installation Consider installing HTML documentation #14916
• shutdown idle slaves Teach buildbot to shutdown EC2 slaves when idle #18151
• losing windows slaves Windows buildslaves are lost and never come back #18703
• nightlies running at wrong time Reschedule nightlies for middle of the night #18912
• stability upload fails incorrectly Failed stability upload causes remainder of build to not execute #19079
• remove metrics infra Remove stability and benchmark metric infrastructure #19145
• doc tools on dist bots Ensure that all dist bots contain build the same set of docs #19200
• mac minis -> macstadium Move automation off mac minis, onto macstadium #19286
• guidelines Publish guidelines somewhere official #19315
• rust-by-example Set up automation for rust-by-example #19455
• release signing Have bots automatically sign distributable artifacts #13252
• snap signing Have bots sign snapshots #13254
• upload rustup.sh Upload rustup.sh to static. automatically #19505
• test 'make install' Buildbot should not only test "make check" but also "make install" #19698
• install an uninstall script Uninstalling Rust downloads the whole Rust again #20283
• fix dist security Fix dist automation security #20629
• package dbg std Package debug libstd with binaries #20630
• directory listings Set up directory listings for the distribution directory on s3 #20838
• update rustup for beta Update rustup.sh for beta #21149
• install source Install source code for debugging #19535
• get rid of cloudfront Invalidate artifacts on CloudFront more reliably #21239
• cargo dist dir Move uploaded cargo bins to dist/ from cargo-dist #21240
• machine readable dist index Add machine-readable dist index #21241
• channel metadata Add additional channel manifest metadata #21243
• windows dist locking bug Windows dist builders hit upload exception due to locking bug in buildbot #21434
• ec2 spot instances Switch to ec2 spot instances #21904
• cargo compatibility bots automated testing against old rustc's cargo#1276
• rust package desyncing rust packages get desynched rust-lang-deprecated/rust-buildbot#7
• cross-platform type system consistency for std portability risk: impls for libstd type can be platform dependent #22228
• slave per build Start a new slave per build #22448
• fix build date Make the 'build date' reported by --version *the exact same* as the archive date #22774
• permission denied on windows Intermittent "Permission denied" on Windows #22628
• beta/stable integration automation Add beta/stable integration testing #23756
• doc translation services Set up documentation translation server #25660
• upload feature list Upload feature list to s3 #25724
• www.rust-lang.org SSL serve the rust-lang.org domain over https #13180
• move from buildbot to taskcluster?
• turn off ssh on play.rlo Turn off public SSH access to play.rust-lang.org #25811
• signing key revcert Add a revocation cert to the Rust signing key #25814
• keybase.io Publish signing key and security team key to keybase.io #25815
• convert playpen sandbox Convert to standard container infrastructure rust-playpen#134

[edunham]

This list represents the unchecked items in brson's list, sorted into categories which roughly correlate to what action they look like they'll need next, and annotated with summaries/updates where necessary. Duplicating mainly so I can have permissions on a copy of the list, with issue numbers turned into links correctly.

Brson's Notes-To-Self

• upload rustup.sh Upload rustup.sh to static. automatically #19505
• channel metadata Add additional channel manifest metadata #21243
• fix build date Make the 'build date' reported by --version *the exact same* as the archive date #22774 closed in favor of Remove build date from --version #25812, which is now closed too
• windows dist locking bug Windows dist builders hit upload exception due to locking bug in buildbot #21434
• doc translation services Set up documentation translation server #25660

Things that got closed or became a different issue

• release channel Add release channel automation #16677 Add beta/stable integration testing #23757 is only surviving child of this issue
• beta/stable integration automation Add beta/stable integration testing #23756 -- closed for
  Add beta/stable integration testing #23757
• more android slaves Add more android slaves #16795 -- will probably fork into enabling more
  android configs, if that doesn't exist yet
• rust package desyncing rust packages get desynched rust-lang-deprecated/rust-buildbot#7 -- is closed now, yay
• cross-platform type system consistency for std portability risk: impls for libstd type can be platform dependent #22228 -- original bug got
  closed; needs new bug?
• remove metrics infra Remove stability and benchmark metric infrastructure #19145 -- it got finished, then closed, then
  reopened by a comment referencing Fix scripts/bench-branches.py servo/html5ever#72

Things that look "easy"

• benchmark bot Add a dedicated benchmark slave #12458 -- looks "easy" if stuff still works
• doc/master -> doc/nightly Rename the doc/master S3 folder to doc/nightly #17398 -- pretty sure i can do this one, but
  is it dev.rust-lang.org or static.rust-lang.org bucket?
• test 'make install' Buildbot should not only test "make check" but also "make install" #19698 -- has caused pile of bugs. maybe not done
  yet because slow? should be "easy"
• guidelines Publish guidelines somewhere official #19315 -- work done in
  Import rust-guidelines #22331 needs to be linked from somewhere
• snap signing Have bots sign snapshots #13254 -- directly from
  Have bots automatically sign distributable artifacts #13252
• upload feature list Upload feature list to s3 #25724 -- looks clearly defined, could be a good
  first "real bug"
• www.rust-lang.org SSL serve the rust-lang.org domain over https #13180 -- kick nginx and the DNS

Buildbot things, or things that'll change a lot if we switch CI

• slave per build Start a new slave per build #22448 -- nice-to-have
• fix dist security Fix dist automation security #20629 -- needs BUILDBOT to become https
• ec2 spot instances Switch to ec2 spot instances #21904 -- looks like a matter of
  read-the-linked-manual
• move from buildbot to taskcluster?

Everything Else (mostly things that need design decisions)

• testing 32 bit on distcheck 32-bit nightlies are not being tested #17344 -- need to add more cases to make
  check. Slowdowns incoming!
• lexer tests Move checking of tests with little x-platform risk to supplemental auto bot, turn on lexer tests #15880 -- hello, yak (hairier than its cousin the yacc)...
• test more OS X need to test with more OSX versions #17269 -- tools are identified; licensing situation
  unknown
• losing windows slaves Windows buildslaves are lost and never come back #18703 -- needs troubleshooting
• doc tools on dist bots Ensure that all dist bots contain build the same set of docs #19200 -- needs design decision
• mac minis -> macstadium Move automation off mac minis, onto macstadium #19286 -- needs research, design, implementation
• package dbg std Package debug libstd with binaries #20630 -- needs design decision
• install source Install source code for debugging #19535 -- needs design decision
• get rid of cloudfront Invalidate artifacts on CloudFront more reliably #21239 -- turned into artifact invalidation. lots
  of noise here; having it broken makes many people very unhappy
• cargo dist dir Move uploaded cargo bins to dist/ from cargo-dist #21240 -- design decision kinda made; needs implementation
  knowledge
• cargo compatibility bots automated testing against old rustc's cargo#1276 -- cargo needs better
  tests