Redirect all 2.7 files removed in Python 3

[verhovsky]

See python/cpython#24195 and python/pythondotorg#1719

Also see https://github.com/verhovsky/py3redirect/blob/master/special-cases.js for a list of anchors that have been moved.

Generated using the following code:

moved_files = {
    "library/fpformat": None,
    "library/mutex": None,
    "library/new": None,
    "library/statvfs": None,
    "library/dircache": None,
    "library/macpath": None,
    "library/dbhash": None,
    "library/bsddb": None,
    # "library/someos": "library/index.html",
    "library/someos": None,
    # "library/popen2": "library/subprocess.html",
    "library/popen2": None,
    # "library/mhlib": "library/mailbox.html",
    "library/mhlib": None,
    # "library/mimetools": "library/email.html",
    "library/mimetools": None,
    # "library/mimewriter": "library/email.html",
    "library/mimewriter": None,
    # "library/mimify": "library/email.html",
    "library/mimify": None,
    # "library/multifile": "library/email.html",
    "library/multifile": None,
    "library/sgmllib": None,
    "library/imageop": None,
    # "library/hotshot": "library/profile.html",
    "library/hotshot": None,
    "library/future_builtins": None,
    "library/user": None,
    "library/fpectl": None,
    "library/restricted": None,
    "library/rexec": None,
    "library/bastion": None,
    "library/imputil": None,
    "library/compiler": None,
    # "library/dl": "library/ctypes.html",
    "library/dl": None,
    "library/posixfile": None,
    # "library/commands": "library/subprocess.html",
    "library/commands": None,
    "library/mac": None,
    "library/ic": None,
    "library/macos": None,
    "library/macostools": None,
    "library/easydialogs": None,
    "library/framework": None,
    "library/autogil": None,
    "library/carbon": None,
    "library/colorpicker": None,
    "library/macosa": None,
    "library/gensuitemodule": None,
    "library/aetools": None,
    "library/aepack": None,
    "library/aetypes": None,
    "library/miniaeframe": None,
    "library/sgi": None,
    "library/al": None,
    "library/cd": None,
    "library/fl": None,
    "library/fm": None,
    "library/gl": None,
    "library/imgfile": None,
    "library/jpeg": None,
    "library/sun": None,
    "library/sunaudio": None,
    # "c-api/int": "c-api/long.html",
    "c-api/int": None,
    # "c-api/string": "c-api/bytes.html",
    "c-api/string": None,
    "c-api/class": None,
    # "c-api/cobject": "c-api/capsule.html",
    "c-api/cobject": None,
    "howto/doanddont": None,
    "howto/webservers": None,
    "library/strings": "library/text.html",
    "library/stringio": "library/io.html#io.StringIO",
    # "library/sets": "library/stdtypes.html#set",
    # "library/sets": None,
    "library/sets": "library/stdtypes.html#set-types-set-frozenset",
    "library/userdict": "library/collections.html#userdict-objects",
    "library/repr": "library/reprlib.html",
    "library/copy_reg": "library/copyreg.html",
    "library/anydbm": "library/dbm.html",
    "library/whichdb": "library/dbm.html#dbm.whichdb",
    "library/dumbdbm": "library/dbm.html#module-dbm.dumb",
    "library/dbm": "library/dbm.html#module-dbm.ndbm",
    "library/gdbm": "library/dbm.html#module-dbm.gnu",
    "library/robotparser": "library/urllib.robotparser.html",
    # "library/md5": "library/hashlib.html",
    "library/md5": None,
    # "library/sha": "library/hashlib.html",
    "library/sha": None,
    "library/thread": "library/_thread.html",
    # Renamed to _dummy_thread but has since been removed completely
    # "library/dummy_thread": "library/_dummy_thread.html",
    "library/dummy_thread": None,
    "library/email-examples": "library/email.examples.html",
    # "library/rfc822": "library/email.html",
    "library/rfc822": None,
    "library/htmlparser": "library/html.parser.html",
    "library/htmllib": None,
    # "library/urllib2": "library/urllib.html",
    "library/urllib2": "library/urllib.request.html",
    "library/httplib": "library/http.client.html",
    "library/urlparse": "library/urllib.parse.html",
    "library/basehttpserver": "library/http.server.html",
    "library/simplehttpserver": "library/http.server.html#http.server.SimpleHTTPRequestHandler",
    "library/cgihttpserver": "library/http.server.html#http.server.CGIHTTPRequestHandler",
    "library/cookielib": "library/http.cookiejar.html",
    "library/cookie": "library/http.cookies.html",
    "library/xmlrpclib": "library/xmlrpc.client.html",
    "library/simplexmlrpcserver": "library/xmlrpc.server.html",
    "library/docxmlrpcserver": "library/xmlrpc.server.html#documenting-xmlrpc-server",
    "library/ttk": "library/tkinter.ttk.html",
    "library/tix": "library/tkinter.tix.html",
    "library/scrolledtext": "library/tkinter.scrolledtext.html",
    "library/__builtin__": "library/builtins.html",
    "library/_winreg": "library/winreg.html",
}

deleted_files = {old for old, new in moved_files.items() if new is None}
moved_files = {old: new for old, new in moved_files.items() if new is not None}

from itertools import groupby

LANGUAGE_AND_VERSION = "([a-z-]*/)?(3|3.5|3.6|3.7|3.8|3.9|3.10)"

for top_level, old_urls in groupby(sorted(deleted_files), key=lambda x: x.split("/")[0]):
    # print(top_level, list(old_urls))
    filenames = [x.split("/")[1] for x in old_urls]

    print(
        "    location ~ ^/"
        + LANGUAGE_AND_VERSION
        + "/"
        + top_level
        + "/("
        + "|".join(sorted(filenames))
        + ").html$ {"
    )
    print(f"        return 301 https://$host/$1$2/;")
    print("    }")


for old, new in moved_files.items():
    print("    location ~ ^/" + LANGUAGE_AND_VERSION + "/" + old + ".html$ {")
    print(f"        return 301 https://$host/$1$2/{new};")
    print("    }")

[verhovsky]

@JulienPalard I didn't see any instructions on how to test this, so let me know if it works. For the language matching, I used a regex group in a regex group, hope that works. Also if you think something should be redirected somewhere else (e.g. c-api/int to c-api/long instead of index.html) feel free to change the PR.

[ewdurbin]

Syntacticly this looks right :)

I'm going to have to defer to @JulienPalard for if the actual URLs and Locations are correct before merging.

[JulienPalard]

Suggested change

	return 301 https://$host/$1$2/;
	return 301 https://$host/$1$2/howto/;

?

[verhovsky]

If I clicked on https://docs.python.org/3/howto/webservers.html I (probably) don't care about other how-to's, I care to know that the link no longer works and I think linking to the homepage of the docs is better for that.

[JulienPalard]

right.

[JulienPalard]

Looks really nice, I this this is the little things I appreciate when reading a doc.

How did you managed to craft the list in the first place? All by hand?

I think we can expect Python 3.11 and a few future versions:

3|3.5|3.6|3.7|3.8|3.9|3.[0-9]{2}

[verhovsky]

You can get the list of files missing by downloading a copy of the documentation and using comm:

cd /tmp

wget https://docs.python.org/2/archives/python-2.7.18-docs-html.zip
unzip python-2.7.18-docs-html.zip

wget https://docs.python.org/3/archives/python-3.9.1-docs-html.zip  # new link at https://docs.python.org/3/download.html
unzip python-3.9.1-docs-html.zip

comm -23 <(find python-2.7.18-docs-html/ | cut -d'/' -f2- | sort) <(find python-3.9.1-docs-html/ | cut -d'/' -f2- | sort)

But I actually got the list with a Python script that reads the documentation and finds all the #id's in the HTML and compares the resulting list of links for all Python 2 minor versions against Python 3. I also made a quick standalone HTML page for viewing side-by-side old and new id's. Then I just greped for all the lines without a # in the old link column in the lines between Python 2.7 and 2.6 in this JSON file.

we can expect Python 3.11 and a few future versions

I did it that way first too, but then I thought it would be weird to see a link to the docs with "/3.99/" as the version number and have that link actually work when the latest version is 3.10. I think it makes more sense to hard code the currently supported versions. You could update it with a sed 's/(3|3.5|3.6|3.7|3.8|3.9|3.10)/(3|3.5|3.6|3.7|3.8|3.9|3.10|3.11)/' or modifying the code in my PR but it will be tedious to do every year until Python 2.7 documentation is removed (if it's ever removed)...

[JulienPalard]

LGTM. @ewdurbin.

[ewdurbin]

OK, these redirects are now live, but will take some time to propagate through the CDN cache.

[JulienPalard]

There's a redirect loop on this page: https://docs.python.org/3/library/dbm.html see #204