Cannot install package with files that have non-ascii filenames

[megies]

For me installation of a python package fails because some test files in it have non-ascii characters in filenames:

$ pip install https://github.com/megies/seishub.core/archive/quakeml.zip
Downloading/unpacking https://github.com/megies/seishub.core/archive/quakeml.zip
  Downloading quakeml.zip (unknown size): 3.5MB downloaded
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/req.py", line 1197, in prepare_files
    do_download,
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/req.py", line 1375, in unpack_url
    self.session,
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/download.py", line 582, in unpack_http_url
    unpack_file(temp_location, location, content_type, link)
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/util.py", line 621, in unpack_file
    unzip_file(filename, location, flatten=not filename.endswith(('.pybundle', '.whl')))
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/util.py", line 492, in unzip_file
    leading = has_leading_dir(zip.namelist()) and flatten
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/util.py", line 232, in has_leading_dir
    prefix, rest = split_leading_dir(path)
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/util.py", line 216, in split_leading_dir
    path = str(path)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 85-87: ordinal not in range(128)

Storing debug log for failure in /home/megies/.pip/pip.log

The problematic files are some test files in there:
https://github.com/megies/seishub.core/tree/quakeml/seishub/core/processor/tests/data/filesystem/special

I'm using pip 1.5.4 on Python 2.7.6.

Installation seems to run smoothly when commenting out this line: https://github.com/pypa/pip/blob/develop/pip/util.py#L228

[dholth]

We ran into a similar issue where the URLs (network) part of pip was operating in Unicode, while the filesystem part was operating in bytes. There may need to be a decode(sys.getdefaultfilesystemencoding()) [spelling?] in there somewhere.

[blueyed]

I am seeing this error when trying to install Django master via tox.

Django now ships a file with a unicode symbol: django/django@bd059e3, and the traceback from pip is:

Collecting https://github.com/django/django/archive/master.zip
  Downloading https://github.com/django/django/archive/master.zip (11.4MB)
  Exception:
  Traceback (most recent call last):
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/basecommand.py", line 232, in main
      status = self.run(options, args)
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/commands/install.py", line 339, in run
      requirement_set.prepare_files(finder)
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/req/req_set.py", line 355, in prepare_files
      do_download, session=self.session,
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/download.py", line 782, in unpack_url
      session,
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/download.py", line 671, in unpack_http_url
      unpack_file(from_path, location, content_type, link)
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/utils/__init__.py", line 648, in unpack_file
      flatten=not filename.endswith('.whl')
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/utils/__init__.py", line 528, in unzip_file
      leading = has_leading_dir(zip.namelist()) and flatten
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/utils/__init__.py", line 253, in has_leading_dir
      prefix, rest = split_leading_dir(path)
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/utils/__init__.py", line 237, in split_leading_dir
      path = str(path)
  UnicodeEncodeError: 'ascii' codec can't encode character u'\u2297' in position 60: ordinal not in range(128)

(Initially reported for tox at https://bitbucket.org/hpk42/tox/issue/221/unicodeencodeerror-when-trying-to-install).

[blueyed]

Duplicate of #1140, and #1473 has a fix.

[piotr-dobrogost]

@blueyed 1473 has a fix for .tar files and this bug is for .zip files. Maybe the fix for .zip is the same or very similar but these are two different bugs from user point of view.