pysam-0.8.4+gx1.tar.gz breaks on TravisCI

[peterjc]

Filing here as per @natefoo on Twitter: https://twitter.com/natefoo/status/748182338006319105

See https://travis-ci.org/peterjc/galaxy_blast/jobs/141112142 which downloads https://wheels.galaxyproject.org/packages/pysam-0.8.4+gx1.tar.gz and tries to install it, but fails with what looks like pysam-developers/pysam#79 suggesting the Galaxy wheel fo pysam is missing some headers?

  Running setup.py install for pysam
    package init file 'htslib/__init__.py' not found (or not a regular file)
    package init file 'htslib/htslib/__init__.py' not found (or not a regular file)
    package init file 'samtools/__init__.py' not found (or not a regular file)
    package init file 'samtools/win32/__init__.py' not found (or not a regular file)
    building 'pysam.csamtools' extension
    gcc -pthread -fno-strict-aliasing -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -D_FILE_OFFSET_BITS=64 -D_USE_KNETFILE= -Isamtools -Ipysam -Ihtslib -I/opt/python/2.7.9/include/python2.7 -c pysam/csamtools.c -o build/temp.linux-x86_64-2.7/pysam/csamtools.o -Wno-error=declaration-after-statement -DSAMTOOLS=1
    gcc: error: pysam/csamtools.c: No such file or directory
    gcc: fatal error: no input files
    compilation terminated.
    error: command 'gcc' failed with exit status 4
    Complete output from command /home/travis/virtualenv/python2.7.9/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-ehLQC8/pysam/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-2wp2Q7-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/travis/virtualenv/python2.7.9/include/site/python2.7:
    running install

    running build

    running build_py

    creating build

    creating build/lib.linux-x86_64-2.7

    creating build/lib.linux-x86_64-2.7/pysam

    copying pysam/Pileup.py -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/__init__.py -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/namedtuple.py -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/version.py -> build/lib.linux-x86_64-2.7/pysam

    creating build/lib.linux-x86_64-2.7/pysam/include

    copying pysam/include/__init__.py -> build/lib.linux-x86_64-2.7/pysam/include

    package init file 'htslib/__init__.py' not found (or not a regular file)

    package init file 'htslib/htslib/__init__.py' not found (or not a regular file)

    package init file 'samtools/__init__.py' not found (or not a regular file)

    package init file 'samtools/win32/__init__.py' not found (or not a regular file)

    copying pysam/calignedsegment.pxd -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/calignmentfile.pxd -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/cbcf.pxd -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/cfaidx.pxd -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/chtslib.pxd -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/csamfile.pxd -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/csamtools.pxd -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/ctabix.pxd -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/ctabixproxies.pxd -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/cutils.pxd -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/cvcf.pxd -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/htslib_util.h -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/pysam_stream.h -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/pysam_util.h -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/samfile_util.h -> build/lib.linux-x86_64-2.7/pysam

    copying pysam/tabix_util.h -> build/lib.linux-x86_64-2.7/pysam

    creating build/lib.linux-x86_64-2.7/pysam/include/htslib

    copying htslib/config.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib

    copying htslib/hfile_internal.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib

    copying htslib/version.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib

    creating build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/bgzf.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/faidx.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/hfile.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/hts.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/hts_defs.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/kfunc.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/khash.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/khash_str2int.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/klist.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/knetfile.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/kseq.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/ksort.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/kstring.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/regidx.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/sam.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/synced_bcf_reader.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/tbx.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/vcf.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/vcf_sweep.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    copying htslib/htslib/vcfutils.h -> build/lib.linux-x86_64-2.7/pysam/include/htslib/htslib

    creating build/lib.linux-x86_64-2.7/pysam/include/samtools

    copying samtools/bam.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools

    copying samtools/bam2bcf.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools

    copying samtools/bam_endian.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools

    copying samtools/bam_lpileup.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools

    copying samtools/bam_plbuf.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools

    copying samtools/bam_tview.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools

    copying samtools/errmod.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools

    copying samtools/kprobaln.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools

    copying samtools/pysam.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools

    copying samtools/sam.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools

    copying samtools/sam_header.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools

    copying samtools/sample.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools

    copying samtools/samtools.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools

    copying samtools/stats_isize.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools

    copying samtools/version.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools

    creating build/lib.linux-x86_64-2.7/pysam/include/samtools/win32

    copying samtools/win32/xcurses.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools/win32

    copying samtools/win32/zconf.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools/win32

    copying samtools/win32/zlib.h -> build/lib.linux-x86_64-2.7/pysam/include/samtools/win32

    running build_ext

    building 'pysam.csamtools' extension

    creating build/temp.linux-x86_64-2.7

    creating build/temp.linux-x86_64-2.7/pysam

    creating build/temp.linux-x86_64-2.7/samtools

    creating build/temp.linux-x86_64-2.7/samtools/misc

    creating build/temp.linux-x86_64-2.7/samtools/test

    creating build/temp.linux-x86_64-2.7/htslib

    creating build/temp.linux-x86_64-2.7/htslib/cram

    gcc -pthread -fno-strict-aliasing -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -D_FILE_OFFSET_BITS=64 -D_USE_KNETFILE= -Isamtools -Ipysam -Ihtslib -I/opt/python/2.7.9/include/python2.7 -c pysam/csamtools.c -o build/temp.linux-x86_64-2.7/pysam/csamtools.o -Wno-error=declaration-after-statement -DSAMTOOLS=1

    gcc: error: pysam/csamtools.c: No such file or directory

    gcc: fatal error: no input files

    compilation terminated.

    error: command 'gcc' failed with exit status 4

    ----------------------------------------

[natefoo]

Thanks for submitting this. I was aware of it but haven't had a chance to fix it yet - typically it does not come up since pip should prefer our wheels over our sdists. I'll fix the sdist ASAP.

Your Travis builds aren't using Galaxy pip, which is why it isn't downloading the wheels. I am not sure why - I know this was already fixed once. If you remove --skip-venv from bash scripts/common_startup.sh --skip-venv --dev-wheels, does this fix it?

[peterjc]

I'm intending to update my TravisCI setup to more closely follow the IUC setup, so problem may go away for me...

[peterjc]

Failing TravisCI was using this:

#Older Galaxy versions using eggs (like 15.10) will ignore the extra arguments:
 - bash scripts/common_startup.sh --skip-venv --dev-wheels

Just switched to this which I was already using with TravisCI on another repository:

#Older Galaxy versions using eggs (like 15.10) will ignore the extra arguments:
 - bash scripts/common_startup.sh --no-create-venv --dev-wheels

Thanks! I'll leave this issue open for the underlying problem with the pysam not building.

[natefoo]

Pysam source builds (even for the older versions) should be fixed now.

It should be less of an issue going forward because we'll be using an unmodified upstream pysam. That said, unless I keep making modified source tarballs as described in galaxyproject/starforge#180 and uploading them to wheels.galaxyproject.org for whatever releases we pin, local builds will fail if you do not have Cython installed.

At least, that will be the case until pip with PEP 518 pyproject.toml support: pypa/pip#4144 is released, and pysam is updated to use pyproject.toml.

But, I'm closing this, as the issue is fixed for now.