add --unzip-exclude

Author: ClericPy

README.md

@@ -406,50 +406,54 @@ Details:
    1. The names which need to be unzipped while running, splited by "," `without ext`, such as `bottle,aiohttp`, or the complete path like `bin/bottle.py,temp.py`. For `.so/.pyd` files(which can not be loaded by zipimport), or packages with operations of static files.
       1. if unzip is set to "*", then will unzip all files and folders.
       2. if unzip is set to **AUTO**, then will add the `.pyd` and `.so` files automatically.
-   2. the `unzip` arg of `zipapps.create_app`
-8. `--unzip-path, -up`
-   1. If `unzip` arg is not null, cache files will be unzipped to the given path while running. Defaults to `zipapps_cache`, support some internal variables as runtime args:
+   2. Can be overwrite with environment variable `ZIPAPPS_UNZIP`
+   3. the `unzip` arg of `zipapps.create_app`
+8. `--unzip-exclude, -ue`
+   1. The opposite of `--unzip` / `-u` which will not be unzipped, should be used with `--unzip` / `-u`.
+   2. Can be overwrite with environment variable `ZIPAPPS_UNZIP_EXCLUDE`
+9.  `--unzip-path, -up`
+   3. If `unzip` arg is not null, cache files will be unzipped to the given path while running. Defaults to `zipapps_cache`, support some internal variables as runtime args:
       1.  `TEMP/HOME/SELF` as prefix, for example `HOME/zipapps_cache`
           1. `TEMP` means `tempfile.gettempdir()`
           2. `HOME` means `Path.home()`
           3. `SELF` means `.pyz` file path.
       2. And you can also **overwrite** it with environment variables:
          1. `ZIPAPPS_CACHE` or `UNZIP_PATH`
-   2. the `unzip_path` arg of `zipapps.create_app`
-9. `-cc, --pyc, --compile, --compiled`
-   1. Compile .py to .pyc for fast import, but zipapp does not work unless you unzip it(so NOT very useful).
-   2. the `compiled` arg of `zipapps.create_app`
-10. ` --cache-path, --source-dir, -cp`
+   4. the `unzip_path` arg of `zipapps.create_app`
+10. `-cc, --pyc, --compile, --compiled`
+   5. Compile .py to .pyc for fast import, but zipapp does not work unless you unzip it(so NOT very useful).
+   6. the `compiled` arg of `zipapps.create_app`
+11. ` --cache-path, --source-dir, -cp`
        1. The cache path of zipapps to store site-packages and `includes` files. If not set, will create and clean-up in TEMP dir automately.
        2. the `cache_path` arg of `zipapps.create_app`
-11. `--shell, -s`
+12. `--shell, -s`
     1.  Only while `main` is not set, used for shell=True in `subprocess.run`.
         1.  *very rarely used*, because extra sub-process is not welcome
     2.  the `shell` arg of `zipapps.create_app`
-12. `--main-shell, -ss`
+13. `--main-shell, -ss`
     1.  Only for `main` is not null, call `main` with `subprocess.Popen`: `python -c "import a.b;a.b.c()"`. This is used for `psutil` ImportError of DLL load.
         1.  *very rarely used* too
     2.  the `main_shell` arg of `zipapps.create_app`
-13. `--strict-python-path, -spp`
+14. `--strict-python-path, -spp`
     1.  `Boolean` value. Ignore global PYTHONPATH, only use `zipapps_cache` and `app.pyz`.
     2.  the `ignore_system_python_path` arg of `zipapps.create_app`
-14. `-b, --build-id`
+15. `-b, --build-id`
     1.  The string to skip duplicate builds, it can be the paths of files/folders which splited by ",", then the modify time will be used as build_id. If build_id contains `*`, will use `glob` function to get paths. For example, you can set requirements.txt as your build_id by `python3 -m zipapps -b requirements.txt -r requirements.txt` when you use pyz as venv.
         1.  *very rarely used* too too
     2.  the `build_id` arg of `zipapps.create_app`
-15. `--zipapps, --env-paths`
+16. `--zipapps, --env-paths`
     1.  Default `--zipapps` arg if it is not given while running. Support TEMP/HOME/SELF prefix.
-16. `--delay, -d, --lazy-pip, --lazy-install, --lazy-pip-install`
+17. `--delay, -d, --lazy-pip, --lazy-install, --lazy-pip-install`
     1.  Install packages with pip while first running, which means requirements will not be install into pyz file.
-17. `--ensure-pip`
+18. `--ensure-pip`
     1.  Add the ensurepip package to your pyz file, works for **embed-python**(windows) or other python versions without `pip` installed but `lazy-install` mode is enabled. [EXPERIMENTAL]
-18. `--layer-mode`
+19. `--layer-mode`
     1. Layer mode for the `serverless` use case, `__main__.py / ensure_zipapps.py / activate_zipapps.py` files will not be set in this mode.
     2. `--layer-mode-prefix`
        1. Only work while `--layer-mode` is set, will move the files in the given prefix folder.
-19. `--clear-zipapps-cache, -czc`
+20. `--clear-zipapps-cache, -czc`
     1.  Clear the zipapps cache folder after running, but maybe failed for .pyd/.so files.
-20. all the other (or `unknown`) args will be used by `pip install`
+21. all the other (or `unknown`) args will be used by `pip install`
     1.  such as `-r requirements.txt`
     2.  such as `bottle aiohttp`
     3.  the `pip_args` arg of `zipapps.create_app`

changelog.md

@@ -2,6 +2,10 @@
 # Changelogs
 
 
+- 2021.02.27
+  - add `--unzip-exclude, -ue`
+    - The opposite of `--unzip` / `-u` which will not be unzipped, should be used with `--unzip` / `-u`.
+    - Can be overwrite with environment variable `ZIPAPPS_UNZIP_EXCLUDE`
 - 2021.01.17
   - add runtime arg `--activate-zipapps` to ensure cache folder and do nothing
   - fix pip install for `-d` (lazy install mode) with `pip.main` function instead of subprocess

test_utils.py

@@ -36,6 +36,18 @@ def _clean_paths():
 
 def test_create_app_function():
 
+    # test unzip_exclude
+    _clean_paths()
+    app_path = create_app(unzip='*', pip_args=['six'], unzip_exclude='')
+    stdout_output, stderr_output = subprocess.Popen(
+        [sys.executable, str(app_path), '--activate-zipapps']).communicate()
+    assert Path('./zipapps_cache/app/six.py').is_file()
+    _clean_paths()
+    app_path = create_app(unzip='*', pip_args=['six'], unzip_exclude='six')
+    stdout_output, stderr_output = subprocess.Popen(
+        [sys.executable, str(app_path), '--activate-zipapps']).communicate()
+    assert not Path('./zipapps_cache/app/six.py').is_file()
+
     # test -czc
     _clean_paths()
     app_path = create_app(clear_zipapps_cache=False, unzip='*')

zipapps/__main__.py

@@ -92,7 +92,15 @@ def main():
         '-u',
         default='',
         help='The names which need to be unzipped while running, splited by "," '
-        '`without ext`, such as `bottle,aiohttp`, or the complete path like `bin/bottle.py,temp.py`. For `.so/.pyd` files(which can not be loaded by zipimport), or packages with operations of static files. if unzip is set to "*", then will unzip all files and folders. if unzip is set to **AUTO**, then will add the `.pyd` and `.so` files automatically.'
+        '`without ext`, such as `bottle,aiohttp`, or the complete path like `bin/bottle.py,temp.py`. For `.so/.pyd` files(which can not be loaded by zipimport), or packages with operations of static files. if unzip is set to "*", then will unzip all files and folders. if unzip is set to **AUTO**, then will add the `.pyd` and `.so` files automatically. Can be overwrite with environment variable `ZIPAPPS_UNZIP`'
+    )
+    parser.add_argument(
+        '--unzip-exclude',
+        '-ue',
+        default='',
+        dest='unzip_exclude',
+        help='The opposite of `--unzip` / `-u` which will not be unzipped, '
+        'should be used with `--unzip` / `-u`. Can be overwrite with environment variable `ZIPAPPS_UNZIP_EXCLUDE`'
     )
     parser.add_argument(
         '--unzip-path',
@@ -246,6 +254,7 @@ def main():
         layer_mode=args.layer_mode,
         layer_mode_prefix=args.layer_mode_prefix,
         clear_zipapps_cache=args.clear_zipapps_cache,
+        unzip_exclude=args.unzip_exclude,
     )
 
 

zipapps/ensure_zipapps_template.py

@@ -10,6 +10,8 @@
 # const
 ignore_system_python_path = {ignore_system_python_path}
 unzip = os.environ.get('ZIPAPPS_UNZIP') or r'''{unzip}'''
+unzip_exclude = os.environ.get(
+    'ZIPAPPS_UNZIP_EXCLUDE') or r'''{unzip_exclude}'''
 _cache_folder = os.environ.get('ZIPAPPS_CACHE') or os.environ.get(
     'UNZIP_PATH') or r'''{unzip_path}'''
 ts_file_name = '_zip_time_{ts}'
@@ -85,12 +87,18 @@ def _remove_cache_folder():
             # rm the folder
             clear_old_cache(_cache_folder_path, LAZY_PIP_DIR_NAME)
             _need_unzip_names = unzip.split(',')
+            if unzip_exclude:
+                _exclude_unzip_names = set(unzip_exclude.split(','))
+            else:
+                _exclude_unzip_names = set()
             _need_unzip_names.append(ts_file_name)
             with ZipFile(zip_file_path, "r") as zf:
                 for member in zf.infolist():
                     file_dir_name = os.path.splitext(
                         member.filename.split('/')[0])[0]
-                    if unzip == '*' or member.filename in _need_unzip_names or file_dir_name in _need_unzip_names:
+                    allow_unzip = unzip == '*' or member.filename in _need_unzip_names or file_dir_name in _need_unzip_names
+                    exclude_unzip = member.filename in _exclude_unzip_names or file_dir_name in _exclude_unzip_names
+                    if allow_unzip and not exclude_unzip:
                         zf.extract(member, path=_cache_folder_path_str)
         if LAZY_PIP_DIR_NAME:
             import platform

zipapps/main.py

@@ -14,7 +14,7 @@
 from pkgutil import get_data
 from zipfile import ZIP_DEFLATED, ZIP_STORED, BadZipFile, ZipFile
 
-__version__ = '2021.01.17'
+__version__ = '2021.02.27'
 
 
 class ZipApp(object):
@@ -51,6 +51,7 @@ def __init__(
         layer_mode: bool = False,
         layer_mode_prefix: str = 'python',
         clear_zipapps_cache: bool = False,
+        unzip_exclude: str = '',
     ):
         """Zip your code.
 
@@ -68,7 +69,7 @@ def __init__(
         :type compressed: bool, optional
         :param shell: whether run python in subprocess, or use runpy if shell is False, defaults to False
         :type shell: bool, optional
-        :param unzip: names to be unzip, using `AUTO` is a better choice, defaults to ''
+        :param unzip: names to be unzip, using `AUTO` is a better choice, defaults to ''. Can be overwrite with environment variable `ZIPAPPS_UNZIP`
         :type unzip: str, optional
         :param unzip_path: If `unzip` arg is not null, cache files will be unzipped to the given path while running. Defaults to `zipapps_cache`, support some internal variables: `TEMP/HOME/SELF` as internal variables, for example `HOME/zipapps_cache`. `TEMP` means `tempfile.gettempdir()`, `HOME` means `Path.home()`, `SELF` means `.pyz` file path, defaults to ''
         :type unzip_path: str, optional
@@ -98,6 +99,8 @@ def __init__(
         :type includes: str, optional
         :param clear_zipapps_cache: Clear the zipapps cache folder after running, but maybe failed for .pyd/.so files..
         :type includes: bool, optional
+        :param unzip_exclude: names not to be unzip, defaults to '', should be used with unzip. Can be overwrite with environment variable `ZIPAPPS_UNZIP_EXCLUDE`
+        :type unzip_exclude: str, optional
         """
         self.includes = includes
         self.cache_path = cache_path
@@ -108,6 +111,7 @@ def __init__(
         self.compressed = compressed
         self.shell = shell
         self.unzip = unzip
+        self.unzip_exclude = unzip_exclude
         self.unzip_path = unzip_path
         self.ignore_system_python_path = ignore_system_python_path
         self.main_shell = main_shell
@@ -129,6 +133,10 @@ def __init__(
 
     def ensure_args(self):
         if not self.unzip:
+            if self.unzip_exclude:
+                self._log(
+                    '[WARN]: The arg `unzip_exclude` should not be with `unzip` but `unzip` is null.'
+                )
             if self.compiled:
                 self._log(
                     '[WARN]: The arg `compiled` should not be True while `unzip` is null, because .pyc files of __pycache__ folder may not work in zip file.'
@@ -273,6 +281,7 @@ def prepare_active_zipapps(self):
             'shell': self.shell,
             'main_shell': self.main_shell,
             'unzip': self.unzip,
+            'unzip_exclude': self.unzip_exclude,
             'output_name': output_name,
             'unzip_path': self.unzip_path,
             'ignore_system_python_path': self.ignore_system_python_path,