Merge pull request #746 from nf-core/dev

Minor release 1.11

Author: ewels

.github/workflows/branch.yml

@@ -24,7 +24,7 @@ jobs:
           message: |
             Hi @${{ github.event.pull_request.user.login }},
 
-            It looks like this pull-request has been made against the ${{github.event.pull_request.head.repo.full_name}} `master` branch. The `master` branch on nf-core repositories should always contain code from the latest release. Beacuse of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch.
+            It looks like this pull-request has been made against the ${{github.event.pull_request.head.repo.full_name}} `master` branch. The `master` branch on nf-core repositories should always contain code from the latest release. Because of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch.
 
             You do not need to close this PR, you can change the target branch to `dev` by clicking the _"Edit"_ button at the top of this page.
 

.github/workflows/sync.yml

@@ -2,6 +2,7 @@ name: Sync template
 on:
   release:
     types: [published]
+  workflow_dispatch:
 
 jobs:
   get-pipelines:
@@ -21,7 +22,6 @@ jobs:
       matrix: ${{fromJson(needs.get-pipelines.outputs.matrix)}}
       fail-fast: false
     steps:
-
       - uses: actions/checkout@v2
         name: Check out nf-core/tools
 
@@ -32,6 +32,7 @@ jobs:
           ref: dev
           token: ${{ secrets.nf_core_bot_auth_token }}
           path: nf-core/${{ matrix.pipeline }}
+          fetch-depth: "0"
 
       - name: Set up Python 3.8
         uses: actions/setup-python@v1
@@ -63,7 +64,6 @@ jobs:
             --username nf-core-bot \
             --repository nf-core/${{ matrix.pipeline }}
 
-
       - name: Upload sync log file artifact
         if: ${{ always() }}
         uses: actions/upload-artifact@v2

.gitignore

@@ -109,3 +109,6 @@ ENV/
 # backup files
 *~
 *\?
+
+# Jetbrains IDEs
+.idea

CHANGELOG.md

@@ -1,5 +1,26 @@
 # nf-core/tools: Changelog
 
+## [v1.11 - Iron Tiger](https://github.com/nf-core/tools/releases/tag/1.11) - [2020-10-27]
+
+### Template
+
+* Fix command error in `awstest.yml` GitHub Action workflow.
+* Allow manual triggering of AWS test GitHub Action workflows.
+* Remove TODO item, which was proposing the usage of additional files beside `usage.md` and `output.md` for documentation.
+* Added a Podman profile, which enables Podman as container.
+* Updated linting for GitHub actions AWS tests workflows.
+
+### Linting
+
+* Made a base-level `Dockerfile` a warning instead of failure
+* Added a lint failure if the old `bin/markdown_to_html.r` script is found
+* Update `rich` package dependency and use new markup escaping to change `[[!]]` back to `[!]` again
+
+### Other
+
+* Pipeline sync - fetch full repo when checking out before sync
+* Sync - Add GitHub actions manual trigger option
+
 ## [v1.10.2 - Copper Camel _(brought back from the dead)_](https://github.com/nf-core/tools/releases/tag/1.10.2) - [2020-07-31]
 
 Second patch release to address some small errors discovered in the pipeline template.

docs/lint_errors.md

@@ -10,8 +10,8 @@ The lint test looks for the following required files:
 
 * `nextflow.config`
   * The main nextflow config file
-* `Dockerfile`
-  * A docker build script to generate a docker image with the required software
+* `nextflow_schema.json`
+  * A JSON schema describing pipeline parameters, generated using `nf-core schema build`
 * Continuous integration tests with [GitHub Actions](https://github.com/features/actions)
   * GitHub Actions workflows for CI of your pipeline (`.github/workflows/ci.yml`), branch protection (`.github/workflows/branch.yml`) and nf-core best practice linting (`.github/workflows/linting.yml`)
 * `LICENSE`, `LICENSE.md`, `LICENCE.md` or `LICENCE.md`
@@ -27,8 +27,14 @@ The following files are suggested but not a hard requirement. If they are missin
 
 * `main.nf`
   * It's recommended that the main workflow script is called `main.nf`
+* `environment.yml`
+  * A conda environment file describing the required software
+* `Dockerfile`
+  * A docker build script to generate a docker image with the required software
 * `conf/base.config`
   * A `conf` directory with at least one config called `base.config`
+* `.github/workflows/awstest.yml` and `.github/workflows/awsfulltest.yml`
+  * GitHub workflow scripts used for automated tests on AWS
 
 The following files will cause a failure if the _are_ present (to fix, delete them):
 
@@ -39,13 +45,18 @@ The following files will cause a failure if the _are_ present (to fix, delete th
 * `parameters.settings.json`
   * The syntax for pipeline schema has changed - old `parameters.settings.json` should be
     deleted and new `nextflow_schema.json` files created instead.
+* `bin/markdown_to_html.r`
+  * The old markdown to HTML conversion script, now replaced by `markdown_to_html.py`
 
 ## Error #2 - Docker file check failed ## {#2}
 
-Pipelines should have a files called `Dockerfile` in their root directory.
+DSL1 pipelines should have a file called `Dockerfile` in their root directory.
 The file is used for automated docker image builds. This test checks that the file
 exists and contains at least the string `FROM` (`Dockerfile`).
 
+Some pipelines, especially DSL2, may not have a `Dockerfile`. In this case a warning
+will be generated which can be safely ignored.
+
 ## Error #3 - Licence check failed ## {#3}
 
 nf-core pipelines must ship with an open source [MIT licence](https://choosealicense.com/licenses/mit/).
@@ -216,22 +227,25 @@ This test will fail if the following requirements are not met in these files:
             { [[ ${{github.event.pull_request.head.repo.full_name}} == <repo_name>/<pipeline_name> ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]]
       ```
 
-4. `awstest.yml`: Triggers tests on AWS batch. As running tests on AWS incurs costs, they should be only triggered on `push` to `master` and `release`.
-    * Must be turned on for `push` to `master` and `release`.
-    * Must not be turned on for `pull_request` or other events.
+4. `awstest.yml`: Triggers tests on AWS batch. As running tests on AWS incurs costs, they should be only triggered on `workflow_dispatch`.
+This allows for manual triggering of the workflow when testing on AWS is desired.
+You can trigger the tests by going to the `Actions` tab on the pipeline GitHub repository and selecting the `nf-core AWS test` workflow on the left.
+    * Must not be turned on for `push` or `pull_request`.
+    * Must be turned on for `workflow_dispatch`.
 
 ### GitHub Actions AWS full tests
 
 Additionally, we provide the possibility of testing the pipeline on full size datasets on AWS.
 This should ensure that the pipeline runs as expected on AWS and provide a resource estimation.
-The GitHub Actions workflow is: `awsfulltest.yml`, and it can be found in the `.github/workflows/` directory.
-This workflow incurrs higher AWS costs, therefore it should only be triggered on `release`.
-For tests on full data prior to release, [https://tower.nf](Nextflow Tower's launch feature) can be employed.
+The GitHub Actions workflow is `awsfulltest.yml`, and it can be found in the `.github/workflows/` directory.
+This workflow incurrs higher AWS costs, therefore it should only be triggered on `release` and `workflow_dispatch`.
+You can trigger the tests by going to the `Actions` tab on the pipeline GitHub repository and selecting the `nf-core AWS full size tests` workflow on the left.
+For tests on full data prior to release, [Nextflow Tower](https://tower.nf) launch feature can be employed.
 
 `awsfulltest.yml`: Triggers full sized tests run on AWS batch after releasing.
 
-* Must be only turned on for `release`.
-* Should run the profile `test_full`. If it runs the profile `test` a warning is given.
+* Must be only turned on for `release` and `workflow_dispatch`.
+* Should run the profile `test_full` that should be edited to provide the links to full-size datasets. If it runs the profile `test` a warning is given.
 
 ## Error #6 - Repository `README.md` tests ## {#6}
 
@@ -298,7 +312,7 @@ If a workflow has a conda `environment.yml` file (see above), the `Dockerfile` s
 to create the container. Such `Dockerfile`s can usually be very short, eg:
 
 ```Dockerfile
-FROM nfcore/base:1.7
+FROM nfcore/base:1.11
 MAINTAINER Rocky Balboa <[email protected]>
 LABEL authors="[email protected]" \
     description="Docker image containing all requirements for the nf-core mypipeline pipeline"

nf_core/__main__.py

@@ -35,11 +35,11 @@ def run_nf_core():
 
     # Print nf-core header to STDERR
     stderr = rich.console.Console(file=sys.stderr)
-    stderr.print("\n[green]{},--.[grey39]/[green],-.".format(" " * 42))
-    stderr.print("[blue]          ___     __   __   __   ___     [green]/,-._.--~\\")
-    stderr.print("[blue]    |\ | |__  __ /  ` /  \ |__) |__      [yellow]   }  {")
-    stderr.print("[blue]    | \| |       \__, \__/ |  \ |___     [green]\`-._,-`-,")
-    stderr.print("[green]                                          `._,._,'\n")
+    stderr.print("\n[green]{},--.[grey39]/[green],-.".format(" " * 42), highlight=False)
+    stderr.print("[blue]          ___     __   __   __   ___     [green]/,-._.--~\\", highlight=False)
+    stderr.print("[blue]    |\ | |__  __ /  ` /  \ |__) |__      [yellow]   }  {", highlight=False)
+    stderr.print("[blue]    | \| |       \__, \__/ |  \ |___     [green]\`-._,-`-,", highlight=False)
+    stderr.print("[green]                                          `._,._,'\n", highlight=False)
     stderr.print("[grey39]    nf-core/tools version {}".format(nf_core.__version__), highlight=False)
     try:
         is_outdated, current_vers, remote_vers = nf_core.utils.check_if_outdated()
@@ -503,7 +503,7 @@ def lint(schema_path):
         try:
             schema_obj.validate_schema_title_description()
         except AssertionError as e:
-            log.warn(e)
+            log.warning(e)
     except AssertionError as e:
         sys.exit(1)
 

nf_core/create.py

@@ -67,8 +67,7 @@ def init_pipeline(self):
         )
 
     def run_cookiecutter(self):
-        """Runs cookiecutter to create a new nf-core pipeline.
-        """
+        """Runs cookiecutter to create a new nf-core pipeline."""
         log.info("Creating new nf-core pipeline: {}".format(self.name))
 
         # Check if the output directory exists
@@ -113,8 +112,7 @@ def run_cookiecutter(self):
         shutil.rmtree(self.tmpdir)
 
     def make_pipeline_logo(self):
-        """Fetch a logo for the new pipeline from the nf-core website
-        """
+        """Fetch a logo for the new pipeline from the nf-core website"""
 
         logo_url = "https://nf-co.re/logo/{}".format(self.short_name)
         log.debug("Fetching logo from {}".format(logo_url))
@@ -135,8 +133,7 @@ def make_pipeline_logo(self):
             fh.write(r.content)
 
     def git_init_pipeline(self):
-        """Initialises the new pipeline as a Git repository and submits first commit.
-        """
+        """Initialises the new pipeline as a Git repository and submits first commit."""
         log.info("Initialising pipeline git repository")
         repo = git.Repo.init(self.outdir)
         repo.git.add(A=True)

nf_core/download.py

@@ -197,8 +197,7 @@ def fetch_workflow_details(self, wfs):
             raise LookupError("Not able to find pipeline '{}'".format(self.pipeline))
 
     def download_wf_files(self):
-        """Downloads workflow files from GitHub to the :attr:`self.outdir`.
-        """
+        """Downloads workflow files from GitHub to the :attr:`self.outdir`."""
         log.debug("Downloading {}".format(self.wf_download_url))
 
         # Download GitHub zip file into memory and extract
@@ -216,8 +215,7 @@ def download_wf_files(self):
                 os.chmod(os.path.join(dirpath, fname), 0o775)
 
     def download_configs(self):
-        """Downloads the centralised config profiles from nf-core/configs to :attr:`self.outdir`.
-        """
+        """Downloads the centralised config profiles from nf-core/configs to :attr:`self.outdir`."""
         configs_zip_url = "https://github.com/nf-core/configs/archive/master.zip"
         configs_local_dir = "configs-master"
         log.debug("Downloading {}".format(configs_zip_url))
@@ -236,8 +234,7 @@ def download_configs(self):
                 os.chmod(os.path.join(dirpath, fname), 0o775)
 
     def wf_use_local_configs(self):
-        """Edit the downloaded nextflow.config file to use the local config files
-        """
+        """Edit the downloaded nextflow.config file to use the local config files"""
         nfconfig_fn = os.path.join(self.outdir, "workflow", "nextflow.config")
         find_str = "https://raw.githubusercontent.com/nf-core/configs/${params.custom_config_version}"
         repl_str = "../configs/"
@@ -294,8 +291,7 @@ def pull_singularity_image(self, container):
                 raise e
 
     def compress_download(self):
-        """Take the downloaded files and make a compressed .tar.gz archive.
-        """
+        """Take the downloaded files and make a compressed .tar.gz archive."""
         log.debug("Creating archive: {}".format(self.output_filename))
 
         # .tar.gz and .tar.bz2 files

nf_core/licences.py

@@ -50,8 +50,7 @@ def run_licences(self):
         return self.print_licences()
 
     def get_environment_file(self):
-        """Get the conda environment file for the pipeline
-        """
+        """Get the conda environment file for the pipeline"""
         if os.path.exists(self.pipeline):
             env_filename = os.path.join(self.pipeline, "environment.yml")
             if not os.path.exists(self.pipeline):
@@ -68,8 +67,7 @@ def get_environment_file(self):
             self.conda_config = yaml.safe_load(response.text)
 
     def fetch_conda_licences(self):
-        """Fetch package licences from Anaconda and PyPi.
-        """
+        """Fetch package licences from Anaconda and PyPi."""
 
         lint_obj = nf_core.lint.PipelineLint(self.pipeline)
         lint_obj.conda_config = self.conda_config