Merge pull request #888 from nf-core/dev

Dev > Master for 1.13 release

Author: ewels

.github/CONTRIBUTING.md

@@ -57,7 +57,7 @@ We aim to write function docstrings according to the [Google Python style-guide]
 You can find this documentation here: [https://nf-co.re/tools-docs/](https://nf-co.re/tools-docs/)
 
 If you would like to test the documentation, you can install Sphinx locally by following Sphinx's [installation instruction](https://www.sphinx-doc.org/en/master/usage/installation.html).
-Once done, you can run `make clean` and then `make html` in the root directory of `nf-core tools`.
+Once done, you can run `make clean` and then `make html` in the `docs/api` directory of `nf-core tools`.
 The HTML will then be generated in `docs/api/_build/html`.
 
 ## Tests

.github/markdownlint.yml

@@ -10,6 +10,7 @@ no-inline-html:
         - kbd
         - details
         - summary
+        - kbd
 # tools only - the {{ jinja variables }} break URLs and cause this to error
 no-bare-urls: false
 # tools only - suppresses error messages for usage of $ in main README

.github/workflows/branch.yml

@@ -23,9 +23,12 @@ jobs:
           message: |
             Hi @${{ github.event.pull_request.user.login }},
 
-            It looks like this pull-request has been made against the ${{github.event.pull_request.head.repo.full_name}} `master` branch. The `master` branch on nf-core repositories should always contain code from the latest release. Because of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch.
+            It looks like this pull-request has been made against the ${{github.event.pull_request.base.repo.full_name}} `master` branch.
+            The `master` branch on nf-core repositories should always contain code from the latest release.
+            Because of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.base.repo.full_name}} `dev` branch.
 
             You do not need to close this PR, you can change the target branch to `dev` by clicking the _"Edit"_ button at the top of this page.
+            Note that even after this, the test will continue to show as failing until you push a new commit.
 
             Thanks again for your contribution!
           repo-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/create-lint-wf.yml

@@ -30,15 +30,16 @@ jobs:
       - name: Run nf-core/tools
         run: |
           nf-core --log-file log.txt create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface"
-          nf-core --log-file log.txt lint nf-core-testpipeline
+          nf-core --log-file log.txt lint nf-core-testpipeline --fail-ignored
           nf-core --log-file log.txt list
           nf-core --log-file log.txt licences nf-core-testpipeline
           nf-core --log-file log.txt sync nf-core-testpipeline/
           nf-core --log-file log.txt schema build nf-core-testpipeline/ --no-prompts
           nf-core --log-file log.txt bump-version nf-core-testpipeline/ 1.1
-          nf-core --log-file log.txt modules install nf-core-testpipeline/ fastqc
+          nf-core --log-file log.txt modules install nf-core-testpipeline/ --tool fastqc
 
       - name: Upload log file artifact
+        if: ${{ always() }}
         uses: actions/upload-artifact@v2
         with:
           name: nf-core-log-file

.github/workflows/tools-api-docs.yml

@@ -19,7 +19,8 @@ jobs:
 
       - name: Install python dependencies
         run: |
-          python -m pip install --upgrade pip Sphinx sphinxcontrib-napoleon
+          pip install --upgrade pip
+          pip install -r ./docs/api/requirements.txt
           pip install .
 
       - name: Build HTML docs
@@ -30,6 +31,7 @@ jobs:
         run: |
           git checkout --orphan api-doc
           git rm -r --cache .
+          rm .gitignore
           git config user.email "[email protected]"
           git config user.name "nf-core-bot"
           git add docs

.gitignore

@@ -2,6 +2,7 @@
 
 .coverage
 .pytest_cache
+docs/api/_build
 
 # Byte-compiled / optimized / DLL files
 __pycache__/
@@ -20,7 +21,6 @@ dist/
 downloads/
 eggs/
 .eggs/
-lib/
 lib64/
 parts/
 sdist/

CHANGELOG.md

@@ -1,5 +1,69 @@
 # nf-core/tools: Changelog
 
+## [v1.13 - Copper Crocodile](https://github.com/nf-core/tools/releases/tag/1.13) - [2021-03-18]
+
+### Template
+
+* **Major new feature** - Validation of pipeline parameters [[#426]](https://github.com/nf-core/tools/issues/426)
+  * The addition runs as soon as the pipeline launches and checks the pipeline input parameters two main things:
+    * No parameters are supplied that share a name with core Nextflow options (eg. `--resume` instead of `-resume`)
+    * Supplied parameters validate against the pipeline JSON schema (eg. correct variable types, required values)
+  * If either parameter validation fails or the pipeline has errors, a warning is given about any unexpected parameters found which are not described in the pipeline schema.
+  * This behaviour can be disabled by using `--validate_params false`
+* Added profiles to support the [Charliecloud](https://hpc.github.io/charliecloud/) and [Shifter](https://nersc.gitlab.io/development/shifter/how-to-use/) container engines [[#824](https://github.com/nf-core/tools/issues/824)]
+  * Note that Charliecloud requires Nextflow version `v21.03.0-edge` or later.
+* Profiles for container engines now explicitly _disable_ all other engines [[#867](https://github.com/nf-core/tools/issues/867)]
+* Fixed typo in nf-core-lint CI that prevented the markdown summary from being automatically posted on PRs as a comment.
+* Changed default for `--input` from `data/*{1,2}.fastq.gz` to `null`, as this is now validated by the schema as a required value.
+* Removed support for `--name` parameter for custom run names.
+  * The same functionality for MultiQC still exists with the core Nextflow `-name` option.
+* Added to template docs about how to identify process name for resource customisation
+* The parameters `--max_memory` and `--max_time` are now validated against a regular expression [[#793](https://github.com/nf-core/tools/issues/793)]
+  * Must be written in the format `123.GB` / `456.h` with any of the prefixes listed in the [Nextflow docs](https://www.nextflow.io/docs/latest/process.html#memory)
+  * Bare numbers no longer allowed, avoiding people from trying to specify GB and actually specifying bytes.
+* Switched from cookiecutter to Jinja2 [[#880]](https://github.com/nf-core/tools/pull/880)
+* Finally dropped the wonderful [cookiecutter](https://github.com/cookiecutter/cookiecutter) library that was behind the first pipeline template that led to nf-core [[#880](https://github.com/nf-core/tools/pull/880)]
+  * Now rendering templates directly using [Jinja](https://jinja.palletsprojects.com/), which is what cookiecutter was doing anyway
+
+### Modules
+
+Initial addition of a number of new helper commands for working with DSL2 modules:
+
+* `modules list` - List available modules
+* `modules install` - Install a module from nf-core/modules
+* `modules remove` - Remove a module from a pipeline
+* `modules create` - Create a module from the template
+* `modules create-test-yml` - Create the `test.yml` file for a module with md5 sums, tags, commands and names added
+* `modules lint` - Check a module against nf-core guidelines
+
+You can read more about each of these commands in the main tools documentation (see `README.md` or <https://nf-co.re/tools>)
+
+### Tools helper code
+
+* Fixed some bugs in the command line interface for `nf-core launch` and improved formatting [[#829](https://github.com/nf-core/tools/pull/829)]
+* New functionality for `nf-core download` to make it compatible with DSL2 pipelines [[#832](https://github.com/nf-core/tools/pull/832)]
+  * Singularity images in module files are now discovered and fetched
+  * Direct downloads of Singularity images in python allowed (much faster than running `singularity pull`)
+  * Downloads now work with `$NXF_SINGULARITY_CACHEDIR` so that pipelines sharing containers have efficient downloads
+* Changed behaviour of `nf-core sync` command [[#787](https://github.com/nf-core/tools/issues/787)]
+  * Instead of opening or updating a PR from `TEMPLATE` directly to `dev`, a new branch is now created from `TEMPLATE` and a PR opened from this to `dev`.
+  * This is to make it easier to fix merge conflicts without accidentally bringing the entire pipeline history back into the `TEMPLATE` branch (which makes subsequent sync merges much more difficult)
+
+### Linting
+
+* Major refactor and rewrite of pipieline linting code
+  * Much better code organisation and maintainability
+  * New automatically generated documentation using Sphinx
+  * Numerous new tests and functions, removal of some unnecessary tests
+* Added lint check for merge markers [[#321]](https://github.com/nf-core/tools/issues/321)
+* Added new option `--fix` to automatically correct some problems detected by linting
+* Added validation of default params to `nf-core schema lint` [[#823](https://github.com/nf-core/tools/issues/823)]
+* Added schema validation of GitHub action workflows to lint function [[#795](https://github.com/nf-core/tools/issues/795)]
+* Fixed bug in schema title and description validation
+* Added second progress bar for conda dependencies lint check, as it can be slow [[#299](https://github.com/nf-core/tools/issues/299)]
+* Added new lint test to check files that should be unchanged from the pipeline.
+* Added the possibility to ignore lint tests using a `nf-core-lint.yml` config file [[#809](https://github.com/nf-core/tools/pull/809)]
+
 ## [v1.12.1 - Silver Dolphin](https://github.com/nf-core/tools/releases/tag/1.12.1) - [2020-12-03]
 
 ### Template