Add new plugin nf-python

[royjacobson]

project here with some documentation

[royjacobson]

I tested it with NXF_PLUGINS_TEST_REPOSITORY:

[user@fedora test]$ export NXF_PLUGINS_TEST_REPOSITORY=https://raw.githubusercontent.com/nextflow-io/plugins/ab8ae7935a0b51fbdd452fe461cda676ad0f1e67/plugins.json
[user@fedora test]$ nextflow run test_flow.nf -plugins 'nf-python'

 N E X T F L O W   ~  version 25.04.2

WARN: =======================================================================
=                                WARNING                                    =
= You are running this script using a un-official plugin repository.        =
=                                                                           =
= https://raw.githubusercontent.com/nextflow-io/plugins/ab8ae7935a0b51fbdd452fe461cda676ad0f1e67/plugins.json
=                                                                           =
= This is only meant to be used for plugin testing purposes.                =
=============================================================================

Launching `test_flow.nf` [confident_khorana] DSL2 - revision: 869cdde4b7

Downloading plugin [email protected]
executor >  local (1)
[4f/3e91ec] PyProcess [100%] 1 of 1 ✔
[TEST] pyFunction standalone returned: [foo:standalone, bar:42]
[TEST] memory roundtrip: [value:10241024, type:memory]
[TEST] path roundtrip: [value:/tmp/example.txt]
.....

[bentsherman]

Hi @royjacobson , thanks for contributing this plugin to the Nextflow community.

This initial release always takes the longest because we essentially have to decide whether to "give away" the plugin name, and you are requesting a "big" plugin name in nf-python. It is one that I expect will attract a lot of users, and one that we might potentially want to own as a core plugin one day.

At the same time, I don't want to hold you up indefinitely, even if users can already use your plugin via NXF_PLUGINS_TEST_REPOSITORY. And I think your plugin is fulfilling the purpose of what an nf-python plugin would be, so I won't ask you to rename the plugin to something more verbose.

In exchange, I'm going to be a bit more opinionated about this plugin than I normally would, since we're going to call it nf-python and it's likely to attract a lot of users. Here are my requests:

1. It looks like pyOperator is essentially a specialization of the map operator that simply calls pyFunction. We generally discourage plugin devs from creating custom operators that are just thin specializations of builtin-operators -- it's more work for you and it's less flexible than just exposing a function. I would prefer to just have the pyFunction() function (maybe even just call it python() ?) and encourage users in your docs to use the map operator or an exec process to make python calls over a channel. The exec process would be used over map only if you want caching, automatic retries, containers, etc.

2. Please provide some API documentation on your nextflow Python package, as well as documenting how the standard Nextflow types are passed into Python (most of them are documented here). It looks like it's basically just JSON ser-de but would be good to include a few examples for numbers, strings, lists, maps, files, etc.

3. Like I said, we may eventually want to provide a native Python integration, and there's a good chance we would want to do it as an "official" nf-python plugin. I think your plugin is a step in the direction we would want to go. For example, I'm interested in using GraalPy to run Python code directly on the JVM instead of using a shell. If/when that day comes, we might want to "take over" this plugin so that we can incorporate it more deeply into Nextflow and provide the best Nextflow-Python experience we can. Will you be open to that? If not, it might be better to use a different name.

[ewels]

Can I request that we call the Python module something other than just nextflow? There is already a package on PyPI with this name: https://pypi.org/project/nextflow/

To avoid conflicts, I'd prefer something more specific.

[bentsherman]

Good point, maybe we can call it nf_python instead

[royjacobson]

Hi, @bentsherman @ewels, thank you for the enthusiasm and the great feedback.

You raised good points, and I hope I'll have a new version that addresses them this week.
I'm happy to hear that this direction is something that you're thinking to support under the main project, and I'll be happy to transfer the plugin name if that becomes the case.

[royjacobson]

Ok! I actually did have enough time this evening to wrap up everything - please let me know what you think :)

I decided at the end to stick with pyFunction and not python, to reduce confusion and conflicts with existing code.

[bentsherman]

Looks great! Happy coding 😄