Skip to content

Add new plugin nf-python #132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 7, 2025
Merged

Conversation

royjacobson
Copy link
Contributor

@royjacobson
Copy link
Contributor Author

I tested it with NXF_PLUGINS_TEST_REPOSITORY:

[user@fedora test]$ export NXF_PLUGINS_TEST_REPOSITORY=https://raw.githubusercontent.com/nextflow-io/plugins/ab8ae7935a0b51fbdd452fe461cda676ad0f1e67/plugins.json
[user@fedora test]$ nextflow run test_flow.nf -plugins 'nf-python'

 N E X T F L O W   ~  version 25.04.2

WARN: =======================================================================
=                                WARNING                                    =
= You are running this script using a un-official plugin repository.        =
=                                                                           =
= https://raw.githubusercontent.com/nextflow-io/plugins/ab8ae7935a0b51fbdd452fe461cda676ad0f1e67/plugins.json
=                                                                           =
= This is only meant to be used for plugin testing purposes.                =
=============================================================================

Launching `test_flow.nf` [confident_khorana] DSL2 - revision: 869cdde4b7

Downloading plugin [email protected]
executor >  local (1)
[4f/3e91ec] PyProcess [100%] 1 of 1 ✔
[TEST] pyFunction standalone returned: [foo:standalone, bar:42]
[TEST] memory roundtrip: [value:10241024, type:memory]
[TEST] path roundtrip: [value:/tmp/example.txt]
.....

@bentsherman
Copy link
Member

Hi @royjacobson , thanks for contributing this plugin to the Nextflow community.

This initial release always takes the longest because we essentially have to decide whether to "give away" the plugin name, and you are requesting a "big" plugin name in nf-python. It is one that I expect will attract a lot of users, and one that we might potentially want to own as a core plugin one day.

At the same time, I don't want to hold you up indefinitely, even if users can already use your plugin via NXF_PLUGINS_TEST_REPOSITORY. And I think your plugin is fulfilling the purpose of what an nf-python plugin would be, so I won't ask you to rename the plugin to something more verbose.

In exchange, I'm going to be a bit more opinionated about this plugin than I normally would, since we're going to call it nf-python and it's likely to attract a lot of users. Here are my requests:

  1. It looks like pyOperator is essentially a specialization of the map operator that simply calls pyFunction. We generally discourage plugin devs from creating custom operators that are just thin specializations of builtin-operators -- it's more work for you and it's less flexible than just exposing a function. I would prefer to just have the pyFunction() function (maybe even just call it python() ?) and encourage users in your docs to use the map operator or an exec process to make python calls over a channel. The exec process would be used over map only if you want caching, automatic retries, containers, etc.

  2. Please provide some API documentation on your nextflow Python package, as well as documenting how the standard Nextflow types are passed into Python (most of them are documented here). It looks like it's basically just JSON ser-de but would be good to include a few examples for numbers, strings, lists, maps, files, etc.

  3. Like I said, we may eventually want to provide a native Python integration, and there's a good chance we would want to do it as an "official" nf-python plugin. I think your plugin is a step in the direction we would want to go. For example, I'm interested in using GraalPy to run Python code directly on the JVM instead of using a shell. If/when that day comes, we might want to "take over" this plugin so that we can incorporate it more deeply into Nextflow and provide the best Nextflow-Python experience we can. Will you be open to that? If not, it might be better to use a different name.

@ewels
Copy link
Member

ewels commented Jul 1, 2025

Can I request that we call the Python module something other than just nextflow? There is already a package on PyPI with this name: https://pypi.org/project/nextflow/

To avoid conflicts, I'd prefer something more specific.

@bentsherman
Copy link
Member

Good point, maybe we can call it nf_python instead

@royjacobson
Copy link
Contributor Author

Hi, @bentsherman @ewels, thank you for the enthusiasm and the great feedback.

You raised good points, and I hope I'll have a new version that addresses them this week.
I'm happy to hear that this direction is something that you're thinking to support under the main project, and I'll be happy to transfer the plugin name if that becomes the case.

@royjacobson
Copy link
Contributor Author

Ok! I actually did have enough time this evening to wrap up everything - please let me know what you think :)

I decided at the end to stick with pyFunction and not python, to reduce confusion and conflicts with existing code.

@bentsherman
Copy link
Member

Looks great! Happy coding 😄

@bentsherman bentsherman merged commit f77f5d4 into nextflow-io:main Jul 7, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants