Use snakemake conda envs with GitHub Actions/cache

[lczech]

Hi there,

I am trying to create CI for a Snakemake workflow that needs to install several conda envs for different rules. Is there a way to use the GitHub actions/cache to eliminate the need to install the envs for every run?

Background

The main problem: Snakemake installs the per-rule conda envs with a name that is a hash of the env file. This makes it hard to know which path to use for the actions/cache. Is there a way to do this?

Secondary question

As a secondary question: If this was solved, how would that work across jobs? I have a matrix of jobs in my GitHub Actions workflow (testing different aspects of my Snakemake workflow), of which several use the same rules and conda envs. It would be ideal if conda envs would be re-used across them.

However, I could not find any information on how race conditions are solved when multiple jobs use the same cache keys. Imagine GitHub Actions job A and job B run in parallel, job A starts with the action/cache empty, and hence creates a conda env to be cached. At the same time, if job B starts before job A is done, it will not yet find the cache. This is because

On a cache miss, the action automatically creates a new cache if the job completes successfully. (source)

Will job B then also start creating the conda env and filling the cache? Or will it wait for job A to finish?

Cheers and thanks in advance
Lucas

[lparsons]

I'm running into the same issue (long wait for repeated conda installations). Does anyone have any insight into getting this setup?

[lczech]

Hi @lparsons,

I have not found a solution yet. But using mamba instead of conda definitely helps to alleviate the problem. In my case, for some GitHub CI jobs, it's going down from 6h (which is already the time limit for GitHub Actions) to 15min.

Lucas

[m-jahn]

Hi Lucas @lczech !
Is this issue still relevant for you?

[lczech]

Hi @m-jahn,

thanks for looking into this! Well, I am still using the above solution: mamba. It's fast enough in practice, although of course wastes a lot of runtime in my CI runs for re-installing the environments over and over again.

So, while I do have a "workable enough" solution, I'd still be interested in a more efficient way, if you see one :-)

All the best
Lucas

[m-jahn]

To be honest, I wanted to check first if this issue is still relevant or if it can be closed considering it has been open for 2 years.
Would an apptainer probably be a solution, where the conda envs are already pre-installed? Then instead of testing --sdm conda you run your tests with sdm apptainer,conda? I have not thought this through, just a wild guess.

Loosely related: I'm currently thinking about making a github action that would automatically deploy 1 apptainer on GHCR so to allow most standardized workflows to run with apptainer, without the hassle of manually building one after every update.

[m-jahn]

OK forget my idea, I was just remembered that an apptainer also gets the conda envs installed during workflow runtime, not at the moment of making/deploying the container image. So that would save no package installs at all.