Concurrency issues when sharing Proto/Moon cache in Dagger #2031
Replies: 2 comments 18 replies
-
|
@arthur-leclerc Yo, this is awesome! Your assumption about the folders is correct, anything in As for your problems.
Does it always hang at Another issue here which I'm going to fix in moon v2, is that the moon binaries are not compressed in the releases: https://github.com/moonrepo/moon/releases/tag/v1.38.0 So each download is about 30-40mb and may take some time based on network speed.
As the error states, this shouldn't be possible... It's failing here, which means there's no package manager: https://github.com/moonrepo/moon/blob/master/legacy/node/tool/src/node_tool.rs#L215 But we literally create one of them in the constructor here: https://github.com/moonrepo/moon/blob/master/legacy/node/tool/src/node_tool.rs#L85 What's your |
Beta Was this translation helpful? Give feedback.
-
|
I was going to come update that I still had the handling when using a locked volume... but I think you got it right from the first try! 👏 For my tests:
Result with your recommendation to run
You helped me a lot, thank you very much! The result is really more than satisfactory! Would it be possible to have more information about what does the For the uncompressed Moon file issue, I think it's not too serious vs a functional toolchain + package manager cache, 40mb vs the size of a Now there remains this mysterious Here's my current node:
version: 22.16.0
packageManager: yarn
yarn:
version: 4.9.2
addEnginesConstraint: true
dedupeOnLockfileChange: true
dependencyVersionFormat: workspace
syncProjectWorkspaceDependencies: true
python:
version: 3.13.5
packageManager: uv
uv:
version: 0.6.14
syncArgs:
- --locked
- --no-dev
- --no-install-project
typescript:
routeOutDirToCache: true
syncProjectReferences: true
syncProjectReferencesToPaths: true
unstable_go:
version: 1.24.4
tidyOnChange: trueI'll need to retest now that everything else is stable now, but I have another question about I don't have a lot of experience with uv, but if I understood correctly, if I don't pass the options I added in With the option I added, I'm able to create separate layer for each cmd: // Build builds the project with Moon (scaffold + setup) and returns a container with the project built.
func (m *Project) Build() *dagger.Container {
skeleton := m.Moon.BaseCtr.
WithMountedDirectory(WorkDir, m.Moon.Source).
WithExec([]string{"moon", "docker", "scaffold", m.ProjectID}).
Directory(filepath.Join(WorkDir, ".moon/docker"))
workspaceDir := skeleton.Directory("workspace")
sourcesDir := skeleton.Directory("sources")
return m.Moon.Builder(m).
WithMountedDirectory(WorkDir, workspaceDir).
WithExec([]string{"moon", "docker", "setup"}).
WithDirectory(WorkDir, sourcesDir).
WithExec([]string{"cat", ".moon/toolchain.yml"}).
WithExec([]string{"moon", "run", fmt.Sprintf("%s:build", m.ProjectID)}).
WithExec([]string{"moon", "docker", "prune"})
}My "hack" make it work the way I want, but the problem I have is that the behavior in local is also modified and thus I need a local task to manually sync uv lock file as Moon will now fail to do it.
|
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
Quiet turns off the TUI rendering stuff, like progress bars. These should not render if not a TTY, but maybe dagger thinks its a TTY?
This is a good question and I don't have an answer really here. Rust kind of has the same problem, where Cargo fails without a I think in the future,
No way right now.
It runs the
Nope, but
There is not, but there should be no difference in CI. |
Beta Was this translation helpful? Give feedback.
-
|
Regarding the Main thread panicked error, here is a complete log when this occurs.
node:
version: 22.16.0
packageManager: yarn
yarn:
version: 4.9.2
addEnginesConstraint: true
dedupeOnLockfileChange: true
dependencyVersionFormat: workspace
syncProjectWorkspaceDependencies: true
python:
version: 3.13.5
packageManager: uv
uv:
version: 0.6.14
syncArgs:
- --locked
- --no-dev
- --no-install-project
typescript:
routeOutDirToCache: true
syncProjectReferences: true
syncProjectReferencesToPaths: true
unstable_go:
version: 1.24.4
tidyOnChange: true |
Beta Was this translation helpful? Give feedback.
-
|
Good news! I figured out the duplicate action node problem, which also resolved that package manager panic. I tried debugging the panic but it was very weird. It would fail with a bus 10 error pretty consistently, so I think this was hitting some weird memory access violation. That's why it was basically reaching this code that should be unreachable. I've ran the ko script about 10x now without a single failure. |
Beta Was this translation helpful? Give feedback.
-
|
Great news! I can't wait to try it! 🚀 |
Beta Was this translation helpful? Give feedback.
-
|
Ok try 1.38.3 🤞 |
Beta Was this translation helpful? Give feedback.
-
|
Hello, for now it seems very promising, I can't reproduce the issue locally or on Dagger. I'll need to test more thoroughly the case where the cache is cleared and there are several PR triggers at the same time, so many parallel builds. I thought I saw some errors during my initial tests, we'll see if I can reproduce this in a local environment. I'll keep you updated on my progress. In any case, with a hydrated cache, the solution is very promising. I'm going to slow down a bit on this project for now, plus I don't want to monopolize all your time. We'll see how it performs in normal usage (without wiping the cache every 5 minutes and forcing 30 builds at once :D). Thank you again for your time and help. |
Beta Was this translation helpful? Give feedback.
-
|
Awesome, glad to hear it. Definitely let me know if you have other questions! |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hey there!
I'm currently working on implementing a Dagger module for Moon.
If you're not familiar with this technology, to put it simply, Dagger allows you to create CI pipelines as code and manipulate containers via an SDK. At its core, I can do the same things that can be achieved in a Dockerfile, and even more.
What I'm trying to do is to be able to run Moon, particularly these commands in a containerized environment:
moon docker scaffoldmoon docker setupmoon docker pruneTo do this, I leverage several features of Dagger/Docker, allowing me to share the cache folders of Proto, Moon, but also package managers like Yarn, UV, Go, etc.
The idea is to create a base container with Proto and Moon installed inside. From what I understand, Proto/Moon mainly use these folders to work:
~/.proto/, the Proto cache, for me it's safe to share between multiple Proto/Moon processes.~/.moon/, I'm not sure about this one, for me it's the equivalent of~/.proto/but for Moon's wasm, and it should also be safe to share between multiple Proto/Moon processes./workspace/.moon/, the Moon cache for a workspace, mainly the cache of tasks etc, for now I don't share it, because I focus on optimizing the setup of toolchains and package managers cache, and I don't think it should be shared but I might be wrong. I'll probably set up the Moon cache directly to share the result of tasks.I won't share the entire code, but some parts to expose my thoughts. Here's how the base is created:
When I run my build function, I call my
baseContainer()which returns a container ready to use, and I can technically call it several times in parallel, it's the responsibility of Dagger to not re-execute what should not be executed. In this container I will then call the commandsmoon docker scaffold,moon docker setupandmoon docker prune, respectively to scaffold a project, setup the needed toolchains, build the project and clean it up, as we would do in a Dockerfile.I come back to a part of my code, in my example, I use
Sharing: dagger.CacheSharingModeLocked, which allowsShares the cache volume amongst many build pipelines, but will serialize the writes, in this configuration, my code works correctly, but I have of course slowdowns when I run my function several times, because a container can't really use the cache at the same time, and therefore each exec waiting for the other to finish, but each process reuses the cache folders, but can't use them at the same time.But if I switch to the
dagger.CacheSharingModeSharedmode (Shares the cache volume amongst many build pipelines), and thus I allow parallel writes, I have the impression of having concurrency problems when several Proto/Moon commands are run at the same time.First when executing several
proto install moon, often the command hangs:Then, when building projects, sometimes I have strange errors like:
From what I understand, I should be able to share these folders, because Proto and Moon use locks, but maybe I'm wrong about the folder mounting, maybe the
/workspace/.moon/folder is necessary?I would like your opinion on this subject, and if you can give me information if such a setup is possible, and if so, how to do it, what is important in these cache folders, etc.
I'll stop here in my explanation, to not make the message too long, but I'm available if you have any questions.
Thank you very much for your help.
Beta Was this translation helpful? Give feedback.
All reactions