bug: singletons fail to relocate on leader leaving a cluster

[TrueBrain]

Bug description
I am trying to get a singleton scheduled on a cluster. I am creating a persistent scheduler (backed by a database), to allow grains to schedule messages at a later date, and the cluster to survive node outages, including full cluster outages.

When experimenting with it, I notice often that my cluster is left without the singleton running. This could full well be that I am doing something wrong initializing the singleton.

It seems this is a composition of several bugs on top of each other, so I am just going to write out what I was trying and what I observed. Hopefully you can make something out of that :D If not, let me know, and I will see if I can write small test applications :) But first, the main issue I was running into.

How to reproduce it?
Running two nodes in a cluster (with quorum of 2, with relocation enabled), and the following snippet on startup:

n.actorSystem.SpawnSingleton(ctx, "scheduler", scheduler.New())

Now when I shut down the node running the scheduler, I get this message:

{"level":"error","ts":"2025-12-27T10:00:14.898023+0100","caller":"actor/relocator.go:106","msg":"cluster rebalancing failed: spawn error: singleton already exists"}

And the cluster is left without a scheduler running.

Expected behavior
Always have the singleton running somewhere on the cluster.

Library Version:

• Go-Akt version: 1b11cfd
• Go version: 1.25.3

Additional context
Initially I was running my cluster with WithoutRelocation. In my use-case, I don't need grains to relocate when a node is shut down. They will just spin up a new instance on next invocation. No need for migration: only wasting CPU.
However, for singletons, if the leader goes away, it doesn't restart the singleton on the new leader either. I somewhat did expect that, as even so I use WithoutRelocation, I did expect singletons to be the exception here. But this was an assumption on my side :D
So now the question becomes: how do I keep a singleton running on a cluster with WithoutRelocation? As for grains, it is very much exactly what I want. Singletons however are the exception in my use-case. But I also can't tell when using client.TellGrain that the grain shouldn't be relocated on node issues.

Another thing I ran into, but this really is a "me" problem: I had a hard time getting singletons working. Initially I just did a n.actorSystem.SpawnSingleton(ctx, "scheduler", scheduler.New()) when each nodes starts. But this errors on the second node (correctly, if you ask me). The error however was difficult to deal with: internal: actor=(scheduler) actor already exists. This is not singleton already exists, and I couldn't differentiate between "there is another error" vs "this singleton already started".
In the end I just added an IsLeader check in front of the SpawnSingleton, once I figured out that "oldest node" means "is-leader" :) So my code became:

	if isLeader, _ := n.actorSystem.IsLeader(ctx); isLeader {
		n.actorSystem.SpawnSingleton(ctx, "scheduler", scheduler.New())
	}

But there might be better ways to ensure a singleton is running on start of a cluster?

[Tochemey]

@TrueBrain I am finding hard to follow the issue. Singleton actor gets relocated automatically on the oldest node of the cluster when the node hosting them goes down and when WithoutRelocation is not used. There is no exception for singleton actors because system-wide relocation feature affect the whole system.

[Tochemey]

@TrueBrain I am happy to have a chat if you are available within the next hour. Let me know and I will send you a link.

[TrueBrain]

Yeah, I should have created an example app before reporting the bug. Sorry about that :)

Here is an example of what I am doing: gist

And if I do the following:

Terminal 1) go run ./main.go
Terminal 2) go run ./main.go 44422

After I see on terminal 1 that it has received two messages, I stop the node on terminal 1. I would expect the singleton to be started on terminal 2 now, right?

Instead, I get this:

{"level":"error","ts":"2025-12-27T13:02:07.465892+0100","caller":"actor/relocator.go:106","msg":"cluster rebalancing failed: spawn error: internal error: deadline exceeded"}

Which is different from what I get when running it in my own application, but at least it gives you a chance to understand what I am running up against :) Just to be clear, this could easily be me not understanding how to start a singleton :)

[Tochemey]

@TrueBrain Are you available for a chat?

[TrueBrain]

Ah, sorry, forgot to answer: sure! Throw me a link :)

[Tochemey]

@TrueBrain I have added a fix: #1038

[TrueBrain]

All these additions are awesome. I really like the new options! And that I can run SpawnSingleton in an idempotent way makes reasoning about singletons a lot easier (for me at least) :)

One question: the new options, do they also apply during rebalancing? As that still fails with the default partition count of 271 (it works when set to 9) for me.
Edit: I can fully understand if they don't apply to rebalance btw; mostly checking if I am understanding it correct :)

[Tochemey]

Let me look at the partition count issue. I am making this options network-aware so that I can use them during relocation.

[TrueBrain]

Did a bit of delving into the code to see if I can trace where the issue comes from. From what I can tell:

• Reallocation happens, and the singleton is send for rescheduling. The new node is marked as IsLeader.
• recreateLocally calls r.pid.ActorSystem().getCluster().RemoveActor(), which calls x.deleteRecord() on the cluster.
• Here it times out on writeTimeout while trying x.dmap.Delete().

If I increase the timeout, it simply fails on connection refused. So it seems the cluster isn't in a state where it can be mutated?

[TrueBrain]

This is only a problem if a node leaves the cluster before Stop() finished its work. And this is because the gist I shared doesn't give any real time for the Stop() at the end to do its job (it seems that NATS and CTRL+C have an interaction, causing the process to exit before Stop() actually finished).

In other words: this works fine (with your latest PR) if a node cleanly removes itself from the cluster, but this goes wrong if a node just disappears.

Edit: this also happens in a cluster of 3, with minquorum of 2 set. There too, actors can't be respawn because the cluster dmap isn't working in time.

[TrueBrain]

Bit more digging, as looking into bugs is fun :D From what I can tell, the relation with partitions is just "luck".

It appears that my singleton actor is assigned to a partition that is owned by node1. And node1 just left the cluster. But on next mutation of that partition, node1 is contacted to make the mutation. But ... it left the cluster. So things fails from there on.

This seems to be in the olric library; I would assume that if a node left, the partitions are reassigned. But it is still in the process of reassigning the partitions when node-left event is fired. So possibly this is a timing issue more than anything else. I will continue to dig a bit more :)

Btw, more a reminder for myself: nodes are only added to rebalancedNodes, but never removed. So I need to test what happens if a node leaves, joins again, and leaves again. It might not rebalance in those case. But that is a problem to look into after I looked some more into this partitioning issue :)

Edit: indeed, setting a time.Sleep(time.Second) in handleNodeLeftEvent solves the issue (but is a non-sense solution ofc). So olric is still repartitioning while the node-left event is emitted, and so any keys on partitions still owned by the now-left node will cause issues. Concurrency. Fun. It feels to me that olric shouldn't emit a node-left event till repartitioning is done, but I don't know enough about olric's design to say anything about that really :) So hopefully you have some clever ideas here.

[Tochemey]

If you don't have enough nodes then when the node holding the partition goes off, the cluster is completely unstable and cannot be recovered.