[noaccOS]

When a node goes down, all the entries in the registry were deleted. This means that if the same node came back up, the result was an empty registry. When spawning processes using via tuples, the result was that the process was still alive, but without an associated name.

As keys are already filtered on alive nodes when queried, explicit deletion is not required.

Note for reviewers: the issue was found and the code has mainly been written by LLMs. The tests have been confirmed as failing before the fix was applied.

[sleipnir]

@noaccOS Thank you. I will review this soon.

[noaccOS]

@sleipnir The PR was updated because we've found in production that an healthy process could receive a name_conflict from other members caused by a previous instance (we use horde to keep a singleton process for RPC calls). Now registries make sure the old entry is alive before sending name_conflict messages.

[derekkraan]

@noaccOS have you considered using highlander_pg (preferred imo) or the classic highlander? These libs are more suited towards keeping a singleton process alive.

[noaccOS]

@noaccOS have you considered using highlander_pg (preferred imo) or the classic highlander? These libs are more suited towards keeping a singleton process alive.

@derekkraan Yes we did see the note in the README but we're already leveraging horde anyway and there is no reason it shouldn't reliably work. If we can't trust it for a singleton we can't trust it for other processes either :)

[sleipnir]

@sleipnir The PR was updated because we've found in production that an healthy process could receive a name_conflict from other members caused by a previous instance (we use horde to keep a singleton process for RPC calls). Now registries make sure the old entry is alive before sending name_conflict messages.

@noaccOS Let's do this: revert the last commit to the PR's previous state and open a new PR so we can discuss the issue that has come up.

[noaccOS]

@sleipnir Reverted the previous commit as requested + added the separate fix on top.
They should be merged together because the "dead process on name conflict" is only an issue with my first commit which allows dead processes in the registry

[noaccOS]

Found a brand new issue, but this time the symptoms were different: the service which started the process has the correct state (both supervision tree and registered names), but other services had an outdated state with a previous process registered in the registry

[sleipnir]

Found a brand new issue, but this time the symptoms were different: the service which started the process has the correct state (both supervision tree and registered names), but other services had an outdated state with a previous process registered in the registry

Hi @noaccOS
What is the network topology of your scenario? How many nodes are we talking about? Do they change frequently? How are the CRDT update settings configured?

[noaccOS]

Hi @noaccOS What is the network topology of your scenario? How many nodes are we talking about? Do they change frequently? How are the CRDT update settings configured?

@sleipnir

There are a total of 6 connected nodes to the same cluster, 2 of which spawn an RPC server which have their own horde registry. Other services start the registries of the RPCs they need.

They do change frequently, pretty much each time we release a new snapshot kubernetes starts the new pod and then kills the old one.

CDRT settings are not tweaked