Riak crashed after nodename adjust

[AndreiBrajnicov]

I was creating nodes from a template, where in riak.conf it was defined as:

nodename = riak@0.0.0.0

I was joining a bunch of nodes to the cluster, and for one of them I forgot to adjust the nodename variable. I noticed this after executing:

riak admin cluster join

But I didn’t hit the cluster plan and cluster commit commands.

After realizing the misconfiguration, I stopped the node, removed it from the cluster (using another node), adjusted the nodename variable, and then noticed that Riak is not starting on the node where I messed up the nodename:

[root@ip-10-228-211-90 ~]# riak start
[root@ip-10-228-211-90 ~]# echo $?
0
[root@ip-10-228-211-90 ~]# riak ping
Node 'riak@10.228.211.90' not responding to pings.
[root@ip-10-228-211-90 ~]#

Expected behaviour - to receive an error or some other meaningful information related to this case.
Actual bevaviour - app crashed

This issue happened in version 3.0.1. Tbh, I wasn’t able to find anything similar in the changelog.

Logs

cat console.log
2025-12-12 12:38:20.342 [info] <0.1224.0> Application lager started on node 'riak@10.228.211.90'
2025-12-12 12:38:20.342 [info] <0.1224.0> Application cuttlefish started on node 'riak@10.228.211.90'
2025-12-12 12:38:20.343 [info] <0.1224.0> Application riak_sysmon started on node 'riak@10.228.211.90'
2025-12-12 12:38:20.343 [info] <0.1224.0> Application clique started on node 'riak@10.228.211.90'
2025-12-12 12:38:20.343 [info] <0.1224.0> Application basho_stats started on node 'riak@10.228.211.90'
2025-12-12 12:38:20.343 [info] <0.1224.0> Application pbkdf2 started on node 'riak@10.228.211.90'
2025-12-12 12:38:20.343 [info] <0.1224.0> Application poolboy started on node 'riak@10.228.211.90'
2025-12-12 12:38:20.343 [info] <0.1224.0> Application cluster_info started on node 'riak@10.228.211.90'
2025-12-12 12:38:20.343 [info] <0.1224.0> Application eleveldb started on node 'riak@10.228.211.90'
2025-12-12 12:38:20.343 [info] <0.1224.0> Application ssl started on node 'riak@10.228.211.90'
2025-12-12 12:38:20.343 [info] <0.1224.0> Application mochiweb started on node 'riak@10.228.211.90'
2025-12-12 12:38:20.344 [info] <0.1224.0> Application webmachine started on node 'riak@10.228.211.90'
2025-12-12 12:38:20.345 [info] <0.1224.0> Application os_mon started on node 'riak@10.228.211.90'
2025-12-12 12:38:20.355 [info] <0.1403.0> msg: "Starting reporters with []\n", options: []
2025-12-12 12:38:20.355 [info] <0.1224.0> Application exometer_core started on node 'riak@10.228.211.90'
2025-12-12 12:38:20.374 [error] <0.1411.0> Supervisor riak_core_sup had child riak_core_broadcast started with riak_core_broadcast:start_link() at undefined exit with reason {'EXIT',{function_clause,[{orddict,fetch,['riak@10.228.211.90',[]],[{file,"orddict.erl"},{line,80}]},{riak_core_broadcast,init_peers,1,[{file,"/home/ec2-user/riak-riak-3.0.1/_build/default/lib/riak_core/src/riak_core_broadcast.erl"},{line,612}]},{riak_core_broadcast,start_link,0,[{file,"/home/ec2-user/riak-riak-3.0.1/_build/default/lib/riak_core/src/riak_core_broadcast.erl"},{line,116}]},{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,365}]},{supervisor,start_children,3,[{file,"..."},...]},...]}} in context start_error
2025-12-12 12:38:20.374 [error] <0.1409.0> CRASH REPORT Process <0.1409.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,riak_core_broadcast,{'EXIT',{function_clause,[{orddict,fetch,['riak@10.228.211.90',[]],[{file,"orddict.erl"},{line,80}]},{riak_core_broadcast,init_peers,1,[{file,"/home/ec2-user/riak-riak-3.0.1/_build/default/lib/riak_core/src/riak_core_broadcast.erl"},{line,612}]},{riak_core_broadcast,start_link,0,[{file,"/home/ec2-user/riak-riak-3.0.1/_build/default/lib/riak_core/src/riak_core_broadcast.erl"},{line,116}]},{supervisor,do_start_child,2,[{file,"supervisor.e..."},...]},...]}}}},...} in application_master:init/4 line 134
2025-12-12 12:38:20.374 [info] <0.1224.0> Application riak_core exited with reason: {{shutdown,{failed_to_start_child,riak_core_broadcast,{'EXIT',{function_clause,[{orddict,fetch,['riak@10.228.211.90',[]],[{file,"orddict.erl"},{line,80}]},{riak_core_broadcast,init_peers,1,[{file,"/home/ec2-user/riak-riak-3.0.1/_build/default/lib/riak_core/src/riak_core_broadcast.erl"},{line,612}]},{riak_core_broadcast,start_link,0,[{file,"/home/ec2-user/riak-riak-3.0.1/_build/default/lib/riak_core/src/riak_core_broadcast.erl"},{line,116}]},{supervisor,do_start_child,2,[{file,"supervisor.e..."},...]},...]}}}},...}

# cat crash.log
2025-12-12 12:38:20 =SUPERVISOR REPORT====
     Supervisor: {local,riak_core_sup}
     Context:    start_error
     Reason:     {'EXIT',{function_clause,[{orddict,fetch,['riak@10.228.211.90',[]],[{file,"orddict.erl"},{line,80}]},{riak_core_broadcast,init_peers,1,[{file,"/home/ec2-user/riak-riak-3.0.1/_build/default/lib/riak_core/src/riak_core_broadcast.erl"},{line,612}]},{riak_core_broadcast,start_link,0,[{file,"/home/ec2-user/riak-riak-3.0.1/_build/default/lib/riak_core/src/riak_core_broadcast.erl"},{line,116}]},{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,365}]},{supervisor,start_children,3,[{file,"supervisor.erl"},{line,348}]},{supervisor,init_children,2,[{file,"supervisor.erl"},{line,314}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,365}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,333}]}]}}
     Offender:   [{pid,undefined},{id,riak_core_broadcast},{mfargs,{riak_core_broadcast,start_link,[]}},{restart_type,permanent},{shutdown,5000},{child_type,worker}]

2025-12-12 12:38:20 =CRASH REPORT====
  crasher:
    initial call: application_master:init/4
    pid: <0.1409.0>
    registered_name: []
    exception exit: {{{shutdown,{failed_to_start_child,riak_core_broadcast,{'EXIT',{function_clause,[{orddict,fetch,['riak@10.228.211.90',[]],[{file,"orddict.erl"},{line,80}]},{riak_core_broadcast,init_peers,1,[{file,"/home/ec2-user/riak-riak-3.0.1/_build/default/lib/riak_core/src/riak_core_broadcast.erl"},{line,612}]},{riak_core_broadcast,start_link,0,[{file,"/home/ec2-user/riak-riak-3.0.1/_build/default/lib/riak_core/src/riak_core_broadcast.erl"},{line,116}]},{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,365}]},{supervisor,start_children,3,[{file,"supervisor.erl"},{line,348}]},{supervisor,init_children,2,[{file,"supervisor.erl"},{line,314}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,365}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,333}]}]}}}},{riak_core_app,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,134}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
    ancestors: [<0.1408.0>]
    message_queue_len: 1
    messages: [{'EXIT',<0.1410.0>,normal}]
    links: [<0.1408.0>,<0.1224.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 1598
    stack_size: 27
    reductions: 238
  neighbours:

cat error.log
2025-12-12 12:38:20.374 [error] <0.1411.0> Supervisor riak_core_sup had child riak_core_broadcast started with riak_core_broadcast:start_link() at undefined exit with reason {'EXIT',{function_clause,[{orddict,fetch,['riak@10.228.211.90',[]],[{file,"orddict.erl"},{line,80}]},{riak_core_broadcast,init_peers,1,[{file,"/home/ec2-user/riak-riak-3.0.1/_build/default/lib/riak_core/src/riak_core_broadcast.erl"},{line,612}]},{riak_core_broadcast,start_link,0,[{file,"/home/ec2-user/riak-riak-3.0.1/_build/default/lib/riak_core/src/riak_core_broadcast.erl"},{line,116}]},{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,365}]},{supervisor,start_children,3,[{file,"..."},...]},...]}} in context start_error
2025-12-12 12:38:20.374 [error] <0.1409.0> CRASH REPORT Process <0.1409.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,riak_core_broadcast,{'EXIT',{function_clause,[{orddict,fetch,['riak@10.228.211.90',[]],[{file,"orddict.erl"},{line,80}]},{riak_core_broadcast,init_peers,1,[{file,"/home/ec2-user/riak-riak-3.0.1/_build/default/lib/riak_core/src/riak_core_broadcast.erl"},{line,612}]},{riak_core_broadcast,start_link,0,[{file,"/home/ec2-user/riak-riak-3.0.1/_build/default/lib/riak_core/src/riak_core_broadcast.erl"},{line,116}]},{supervisor,do_start_child,2,[{file,"supervisor.e..."},...]},...]}}}},...} in application_master:init/4 line 13

[martinsumner]

A node at startup is a cluster of one, so when a node starts, it will define a ring with one member - before joining a cluster and adopting the new ring gossiped about that cluster.

The ring contains the nodename. When you want to rename a node, it has to be renamed in the config, but also in the ring that was saved at the previous startup.

The formal way of doing this is using reip_manual - see riak admin reip_manual --help. So as well as giving its new correct name in riak.conf before you start you have to reip_manual so that name matches that of the node in the ring.

Given this node is a fresh node, not yet joined to anything, you can just wipe the data directory, and rename it in riak.conf. Deleting the old ring rather than correcting. It will then start with a fresh ring with the correct node name.

It is expected behaviour that it is not sufficient to change riak.conf to update a nodename (once the node has been started). I agree the error is unhelpful though.

Not sure how easy it is to address, but I agree it should be at least a clearer error in the logs to state that the node has been restarted after a name change, and a pointer to use reip_manual

[martinsumner]

The ideal way to solve this would be through a pre_start script: https://github.com/OpenRiak/riak/blob/openriak-3.4/rebar.config#L115-L120.

However, I don't think that pre_start it would be possible to know what node to check for in the ring file, as that is in the configuration.

So the best change is probably to alter the riak_core_app:start/2 function, prior to it starting the supervisor riak_core_sup. There is presently a check for the ring folder within this function: this could go further an check for a ring file, and if one is present check that the nodename is within the file. If there is a file and it does not contain a nodename, throw an exception with a hint to use reip manual.