`erleans` with `plumdb` interoperability issue with other `partisan` apps

[mcesaro]

Hi,
I keep getting this crashes:

=ERROR REPORT==== 27-Oct-2023::17:41:21.504285 ===
** Generic server partisan_plumtree_broadcast terminating 
** Last message in was {'$gen_cast',
                           {broadcast,
                               {{{erleans_pm,registry},
                                 {'pebble@max-a5',
                                     {<<"01HDNZBQH8E17KWKSXGFVBCHXG">>,
                                      pebble}}},
                                [{{12,
                                   '09a4965a-74d9-11ee-8d17-1d71807ddbb4@max-a5'},
                                  2},
                                 {{12,
                                   '33dd009a-74db-11ee-b011-1d71764ec79c@max-a5'},
                                  2},
                                 {{12,
                                   '5fd8bdc8-74dc-11ee-b48c-1d71764ec79c@max-a5'},
                                  2},
                                 {{12,
                                   '7b2896d2-7411-11ee-b434-1d71764ec79c@max-a5'},
                                  4},
                                 {{12,
                                   '9613b588-74db-11ee-a4af-1d71807ddbb4@max-a5'},
                                  4},
                                 {{12,
                                   'a55499b2-74d7-11ee-a9df-1d71807ddbb4@max-a5'},
                                  2},
                                 {{12,
                                   'b794aeda-74de-11ee-b5e2-1d71764ec79c@max-a5'},
                                  2},
                                 {{12,
                                   'f041ca12-74dc-11ee-88b3-1d71807ddbb4@max-a5'},
                                  2},
                                 {{12,
                                   'f575b378-73fa-11ee-875f-1d718ad5fce5@max-a5'},
                                  4},
                                 {{12,
                                   'fb2a5b2c-74de-11ee-9c8a-1d71764ec79c@max-a5'},
                                  2}]},
                               {object,
                                   {[{{12,
                                       '09a4965a-74d9-11ee-8d17-1d71807ddbb4@max-a5'},
                                      2,[]},
                                     {{12,
                                       '33dd009a-74db-11ee-b011-1d71764ec79c@max-a5'},
                                      2,[]},
                                     {{12,
                                       '5fd8bdc8-74dc-11ee-b48c-1d71764ec79c@max-a5'},
                                      2,[]},
                                     {{12,
                                       '7b2896d2-7411-11ee-b434-1d71764ec79c@max-a5'},
                                      4,[]},
                                     {{12,
                                       '9613b588-74db-11ee-a4af-1d71807ddbb4@max-a5'},
                                      4,[]},
                                     {{12,
                                       'a55499b2-74d7-11ee-a9df-1d71807ddbb4@max-a5'},
                                      2,[]},
                                     {{12,
                                       'b794aeda-74de-11ee-b5e2-1d71764ec79c@max-a5'},
                                      2,[]},
                                     {{12,
                                       'f041ca12-74dc-11ee-88b3-1d71807ddbb4@max-a5'},
                                      2,[]},
                                     {{12,
                                       'f575b378-73fa-11ee-875f-1d718ad5fce5@max-a5'},
                                      4,[]},
                                     {{12,
                                       'fb2a5b2c-74de-11ee-9c8a-1d71764ec79c@max-a5'},
                                      2,
                                      [{'$deleted',{1698,421281,501793}}]}],
                                    []}},
                               plum_db,0,'pebble@max-a5','pebble@max-a5'}}
** When Server state == {state,['clockwork@max-a5','pebble@max-a5'],
                               [],#{},#{},
                               [partisan_plumtree_backend],
                               [],
                               ['clockwork@max-a5','pebble@max-a5',
                                'ska@max-a5'],
                               1000,10000}
** Reason for termination ==
** {'module could not be loaded',
       [{plum_db,merge,
            [{{{erleans_pm,registry},
               {'pebble@max-a5',{<<"01HDNZBQH8E17KWKSXGFVBCHXG">>,pebble}}},
              [{{12,'09a4965a-74d9-11ee-8d17-1d71807ddbb4@max-a5'},2},
               {{12,'33dd009a-74db-11ee-b011-1d71764ec79c@max-a5'},2},
               {{12,'5fd8bdc8-74dc-11ee-b48c-1d71764ec79c@max-a5'},2},
               {{12,'7b2896d2-7411-11ee-b434-1d71764ec79c@max-a5'},4},
               {{12,'9613b588-74db-11ee-a4af-1d71807ddbb4@max-a5'},4},
               {{12,'a55499b2-74d7-11ee-a9df-1d71807ddbb4@max-a5'},2},
               {{12,'b794aeda-74de-11ee-b5e2-1d71764ec79c@max-a5'},2},
               {{12,'f041ca12-74dc-11ee-88b3-1d71807ddbb4@max-a5'},2},
               {{12,'f575b378-73fa-11ee-875f-1d718ad5fce5@max-a5'},4},
               {{12,'fb2a5b2c-74de-11ee-9c8a-1d71764ec79c@max-a5'},2}]},
             {object,
                 {[{{12,'09a4965a-74d9-11ee-8d17-1d71807ddbb4@max-a5'},2,[]},
                   {{12,'33dd009a-74db-11ee-b011-1d71764ec79c@max-a5'},2,[]},
                   {{12,'5fd8bdc8-74dc-11ee-b48c-1d71764ec79c@max-a5'},2,[]},
                   {{12,'7b2896d2-7411-11ee-b434-1d71764ec79c@max-a5'},4,[]},
                   {{12,'9613b588-74db-11ee-a4af-1d71807ddbb4@max-a5'},4,[]},
                   {{12,'a55499b2-74d7-11ee-a9df-1d71807ddbb4@max-a5'},2,[]},
                   {{12,'b794aeda-74de-11ee-b5e2-1d71764ec79c@max-a5'},2,[]},
                   {{12,'f041ca12-74dc-11ee-88b3-1d71807ddbb4@max-a5'},2,[]},
                   {{12,'f575b378-73fa-11ee-875f-1d718ad5fce5@max-a5'},4,[]},
                   {{12,'fb2a5b2c-74de-11ee-9c8a-1d71764ec79c@max-a5'},
                    2,
                    [{'$deleted',{1698,421281,501793}}]}],
                  []}}],
            []},
        {partisan_plumtree_broadcast,handle_cast,2,
            [{file,
                 "/home/max/work/ska/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl"},
             {line,549}]},
        {gen_server,try_handle_cast,3,[{file,"gen_server.erl"},{line,1103}]},
        {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1165}]},
        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]}

=CRASH REPORT==== 27-Oct-2023::17:41:21.505225 ===
  crasher:
    initial call: partisan_plumtree_broadcast:init/1
    pid: <0.1101.0>
    registered_name: partisan_plumtree_broadcast
    exception error: undefined function plum_db:merge/2
      in function  partisan_plumtree_broadcast:handle_cast/2 (/home/max/work/ska/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl, line 549)
      in call from gen_server:try_handle_cast/3 (gen_server.erl, line 1103)
      in call from gen_server:handle_msg/6 (gen_server.erl, line 1165)
    ancestors: [partisan_sup,<0.1007.0>]
    message_queue_len: 0
    messages: []
    links: [<0.1008.0>,<0.1019.0>]
    dictionary: [{rand_seed,{#{type => exsss,next => #Fun<rand.0.65977474>,
                                bits => 58,uniform => #Fun<rand.1.65977474>,
                                uniform_n => #Fun<rand.2.65977474>,
                                jump => #Fun<rand.3.65977474>},
                              [281454710417369215|78408885658432356]}}]
    trap_exit: false
    status: running
    heap_size: 28690
    stack_size: 28
    reductions: 38827
  neighbours:

=SUPERVISOR REPORT==== 27-Oct-2023::17:41:21.506012 ===
    supervisor: {local,partisan_sup}
    errorContext: child_terminated
    reason: {undef,
                [{plum_db,merge,
                     [{{{erleans_pm,registry},
                        {'pebble@max-a5',
                            {<<"01HDNZBQH8E17KWKSXGFVBCHXG">>,pebble}}},
                       [{{12,'09a4965a-74d9-11ee-8d17-1d71807ddbb4@max-a5'},2},
                        {{12,'33dd009a-74db-11ee-b011-1d71764ec79c@max-a5'},2},
                        {{12,'5fd8bdc8-74dc-11ee-b48c-1d71764ec79c@max-a5'},2},
                        {{12,'7b2896d2-7411-11ee-b434-1d71764ec79c@max-a5'},4},
                        {{12,'9613b588-74db-11ee-a4af-1d71807ddbb4@max-a5'},4},
                        {{12,'a55499b2-74d7-11ee-a9df-1d71807ddbb4@max-a5'},2},
                        {{12,'b794aeda-74de-11ee-b5e2-1d71764ec79c@max-a5'},2},
                        {{12,'f041ca12-74dc-11ee-88b3-1d71807ddbb4@max-a5'},2},
                        {{12,'f575b378-73fa-11ee-875f-1d718ad5fce5@max-a5'},4},
                        {{12,'fb2a5b2c-74de-11ee-9c8a-1d71764ec79c@max-a5'},
                         2}]},
                      {object,
                          {[{{12,
                              '09a4965a-74d9-11ee-8d17-1d71807ddbb4@max-a5'},
                             2,[]},
                            {{12,
                              '33dd009a-74db-11ee-b011-1d71764ec79c@max-a5'},
                             2,[]},
                            {{12,
                              '5fd8bdc8-74dc-11ee-b48c-1d71764ec79c@max-a5'},
                             2,[]},
                            {{12,
                              '7b2896d2-7411-11ee-b434-1d71764ec79c@max-a5'},
                             4,[]},
                            {{12,
                              '9613b588-74db-11ee-a4af-1d71807ddbb4@max-a5'},
                             4,[]},
                            {{12,
                              'a55499b2-74d7-11ee-a9df-1d71807ddbb4@max-a5'},
                             2,[]},
                            {{12,
                              'b794aeda-74de-11ee-b5e2-1d71764ec79c@max-a5'},
                             2,[]},
                            {{12,
                              'f041ca12-74dc-11ee-88b3-1d71807ddbb4@max-a5'},
                             2,[]},
                            {{12,
                              'f575b378-73fa-11ee-875f-1d718ad5fce5@max-a5'},
                             4,[]},
                            {{12,
                              'fb2a5b2c-74de-11ee-9c8a-1d71764ec79c@max-a5'},
                             2,
                             [{'$deleted',{1698,421281,501793}}]}],
                           []}}],
                     []},
                 {partisan_plumtree_broadcast,handle_cast,2,
                     [{file,
                          "/home/max/work/ska/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl"},
                      {line,549}]},
                 {gen_server,try_handle_cast,3,
                     [{file,"gen_server.erl"},{line,1103}]},
                 {gen_server,handle_msg,6,
                     [{file,"gen_server.erl"},{line,1165}]},
                 {proc_lib,init_p_do_apply,3,
                     [{file,"proc_lib.erl"},{line,241}]}]}
    offender: [{pid,<0.1101.0>},
               {id,partisan_plumtree_broadcast},
               {mfargs,{partisan_plumtree_broadcast,start_link,[]}},
               {restart_type,permanent},
               {significant,false},
               {shutdown,5000},
               {child_type,worker}]

The problem is that my 3 nodes are just

['clockwork@max-a5','pebble@max-a5','ska@max-a5']

while all the others like 09a4965a-74d9-11ee-8d17-1d71807ddbb4@max-a5 shouldn't be there as they seem to be remnants from my previous tests.
Note that ska and clockwork are using partisan{tag, "v5.0.0-rc.11"} and pebble is using erleans {tag, "leapsight-1.5.1"} which also uses partisan{tag, "v5.0.0-rc.11"}.
I was wondering if they are there because they were persisted in the past, or if they are generated in a way that is not apparent to me. In the first case, I would like to try and erase persisted data and start from scratch, since at the moment the system/cluster is unstable.

[mcesaro]

Update:
I zapped the data directory use by plum_db and ran the test again.
I noticed that there are no problems using partisan on the three clusters node. Two nodes, clockwork and ska, are using partisan_rpc and this works as expected.
If pebble node only calls partisan_rpc like the other two nidem then all works fine; as soon as pebbledoes the first request to erleans with the erleans_grain:call/2 the other 2 nodes display the error:

=ERROR REPORT==== 27-Oct-2023::20:20:23.880437 ===
** Generic server partisan_plumtree_broadcast terminating 
** Last message in was {'$gen_cast',
                           {broadcast,
                               {{{erleans_pm,registry},
                                 {'pebble@max-a5',
                                     {<<"01HDNZBQH8E17KWKSXGFVBCHXG">>,
                                      pebble}}},
                                [{{12,
                                   '3ac5b58e-7506-11ee-a9a2-1d71764ec79c@max-a5'},
                                  1}]},
                               {object,
                                   {[{{12,
                                       '3ac5b58e-7506-11ee-a9a2-1d71764ec79c@max-a5'},
                                      1,
                                      [{['pebble@max-a5'|<<"#Pid<0.1104.0>">>],
                                        {1698,438023,871848}}]}],
                                    []}},
                               plum_db,0,'pebble@max-a5','pebble@max-a5'}}
** When Server state == {state,['pebble@max-a5','ska@max-a5'],
                               [],#{},#{},
                               [partisan_plumtree_backend],
                               [],
                               ['clockwork@max-a5','pebble@max-a5',
                                'ska@max-a5'],
                               1000,10000}
** Reason for termination ==
** {'module could not be loaded',
       [{plum_db,merge,
            [{{{erleans_pm,registry},
               {'pebble@max-a5',{<<"01HDNZBQH8E17KWKSXGFVBCHXG">>,pebble}}},
              [{{12,'3ac5b58e-7506-11ee-a9a2-1d71764ec79c@max-a5'},1}]},
             {object,
                 {[{{12,'3ac5b58e-7506-11ee-a9a2-1d71764ec79c@max-a5'},
                    1,
                    [{['pebble@max-a5'|<<"#Pid<0.1104.0>">>],
                      {1698,438023,871848}}]}],
                  []}}],
            []},
        {partisan_plumtree_broadcast,handle_cast,2,
            [{file,
                 "/home/ubuntu/clockwork/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl"},
             {line,549}]},
        {gen_server,try_handle_cast,3,[{file,"gen_server.erl"},{line,1103}]},
        {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1165}]},
        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]}

=CRASH REPORT==== 27-Oct-2023::20:20:23.880767 ===
  crasher:
    initial call: partisan_plumtree_broadcast:init/1
    pid: <0.854.0>
    registered_name: partisan_plumtree_broadcast
    exception error: undefined function plum_db:merge/2
      in function  partisan_plumtree_broadcast:handle_cast/2 (/home/ubuntu/clockwork/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl, line 549)
      in call from gen_server:try_handle_cast/3 (gen_server.erl, line 1103)
      in call from gen_server:handle_msg/6 (gen_server.erl, line 1165)
    ancestors: [partisan_sup,<0.839.0>]
    message_queue_len: 0
    messages: []
    links: [<0.840.0>,<0.851.0>]
    dictionary: [{rand_seed,{#{type => exsss,next => #Fun<rand.0.65977474>,
                                bits => 58,uniform => #Fun<rand.1.65977474>,
                                uniform_n => #Fun<rand.2.65977474>,
                                jump => #Fun<rand.3.65977474>},
                              [209518163594830093|7720486536226569]}}]
    trap_exit: false
    status: running
    heap_size: 610
    stack_size: 28
    reductions: 16735
  neighbours:

=SUPERVISOR REPORT==== 27-Oct-2023::20:20:23.885797 ===
    supervisor: {local,partisan_sup}
    errorContext: child_terminated
    reason: {undef,
                [{plum_db,merge,
                     [{{{erleans_pm,registry},
                        {'pebble@max-a5',
                            {<<"01HDNZBQH8E17KWKSXGFVBCHXG">>,pebble}}},
                       [{{12,'3ac5b58e-7506-11ee-a9a2-1d71764ec79c@max-a5'},
                         1}]},
                      {object,
                          {[{{12,
                              '3ac5b58e-7506-11ee-a9a2-1d71764ec79c@max-a5'},
                             1,
                             [{['pebble@max-a5'|<<"#Pid<0.1104.0>">>],
                               {1698,438023,871848}}]}],
                           []}}],
                     []},
                 {partisan_plumtree_broadcast,handle_cast,2,
                     [{file,
                          "/home/ubuntu/clockwork/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl"},
                      {line,549}]},
                 {gen_server,try_handle_cast,3,
                     [{file,"gen_server.erl"},{line,1103}]},
                 {gen_server,handle_msg,6,
                     [{file,"gen_server.erl"},{line,1165}]},
                 {proc_lib,init_p_do_apply,3,
                     [{file,"proc_lib.erl"},{line,241}]}]}
    offender: [{pid,<0.854.0>},
               {id,partisan_plumtree_broadcast},
               {mfargs,{partisan_plumtree_broadcast,start_link,[]}},
               {restart_type,permanent},
               {significant,false},
               {shutdown,5000},
               {child_type,worker}]

It is not clear where '3ac5b58e-7506-11ee-a9a2-1d71764ec79c@max-a5 comes from, as the three nodes were communicating fine before the first erleans invocation, although my impression is that it refers to a grain managed by erleans.
Thus it seems that the problem is related to erleans rather than partisan, because the plum_db merge attempt on {erleans_pm, registry} happens in a node which is clustered with partisan but it is not using erleans (I mean sharing grains) at all.
Since the issue is propagated as a partisan broadcast on all the cluster notes, this might explain the issue, i.e. the erleans process managements expects plum_db on all the node, but nodes using partisan without plum_db will report the crash.