Decoding/Encoding issues

[pdawczak]

Hi!

First of all - thank you for building those two libraries!

That's absolutely amazing to see some activity in this are, and it makes me super excited to be able to build a node that can easily communicate with other BEAM based technologies!

When playing with the code, I've discovered a couple of problems.

Let me start with my Environment/setup - initially, I starting with Elixir, but then, I've double confirmed with "pure" Erlang.

Environment:

elixir --version
Erlang/OTP 21 [erts-10.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Elixir 1.7.3 (compiled with Erlang/OTP 19)

Setup

Given I have a node implemented with Pyrlang:

from pyrlang import Node, GeventEngine, GenServer
from term import Atom
import pickle


class PyGenServer(GenServer):
    def __init__(self, node):
        GenServer.__init__(self,
                           node_name=node.node_name_,
                           accepted_calls=["handle_call"])

        node.register_name(self, Atom("py_process"))

    def handle_call(self, args):
        print(type(args))
        print(args)
        return "foo"


def main():
    eng = GeventEngine()
    node = Node(node_name="py@127.0.0.1", cookie="COOKIE", engine=eng)

    PyGenServer(node)

    eng.run_forever()


if __name__ == "__main__":
    main()

and start it:

$ PYTHONPATH=./Pyrlang PYRLANG_ENABLE_LOG_FORMAT=1 PYRLANG_LOG_LEVEL=DEBUG python3 py_app/first.py

I'll have an iex shell running, using same cookie, and started like:

iex --name ex@127.0.0.1 --cookie COOKIE

I'll have receiver defined in my Elixir side like:

receiver = {:py_process, :"py@127.0.0.1"}

Problematic entries

1 - Returned value

iex(ex@127.0.0.1)15> r = GenServer.call(receiver, {:handle_call, 5})
'foo'
iex(ex@127.0.0.1)17> i r
Term
  'foo'
Data type
  List
Description
  This is a list of integers that is printed as a sequence of characters
  delimited by single quotes because all the integers in it represent valid
  ASCII characters. Conventionally, such lists of integers are referred to
  as "charlists" (more precisely, a charlist is a list of Unicode codepoints,
  and ASCII is a subset of Unicode).
Raw representation
  [102, 111, 111]
Reference modules
  List
Implemented protocols
  IEx.Info, Inspect, String.Chars, List.Chars, Collectable, Enumerable

Even though, in Python code I'm returning string "foo", this is visible to Elixir as list of code points, while string/binary was expected.

iex(ex@127.0.0.1)24> is_binary(r)
false

2 - Recognising sent data

iex(ex@127.0.0.1)18> GenServer.call(receiver, {:handle_call, [1, 2, 3]})

I was struggling to implement handler for the data, then discovered, what Python sees is not exactly what I was expecting it to see, from the logs:

2018-11-07 08:29:30,552 [pyrlang] node:247: Send local reg=py_process receiver=<__main__.PyGenServer object at 0x10d343128> msg=(atom'$gen_call', (<3.111.0 @ ex@127.0.0.1>,Ref<3,202040,3575644163,1815979671>@ex@127.0.0.1), (atom'handle_call', '\x01\x02\x03'))
2018-11-07 08:29:30,553 [pyrlang.OTP] gen_server:84: In call GenIncomingMessage((atom'handle_call', '\x01\x02\x03'))
2018-11-07 08:29:30,553 [pyrlang.OTP] gen_server:106: method=<bound method PyGenServer.handle_call of <__main__.PyGenServer object at 0x10d343128>>
<class 'str'> // Please note, it recognised a list of integers sent from Elixir as a stringm and...
// ... Python was unable to print the value of the argument, and this line is empty
2018-11-07 08:29:30,553 [pyrlang.OTP] gen_server:108: Replying with result=foo

as contrary, when I send a tuple with the same data, like:

iex(ex@127.0.0.1)18> GenServer.call(receiver, {:handle_call, {1, 2, 3}})

on Python side, it looks correct:

2018-11-07 08:29:30,575 [pyrlang] node:494: demonitor orig=<3.111.0 @ ex@127.0.0.1> target=<1.0.2 @ py@127.0.0.1> ref=Ref<3,202040,3575644163,1815979671>@ex@127.0.0.1
2018-11-07 08:32:44,895 [pyrlang] node:247: Send local reg=py_process receiver <__main__.PyGenServer object at 0x10d343128> msg=(atom'$gen_call', (<3.111.0 @ ex@127.0.0.1>,Ref<3,202050,3575644163,1815979671>@ex@127.0.0.1), (atom'handle_call', (1, 2, 3)))
2018-11-07 08:32:44,895 [pyrlang.OTP] gen_server:84: In call GenIncomingMessage((atom'handle_call', (1, 2, 3)))
2018-11-07 08:32:44,895 [pyrlang.OTP] gen_server:106: method=<bound method PyGenServer.classify of <__main__.PyGenServer object at 0x10d343128>>
<class 'tuple'> // this time it recognises properly, and...
(1, 2, 3) // ... it printed the arg correctly
2018-11-07 08:32:44,895 [pyrlang.OTP] gen_server:108: Replying with result=foo

3 - Decoding data, that crashes

In Elixir:

iex(ex@127.0.0.1)22> GenServer.call(receiver, {:handle_call, [3.5]})
** (exit) exited in: GenServer.call({:py_process, :"py@127.0.0.1"}, {:handle_call, [3.5]}, 5000)
    ** (EXIT) no connection to py@127.0.0.1
    (elixir) lib/gen_server.ex:924: GenServer.call/3

and in Python logs:

2018-11-07 09:19:58,349 [pyrlang] node:162: Node ex@127.0.0.1 connected
2018-11-07 09:19:58,382 [pyrlang] gevent_engine:176: Exception: Traceback (most recent call last):
  File "/Users/.../pyex_project/Pyrlang/pyrlang/async/gevent_engine.py", line 173, in _serve_loop
    _read_loop(proto=proto, sock=sock)
  File "/Users/.../pyex_project/Pyrlang/pyrlang/async/gevent_engine.py", line 206, in _read_loop
    collected1 = proto.on_incoming_data(collected)
  File "/Users/.../pyex_project/Pyrlang/pyrlang/dist/base_dist_protocol.py", line 154, in on_incoming_data
    if self.on_packet(packet):
  File "/Users/.../pyex_project/Pyrlang/pyrlang/dist/in_dist_protocol.py", line 50, in on_packet
    return self.on_packet_connected(data)
  File "/Users/.../pyex_project/Pyrlang/pyrlang/dist/base_dist_protocol.py", line 366, in on_packet_connected
    (msg_term, tail) = codec.binary_to_term(tail)
  File "/Users/.../pyex_project/pyex_env/lib/python3.6/site-packages/term/py_codec_impl.py", line 113, in binary_to_term
    return binary_to_term_2(data[1:], options)
  File "/Users/.../pyex_project/pyex_env/lib/python3.6/site-packages/term/py_codec_impl.py", line 262, in binary_to_term_2
    term1, tail = binary_to_term_2(tail)
  File "/Users/.../pyex_project/pyex_env/lib/python3.6/site-packages/term/py_codec_impl.py", line 262, in binary_to_term_2
    term1, tail = binary_to_term_2(tail)
  File "/Users/.../pyex_project/pyex_env/lib/python3.6/site-packages/term/py_codec_impl.py", line 245, in binary_to_term_2
    term1, tail = binary_to_term_2(tail)
  File "/Users/.../pyex_project/pyex_env/lib/python3.6/site-packages/term/py_codec_impl.py", line 405, in binary_to_term_2
    raise PyCodecError("Unknown tag %d" % data[0])
term.py_codec_impl.PyCodecError: Unknown tag 99

I know this is quite lenghty, but I was hoping to provide as much information as I could!

Let me know if there is anything else I could help with this issue!

Best regards,
Paweł

[s2hc-johan]

Hi,

Sorry for the late response, I haven't managed to set notifications on this repo.

I've glanced through your issue, update me if I missed something.

I think that it's that a list of integers with values under 255 will be represented as a string. I'm not sure, but in erlang strings ar just list of integers:

2> [99,100].
"cd"
3>

could you try with a list of integers with higher values then 255 to see if I'm correct.

[pdawczak]

I see - thank you for responding @s2hc-johan. I'll give it a try within the next two days and get back to you 👍