Request: normalization and `depythonize`

[jamesbraza]

When upgrading pythonize from version 0.21.0 to version 0.23.0 in tantivy-py's quickwit-oss/tantivy-py#401, we switched from extract to extract_bound (per PyO3/pyo3#3916).

Now, we are hitting an issue where I64 values are becoming U64 after depythonize.

I came up with a solution in jamesbraza/tantivy-py#1 that involves re-casting U64 back to I64 if it fits, but this is pretty hacky.

Is there some way to preserve integer types across serialization/deserialization?

• I64 values remaining I64
• U64 values remaining U64

I am wondering if there is some way pythonize can accommodate this, maybe by adding an opt-in flag somewhere?

Also, I am not 100% sure if this is a pythonize vs pyo3 issue, but it seems to be a deserialization issue, so I opened this here.

[davidhewitt]

Thanks for the report & sorry for the slow reply.

I suspect that #69 was the cause here; I see now that it was a behavioural change where positive values previously in i64 range will now be deserialized as u64 values.

I think it will be hard to guarantee round-tripping of i64 and u64 values, because as soon as the data goes from Rust to Python, for positive integers we don't know whether they came from a signed or unsigned value.

What we might be able to do is add a config option as to whether you prefer signed or unsigned types, and make depythonize select according to your preference. Does that potentially help?

[jamesbraza]

Hi @davidhewitt thank you for the detailed response, #69 was indeed the pythonize change, sharing that really helped us.

What we might be able to do is add a config option as to whether you prefer signed or unsigned types, and make depythonize select according to your preference.

What the tantivy-py team is doing is recursively forcing I64/F64 after deserialization, effectively eliminating the U64 vs I64 ambiguity.

I think an option to prefer or force signed/unsigned is useful, it would enable tantivy-py to remove the new forcing logic. Up to you though, also feel free to close this out as resolved.

Thanks again for the help here btw, much appreciated.