Skip to content

Python 3: Kiko writes empty files #10

@ben-hawkyard-absolute

Description

@ben-hawkyard-absolute

Hi there, I've been having trouble saving .kiko files correctly with python 3. As far as I can tell, when a kiko file is saved the data files that get added to the underlying archive are truncated at zero bytes. You can reproduce this with the following code snippet:

import os
import tempfile
from kiko.io.kikofile import KikoFile


handle, kiko_path = tempfile.mkstemp(suffix=".kiko")
os.close(handle)
kiko_file = KikoFile(kiko_path)
kiko_file.set_data({"test": "data"})
kiko_file.save()
kiko_file.parse()

When I do this I get an error that looks something like:

Traceback (most recent call last):
  File "/home/ben.hawkyard/Documents/scripts/testing/kiko_metadata_testing.py", line 11, in <module>
    kiko_file.parse()
  File "/opt/rez/rez_package_cache/kiko/1.0.0/a99e/a/python/kiko/io/kikofile.py", line 145, in parse
    self._metadata = json.load(v)
                     ^^^^^^^^^^^^
  File "/opt/rez/rez_package_cache/python/3.11.8/3e7f/a/lib/python3.11/json/__init__.py", line 293, in load
    return loads(fp.read(),
           ^^^^^^^^^^^^^^^^
  File "/opt/rez/rez_package_cache/python/3.11.8/3e7f/a/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rez/rez_package_cache/python/3.11.8/3e7f/a/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rez/rez_package_cache/python/3.11.8/3e7f/a/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I think this is caused by this bit of code in io/kikofile.py:

    @staticmethod
    def _add_to_tar(tar_file, name, f_obj):
        info = tarfile.TarInfo(name=name)
        if sys.version_info.major == 2:
            info.size = len(f_obj.buf)
            info.time = time.time()
        else:
            info.size = f_obj.tell()
            info.mtime = time.time()
        tar_file.addfile(tarinfo=info, fileobj=f_obj)

    @classmethod
    def _add_to_tar_from_dict(cls, tar_file, name, data):
        io = StringIO()
        json.dump(data, io)
        io.seek(0)
        cls._add_to_tar(tar_file, name, io)

calling io.seek(0) in _add_to_tar_from_dict means that f_obj.tell() in _add_to_tar returns zero and this causes us to write empty files to our kiko file when adding dicts to it.

I've done some testing and I think this could be fixed by changing the else branch in _add_to_tar:

        else:
            f_obj = BytesIO(f_obj.read().encode("utf-8"))  # TarFile expects file objects to be opened in binary mode.
            info.size = f_obj.seek(0, os.SEEK_END)
            f_obj.seek(0)
            info.mtime = time.time()

I'm happy to put in a pull request for this if there's interest. I can also see that mottosse fixes this in their pull request for kiko here: #6. Is this likely to be accepted, do you think?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions