Skip to content

Large files break serial-json output #790

@kwalcock

Description

@kwalcock

They seem to fail in SerialJsonOutput.scala at

MentionsOps(mentions).json(pretty = true)

in which the entire output is converted into a single string. That string may be over 2GB in length. The exceptions thrown start complaining about negative numbers which are probably integers overflowing.

If that doesn't happen, then it is during the next f.writeString where it can't encode the large string to get it into the file. The error is "java.lang.OutOfMemoryError: Requested array size exceeds VM limit".

There may be a way to send formatted output directly and piecewise to a file without the intermediate string. That should fix the problem. The input file is only 560KB and there are some as large as 3.5 MB that need to be processed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions