Enhancement: add `jprint(1)` tool to output valid JSON in canonical format

[lcn2]

Is there an existing issue for this?

• I have searched for existing issues and did not find anything like this

Describe the feature

We recommend adding a jprint(1) tool that will output JSON that has been parsed and found to be valid, in canonical format.

For the canonical output format model, we recommend that bin/jprint-wrapper.sh prints.

Relevant images, screenshots or other files

We recommend the following command line for jprint(1):

usage: jprint [-h] [-v level] [-V] [--use-jparse] [-S] [-i ext] file.json

    -h		    print help message and exit
    -v level	    set verbosity level (def level: 0)
    -V		    print version string and exit

    --use-jparse    Do nothing: this option is a simple way to verify this tool came from the jparse toolset.

    -S              Reserved for a future print JSON semantic strings facility (def: print values)

    -i  ext         Edit files in-place, saving backups with the specified ext extension, and do not write to stdout.
                    If a zero-length ext is given, no backup will be saved.  (def: do not edit files)
                    If file.json is - (dash) or . (dot) then this option is ignored.

    -c              Compare new contents with the old contents and only update is different
                    The use of -c requires the use of -i ext
                    If file.json is - (dash) or . (dot) then this option is ignored.

    file.json       JSON input file
                    If file.json is - (dash) read from standard input.
                    If file.json is . (dot) do not read any file, perform an internal test and exit accordingly.

Exit codes:
     0         all OK
     1         file is not valid JSON 
     2         cannot read file.json, or file.json does not exist, or file.json is not a file
     3         internal test failed
     4         invoked a reserved facility that is not yet implemented=
 >= 10         internal error 

Relevant links

The bin/jprint-wrapper.sh tool is the model for how to print JSON canonically.

Anything else?

We would not expect another jparse command to be able to undestant --use-jparse as a command line option. Again, the --use-jparse command line option does nothing (other than to be parsed as a command line option).

When file.json is - (dash), read from standard input as in:

    /some/command | jprint -

When file.json is . (dot), perform an internal test by parsing an internal (static array in memory) JSON. If the test of the internal (static array in memory) JSON fails, when jprint(1) must exit 3.

HINT: While the internal (static array in memory) JSON should NOT be huge, the contents should be a mix of various JSON elements. Something fun along the lines of jparse/test_jparse/test_JSON/good/party.json might serve well.

The purposes of jprint --use-jparse is to "verify" that the jprint(1) tool came from this jparse repo toolset. There are likely to be more than 1 jprint command out there. To "verify" (somewhat) that the jprint tool found on $PATH is the proper jparse tool version we use:

    jprint --use-jparse .

If there is someone else's jprint found (such as along the $PATH), then one may expect that jprint --use-jparse . will fail, either because -use-jparse is NOT a valid argument for that command, or because . is not a file.

The -i ext allow one to update a file with the canonically formatted JSON and save the old contents of the file. For example:

    jprint -i .old file.json

The previous contents is saved as file.json.old and file.json is replaced with the canonically formatted JSON contents. The actions to move files must be performed using the rename(2) system call. The effect of the above command is as if this was performed:

    jprint file.json > .tmp.file.json
    ln -f file.json file.json.old
    mv -f .tmp.file.json file.json

NOTE: In these examples, files of the form .tmp.file.json represent a temporary file. One should use mkstemp(3) call to crate a temporary file in the same directory as file.json so that the rename(2) system call will be atomic.

If the ext is an empty string, no old file contents is saved:

    jprint -i "" file.json

The file.json contents is replaced with the canonical JSON contents canonically formatted JSON contents. The actions to move files must be performed using the rename(2) system call. The effect of the above command is as if this was performed:

    jprint file.json > .tmp.file.json
    mv -f .tmp.file.json file.json

When the file.json is - (dash) or . (dot), the -i ext is ignored.

The -c option must be used with -i ext. The file.json file is ONLY touched if the contents of the file is updated. If the contents of the file.json file is not changed, no old file is saved.

This command:

    jprint -i .old -c file.json

has the effect of:

    jprint file.json > .tmp.file.json
    if cmp -s file.json .tmp.file.json; then
        rm -f .tmp.file.json
    else
        ln -f file.json file.json.old
        mv -f .tmp.file.json file.json
    fi

This command:

    jprint -i "" -c file.json

has the effect of:

    jprint file.json > .tmp.file.json
    if cmp -s file.json .tmp.file.json; then
        rm -f .tmp.file.json
    else
        mv -f .tmp.file.json file.json
    fi

The contents of file.json should be read into memory. If the file.json is not a file, or if file.json does not exist, or if there is a read error while reading the file into memory, or if the command runs out of memory, then the jprint(1) command must exit 2.

Once the contents of the file.json is in memory, then jparse_json(3) (or related function) should be called to parse the contents. If the contents is NOT valid JSON (or the JSON parser function returns an error), then the jprint(1) command must exit 1.

UPDATE 0

We will save this issue now, and return in the morning to review and if need edit/revise it. This is just a draft.

[xexyl]

Thanks for opening this. Please let me know when you've updated it so I can read it proper.

[lcn2]

Thanks for opening this. Please let me know when you've updated it so I can read it proper.

Going to sleep now. Will consider this perhaps tomorrow or so as this is not a high priority.

UPDATE 0

We plan to describe -S as well during this draft enhancement period that might last a few days.

[xexyl]

Thanks for opening this. Please let me know when you've updated it so I can read it proper.

Going to sleep now. Will consider this perhaps tomorrow or so as this is not a high priority.

Quite all right .. sleep tight!

[lcn2]

An UPDATE

We are focusing on compelting "bin tools" in the other repo, as well as how to open up a new issue in this repo related to a jval(1) tool as discussed in GH-. As such it will be even longer before we address the -S concept. We realize this impacts the chkentry(1) related issue, however given priorities we need to delay addressing this further.

The BELOW may be edited without warning

We will print this now, and edit things as we think / re-read the text. This is a "work in progress" as the expression goes.

Please hold off on asking too many questions about this next section.

Nevertheless to give you a preliminary concept on what are "JSON semantic strings". We do this in the hope that it will NOT DISTRACT YOU FROM HIGHER PRIORITIES and instead if you an idea about the general concepts that are being architected.

Consider this valid JSON:

{
    "curds" : 23209,
    "foo" : [
        "bar",
        true,
        42,
        null
    ]
}

With jprint -S, each JSON value (JSON strong, JSON number, JSON boolean, JSON null) would be printed as a JSON string:

{
    "description of JSON name member, curds" : "description of JSON integer number as a JSON member value of curds",
    "description of JSON name member, foo" : [
        "description of JSON string as part of an array that is a JSON member value of foo",
        "description of JSON boolean as part of an array that is a JSON member value of foo",
        "description of JSON integer number as part of an array that is a JSON member value of foo",
        "description of JSON null as part of an array that is a JSON member value of foo"
    ]
}

The "description of ..." would be a specific, program parsable friendly string that describe details about the semantics of the JSON "thing" is represented, including, but not limited to:

• item_type
• sub-types (such as jSON number information, JSON string encoding, etc.)
• min repeat count
• max repeat count
• where applicable a "chk_name" that can refer to a specific check function name to call
• etc.

NOTE: The above bullet list is TBD and may be slightly modified as the architecture of the JSON semantics continues to be developed.

The JSON produced by jprint -S could be parched / edited to account for variations. Consider an JSON document that recorded test scores. One could start with a simple minimum reference example:

[
    100
]

With jprint -S:

[
    "description of JSON integer number"
]

The "description of JSON integer number" would initial indicate a "min repeat count" of 1 and a "max repeat count" of 1. However if the indent is to have recorded test scores allow for a minimum of 1 and no limit to the number, then the "description of JSON integer number" could easily be patched (modified) to indicate 1+ repeats.

The idea is that a tool such as chkentry(1), when given a JSON document at a JSON semantic document containing JSON semantic strings, would "walk" both trees in parallel, checking data in the JSON parse tree and potentially calling check functions as needed.

For JSON such as:

    "email" : "mame@ruby-lang.org",

With jprint -S:

    "description of JSON name member, email" : "description of JSON integer number as a JSON member value of email",

The "description of JSON integer number as a JSON member value of email" patched (modified) to indicate that the value may be a JSON string or a JSON null. The "where applicable a "chk_name" that can refer to a specific check function name to call" part of the ascription would indicate a "chk_email" function name that when called, would perform further "does it look like a valid Email address or is it JSON null?" check.

In conclusion

The general concept of having 2 parallel JSON trees, one whose JSON semantics is being checked, and the other that describes the scope of the JSON semantics is being architected to be fairly easy for you to code the general JSON semantics checking function AND very flexible to accommodate a wide range of semantics requirements AND fairly easy for some programmer who wants to "validate some JSON" beyond a simple "does it match the JSON grammar" check.

The functionally of JSON semantic checking will be an extremely important contribution to the overall jparse repo and will be something very unique in the world of JSON tools.

The chkentry(1) program tried to do some of this, but it was coded too early and the whole soup/chk.* mess was an hack to get something working instead of focusing on the more general JSON semantics case.

We intend to finalize the format of the "JSON semantic strings", perhaps post-Great Fork Merge. We intend to produce by hand, what jprint -S should produce for a reference .info.json, a reference .auth.json, and a reference .entry.json document.

REQUEST

Try to not let this above "distract" to, nor "alarm" you. The architecture is a work in progress, one that is fairly far along but needs some polishing. It is also something we believe you are very capable of coding. And even more importably for its contribution to the overall value of the jparse repo](https://github.com/xexyl/jparse), something you should be able to document so that others may be able to use the JSON semantics facility w/o having to deal with much of the complexity of JSON parse tree data structures.

[lcn2]

JPRINT SUGGESTON

The jprint -S as described in GH-issuecomment-2323573780 is a LOWER PRIORTY than the general jprint(1) tool.

We recommend you focus on coding the general jprint(1) tool such that it someone invokes jprint -S ... the tool simply prints to stderr, via the err(3) facility an error message that includes something like:

The -S facility has not yet been implemented

and then does an exit(4) via the err(3) facility.

See the command line usage message in the top level TODO.

Focus on the general jprint(1) tool, leave a command like parsing hook for -S and focus on the general tool first.

... post Great Fork Merge of course.

[lcn2]

Priority of jval vs JSON semantic checking

We believe that implementing jval(1) may be a higher priority than implementing the library functions needed to implement JSON semantic checking. This is because jval(1), using a facility similar to JSONPath.sh, would be useful to modify JSON documents (see comment regarding "could be parched / edited" in GH-issuecomment-2323573780).

So the overall priority might be:

1. Work to start the Great Fork Merge
2. code jprint(1) with -S returning an error message and exit 4
3. open the jval(1) issue
4. document the JSON semantic string format
5. code jprint -S
6. code jval(1)
7. code the JSON semantic library facility

The above is just a guess at the overall priorities. Somewhere within that list is to hold the IOCCC28 as well. :-)

[lcn2]

A mis-feature or BUG in JSONPath.sh

In the other repo, the bin/jfmt-wrapper.sh tool uses JSONPath.sh (see our JSONPath.sh repo) to format JSON in a canonical format.

Consider, however, this valid JSON:

{"foo":[100]}

When processed the bin/jfmt-wrapper.sh tool in the other repo, the tool (via JSONPath.sh) produces:

{
    "foo" :     [
        100
    ]
}

Notice the excessive whitespace between the ":" and the "[". That should NOT be considered canonical.

QUESTION

How should this JSON be formatted canonically?

{"foo":[100]}

UPDATE 0

We ask this because the answer will help is "bug fix / improve" our [JSONPath.sh repo].

[xexyl]

A mis-feature or BUG in JSONPath.sh

Are you sure the problem is not JSON itself ? 😀

[xexyl]

QUESTION

What is the order of priority of all the tools (including chkentry mods) versus the other website, please?

[xexyl]

SUGGESTION

For the tools being discussed here why don't you update the JSON_util_README.md (if that is what it was called) to help organise the thoughts? We can then discuss it in more detail here. We might include information here too of course but it might be nice to have a document? Just a thought: maybe not worth it, maybe worth it.