JSON streaming array updates

[mikee47]

Current streaming update (writing) behaviour is to overwrite only those values received.
This allows selective updating of properties. For example:

{
    "security": {
        "api_secured": "false"
    }
}

This updates one value in the database, leaving everything else unchanged.

Arrays are overwritten entirely:

{
    "general": {
        "supported_color_models": [
            "RGB",
            "RAW"
        ]
    }
}

replaces everything in general.supported_color_models, and this::

{
    "general": {
        "channels": [
            {
                "pin": 1,
                "name": "dummy"
            }
        ]
    }
}

Deletes all existing entries in general.channels and replaces it with the one object provided.

As mentioned in the discussion #17, this is really the simplest way to handle updates, but does not allow updating of individual array items. This is desirable so that update messages do not have to contain data which doesn't change.

Possible modes of update operation may include:

• Overwrite, as above - this includes clearing/deleting the entire array by setting it to []
• Wipe everything and replace with provided JSON data
• Overwrite an array item at a given position
• Insert an array item at a given position
• Delete an array item
• Overwrite a property or properties of an object in an object array

This is basically XPath, JSONPath, whatever, so a selector expression is used to identify the item to update.

Given that this is focused on configuration data, the dataset is not going to be particularly large.
The benefits of providing an elaborate query language in a resource-limited device is interesting, but of doubtful value for this application.

Proposed solution

A simple selector mechanism could be implemented by modifying JSON key values.

For simplicity I'll use an array of integers for demonstration.

Overwrite array

"x": [1, 2, 3, 4]

Clear array

"x": []

Note: The array itself cannot be deleted as it's fixed by the schema.

Indexed operations

Python list operations provide a good working model for this.
So x[i] corresponds to a single element at index i, x[i:j] is a 'slice' starting at index i and ending with index (j-1).

Note that x[] = [8, 9] isn't valid python but we can support it as shorthand for 'append'. Python would require x[len(x):] or x[10000000:0], for example.

The following operations assume an initial value of x = [1, 2, 3, 4]

Overwrite array
  "x"         : [1, 2, 3, 4]     [1, 2, 3, 4]
  "x[0:]"     : [8, 9]           [8, 9]
Clear array
  "x"         : []               []
Update single item
  "x[0]"      : 8                [8, 2, 3, 4]
  "x[2]"      : 8                [1, 2, 8, 4]
  "x[-1]"     : 8                [1, 2, 3, 8]
  "x[4]"      : 8                list assignment index out of range
Update multiple items
  "x[0:2]"    : [8, 9]           [8, 9, 3, 4]
  "x[1:1]"    : [8, 9]           [1, 8, 9, 2, 3, 4]
  "x[1:2]"    : [8, 9]           [1, 8, 9, 3, 4]
  "x[2:]"     : [8, 9]           [1, 2, 8, 9]
  "x[0:1]"    : 8                can only assign an iterable
  "x[pin]"    : 8                name 'pin' is not defined
Insert item
  "x[3:0]"    : [8]              [1, 2, 3, 8, 4]
  "x[3:3]"    : [8]              [1, 2, 3, 8, 4]
  "x[-1:]"    : [8, 9]           [1, 2, 3, 8, 9]
Append items (insert at end)
  "x[]"       : [8, 9]           invalid syntax (<string>, line 1)
  "x[10:]"    : [8, 9]           [1, 2, 3, 4, 8, 9]
  "x[10:]"    : 8                can only assign an iterable

Update Object array item by value

This is an invention (I think) so we can select items by field.

"channels[name=dummy]": {
    "pin": 1
}

or

"channels[pin=1]": {
    "name": "new name"
}

Note: We could support simple arrays like this:

"x[=1]": 5

So if x is [3, 1, 4, 1] it would be updated to [3, 5, 4, 5].
Not sure there's a particular use-case for this so might leave it.

[mikee47]

@pljakobs In your comment #17 (comment) you suggest using schema annotations to identify index fields. This seems similar to defining indices in a transactional database system, so is a sensible suggestion. It also suggests that ConfigDB would be responsible for enforcing indexing integrity. Implementing all that is a fair bit of work.

If I understand correctly, the purpose would be to enable updates like this:

{
    "general": {
        "channels[\"13\"]": {
            "name": "blue"
        }
    }
}

Using the selector approach we'd do this:

{
    "general": {
        "channels[pin=13]": {
            "name": "blue"
        }
    }
}

Which is more descriptive and simpler to implement. You could also do this:

{
    "general": {
        "channels[name=red]": {
            "pin": 25,
            "name": "green"
        }
    }
}

Or whatever. Would that work for you? If so, I'll go ahead and implement it.

[pljakobs]

I'm perfectly happy with the selectors as you describe them. After all, it's reasonable to expect the application to have the dataset, too, and to know what it "wants" to update.
I don't want to create any extra effort here.

btw, without a selector, the default operation would be to append, I assume?

I believe that, to be really true to arrays, at some point an "update by Index" or even an "insert before index" might be desireable (after all, an array implies some sort of order, just by the fact that it can be accessed by index) but for now, the front-end can get around this if it just manipulates the array locally and overwrites it in ConfigDB

[mikee47]

I don't want to create any extra effort here.

Any effort spent here is better than time spent later fixing/updating lots of application code if the API changes! So this is one of those things that needs a minimal, simple implementation to make the library useable.

btw, without a selector, the default operation would be to append, I assume?

No, overwrite - see the opening post. I'll edit that to clarify the use-cases.

[pljakobs]

I don't want to create any extra effort here.

Any effort spent here is better than time spent later fixing/updating lots of application code if the API changes! So this is one of those things that needs a minimal, simple implementation to make the library useable.

well, let me rephrase then: for my use case, the selector approach is perfectly suitable.
It is, in fact, very much what I have suggested - albeit clearer.
I can, however, see that at some point, someone might want to edit the order of elements in an array - given the fact that an array has a natural order. That doesn't necessarily need any code to implement a database-like index, but it would require functions to insert, edit and remove specific elements.

btw, without a selector, the default operation would be to append, I assume?

No, overwrite - see the opening post. I'll edit that to clarify the use-cases.

ah yes, stupid me, forgot that.

[mikee47]

I've updated the head comment. Below is the python script I used to generate the test cases. I'll use that as a basis for the library tests.

'''Script to verify appropriate array selector expressions
'''

TEST_CASES = {
    'Overwrite array': [
        'x = [1, 2, 3, 4]',
        'x[0:] = [8, 9]',
    ],
    'Clear array': [
        'x = []',
    ],
    'Update single item': [
        'x[0] = 8',
        'x[2] = 8',
        'x[-1] = 8',
        'x[4] = 8',
    ],
    'Update multiple items': [
        'x[0:2] = [8, 9]',
        'x[1:1] = [8, 9]',
        'x[1:2] = [8, 9]',
        'x[2:] = [8, 9]',
        'x[0:1] = 8',
        'x[pin] = 8',
    ],
    'Insert item': [
        'x[3:0] = [8]',
        'x[3:3] = [8]',
        'x[-1:] = [8, 9]',
    ],
    'Append items (insert at end)': [
        'x[] = [8, 9]',
        'x[10:] = [8, 9]',
        'x[10:] = 8',
    ]
}

for title, expressions in TEST_CASES.items():
    print(title)
    for expr in expressions:
        try:
            vars = {'x': [1, 2, 3, 4]}
            exec(expr, None, vars)
            x = vars['x']
        except Exception as e:
            x = e
        a, _, b = expr.partition(' = ')
        a = f'"{a}"'
        print(f'  {a:12s}: {b:16s} {x}')

[mikee47]

I can, however, see that at some point, someone might want to edit the order of elements in an array - given the fact that an array has a natural order.
That doesn't necessarily need any code to implement a database-like index, but it would require functions to insert, edit and remove specific elements.

Re-ordering is an interesting case. A thought here. If we have an array:

{
    "animals": ["rabbit", "fox", "donkey", "cat"]
}

And want to move cat into position 1:

{
    "animals": ["rabbit", "cat", "fox", "donkey"]
}

We could define an ordering operation * like this:

{
    "animals[*]": [0, 3, 1, 2]
}

We could come up with more concise operations, like:

{
    "animals[*1:2]": [2]
}

With the * (or whatever) indicating that the value isn't the actual value to use but refers to an existing value in the array.

It's not very natural language though so something to chew over.