How to transform nested array objects

[Sudhirmiglani]

For example,

Input : ```
{
"obj": {
"y": "s",
"arr": [
{
"clientId" : 1,
"id" : 1
},
{
"clientId" : 2,
"id" : 2
}
]
}
}

TransformationMap : ```
"item":{
			"x" : "y",
			"z" : [
				{
					"a" : "arr.clientId",
					"b" : "arr.id"
				}
				]
		}

This is not working.

[lingwooc]

You can't do that with the current version. However I've got a pull request pending for my fork https://github.com/lingwooc/node-json-transform. This allows the complex output templates you need. Hopefully @bozzltron will merge it into the main version once they've had time to have a look/test it. In the meantime you could just fetch that and install it as a local project.

[tgilkerson]

I am really looking forward to the enhancements, any ETA?

[tgilkerson]

@lingwooc I did fetch your fork and I can do more with complex types but I still can do what was described in the original issue description. I want to map nested arrays and I dont see how to accomplish his. Maybe I am just missing something. Below is have sample input file and desired output file. Can you let me know if this is possible? thanks.

///////////////////////////////////////////
// sample input file
///////////////////////////////////////////
products:[
{
id: 'books0',
zero: 0,
sku:'00234-12312',
subitems:[
{subid:"0.0", subsku: "subskuvalue0.0"},
{subid:"0.1", subsku: "subskuvalue0.1"}
]
},
{
id: 'books1',
zero: 1,
sku:'10234-12312',
subitems:[
{subid:"1.0", subsku: "subskuvalue1.0"},
{subid:"1.1", subsku: "subskuvalue1.1"}
]
}
]

/////////////////////////////////////////////////////////////////////////
// sample output file that I am trying to generate
// I am having issues formatting mysubitems as shown below
////////////////////////////////////////////////////////////////////////
[
{
"myid": "books0",
"mysku": "00234-12312",
"mysubitems": [
{
"subid": "0.0",
"subsku": "subskuvalue0.0"
},
{
"subid": "0.1",
"subsku": "subskuvalue0.1"
}
]
},
{
"myid": "books1",
"mysku": "10234-12312",
"mysubitems": [
{
"mysubid": "1.0",
"mysubsku": "subskuvalue1.0"
},
{
"mysubid": "1.1",
"mysubsku": "subskuvalue1.1"
}
]
}
]

[lingwooc]

Yeah I think you should be able to.

I've not got time to write a full template or test it to be sure (the dog needs walking). However a quick example of how to do the sub lists is:

"mysku": "sku",
"mysubitems": [{
	"mysubid": "subitems.0.subid",
	"mysubsku": "subitems.0.subid"
},
{
	"mysubid": "subitems.1.subid",
	"mysubsku": "subitems.1.subid"
}]

That needs to be in the context of a valid template (that is a snippet). It isn't clever enough to do arbitrary length lists, only fixed (I may resolve that later if anyone cares). The nested part works though. There is an example of complex templates in the tests folder of my fork. The syntax for reading nested lists is the same as the main version. The main version just misses writing lists.

Oh and you need to use an operation for default values (also an example in my tests file) like 0. Your list also needs a name as defined in the template (not sure if you missed that for brevity or not).

Hope that helps. If it still doesn't work for you make a issue in my repo and I'll take a look when I've got more time.

[tgilkerson]

@lingwooc thanks for your response. I should have been more detailed on my question. I do realize that you can do "subitems.0.subid" and "subitems.1.subid" to map the first and second entries of "subitems", but I want to do something like "subitems..subid" where "" stand for "all" entries. In my use case the number of entries in "subitems" is unknown and can vary.

It is as-if I need nested "list:" The first "list: 'products'" and the second nested "list: 'subitems'". In your example I need to know how many entries are in "subitems" when I am creating the map and I won't.

Any help or ideas is appreciated.

[lingwooc]

Oh I understand. Yes that is a limitation of the current version. You want to be able to write subitems.x.subid (or equivalent). I'll probably add that to my fork at some stage. However there is a way round it that I've used which works well.

Create a template for the main list including the line "mysubitems": "subitems". That will send the whole nested list of subitems to the new node. Then create an operation for that node that calls another template (and sends the sublist through this library again) which deals with the ids and skus. Then return the transformed list from the operation. Works well if potentially a little inefficient.

So you need two templates, one for the main items and one for the sublist items. Then process the sub list from an operation.

Hope that makes sense.

[tgilkerson]

Do you mean something like this?

operate: [
    {
        run: function(sublist) { 
        	var dataTransform = DataTransform(sublist, map);
	var result = dataTransform.transform();
        	return result;
        }, on: "subitems"
    }
]

[lingwooc]

Yup. You may need to label the list. So:
var dataTransform = DataTransform({"listName": sublist}, map);

[bozzltron]

@tgilkerson With the current version, I agree with @lingwooc that using two templates is the best way to go. Let me know if you're still having any issues. Sorry for the late response.

[bozzltron]

@tgilkerson I am writing a test to confirm the multiple template use case. I noticed that there are inconsistencies in your example output. In the first item the keys in the array are "subid" and "subsku". In the second item they are "mysubid" and "mysubsku". I'm assuming you want the sub keys to be consisting tranformed from "subid" to "mysubid" and "subsku" to "mysubsku". Let me know if that is correct.

[bozzltron]

Here is the passing test.

`
var DataTransform = require('../index.js').DataTransform,
_ = require("lodash");

var data = {
products: [{
id: 'books0',
zero: 0,
sku: '00234-12312',
subitems: [
{ subid: "0.0", subsku: "subskuvalue0.0" },
{ subid: "0.1", subsku: "subskuvalue0.1" }
]
}, {
id: 'books1',
zero: 1,
sku: '10234-12312',
subitems: [
{ subid: "1.0", subsku: "subskuvalue1.0" },
{ subid: "1.1", subsku: "subskuvalue1.1" }
]
}]
};

var baseMap = {
'list': 'products',
'item' : {
'myid': 'id',
'mysku': 'sku',
'mysubitems': 'subitems'
},
operate: [
{
'run': function(ary) {
return DataTransform({list:ary}, nestedMap).transform();
},
'on': 'mysubitems'
}
]
};

var nestedMap = {
'list': 'list',
'item' : {
'mysubid': 'subid',
'mysubsku': 'subsku'
}
};

var expected = [
{
"myid": "books0",
"mysku": "00234-12312",
"mysubitems": [
{ "mysubid": "0.0", "mysubsku": "subskuvalue0.0" },
{ "mysubid": "0.1", "mysubsku": "subskuvalue0.1"}
]
},
{
"myid": "books1",
"mysku": "10234-12312",
"mysubitems": [
{ "mysubid": "1.0", "mysubsku": "subskuvalue1.0" },
{ "mysubid": "1.1", "mysubsku": "subskuvalue1.1" }
]
}
];

describe("node-json-transform", function() {

it("should copy arrays", function() {

	var dataTransform = DataTransform(data, baseMap);

	expect(dataTransform.transform()).toEqual(expected);

});

});
`

[tgilkerson]

@bozzltron, yes I want the sub keys, sorry that was my mistake. Thanks for putting this together so fast! I will not be able to check it out or try it myself until later tonight, (at my day job now :-)

[lingwooc]

@bozzltron how do you feel about the notation list.x.property to support reading for arbitrary length lists? I'm pretty sure I could do it without much extra code. It just needs to set x with bind and use it for lookups. It shouldn't mask any nodes called x as you can check for arrays. It should be possible to allow for nested, nested lists as well.

Sometimes double templates just seems overkill. Although other times it is actually better.

[tgilkerson]

@bozzltron my first thought was to use the splat character "*", or if you want to go crazy use "{n}" where n is a number 1 to n, so if I want the first 10 occurrences I would say {10} if I want them all I would say {n}. Really it is up to use if x is what you like I will take it! thanks for your help.

[bozzltron]

I regret the existing syntax because I should have just used the JSONPath. My chief concern is that maps are human readable and therefore easy to maintain. Introducing a many-to-one syntax might be confusing. Currently, maps use "list" and "item" to know how to process an array. Another option would be allow the destination keys to be set to a nested "list" and "item" assignment.

var baseMap = {
	'list': 'products',
	'item' : {
		'myid': 'id',
		'mysku': 'sku',
		'mysubitems': {
                      'list': 'subitems',
                      'item': {
                           'mysubid': 'subid',
                           'mysubsku': 'subsku'
                       }
                }
	}
};