Contracts between core and aggregation

[barbaracassani]

The Aggregation engine should have an API to consume from the Core in order to get the raw data. I don't think there's much left to the imagination here - the only thing that comes to my mind at present, being that I don't remember exactly the rationale for not allowing the Aggregation to interrogate the database directly.

Anyway, I think we can agree on a GET call to obtain reports that satisfy given criteria, i.e.:

/reports/tags/[tag or list of tags]
/reports/[report slug]
/reports/location/[coordinates, radius]
/reports/time/[timestamp, time range]

Would that be suitable? Or we can use streams and just initialise the stream with the required criteria, using always the same endpoint.

[padolsey]

I thought we were calling it conglomeration -- ? -- As 'aggregation' seems more appropriate for the eventual collection of data (from multiple sources, Twitter etc.)

The reason for separating them is so changes can be made to the core and how it stores data without having to change those consuming the data -- the API contracts will remain the same.

I'll give some thought to the API -- haven't been able to yet. My initial thoughts were for the conglomeration engine to deal directly with a library called 'core' (or whatever). It would be something like this:

var coreData = require('core');

coreData.on('report', function(report) {
  report.location; // the location, latlng + geocoded data etc.
  report.created; // time of creation
  // report.[...] // various other fields, yet to be defined
  doConglomeration(report);
});

This is how simple I want it to be for the conglomeration engine to consume data. I don't want consumers of the report data to have to worry about REST contracts or sockets or streams etc. ... it should all be abstracted IMHO.

[barbaracassani]

Sorry for misnaming the Conglomeration; indeed that was the agreed name.

• Conglomerations will therefore only receive the reports as they come in, it won't ask for them
• What it will create and store (I suppose, in the db) are object that references Reports (but basically duplicate most of the report content, as Conglomerations need to be made and unmade as time elapses and in order to make this, Conglomeration needs either to store the report content, at least the 'interesting', 'conglomerable' bits - either to be able to reinterrogate Core at will for specific Reports and ranges of reports

Is that correct?

[padolsey]

Yep, that's all correct. I think the cnglms will want to refresh their various reports every so often -- ideally very often. But maybe this is in-fact why we need additional events like:

core.on('report:change', function(report) {
  //...
});

[barbaracassani]

When one report changes or get added, Conglomeration might need to revisit the previous ones that respond to some particular criteria (same time etc). Where those will be found? Shall Conglomeration keep a copy? Re-interrogate Core?

[padolsey]

@barbaracassani Good point. Worth discussing more I think. On reflection... I'm thinking that 'core' should be treated as a reliable and fast way to retrieve reports & their internal updates -- so there's no need for you to store them directly. I imagine that what the conglomerator actually stores will effectively be:

// single conglomeration:
{
  reports: [reportId1, reportId2, reportId3,...]
}

And when there is an update to one of the reports I imagine the conglomerator will want to profile the entire conglomeration to make sure it's all still valid. And I guess this should happen asynchronously, at some point in the future. As in, I imagine there being a queueOfConglomerationsWeHaveToReProfile. This is just my take on it. What do you think?