Configuration Framework: what about persistence?

[saiedt]

The documentation under https://github.com/universAAL/platform/wiki/Configuration-Framework does not explain anything about the persistence, although one of the most frequent reasons for defining connf parameters is to give the possibility to system deployers to feed the conf parameters needed by a module to the system by editing related configuration files that are stored persistently. The module reads the conf file and knows how the relevant part of the environment is configured.

The documentation says that using ModuleScope would be equivalent to "OSGi configuration service, in the sense of a single module having a locally managed configuration space". However, OSGi confadmin supports config files for persistence.

I wonder why the container methods for registering config files have been declared as deprecated if this framework does not have any support for persistence; however, maybe I'm wrong and the issue here is only the bad documentation...

[amedranogil]

The configuration framework has persistente, every configuration change is recorded in the configuration module (turtle) database.
The equivalency with OSGi configuration services is partially true, in the sense there is a service to configure, and automatically restore the last configuration, but there are other issues like the distribution and semantics which make it different.
Also for some time I noticed there is one feature missing from the config framework: instance management (see ManagedServiceFactory)

[saiedt]

So, the questions that arise now are:

1. Where can these parameters be found on Disk?
2. Can deployers edit them manually (if there is no editor with GUI)?
3. Will they be read automatically when the system starts?
4. Will a subscriber (that was loaded before the conf manager) be notified when these parameters are read?
5. Will a subscriber (that is loaded after the conf mgr has read the existing values) be notified with the values read?

[saiedt]

Assuming that all the above questions have satisfying answers, my next questions / comments would be:

1. The conf framework rightly models a conf param as a key-value pair; however in my opinion, storing this simple thing in terms of turtle makes it too hard for the deployers to deal with them (e.g., edit them manually).
2. Consider the following configuration:

cgw.noOfComponents=2
cgw.component.1.type=lamp
cgw.component.1.location=bathroom
cgw.component.2.type=onOffSwitch
cgw.component.2.location=kichen
I assume that when using the conf framework I should define this set of parameters the following way:
cgw.components.types=lamp,onOffSwitch
cgw.components.locations=bathroom,kichen
If I do so, is there any guarantee that the order of the elements in the list is preserved (so that I know that the lamp is in the bathroom and the onOffSwitch is in the kichen)?

3. I'll update the documentation with these questions and answers :-)

[amedranogil]

Where can these parameters be found on Disk?

They are stored in a TTL file under the config.manager config folder.

Can deployers edit them manually (if there is no editor with GUI)?

yes & no. Yes, the configuration framework envisions configuration editors, which are modules that enable configuration edition. These editors may be GUI, or even remote (maybe connected through the MTGW in the future). Currently there is a very simple configuration (text) editor which is managed through the karaf shell.

No, the configuration manager data base can be edited manually, but it is NOT recommended (unless you know what you're doing).

BTW @cstockloew and I have a 2014 email conversation about how to secure the capabilities of configuration editors, using similar authorisation mechanism as the functional manifests, so that malware editors can be controlled.

Will they be read automatically when the system starts?

Yes, when the module registers as a configurable entity, the CM will look for existing configuration in the database (& remote nodes' DBs), if there is configuration corresponding with the given scope, the CM will send that configuration, if not it will generate the default configuration, and send that back.

The local (and remote when applicable) DB is persistently updated with every configuration change request; whether this comes form a configuration editor, the default value, or the dynamic description entity (when the entity has alternative modes of configuraiton).

Will a subscriber (that was loaded before the conf manager) be notified when these parameters are read?

The CM is loaded with the MW, pretty early on (and this causes some issues, see my later comment), it is only dependent on the "control bus" broker, mw.data.* & API modules. There is no chance for application level subscribers to be loaded before the config manager, but if they are, they should use the proper subscription mechanism, and then yes, as soon as the config manager comes online it will receive the request and go through the registry process described before.

Will a subscriber (that is loaded after the conf mgr has read the existing values) be notified with the values read?

Yes, as soon as a Configurable Entity is registered it will be notified with any existing configuration, o default config if none exist.

I have to say that there is no distinction between reading from the DB, loading defaults, or configuration change request; for the configurable entity they are all the same (except in terms of chronology, default and DB are sent just after registry while the rest are sent asynchronously).

The conf framework rightly models a conf param as a key-value pair; however in my opinion, storing this simple thing in terms of turtle makes it too hard for the deployers to deal with them (e.g., edit them manually).

Actually it is a bit more complex than a key-value, the key is the configuration scope (explained in the wiki), where the value can take any RDF form.

Ideally the configuration framework could be used similarly to the UI framework, in the sense that you declare your configuration elements in an ontology (including the restrictions of said configuration), and have your configuration as RDF instances (in fact this is how the internals of the Configuration Manager, and its protocol, work). There are simplification mechanism for simple key-value for simple parameters.

There is a problem though with non standard type values, text (and even GUI) editors will have trouble presenting the user with a usable interface to edit these configurations. for example current terminal configuration editor does not deal with these types (it does not even list them). But purposely built configuration editors can manage these, as the interfaces allow for it.

As a feature request for tools.runtime there is the need to create the GUI for configuraiton, including the RDF editor (as part also of the UCC2.0 discussions, as this editor will help in other tools' functions).

As said before it is not intended for deployers to edit the configuration manually, it is always envisioned a tool for this.

Consider the following configuration (...)

this type of problem is the kind I was trying to address with the "instance issue" (see Scope Instantiation). With that function you can create as many instances of component as needed, and add or remove dynamically.

If I do so, is there any guarantee that the order of the elements in the list is preserved (so that I know that the lamp is in the bathroom and the onOffSwitch is in the kichen)?

When dealing with lists, remember the CM is always using the mw.data.serialization component for storing. The list order question should be asked there.

Personally I would not model this configuration this way; currently you have the option of defining an RDF graph, you also have very fine tuned Scope Id (keys) which can help better define each component; or in the future with the "instantiation feature" you will be able to create dynamic components.

[amedranogil]

Please note there is a sample module showcasing the configuration framework, and how to program your modules to be configurable.

[cstockloew]

Will a subscriber (that was loaded before the conf manager) be notified when these parameters are read?

The CM is loaded with the MW, pretty early on (and this causes some issues, see my later comment), it is only dependent on the "control bus" broker, mw.data.* & API modules. There is no chance for application level subscribers to be loaded before the config manager, but if they are, they should use the proper subscription mechanism, and then yes, as soon as the config manager comes online it will receive the request and go through the registry process described before.

A concrete example: data.rep has some config files, as can be seen here:
https://github.com/universAAL/middleware/blob/master/middleware.core/mw.data.representation.core/src/main/java/org/universAAL/middleware/datarep/SharedResources.java#L103
It currently uses the old deprecated method registerConfigFile. How would this be done? Should I start a new thread that tries to fetch the CM until its available?

[amedranogil]

@cstockloew As far as I see it, there are some options:

1. have 2 paralel configuration frameworks, as we have now, where modules can acess their own configration folder and have their configuration there (the ConfigFramework allows this, by "advaced" configuration interfaces). Keeping pre-CM modules on this other framework will mean that CM can not confure them (unless the advanced integration is built into them, there might be looping dependencies though). The idea behind CM is that we could provide the necesary tools to configure remotely. The CM was also thought to provide configuration for some parts of the MW too.

2. pre-CM modules, should have a default configuration until CM comes online, and can serve the latest config from its data base. Of course this can lead to strange conditions.

3. Have the CM lose all its dependencies with mw.data.*. This is the nuclear option and will leave the Config-framework without semantic power.

[uzi0espil]

Thanks for the detailed explanation of CM.. within wiki I have installed the karaf command line configuration editor: mvn:org.universAAL.middleware/mw.karaf.shell.universAAL.config.osgi.. However, I couldn't find the corresponding code available in the github/middleware..

Having the the code would be useful to understand how the configurations are edited. (Sorry wasn't sure whether to continue with this thread or open a new one)

[amedranogil]

true, it is not there.
It is developed though, I think I may have it in a local branch, I need to check.

[amedranogil]

I found it in my old sandbox (uuuf back to when we used SVN!)
https://gitlab.com/universAAL/middleware/tree/sandboxes/amedrano
I need to integrate it into current master (probably 3.4.1-SNAPSHOT too). It will take an afternoon ( I should be able to include it with history, but it requires finesse and a load of knowledge of GIT).

At least you have there the source...

[cstockloew]

It's not necessary to include the history as well; it's a "new" module.. "simply" copying it to master (and perhaps 3.4.1-S) is OK.

[amedranogil]

It is integrated w/history