The Atom-IT server provides a simple framework to define IoT workflow. It is entirely built upon three basic concepts:
These three concepts are described on this page.
A time series is a time-ordered set of messages:
+---+---+----+----+----+----+----+----+----+----+
==> | 5 | 9 | 10 | 18 | 22 | 30 | 31 | 35 | 40 | 43 | ==>
+---+---+----+----+----+----+----+----+----+----+
Each message is made of:
- A timestamp, that is a signed 64-bit integer.
- A value, that is a string of arbitrary length that can contain binary characters.
- A metadata, that is a string providing information about the value (such as a MIME type or a EUI hardware identifier).
Within a single time series, timestamps must increase monotonically. Messages can only be appended to the end of the time series.
The interpretation of the timestamp is application-specific. For instance, if the timestamps are interpreted as the number of nanoseconds since the Epoch, any nanosecond between years 1678 and 2261 can be indexed (*). Most commonly, timestamps will consist of a sequence of numbers that is automatically incremented by the Atom-IT server as messages are appended.
Once inserted in a time series, the content of the messages cannot be updated. It is only possible to delete messages, by providing a range of timestamps to remove. This behavior could be summarized as "CRD operations", i.e. CRUD without Update. The Atom-IT server will ensure proper mutual exclusion if several threads are simultaneously accessing the same time series, according to the readers/writers paradigm. It is also possible for a reading thread to wait until a change is made by some writer thread on a given time series.
The Atom-IT server can hold several time series at once. The stored time series are all independent from each other (in particular, the interpretation of the timestamps can be different for different time series).
NB: From a developer perspective, the content of the time series are
accessed through the
AtomIT::ITimeSeriesAccessor
C++ interface that are constructed by the
AtomIT::ITimeSeriesManager
object that is unique throughout the Atom-IT server.
(*) Rough computation with
GNU Octave: 1970 ± (2**63-1)/(1000*1000*1000*60*60*24*365.2425), where 365.2425 is the
mean number of days in a year according to the
Gregorian calendar.
The backends are the software components of the Atom-IT server that are responsible for storing and accessing the time series. They must notably implement database transactions to ensure the consistency of data.
For the time being, backends are provided to store time series either directly in RAM or onto local SQLite databases. In the future, backends supporting OpenTSDB, MySQL, PostgreSQL or HBase databases might be developed in order to improve scalability and to transparently share time series between Atom-IT servers.
Backends are also responsible for assigning quotas to their time series in order to recycle space by removing oldest messages as new messages are appended. These quotas are expressed as the maximum number of messages in the time series, and as the maximum total size of the values of the messages in the time series.
One single Atom-IT server can manage several backends at once, each of them storing several time series in different places. This is illustrated in the following figure:
+---------------+
+-- | Time series 1 |
+-------------------+ | +---------------+
+-- | Memory backend | --+
| +-------------------+ | +---------------+
+----------------+ | +-- | Time series 2 |
| Atom-IT server | --+ +---------------+
+----------------+ |
| +---------------+
| +-- | Time series 3 |
| +-------------------+ | +---------------+
+-- | SQLite "atom1.db" | --+
| +-------------------+ | +---------------+
| +-- | Time series 4 |
| +---------------+
|
| +-------------------+ +---------------+
+-- | SQLite "atom2.db" | ----- | Time series 5 |
+-------------------+ +---------------+
NB 1: From a developer perspective, the backends must implement the
AtomIT::ITimeSeriesBackend
C++ interface, and are constructed through
AtomIT::ITimeSeriesFactory
factories.
NB 2: In the currently available SQLite backend, the messages are expected to be relatively small (say, up to about 1KB). An improved data storage scheme to store larger messages (e.g. images for video streams) is work-in-progress.
A filter is an object that processes input messages coming from different time series, and that combines them to compute output messages that are pushed into other time series:
+---------------------+
| Input time series 1 | >--+ +----------------------+
+---------------------+ | +--> | Output time series 1 |
| | +----------------------+
+---------------------+ | +--------+ |
| Input time series 2 | >--+--> | Filter | >--+ ...
+---------------------+ | +--------+ |
| | +----------------------+
... | +--> | Output time series M |
| +----------------------+
+---------------------+ |
| Input time series N | >--+
+---------------------+
The number of inputs of a filter can be different to the number of its outputs. Some special cases of filters commonly arise:
- A source is a filter with no input time series.
- A sink is a filter with no output time series.
- An adapter is a filter with exactly one input time series.
Sources and sinks will most often correspond to a file, a network adapter, or a hardware device. As far as adapters are concerned, they would convert the value of messages, and/or implement demultiplexing to output time series.
Filters must implement a very simple state machine:
Step() => In one thread for each filter
+------+
/ \
/ \
\ /
+---------+ +---------+ \ / +--------+ +------------+
| Factory | --> | Start() | ---+------+---> | Stop() | --> | Destructor |
+---------+ +---------+ +--------+ +------------+
As depicted above:
- A global factory function creates all the filter objects given the configuration file of the Atom-IT server.
- Once all the filters are created, the Atom-IT server invokes the
Start()method of each of them. If some filter fails to start, the Atom-IT server does not start at all. - Once all the filters are started, one separate thread is created
for each of them. This thread will repeatedly invoke the
Step()method of the filter in an infinite loop. TheStep()method is assumed to have a bounded execution time (ideally, no more than 500ms). - Once the Atom-IT server is asked to stop (typically with a
Control-C keypress or any
kill command), all
the threads are stopped by canceling the infinite loop. Once each
filter has stopped, the Atom-IT server invokes the
Stop()method to give each filter a chance to cleanly release its allocated resources, and destruct each filter object.
NB: From a developer perspective, filters must implement the
AtomIT::IFilter C++ interface, and
are constructed through the
AtomIT::CreateFilter() global
factory function.