Skip to content

README.md

Latest commit

 

History

History
148 lines (96 loc) · 7.2 KB

README.md

File metadata and controls

148 lines (96 loc) · 7.2 KB

SensApp

SensApp is an open-source sensor data platform developed by SINTEF.

It handles time-series data ingestion, storage, and retrieval. From small edge devices to big data digital twins, SensApp may be useful.

SensApp allows you to process years of sensor data efficiently

SensApp is compatible with Prometheus and InfluxDB, but with an alternative architecture that prioritise data analysis and long-term storage over ingestion performance and real-time monitoring.

Dealing with system statistics for the last 24 hours? InfluxDB or Prometheus are excellent choices. Fetching average bathroom temperatures over the last 10 years grouped by day? SensApp will compute that instantly while InfluxDB or Prometheus will take a little while.

But you don't have to chose, both InfluxDB and Prometheus can replicate their data to SensApp for long-term storage and analysis. So you get the best of both worlds.

You can also use Sensapp as a standalone time-series database.

Quickstart

The quickest way to run SensApp is with SQLite so no external database is required.

Start SensApp with SQLite:

SENSAPP_STORAGE_CONNECTION_STRING=sqlite://sensapp.db \
cargo run

By default, SensApp listens on http://127.0.0.1:3000.

Ingest one sample:

curl --json '[{"n": "temperature", "v": 21.5}]' \
  http://127.0.0.1:3000/publish

Query the stored data:

curl 'http://127.0.0.1:3000/api/v1/query?query=temperature'
# or
curl 'http://127.0.0.1:3000/api/v1/query?query=temperature&format=csv'

Python Quickstart

Check the python/sensapp documentation for more details.

import asyncio
from sensapp import SensAppClient

async def main():
    async with SensAppClient() as client:
        await client.publish("temperature", 21.5)
  [series] = await client.query("temperature[1h]")
  print(series.frame)

asyncio.run(main())

Using Containers

docker compose up

You can deploy SensApp on Kubernetes using the included Helm chart.

helm install sensapp ./charts/sensapp

Features

  • HTTP REST API
  • Prometheus Compatibility
    • Prometheus Remote Write: Prometheus can push data to SensApp.
    • Prometheus Remote Read: Prometheus can also read data from SensApp.
    • Prometheus Scrape Endpoint: SensApp exposes internal service metrics on /prometheus/metrics.
  • InfluxDB Compatibility:
  • Data formats:
    • JSON: Simple and widely used format for data interchange.
    • CSV: The classic.
    • SenML: Standardized format for sensor data representation, that is almost unheard of but actually pretty good.
    • Apache Arrow IPC Support: Efficient IPC format for high-performance data interchange.
  • Flexible Time Series DataBase Storage:
    • SQLite: Lightweight embedded database for edge deployments.
    • DuckDB: Alternative to SQLite, potentially faster for analytical queries. Not enabled by default.
    • PostgreSQL: Robust relational database for medium to large deployments, with optional TimeScaleDB plugin for enhanced time-series capabilities.
    • ClickHouse: Columnar database management system for high-performance analytical queries on large volumes of data.
    • BigQuery: Fully-managed serverless data warehouse for scalable analysis. Not enabled by default.
    • RRDCached: Integration with RRDtools, mostly implemented for fun. Not enabled by default.

Architecture

SensApp's architecture is relatively simple as the complex problems are delegated to existing databases. It's a stateless adapter between HTTP clients and the chosen time-series database(s).

Most of the complexity lies in the database schema design. After that, it's mostly some code glue.

  • On the edge, SensApp can be deployed as a single lightweight instance with an embedded SQLite database.
  • For medium deployments, SensApp can be deployed with a single message broker and a PostgreSQL database.
  • For larger deployments, many SensApp instances can be deployed behind a load balancer, connected to a ClickHouse database cluster.

SensApp storage is based on the findings of the paper TSM-Bench: Benchmarking Time Series Database Systems for Monitoring Applications. ClickHouse also released an experimental time-series engine that is somewhat similar to SensApp's storage schema.

Check the ARCHITECTURE.md file for more details.

Authentication

SensApp supports optional JWT authentication. By default, all endpoints are open. Visit docs/JWT_AUTH.md for the authentication documentation.

Built With Rust™️

SensApp is developed using Rust, a language known for its performance, memory safety, and annoying borrow checker. SensApp used to be written in Scala, but the new author prefers Rust.

Another reason is from the results from the paper Energy efficiency across programming languages: how do energy, time, and memory relate?, which shows Rust as one of the most energy-efficient programming languages while having memory safety.

Not only the language, it's also the extensive high quality open-source ecosystem that makes Rust a great choice for SensApp.

Here ends the mandatory Rust promotion paragraph.

Contributing

We appreciate your interest in contributing to SensApp! Contributing is as simple as submitting an issue or a merge/pull request. Please read the CONTRIBUTING.md file for more details.

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

The SensApp software is provided "as is," with no warranties, and the creators of SensApp are not liable for any damages that may arise from its use.

You may not want to use it in production (yet)

SensApp is currently under development. It is not ready for production.

Acknowledgments

We thank the historical authors of SensApp who created the first version a decade ago.

SensApp is developed by SINTEF (Digital division, Sustainable Communication Technologies department, Smart Data research group).

It is made possible thanks to the research and development of many research projects, founded notably by the European Commission and the Norwegian Research Council.

We also thank the open-source community for all the tools they create and maintain that allow SensApp to exist.