deployment.md

Deployment

Build artifacts

Building and publishing artifacts are executed by GitHub Actions triggered by events in the siembol repository.

Java artifacts

Java artifacts are published to Central Maven Repository - Sonatype OSS Repository Hosting

• Snapshots - They are built if the version in POM contains SNAPSHOT. Snapshots are usually not stable and we suggest to use releases in a production environment
• Releases - They are built if the version in POM does not contain SNAPSHOT and are published in Central Maven Repository

Docker images

Docker images are built both from snapshots and releases.

• The images are tagged by two tags:

  • latest for tagging the latest image
  • The version of the application from POM

• Springboot applications

  • An application is loaded using springboot properties launcher
  • dockerfile
  • storm-topology-manager
  • config-editor-rest
  • responding-stream

• Config editor UI

  • A Single page Angular application
  • nginx-server with configurations
  • dockerfile

• Storm topologies

  • Images are used for launching storm topologies by storm topology manager
  • dockerfile
  • Storm cli
  • Siembol java storm topology artifact
  • parsing-storm
  • enriching-storm
  • alerting-storm

Infrastructure dependencies

• k8s cluster - environment to deploy siembol ui and related micro services for managements and orchestration of siembol services configurations

  • siembol ui - It combines config editor rest and config editor ui. This deployment also contains synchronisation service for synchronisation of releases, admin configurations and storm topologies
  • Siembol response
  • Storm topology manager - It is used for releasing siembol storm topologies that are specified by config editor rest synchronisation service from siembol configurations

• Github

  • A git repository store for a service configurations that are used in siembol ui
  • Pull Requests API for deploying releases and admin configuration in siembol UI
  • Web hooks for triggering config editor rest synchronisations of releases, admin configurations and storm topologies

• Zookeeper - synchronisation cache for

  • Updating service deployments configurations from git repositories to services
  • Updating desired state of storm topologies that should be released
  • Storing a state of storm topology manager in order to save information about topologies there were already released
  • how to set-up zookeeper nodes
  • See how to set-up kerberos if you need to use kerberised storm cluster in your siembol deployment

• Kafka - message broker for data pipelines. See how to set-up kerberos if you need to use kerberised kafka cluster in your siembol deployment

• Storm - stream processing framework for siembol services except siembol response integrated in kafka streaming and deployed on k8s. See how to set-up kerberos if you need to use kerberised storm cluster in your siembol deployment

• Identity provider - identity povider (oauth2/oidc) used for siembol ui for

  • Authentication - Only authenticated users can access siembol ui and all commits in git repositories which are performed by siembol ui contain user profile including username and email

  • Authorisation - You can specify oidc groups for managing authorisation access to services, see how to set-up a service authorisation in config editor rest

  • See how to set-up oauth2

Deployment scenarios

Helm charts