Technology Stack and Approach

• Agile development
• Data Model is being developed
• Continuous Integration: Jenkins

See the entire technical stack below along with the reasoning behind choosing particular components in italics. Unless specified, all software is Free and/or Open Source.

Table of Contents DBMSes ETL API Tools and Infrastructure

DBMSes

• RDBMS: PostgreSQL 12
  • HBase HDInsight cluster as an alternative was not found to have a lot of value-add.
• Graph DB: Neo4j 3.5

ETL

• Python 3 (Anaconda3 distribution)
  • New code should use Python 3
  • Updated modules should be upgraded to Python 3 and tested
  • More flexibility than provided by specialized ETL tools, e.g. Pentaho Community Edition ETL. Community edition has significant limitations. However, other specialized ETL tools could be evaluated for potential use.
• Legacy: Python 2.7 (Anaconda2 distribution)

API

• GraphQL
• Python 2
  • Web server: Flask
  • GraphQL service: flask_graphql + graphene + graphene_sqlalchemy
  • ORM: SQLAlchemy

Tools and Infrastructure

• OS: CentOS 7 Linux
• Cloud hosting: Microsoft Azure
• Monitoring
  • [] TBD. Server resources and performance: Azure Linux Diagnostic Extension
  • Service heartbeat and automatic restarts: monit
• Version Control: Git
• Continuous Integration: Jenkins
• Developers' IDE: JetBrains DataGrip
• SSH tunneling: (Mac) SSH Tunnel Manager
• Tickets: JIRA
• Automated Testing:
  • pgTap
  • Jenkins TAP Plugin
• Anti-virus: ClamAV