This repository contains the CVE Processor microservice, a core component of a comprehensive, scalable cloud-native system for processing and querying CVE (Common Vulnerabilities and Exposures) data. The complete system leverages Kubernetes, microservices architecture, Kafka messaging, and an LLM-powered RAG application to deliver intelligent CVE insights.
System Overview: The end-to-end solution continuously monitors the official CVE GitHub repository, processes new releases, stores data in PostgreSQL, and provides an AI-powered question-answering interface. Built on AWS EKS with Infrastructure-as-Code (Terraform), the system features automated CI/CD pipelines, service mesh security (Istio), comprehensive monitoring (Prometheus/Grafana), and centralized logging (FluentBit/CloudWatch).
The CVE Processor is a Go application that:
- Downloads and extracts CVE JSON files from the CVEProject repository
- Processes CVE data and publishes messages to Kafka topics
- Integrates with Flyway for database schema migrations
- Runs as a containerized Kubernetes Job triggered by a custom Kubernetes Operator
This project is part of a complete microservices ecosystem:
- CVE Processor - Downloads and processes CVE data (this repo)
- CVE Processor Helm Chart - Kubernetes deployment configuration
- CVE Consumer - Consumes Kafka messages and stores in PostgreSQL
- CVE Consumer Helm Chart - Consumer deployment configuration
- CVE Operator - Custom Kubernetes operator for monitoring CVE releases
- CVE Operator Helm Chart - Operator deployment configuration
- AWS EKS Infrastructure - Terraform code for EKS cluster provisioning
- Jenkins Infrastructure - Terraform code for CI/CD infrastructure
- Jenkins AMI - Packer templates for Jenkins AMI
- EKS Cluster Autoscaler Helm Chart - Auto-scaling configuration
- FluentBit Helm Chart - Centralized logging setup
- LLM Application - Flask-based RAG application with LLaMA3 integration
- LLM Application Helm Chart - LLM service deployment
Before you begin, ensure you have the following installed on your local machine:
- Go (1.18 or newer): Download Go
- PostgreSQL (latest version): Download PostgreSQL
- Flyway for database migration: Download Flyway
- Git (for version control): Download Git
Ensure the GOPATH and GOROOT environment variables are set according to your Go installation.
-
Clone the repository:
git clone https://github.com/eashanroy7/webapp-cve-processor.git cd webapp-cve-processor -
Set up environment variables:
# Copy the .env.example to .env and adjust the database connection strings and other configurations as necessary cp .env.example .env -
Install dependencies:
go mod tidy
-
Build the application:
go build -o cve-processor ./cmd
-
Database Setup:
- Start your PostgreSQL server and create a database named cve.
- Modify the flyway.conf to match your database connection details.
-
Run Database Migrations:
- Navigate to the migrations directory.
- Execute Flyway migrations to set up your database schema:
flyway migrate
-
Run the application:
./cve-processor
This command starts the CVE Processor, which will begin processing the CVE JSON files from the specified directory.
-
Directory Structure: The
cmddirectory contains the main application logic. Thepkgdirectory includes essential packages for CVE processing. -
Deployment: In production, this application runs as a Kubernetes Job triggered by the custom CVE Operator when new CVE releases are detected.
-
CI/CD: Automated pipelines via Jenkins build multi-architecture Docker images (ARM/x86) with semantic versioning.
- Language: Go (Golang)
- Messaging: Kafka
- Database: PostgreSQL with Flyway migrations
- Containerization: Docker
- Orchestration: Kubernetes (AWS EKS)
- Infrastructure: Terraform, Helm
For support, please submit an issue on GitHub.