Cyber Threat Intelligence Monitoring System
Final Year Project • Bachelor of Computer Science (Hons) (Artificial Intelligence)
CyberHolmes is a full-stack, web-based Cyber Threat Intelligence (CTI) monitoring system developed as part of Final Year Project (FYP) for the completion of Bachelor of Computer Science (Honours) (Artificial Intelligence).
The project focuses on detecting and analysing posts that may pose cyber threats by applying advanced natural language processing techniques. It automates the collection of textual data from both clear and dark web forums, followed by prediction and analysis of potential threats using machine learning and deep learning models. The goal is to provide cybersecurity professionals and researchers with actionable intelligence through an interactive dashboard and evaluation, enabling more effective threat monitoring and response.
1. Data Collection
- Crawls posts and discussions from both clear and dark web forums using
ScrapywithTorproxy integration to ensure anonymity.
2. Threat Prediction
- Applies machine learning model (Ensemble = LR + SGD + XGBoost) built using
scikit-learnto classify text-based data and identify potential cyber threats.
3. Threat Analysis
- Utilises deep learning model, SecBERT, imported from
Hugging Faceto perform advanced text analysis on collected data. - Enriches threat context using the Common Vulnerability Scoring System (CVSS) via the
cvsspackage and MITRE ATT&CK framework data via theattackctipackage.
| Layer | Technology |
|---|---|
| Frontend | Next.js |
| Backend | Django |
| Database | SQLite |
| Web Server | NGINX |
/CyberHolmes
│
├── backend/ # Django backend application
│ |── analysis/ # Threat prediction and analysis processes (ML + DL)
│ |── config/ # Django project configuration
│ |── crawler/ # Scrapy-based crawlers for data collection
│ |── main/ # Core app for main logics and API endpoints
│ |── .gitignore/ # Git ignore rules for backend
│ |── Dockerfile/ # Docker setup for backend service
│ |── manage.py/ # Django project management script
│ |── requirements.txt/ # Python dependencies
│ └── torrc/ # Tor configuration
|
├── frontend/ # Next.js frontend application
│ |── app/ # App routing and page definitions
│ |── components/ # Reusable React UI components
│ |── config/ # Frontend configuration
│ |── lib/ # Utility functions and hooks
│ |── styles/ # Global and component-specific CSS styles
│ |── .gitignore/ # Git ignore rules for frontend
│ └── Dockerfile/ # Docker setup for frontend service
|
├── nginx/ # nginx configuration for reverse proxy
├── docker-compose.yml # Docker container orchestration
├── LICENSE
└── README.md
- Debian-based system
- Git
- Docker
- Docker Compose
git clone https://github.com/rydzze/CyberHolmes.git
cd CyberHolmesCreate .env files for both backend and frontend. Use the provided templates or set your own keys.
backend/.env
DEBUG=True
SECRET_KEY=your-secret-key ¯\_(ツ)_/¯frontend/.env
# If run locally:
NEXT_PUBLIC_API_URL=http://localhost:3001/api
# If run remotely:
NEXT_PUBLIC_API_URL=/apiFrom the root of the project directory:
docker-compose up --build -dThe frontend can be accessed at http://localhost:3000
The backend API can be accessed at http://localhost:3001/api
Figure 1: Dashboard overview (light theme)
Figure 2: Dashboard overview (dark theme)
Figure 3: Deploying crawlers to collect threat intelligence
Figure 4: Viewing collected crawler records
Figure 5: Analysis result for a non-threatening post (light theme)
Figure 6: Analysis result for a non-threatening post (dark theme)
Figure 7: Threat detected with enriched threat context (light theme)
Figure 8: Threat detected with enriched threat context (dark theme)