Log Classification with Hybrid Classification Framework

An advanced log classification system that combines three complementary approaches to handle varying levels of complexity in log patterns. This project provides a flexible and effective solution for processing predictable, complex, and poorly-labeled data patterns with real-time capabilities and analytics insights.

Features

Core Classification Methods

1. Regular Expression (Regex): Handles simplified and predictable patterns using predefined rules
2. Sentence Transformer + Logistic Regression: Manages complex patterns with sufficient training data using embeddings
3. LLM (Large Language Models): Handles complex patterns when labeled training data is insufficient using Groq API 4.Real-time Log Streaming: WebSocket support for live log classification as logs arrive 5.Analytics Dashboard: Comprehensive statistics, trends, and insights about classified logs
4. Confidence Scores: Get classification confidence levels for better decision-making

Setup Instructions

Prerequisites

• Python 3.8 or higher
• pip package manager

Installation

1. Clone the repository:

   git clone https://github.com/labdhiongithub7/log_classification.git
   cd log_classification

2. Create a virtual environment (recommended):

   python -m venv venv
   
   # On Windows
   venv\Scripts\activate
   
   # On Linux/Mac
   source venv/bin/activate

3. Install dependencies:

   pip install -r requirements.txt

4. Set up environment variables: Create a .env file in the root directory:

   GROQ_API_KEY=your_groq_api_key_here

   Get your API key from Groq Console

5. Download models (if needed): The BERT model will be downloaded automatically on first use. Ensure the models/ directory contains log_classifier.joblib.

Running the Server

Start the FastAPI server:

uvicorn server:app --reload

The server will be available at:

• Main endpoint: http://127.0.0.1:8000
• Interactive API docs: http://127.0.0.1:8000/docs
• Alternative docs: http://127.0.0.1:8000/redoc