MAILGENIE

1. Project Overview

MAILGENIE is a feature-rich, self-hosted email management system designed to act as a personal unified inbox. It connects to multiple IMAP accounts, synchronizes emails in real-time, and provides a powerful interface for searching, filtering, and managing your emails. The system leverages AI for intelligent email categorization and smart replies, and it is built as a full-stack application with a Node.js backend and a React frontend.

2. Features

Backend

• User Authentication: Secure user signup, login, and logout using JWTs.
• IMAP Account Management: Add, view, update, and delete IMAP account configurations.
• Real-time Email Synchronization: Connects to multiple IMAP accounts and uses the IDLE command to listen for new emails in real-time. It fetches the last 30 days of emails on initial sync.
• AI-Powered Email Categorization: Uses the Google Gemini API to automatically categorize emails into Interested, Meeting Booked, Not Interested, Spam, or Out of Office.
• AI-Powered Smart Replies: Generates context-aware smart replies using a Retrieval-Augmented Generation (RAG) system with a Faiss vector store.
• Searchable Email Storage: Indexes all emails in a locally hosted Elasticsearch instance for powerful full-text search and filtering by account, folder, or category.
• Automated Notifications: Sends Slack notifications and triggers external webhooks for new emails categorized as "Interested".

Frontend

• Unified Email Dashboard: Displays an aggregated list of emails with their subject, sender, and AI-generated category.
• Advanced Filtering and Searching: Allows users to filter emails by category and perform full-text searches across all accounts.
• AI Reply Generation: A "Generate Reply" button provides AI-suggested replies for any email.
• Account Management: A settings page to view and manage configured email accounts.

3. Tech Stack

Area	Technology
Backend	Node.js, TypeScript, Express.js, PostgreSQL, Drizzle ORM, JWT, imapflow
Frontend	React, TypeScript, Chakra UI, react-query, axios, Vite
DevOps	Docker, Docker Compose

4. Getting Started

Prerequisites

• Node.js (v18 or later)
• Docker and Docker Compose
• npm

Setup Instructions

1. Clone the repository:

   git clone <your-repository-url>
   cd MailGenie

2. Install dependencies:

   npm install

3. Set up environment variables:

   This project uses a .env file for environment variables. You can create your own by copying the example file:

   cp .env.example .env

   Then, update the newly created .env file with your specific configurations for DATABASE_URL, ELASTICSEARCH_URL, ELASTICSEARCH_USERNAME, ELASTICSEARCH_PASSWORD, GEMINI_API_KEY, SLACK_WEBHOOK_URL, and WEBHOOK_URL. Refer to .env.example for the list of required variables.

4. Start the database and Elasticsearch:

   Run the following command to start the PostgreSQL and Elasticsearch containers in the background:

   docker compose up -d

5. Run database migrations:

   Apply the database schema to your PostgreSQL instance:

   npm run db:generate
   npm run db:push

6. Run the application:

   Start the backend server in development mode:

   npm run dev

   The server will be running at http://localhost:3000.

5. API Endpoints

Here are some of the available API endpoints:

Authentication

• POST /api/v1/auth/signup: Create a new user.
• POST /api/v1/auth/login: Log in a user.
• POST /api/v1/auth/logout: Log out a user.

IMAP Accounts

• POST /api/v1/imap: Add a new IMAP account.
• GET /api/v1/imap: Get all IMAP accounts for the user.
• GET /api/v1/imap/:id: Get a specific IMAP account.
• PUT /api/v1/imap/:id: Update an IMAP account.
• DELETE /api/v1/imap/:id: Delete an IMAP account.

Emails

• GET /api/v1/emails: Get all emails for the logged-in user. This endpoint also supports search and filtering via query parameters:

  • q: A string for full-text search across email subjects, bodies, and senders.
  • folder: Filter emails by a specific folder (e.g., INBOX).
  • accountId: Filter emails by a specific IMAP account ID.

  Examples:

  • GET /api/v1/emails?q=meeting
  • GET /api/v1/emails?folder=INBOX&accountId=1
  • GET /api/v1/emails?q=urgent&folder=Sent

6. IMAP Account Management

Working and Synchronization

MailGenie provides robust IMAP account management, allowing you to connect multiple email accounts to a single, unified inbox. Here’s how it works:

1. Adding an Account: When you add a new IMAP account, MailGenie securely stores your credentials and establishes a connection to the IMAP server. It then performs an initial synchronization by fetching all emails from the last 30 days.

2. Real-time Updates: After the initial sync, MailGenie uses the IDLE command to listen for new emails in real-time. This ensures that your unified inbox is always up-to-date without constantly polling the server.

3. Email Processing: As new emails arrive, they are processed and indexed in Elasticsearch, making them available for full-text search. The AI-powered categorization and smart reply features are also applied at this stage.

Updating an Account

You can update your IMAP account settings at any time. When you modify your credentials, MailGenie will re-authenticate and re-establish the connection to ensure uninterrupted service.

7. Concurrency for Multiple IMAP Accounts

Can we run multiple threads for multiple IMAP accounts?

Yes, it is possible to run multiple threads (or worker processes) to handle multiple IMAP accounts concurrently. In a Node.js environment, this can be achieved using worker threads or by forking child processes.

Benefits

• Improved Performance: Concurrent processing would allow MailGenie to fetch and synchronize emails from multiple accounts simultaneously, leading to faster updates and a more responsive user experience.

• Enhanced Scalability: By isolating each IMAP connection in its own thread, the system can better handle a growing number of accounts without performance degradation.

Challenges

• Increased Complexity: Managing multiple threads adds complexity to the codebase, particularly concerning state management, error handling, and resource allocation.

• Resource Consumption: Each thread consumes additional memory and CPU resources. A poorly managed multi-threaded system could lead to higher operational costs and potential stability issues.

While MailGenie currently processes IMAP accounts sequentially, a future update could introduce a multi-threaded approach to further enhance its performance and scalability.