Fix Dockerization and backend

README.md

@@ -25,11 +25,13 @@ Explore the project Wiki for detailed documentation and guides: [emon_tools Wiki
 
     - [emon_api](https://github.com/vemonitor/emon_tools/blob/main/README.md#emon_api)
 
-3. [Running Tests](https://github.com/vemonitor/emon_tools/blob/main/README.md#running-tests)
-
-4. [Contributing](https://github.com/vemonitor/emon_tools/blob/main/README.md#contributing)
-
-5. [License](https://github.com/vemonitor/emon_tools/blob/main/README.md#license)
+3. [Optional Full-Stack Visualization App](#optional-full-stack-visualization-app)
+    - [Overview](#overview)
+    - [Local Development Setup](#local-development-setup)
+    - [Docker Compose Deployment](#docker-compose-deployment)
+4. [Running Tests](#running-tests)
+5. [Contributing](#contributing)
+6. [License](#license)
 
 ## Installation
 
@@ -44,10 +46,10 @@ The `emon-tools` package offers flexible installation options tailored to variou
 3. **Check Requirements**: Review the module's requirements in the `setup.cfg` file or on the PyPI page to ensure compatibility with your system.
 
 ### Global Installation
-To install the entire emon-tools package along with all dependencies, run the following command:
+To install the entire emon-tools package along with all dependencies—and to ensure you get the latest version—run:
 
 ```
-pip install emon-tools["all"]
+pip install emon-tools["all"] --upgrade
 ```
 
 Included Dependencies:
@@ -64,31 +66,31 @@ You can install specific modules and their dependencies as needed. For example:
 - To enable `emon_fina` module:
 
 ```
-pip install emon-tools["fina"]
+pip install emon-tools["fina"] --upgrade
 ```
 
 - To enable pandas time-series output functionality:
 
 ```
-pip install emon-tools["fina, time_series"]
+pip install emon-tools["fina, time_series"] --upgrade
 ```
 
 - To include graph plotting capabilities:
 
 ```
-pip install emon-tools["fina, plot"]
+pip install emon-tools["fina, plot"] --upgrade
 ```
 
 - To enable `emon_api` module:
 
 ```
-pip install emon-tools["api"]
+pip install emon-tools["api"] --upgrade
 ```
 
 - To enable `emon_fina` and `emon_api` modules:
 
 ```
-pip install emon-tools["api, fina"]
+pip install emon-tools["api, fina"] --upgrade
 ```
 
 ## Modules
@@ -166,6 +168,102 @@ print("Feeds: ", feeds)
 - **Wiki**: See `emon_api` module [wiki](https://github.com/vemonitor/emon_tools/wiki/emon_api) section.
 - **Examples**: Explore [api_bulk_structure](https://github.com/vemonitor/emon_tools/blob/main/examples/emon_api.py) for input and feed supervision, as well as posting bulk dummy data.
 
+## 🚀 Full-Stack Visualization App Deployment
+
+This optional full-stack application offers a graphical interface to visualize data from any EmonCMS instance and explore archived PhpFina file backups. It comprises:
+
+- FastAPI Backend: Exposes the `emon-tools` functionalities via a RESTful API.
+- Vite React Frontend: A modern dashboard built using React, TypeScript, and Tailwind CSS.
+
+### Overview
+
+The full-stack app enables users to:
+
+- Interactively browse and monitor live data from EmonCMS instances.
+- Visualize historical time-series data from archived PhpFina files.
+- Manage EmonCMS feeds and inputs via an intuitive web interface.
+
+### 🧰 Prerequisites
+
+Ensure the following are installed on your system:
+- Docker
+- Docker Compose
+
+### 📦 Deployment Steps
+This repository now includes a Docker Compose example that leverages the provided Dockerfiles for both backend and frontend. This approach is ideal for quickly deploying the full-stack app in a containerized environment.
+
+1. Clone the Repository
+
+```bash
+git clone https://github.com/vemonitor/emon_tools.git
+cd emon_tools
+```
+
+2. Set Up Environment Variables
+
+Navigate to the Docker Compose development directory and create a `.env` file by copying the provided example:
+
+```bash
+cd docker-compose\dev
+copy .example_env .env
+```
+
+Edit the .env file to replace placeholder values (changethis) with your actual configuration:
+
+```env
+MYSQL_HOST=your_mysql_host
+MYSQL_PORT=3306
+MYSQL_DB=your_database_name
+MYSQL_USER=your_username
+MYSQL_PASSWORD=your_password
+MYSQL_ROOT_PASSWORD=your_root_password
+```
+
+> Note: The `.env` file is utilized by Docker Compose for variable substitution in the `docker-compose.yml` file. Ensure all required variables are defined to prevent runtime errors. 
+
+3. Build containers and start the containers:
+
+From the `docker-compose/dev` directory, execute:
+
+```bash
+docker-compose up --build
+```
+
+This command builds the Docker images and starts the containers as defined in the `docker-compose.yml` file.
+
+4. Access the Application
+
+Once the containers are running:
+
+- Frontend: Access the React dashboard at http://localhost:3000
+- Backend: Access the FastAPI backend at http://localhost:8000
+    - Swagger UI: http://localhost:8000/docs
+    - ReDoc: http://localhost:8000/redoc
+
+### 🪟 Accessing from Windows Host (WSL2 Users)
+
+If you're running Docker within WSL2 and need to access the application from your Windows host:
+
+ 1. Determine the IP address of your WSL2 instance:
+
+    ```bash
+    ip addr show eth0 | grep inet
+    ```
+
+    Look for an output similar to:
+
+    ```ccp
+    inet 172.20.39.89/20 brd 172.20.47.255 scope global eth0
+    ```
+
+ 2. Use the extracted IP address to access the application from your Windows browser:
+
+ - Frontend: http://172.20.39.89:3000​
+
+ - Backend: http://172.20.39.89:8000
+
+ > Note: WSL2 has a separate network interface, so `localhost` on Windows does not directly map to `localhost` within WSL2. Using the WSL2 IP address bridges this gap. ​
+
 ## Running Tests
 
 To ensure everything is functioning correctly, run the test suite:

backend/.dockerignore

@@ -1,2 +1,3 @@
 Dockerfile
-/alembic/versions/*
+/alembic/versions/*
+.env*

backend/Dockerfile

@@ -43,9 +43,6 @@ RUN pip install --no-cache-dir --user --upgrade pip && \
 # Copy the backend source code
 COPY --chown=appuser:appuser . ./backend
 
-# Replace env file with the one for Docker
-COPY --chown=appuser:appuser .env.docker ./backend/.env
-
 RUN dos2unix /opt/emon_tools/backend/scripts/docker_start.sh
 RUN dos2unix /opt/emon_tools/backend/scripts/pre_start.sh
 RUN dos2unix /opt/emon_tools/backend/scripts/wait_for_db.sh

backend/alembic/script.py.mako

@@ -9,6 +9,7 @@ from typing import Sequence, Union
 
 from alembic import op
 import sqlalchemy as sa
+import sqlmodel
 ${imports if imports else ""}
 
 # revision identifiers, used by Alembic.

backend/core/config.py

@@ -195,35 +195,40 @@ def validate_mysql_port(cls, v: int) -> int:
             raise ValueError("MYSQL_PORT must be between 1 and 65535")
         return v
 
-    @classmethod
-    @field_validator("MYSQL_PASSWORD", mode="before")
-    def validate_mysql_password(cls, v: SecretStr) -> SecretStr:
+    @staticmethod
+    def _validate_password(field_name: str, secret: SecretStr) -> SecretStr:
         """
-        Validate the MYSQL_PASSWORD environment variable.
-        Ensure it meets the password policy.
+        Shared validator for password fields.
         """
-        raw = v.get_secret_value()
-        # Enforce a password policy similar to FIRST_SUPERUSER_PASSWORD
+        raw = secret.get_secret_value()
         if not ValidationConstants.PASSWORD_REGEX.match(raw):
             raise ValueError(
-                "MYSQL_PASSWORD must be at least 8 characters "
+                f"{field_name} must be at least 8 characters "
                 "and include at least one lowercase letter, "
                 "one uppercase letter, one digit, and one special character."
             )
-        return v
+        return secret
+
+    @classmethod
+    @field_validator("MYSQL_PASSWORD", mode="before")
+    def validate_mysql_password(cls, v: SecretStr) -> SecretStr:
+        """
+        Validate the MYSQL_PASSWORD environment variable.
+        Ensure it meets the password policy.
+        """
+        return cls._validate_password("MYSQL_PASSWORD", v)
 
-    # type: ignore[prop-decorator, C0103]
     @computed_field
     @property
     # Union[PostgresDsn, MySQLDsn]:
     def SQLALCHEMY_DATABASE_URI(self) -> MySQLDsn:
         """Set SqlAlchemy db url"""
-        encoded_password = quote_plus(
-            self.MYSQL_PASSWORD.get_secret_value()
-        )
+        validated_password = self._validate_password("MYSQL_PASSWORD", self.MYSQL_PASSWORD)
+        encoded_password = quote_plus(validated_password.get_secret_value())
         return (
             f"mysql+pymysql://{self.MYSQL_USER}:{encoded_password}@"
             f"{self.MYSQL_HOST}:{self.MYSQL_PORT}/{self.MYSQL_DB}"
+            f"{self.MYSQL_HOST}:{self.MYSQL_PORT}/{self.MYSQL_DB}"
         )
 
     @computed_field  # type: ignore[prop-decorator]

backend/scripts/docker_start.sh

@@ -2,7 +2,7 @@
 set -e
 
 # wait for the database service to be available
-./backend/scripts/wait_for_db.sh "${MYSQL_SERVER}" "${MYSQL_PORT}" "${MYSQL_USER}" "${MYSQL_PASSWORD}"
+./backend/scripts/wait_for_db.sh "${MYSQL_HOST}" "${MYSQL_PORT}" "${MYSQL_USER}" "${MYSQL_PASSWORD}"
 
 # Run migrations
 # Check if alembic/versions is empty

backend/utils/paths.py

@@ -1,9 +1,10 @@
+"""
+Utility functions for handling file paths in the backend.
+"""
 # backend/utils/paths.py
-
 from pathlib import Path
 from os.path import join as join_path
 
-# This assumes `backend/` is your top-level package
 ROOT_DIR = Path(__file__).resolve()
 
 

docker-compose/dev/.example_env

@@ -1,7 +1,36 @@
-# all those values must same as .env.docker
-MYSQL_HOST="changethis"
+# project envs
+# Domain
+# This would be set to the production domain with an env var on deployment
+# used by Traefik to transmit traffic and aqcuire TLS certificates
+DOMAIN="localhost"
+# To test the local Traefik config
+# DOMAIN=localhost.tiangolo.com
+
+# Environment: local, staging, production
+ENVIRONMENT="local"
+
+API_V1_STR="/api/v1"
+PROJECT_NAME="EmonTools"
+STACK_NAME="full-stack-emontools-project"
+
+# Data Path
+DATA_BASE_PATH="/opt/emon_tools/data"
+STATIC_BASE_PATH="/opt/emon_tools/static"
+
+# Backend
+# Used by the backend to generate links in emails to the frontend
+FRONTEND_HOST="http://localhost:5173"
+# In staging and production, set this env var to the frontend host, e.g.
+# FRONTEND_HOST=https://dashboard.example.com
+BACKEND_CORS_ORIGINS="http://localhost,http://localhost:5173,https://localhost,https://localhost:5173"
+
+SECRET_KEY=changethis
+FIRST_SUPERUSER=changethis
+FIRST_SUPERUSER_PASSWORD=changethis
+
+# Mysql
+MYSQL_HOST="127.0.0.1"
 MYSQL_PORT=3306
-MYSQL_DB="changethis"
-MYSQL_USER="changethis"
-MYSQL_PASSWORD="changethis"
-MYSQL_ROOT_PASSWORD="changethis"
+MYSQL_DB="emontools"
+MYSQL_USER="emontools"
+MYSQL_PASSWORD=changethis

docker-compose/dev/docker-compose.yml

@@ -17,6 +17,8 @@ services:
     build:
       context: ../../backend
       dockerfile: Dockerfile
+    env_file:
+      - ./.env
     volumes:
       - ../../backend:/opt/emon_tools/backend
       - ./alembic:/opt/emon_tools/backend/alembic/versions