2025 10 working branch #72

simonvanlierde · 2025-11-08T10:32:06Z

Backend:
- Fixed backup scripts
- Move to mjml email templates and fastapi-mail over custom smtp setup
- Using Redis for disposable email checking cache, can use later for session mgmt
- Fixed pydantic 2.12 compatibility issues
- Add order_by in products endpoint
- Linting and other small fixes
Frontend
- Fix registration flow
- Remove redundant hardcoded data (products.json, data.json)
- Fix issue where product cannot be saved even on valid product
- Add tooltip on save button showing validation issues

backend/app/api/auth/utils/programmatic_emails.py

Copilot

Pull Request Overview

This PR introduces significant infrastructure improvements and refactoring across the ReLab application stack:

Purpose: Add Redis caching support, improve email handling with MJML templates, enhance validation, update dependencies, and refactor database models for Pydantic 2.12+ compatibility.

Key Changes:

Infrastructure: Added Redis cache service with health checks and persistence
Email System: Migrated from plain text to MJML-compiled HTML templates with FastAPI-Mail integration
Dependencies: Updated Expo/Metro/React ecosystem, upgraded Pydantic constraints, added Redis/MJML libraries
Database: Fixed SQLModel relationship issues for Pydantic 2.12+ compatibility
Validation: Improved frontend user/product validation with better error messages

Reviewed Changes

Copilot reviewed 87 out of 92 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
renovate.json	Formatting changes and added `:preserveSemverRanges` preset
frontend-web/package.json	Updated expo-image from ~2.3.0 to ~2.4.0
frontend-web/package-lock.json	Dependency updates for Expo/Metro ecosystem and peer dependencies
frontend-app/package.json	Updated expo from 54.0.13 to 54.0.15
frontend-app/package-lock.json	Updated Expo dependencies and added yaml package
frontend-app/src/services/api/validation/*.ts	New validation utilities with structured error messages
frontend-app/src/components/product/ProductComponents.tsx	Updated to use new validation functions
frontend-app/src/app/products/[id]/index.tsx	Added tooltip for validation errors and useMemo
frontend-app/src/app/(auth)/new-account.tsx	Complete refactor with real-time validation and improved UX
compose.yml	Added Redis service with health checks and updated image digests
compose.prod.yml	Added cache volume persistence and updated backup compression
compose.override.yml	Exposed Redis port 6379 for development
backend/pyproject.toml	Added Redis, FastAPI-Mail, MJML; upgraded Pydantic/SQLModel
backend/app/core/config.py	Added Redis settings and converted passwords to SecretStr
backend/app/core/redis.py	New Redis connection management with graceful degradation
backend/app/main.py	Added lifespan manager for Redis and email checker initialization
backend/app/templates/emails/src/*.mjml	New MJML email templates for all email types
backend/app/templates/emails/build/*.html	Compiled HTML email templates
backend/tests/conftest.py	Added email testing fixtures and mock utilities
backend/tests/tests/emails/*.py	New comprehensive email tests
backend/scripts/seed/migrations_entrypoint.sh	Improved environment variable handling with lowercase helper
backend/scripts/create_superuser.py	Fixed to use SecretStr.get_secret_value()
backend/scripts/compile_email_templates.py	New script to compile MJML templates
backend/scripts/backup/*.sh	New backup scripts for PostgreSQL and user uploads with rsync/rclone support
backend/app/api/*/models.py	Added explicit relationship kwargs for Pydantic 2.12+ compatibility

Files not reviewed (2)

frontend-app/package-lock.json: Language not supported
frontend-web/package-lock.json: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

backend/app/api/common/schemas/custom_fields.py

backend/app/api/auth/utils/programmatic_emails.py

Copilot

Pull Request Overview

Copilot reviewed 87 out of 92 changed files in this pull request and generated 2 comments.

Files not reviewed (2)

frontend-app/package-lock.json: Language not supported
frontend-web/package-lock.json: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

backend/app/api/common/schemas/custom_fields.py

Copilot

Pull Request Overview

Copilot reviewed 87 out of 92 changed files in this pull request and generated 3 comments.

Files not reviewed (2)

frontend-app/package-lock.json: Language not supported
frontend-web/package-lock.json: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

backend/app/core/redis.py

backend/scripts/backup/backup_pg_database.sh

backend/scripts/seed/migrations_entrypoint.sh

simonvanlierde · 2025-11-17T12:41:11Z

@mrvisscher I just added a basic circularity_properties model (7e614f5), I think we can start with implementing this in the frontend.

…endpoint

…docker)

…idation status

… product get endpoint

…implify email setup

- Add CircularityPropertiesBase and CircularityProperties database models with fields for recyclability, repairability, and remanufacturability - Each property has observation (required), comment (optional), and reference (optional) fields - Add one-to-one relationship between Product and CircularityProperties - Create full CRUD operations for circularity properties - Add REST API endpoints (GET, POST, PATCH, DELETE) for managing circularity properties - Update product schemas to support circularity_properties in create/read/update operations - Update API documentation and examples to include circularity properties - Add alembic migration script for new circularity properties model

…eation

…erties Frontend app circularity properties

…igrations Docker image

backend/app/api/plugins/rpi_cam/routers/camera_interaction/utils.py

+        except RequestError as e:
+            # Network-level errors (DNS, connection refused, timeouts).
+            logger = logging.getLogger(__name__)
+            logger.warning("Network error contacting camera %s%s: %s", camera.url, endpoint, e)


To fix the log injection vulnerability, we must ensure any user-provided input logged is sanitized. For text logs, this typically involves stripping line breaks and carriage returns from the logged representation. Thus, in utils.py inside fetch_from_camera_url, before logging the network error with the endpoint, we should sanitize endpoint by removing or replacing newline (\n) and carriage return (\r) characters. The fix requires changing line 80 so that the logged value for endpoint contains only a single line—i.e., replacing \r and \n with empty strings. We do this inline before logging. Only code inside backend/app/api/plugins/rpi_cam/routers/camera_interaction/utils.py needs to be updated, on line 80 of the function.

…ed sqladmin interface

…s Patch update schema

…RM issues

Copilot AI review requested due to automatic review settings November 8, 2025 10:32

github-advanced-security bot found potential problems Nov 8, 2025

View reviewed changes

backend/app/api/auth/utils/programmatic_emails.py Fixed Show fixed Hide fixed

backend/app/api/auth/utils/programmatic_emails.py Fixed Show fixed Hide fixed

Copilot AI reviewed Nov 8, 2025

View reviewed changes

backend/app/api/common/schemas/custom_fields.py Outdated Show resolved Hide resolved

backend/app/api/common/schemas/custom_fields.py Outdated Show resolved Hide resolved

github-advanced-security bot found potential problems Nov 8, 2025

View reviewed changes

backend/app/api/auth/utils/programmatic_emails.py Fixed Show fixed Hide fixed

backend/app/api/auth/utils/programmatic_emails.py Fixed Show fixed Hide fixed

simonvanlierde requested review from Copilot and mrvisscher November 8, 2025 11:07

Copilot AI reviewed Nov 8, 2025

View reviewed changes

backend/app/api/common/schemas/custom_fields.py Outdated Show resolved Hide resolved

backend/app/api/common/schemas/custom_fields.py Outdated Show resolved Hide resolved

simonvanlierde requested a review from Copilot November 8, 2025 11:15

Copilot AI reviewed Nov 8, 2025

View reviewed changes

backend/app/core/redis.py Outdated Show resolved Hide resolved

backend/scripts/backup/backup_pg_database.sh Show resolved Hide resolved

backend/scripts/seed/migrations_entrypoint.sh Outdated Show resolved Hide resolved

simonvanlierde added 20 commits November 27, 2025 10:14

fix(cicd): update renovate.json to preserve semver ranges

f8a2b23

fix(backend): Include disposable email check in backend registration …

30d992b

…endpoint

fix(cicd): Fix versioning issues across the repo

6b7ca36

fix(backend): Don't seed dummy data by default

793b2a5

fix(backend): improve seeding scripts

eaf6b42

fix(docker): Fix postgres-backup-local config

6a59854

feat(backend): add manual pg backup script and rclone backup sync script

8774ecf

feat(backend): Print backup stats after rclone

c68890f

fix(backend): Use rclone symlink files for symlinked backups

f2e1eaf

fix(backend): Move username validation from model to schema

8755867

refactor(frontend-app): Deduplicate product name validation

4e7a0ac

chore(deps): update pre-commit yaml

93a938e

feat(frontend-app): update registration validation and error handling

5875e1e

fix(cicd): Format frontend instead of just linting

acdd1f0

chore: linting

4b5e80b

fix(backend): remove logging info

cd06b02

docs(backend): Clarify config setup (local .env file vs. env vars in …

8b6f233

…docker)

feature(frontend-app): Add tooltip to product save button showing val…

2fe6e36

…idation status

feature(backend): add order_by, created_at and updated_at filters for…

ac2ab05

… product get endpoint

feature(backend): Move to mjml email templates and fastapi-email to s…

d0f5247

…implify email setup

simonvanlierde added 11 commits November 27, 2025 10:21

fix(backend): Simplify URL serializing to DB

7a43515

fix(backend): Simplify dummy data checking logic

db2e6e5

fix(backend): Spellcheck

fbb34dd

fix(backend): Don't initialize email checker for on synthetic user cr…

d9d2edb

…eation

feature(backend): Allow superuser to remove any product

3b49519

feature(backend): Move from kg to g for physical_properties weight field

e66f286

feature: paginate users/me/products

0cd5def

fix(backend): make circularity properties optional

100039d

chore(backend): Update to python 3.14

71a680e

chore: linting

04715fe

simonvanlierde force-pushed the 2025-10_working_branch branch from 69a7725 to 04715fe Compare November 27, 2025 10:26

simonvanlierde added 7 commits November 27, 2025 10:27

feat(backend): Allow patching videos

5e69941

Merge pull request #80 from CMLPlatform/frontend-app-circularity-prop…

93a469e

…erties Frontend app circularity properties

chore: linting

89c64ce

chore(backend): Simplify circularity properties model

ba4017b

fix(docker): Update runtime stage python version to 3.14 in backend_m…

0223b0e

…igrations Docker image

fix(frontend-app): omit circularityproperties if all fields are null

671c006

fix(backend): Add some logging on rpi-cam related network errors

6cf7e9f

github-advanced-security bot found potential problems Nov 27, 2025

View reviewed changes

backend/app/api/plugins/rpi_cam/routers/camera_interaction/utils.py Fixed Show fixed Hide fixed

simonvanlierde added 4 commits November 27, 2025 15:09

fix(backend): Remove unused docker arg

f68a950

fix(backend): update ruff target python version

ef7deb5

fix(backend): Update python target version for ruff to 3.14

1ef6d7a

fix(backend): Move logging module

cbdfbc3

github-advanced-security bot found potential problems Dec 2, 2025

View reviewed changes

simonvanlierde added 5 commits December 2, 2025 11:01

fix(backend): Lint backend with ruff targeted to 3.14 and remove unus…

324ce8e

…ed sqladmin interface

fix(backend): Remove min length constraints from CircularityPropertie…

c4eb2be

…s Patch update schema

fix(backend): Hotfix for SQLmodel issues

88a6cbc

fix(backend): Revert to old-school sqlalchemy typing to fix runtime O…

d74517d

…RM issues

feat(backend): Add timeout and use-cookies vars to rclone backup script

c6665bd

@@ -77,7 +77,8 @@
                     except RequestError as e:
                         # Network-level errors (DNS, connection refused, timeouts).
                         logger = logging.getLogger(__name__)
-                        logger.warning("Network error contacting camera %s%s: %s", camera.url, endpoint, e)
+                        safe_endpoint = endpoint.replace("\r", "").replace("\n", "")
+                        logger.warning("Network error contacting camera %s%s: %s", camera.url, safe_endpoint, e)
                         raise HTTPException(
                             status_code=503,
                             detail={

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

2025 10 working branch #72

2025 10 working branch #72

Uh oh!

simonvanlierde commented Nov 8, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

simonvanlierde commented Nov 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

Check failure

Copilot Autofix

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

2025 10 working branch #72

Are you sure you want to change the base?

2025 10 working branch #72

Uh oh!

Conversation

simonvanlierde commented Nov 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

simonvanlierde commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Check failure

Uh oh!

Copilot Autofix

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

simonvanlierde commented Nov 8, 2025 •

edited

Loading

simonvanlierde commented Nov 17, 2025 •

edited

Loading