Skip to content

Feat/automate unified doc creation#3

Merged
RepoRover merged 6 commits intodevfrom
feat/automate-unified-doc-creation
Aug 18, 2025
Merged

Feat/automate unified doc creation#3
RepoRover merged 6 commits intodevfrom
feat/automate-unified-doc-creation

Conversation

@Matheus-OAMK
Copy link
Contributor

This pull request introduces the initial setup for the Database Schema Spec Generator project, establishing a robust Python package structure, development tooling, and CI/CD workflows. The changes include a complete project scaffold with configuration management, command-line interface, core logic, and supporting documentation. Additionally, it sets up automated testing, linting, formatting, and type checking using modern Python tools.

Project Structure and Core Implementation:

  • Introduced a modular Python package layout under database_schema_spec/ with clear separation of CLI, core logic, I/O, logging, schema resolution, and validation modules. The main orchestration is handled by the SchemaGenerator class, which coordinates schema extraction, conditional logic, validation, and output generation. [1] [2] [3] [4]
  • Implemented robust configuration management using Pydantic models in core/config.py, supporting environment variable overrides and custom error handling for missing required configuration (notably BASE_URL).
  • Defined custom exception classes for all major error conditions in the schema generation process, improving error reporting and maintainability.

Development Tooling and Automation:

  • Added a comprehensive pre-commit configuration for dependency locking, linting, formatting, import sorting, and type checking using uv, ruff, and pyright.
  • Established a reusable GitHub Actions workflow for CI, covering dependency installation, linting, formatting, type checking, and running tests on PRs and key branches.
  • Created a composite GitHub Action for setting up Python and uv to standardize the development environment.

Documentation and Environment Configuration:

  • Revamped the README.md to provide clear instructions for setup, running, development, and output expectations. Added detailed environment variable and configuration documentation.
  • Added .env.example with documentation for the required BASE_URL configuration.
  • Specified Python 3.13 as the required version in .python-version.

Miscellaneous:

  • Added __init__.py files to ensure correct package discovery and testability.

Most Important Changes:

1. Core Package and Application Logic

  • Created a modular package structure in database_schema_spec/ with a central SchemaGenerator class to orchestrate schema extraction, resolution, conditional logic, validation, and output. [1] [2] [3] [4]
  • Implemented configuration management using Pydantic in core/config.py, requiring a BASE_URL environment variable and supporting robust error handling for missing configuration.
  • Added comprehensive custom exception classes for all major error scenarios in core/exceptions.py.

2. Development and CI Tooling

  • Added .pre-commit-config.yaml to enable pre-commit hooks for dependency locking, linting, formatting, import sorting, and type checking.
  • Established a reusable GitHub Actions CI workflow for dependency installation, linting, formatting, type checking, and testing.
  • Added a composite GitHub Action for setting up Python and uv for consistent local and CI environments.

3. Documentation and Environment

  • Overhauled README.md with detailed setup, usage, development, and output instructions.
  • Added .env.example to document required environment variables, and .python-version to pin the Python version. [1] [2]

4. Miscellaneous

  • Added __init__.py files to ensure package discovery and testability.

…documents

- Removed `$schema` and `$id` from nested .json documents
- Fixed the reference inside the oneOf block for PSQL v15.0 in the file `specs.json` to point to the correct `spec.json`
…nal resolution

Implement a comprehensive database schema specification generator that processes JSON Schema files with conditional oneOf logic for different database engines and versions.

### Core Features

**Schema Resolution System:**
- Add JSONRefResolver for handling $ref references with circular detection
- Implement ConditionalMerger for oneOf conditional logic resolution
- Support both if/then and direct property constraint formats
- Add VariantExtractor for database variant discovery from oneOf blocks

**Validation Framework:**
- Add SchemaValidator with JSON Schema Draft 7 compliance checking
- Implement custom validation rules for project-specific requirements
- Include validation for required fields and schema structure

**Logging Infrastructure:**
- Add centralized logging configuration with JSON-based setup
- Support both console and file handlers with detailed formatting
- Include queue-based logging for performance

**CLI Generator:**
- Implement SchemaGenerator for end-to-end schema processing
- Support multiple database variants (PostgreSQL, MySQL)
- Generate resolved schemas without oneOf conditionals
…t/vmap.json`

- Introduced new functionality to generate a .json file that maps all available engines and their corresponndiing versions to an url. The map is saved in the output folder in a singleton file.
- Introduced a new configuration module to manage application settings.
- Added support for loading configuration from a .env file.
- Updated the schema generator and output manager to utilize the new config.
- Created example .env file and updated README with configuration instructions.
- Implemented tests for configuration handling and output manager functionality.
@Matheus-OAMK Matheus-OAMK self-assigned this Aug 6, 2025
@Matheus-OAMK Matheus-OAMK linked an issue Aug 6, 2025 that may be closed by this pull request
@RepoRover RepoRover requested review from RepoRover and Copilot August 15, 2025 16:37
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces the initial setup for the Database Schema Spec Generator project, establishing a complete Python package structure with configuration management, CLI interface, core schema processing logic, and supporting development tooling. The changes create a robust foundation for generating unified JSON Schema documentation from modular database specifications.

Key Changes

  • Complete project scaffold: Modular Python package with CLI, core logic modules for schema resolution, validation, and output management
  • Configuration and environment management: Pydantic-based configuration with required BASE_URL environment variable and comprehensive error handling
  • Development tooling and CI/CD: Pre-commit hooks, GitHub Actions workflows, and automated testing/linting setup using modern Python tools

Reviewed Changes

Copilot reviewed 47 out of 52 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
database_schema_spec/ Core package implementation with CLI, resolution, validation, and I/O modules
tests/ Comprehensive test suite with unit, integration, and production-quality tests
pyproject.toml Project configuration with dependencies and development tooling setup
main.py Application entry point
.github/ CI/CD workflows and reusable setup actions
Configuration files Pre-commit hooks, environment examples, and Python version specification

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

line-length = 88
indent-width = 4

# Assume Python 3.9
Copy link

Copilot AI Aug 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The target-version should be 'py313' but the comment on line 59 says 'Assume Python 3.9'. This inconsistency should be corrected to match the actual Python version requirement.

Suggested change
# Assume Python 3.9
# Assume Python 3.13

Copilot uses AI. Check for mistakes.
Returns:
True if condition matches, False otherwise
"""
# Look for database properties in the if condition
Copy link

Copilot AI Aug 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment mentions 'database properties' but the logic also handles direct property constraints. The comment should clarify that it handles both nested database properties and direct engine/version properties.

Suggested change
# Look for database properties in the if condition
# Check for both nested database properties and direct property constraints in the if condition

Copilot uses AI. Check for mistakes.
Copy link
Contributor

@RepoRover RepoRover left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that I have lost my ability to uderstand python after all... but as far as I'm concerned, the code works and very well! Code will prove itself in practice too.

@RepoRover RepoRover merged commit 4217530 into dev Aug 18, 2025
5 checks passed
@RepoRover RepoRover deleted the feat/automate-unified-doc-creation branch August 18, 2025 07:16
RepoRover added a commit that referenced this pull request Sep 2, 2025
* Feat/automate unified doc creation (#3)

* refactor: Move all files located in the root to a dedicated  folder

* chore: Setup development tools and packages

* refactor/fix: Remove redundant metadata and fix reference from .json documents

- Removed `$schema` and `$id` from nested .json documents
- Fixed the reference inside the oneOf block for PSQL v15.0 in the file `specs.json` to point to the correct `spec.json`

* feat: implement database schema specification generator with conditional resolution

Implement a comprehensive database schema specification generator that processes JSON Schema files with conditional oneOf logic for different database engines and versions.

### Core Features

**Schema Resolution System:**
- Add JSONRefResolver for handling $ref references with circular detection
- Implement ConditionalMerger for oneOf conditional logic resolution
- Support both if/then and direct property constraint formats
- Add VariantExtractor for database variant discovery from oneOf blocks

**Validation Framework:**
- Add SchemaValidator with JSON Schema Draft 7 compliance checking
- Implement custom validation rules for project-specific requirements
- Include validation for required fields and schema structure

**Logging Infrastructure:**
- Add centralized logging configuration with JSON-based setup
- Support both console and file handlers with detailed formatting
- Include queue-based logging for performance

**CLI Generator:**
- Implement SchemaGenerator for end-to-end schema processing
- Support multiple database variants (PostgreSQL, MySQL)
- Generate resolved schemas without oneOf conditionals

* feat: Add generation of all available files in .json format to `output/vmap.json`

- Introduced new functionality to generate a .json file that maps all available engines and their corresponndiing versions to an url. The map is saved in the output folder in a singleton file.
- Introduced a new configuration module to manage application settings.
- Added support for loading configuration from a .env file.
- Updated the schema generator and output manager to utilize the new config.
- Created example .env file and updated README with configuration instructions.
- Implemented tests for configuration handling and output manager functionality.

* feat: Add CI workflow and setup actions for Python and uv

* Feature/versioned releases (#6)

* fix: enforce required environment variables in configuration & pass them into generated schemas dynamically

* fix: reorder top-level keys in generated schemas to prioritize $id and $schema fields

* fix: moved description field from snapshot to schema definition

* fix: enable overriding of environment variables when loading .env file

* feat: add Dockerfile, .dockerignore for schema generation and workflow to create images

* feat: added workflow to create production images & cleaned workflow to create staging images

* refactor(config): Remove unecessary python-dotenv dependency

- Removed python-dotenv dependency and lines utilizing it. The pydantic_settings managed the environment and should fail on missing dependencies.

* refactor(Dockerfile): Pin UV version and simplify building stage

- Refactiored the dockerfile to have a pinned version of UV for consistent builds.
- Simplified dockerfile by utilizing UV's prebuilt image

* refactor(config): Remove leftover comments

---------

Co-authored-by: Matheus-OAMK <t2fema00@students.oamk.fi>

* feat: consolidate CI/CD workflows by creating service version images and removing staging image workflow (#9)

* feat: consolidate CI/CD workflows by creating service version images and removing staging image workflow

* feat: remove staging branch from CI workflow triggers

---------

Co-authored-by: Matheus Oliveira <t2fema00@students.oamk.fi>
RepoRover added a commit that referenced this pull request Dec 10, 2025
* Feat/automate unified doc creation (#3)

* refactor: Move all files located in the root to a dedicated  folder

* chore: Setup development tools and packages

* refactor/fix: Remove redundant metadata and fix reference from .json documents

- Removed `$schema` and `$id` from nested .json documents
- Fixed the reference inside the oneOf block for PSQL v15.0 in the file `specs.json` to point to the correct `spec.json`

* feat: implement database schema specification generator with conditional resolution

Implement a comprehensive database schema specification generator that processes JSON Schema files with conditional oneOf logic for different database engines and versions.

### Core Features

**Schema Resolution System:**
- Add JSONRefResolver for handling $ref references with circular detection
- Implement ConditionalMerger for oneOf conditional logic resolution
- Support both if/then and direct property constraint formats
- Add VariantExtractor for database variant discovery from oneOf blocks

**Validation Framework:**
- Add SchemaValidator with JSON Schema Draft 7 compliance checking
- Implement custom validation rules for project-specific requirements
- Include validation for required fields and schema structure

**Logging Infrastructure:**
- Add centralized logging configuration with JSON-based setup
- Support both console and file handlers with detailed formatting
- Include queue-based logging for performance

**CLI Generator:**
- Implement SchemaGenerator for end-to-end schema processing
- Support multiple database variants (PostgreSQL, MySQL)
- Generate resolved schemas without oneOf conditionals

* feat: Add generation of all available files in .json format to `output/vmap.json`

- Introduced new functionality to generate a .json file that maps all available engines and their corresponndiing versions to an url. The map is saved in the output folder in a singleton file.
- Introduced a new configuration module to manage application settings.
- Added support for loading configuration from a .env file.
- Updated the schema generator and output manager to utilize the new config.
- Created example .env file and updated README with configuration instructions.
- Implemented tests for configuration handling and output manager functionality.

* feat: Add CI workflow and setup actions for Python and uv

* Feature/versioned releases (#6)

* fix: enforce required environment variables in configuration & pass them into generated schemas dynamically

* fix: reorder top-level keys in generated schemas to prioritize $id and $schema fields

* fix: moved description field from snapshot to schema definition

* fix: enable overriding of environment variables when loading .env file

* feat: add Dockerfile, .dockerignore for schema generation and workflow to create images

* feat: added workflow to create production images & cleaned workflow to create staging images

* refactor(config): Remove unecessary python-dotenv dependency

- Removed python-dotenv dependency and lines utilizing it. The pydantic_settings managed the environment and should fail on missing dependencies.

* refactor(Dockerfile): Pin UV version and simplify building stage

- Refactiored the dockerfile to have a pinned version of UV for consistent builds.
- Simplified dockerfile by utilizing UV's prebuilt image

* refactor(config): Remove leftover comments

---------

Co-authored-by: Matheus-OAMK <t2fema00@students.oamk.fi>

* feat: consolidate CI/CD workflows by creating service version images and removing staging image workflow (#9)

* feat: consolidate CI/CD workflows by creating service version images and removing staging image workflow

* feat: remove staging branch from CI workflow triggers

---------

Co-authored-by: Matheus Oliveira <t2fema00@students.oamk.fi>
RepoRover added a commit that referenced this pull request Dec 10, 2025
…djusted the schema spec generation logic

* Feat/automate unified doc creation (#3)

* refactor: Move all files located in the root to a dedicated  folder

* chore: Setup development tools and packages

* refactor/fix: Remove redundant metadata and fix reference from .json documents

- Removed `$schema` and `$id` from nested .json documents
- Fixed the reference inside the oneOf block for PSQL v15.0 in the file `specs.json` to point to the correct `spec.json`

* feat: implement database schema specification generator with conditional resolution

Implement a comprehensive database schema specification generator that processes JSON Schema files with conditional oneOf logic for different database engines and versions.

### Core Features

**Schema Resolution System:**
- Add JSONRefResolver for handling $ref references with circular detection
- Implement ConditionalMerger for oneOf conditional logic resolution
- Support both if/then and direct property constraint formats
- Add VariantExtractor for database variant discovery from oneOf blocks

**Validation Framework:**
- Add SchemaValidator with JSON Schema Draft 7 compliance checking
- Implement custom validation rules for project-specific requirements
- Include validation for required fields and schema structure

**Logging Infrastructure:**
- Add centralized logging configuration with JSON-based setup
- Support both console and file handlers with detailed formatting
- Include queue-based logging for performance

**CLI Generator:**
- Implement SchemaGenerator for end-to-end schema processing
- Support multiple database variants (PostgreSQL, MySQL)
- Generate resolved schemas without oneOf conditionals

* feat: Add generation of all available files in .json format to `output/vmap.json`

- Introduced new functionality to generate a .json file that maps all available engines and their corresponndiing versions to an url. The map is saved in the output folder in a singleton file.
- Introduced a new configuration module to manage application settings.
- Added support for loading configuration from a .env file.
- Updated the schema generator and output manager to utilize the new config.
- Created example .env file and updated README with configuration instructions.
- Implemented tests for configuration handling and output manager functionality.

* feat: Add CI workflow and setup actions for Python and uv

* Feature/versioned releases (#6)

* fix: enforce required environment variables in configuration & pass them into generated schemas dynamically

* fix: reorder top-level keys in generated schemas to prioritize $id and $schema fields

* fix: moved description field from snapshot to schema definition

* fix: enable overriding of environment variables when loading .env file

* feat: add Dockerfile, .dockerignore for schema generation and workflow to create images

* feat: added workflow to create production images & cleaned workflow to create staging images

* refactor(config): Remove unecessary python-dotenv dependency

- Removed python-dotenv dependency and lines utilizing it. The pydantic_settings managed the environment and should fail on missing dependencies.

* refactor(Dockerfile): Pin UV version and simplify building stage

- Refactiored the dockerfile to have a pinned version of UV for consistent builds.
- Simplified dockerfile by utilizing UV's prebuilt image

* refactor(config): Remove leftover comments

---------



* feat: consolidate CI/CD workflows by creating service version images and removing staging image workflow (#9)

* feat: consolidate CI/CD workflows by creating service version images and removing staging image workflow

* feat: remove staging branch from CI workflow triggers

---------

Co-authored-by: Matheus Oliveira <t2fema00@students.oamk.fi>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Chores] Automate Document Generation from JSON Schema Files

2 participants