Statement Logging #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

TobiasHafner merged 11 commits into main from dev

Sep 2, 2025

docs/application.md

-Original file line number
+Diff line change
@@ Expand Up / @@ -29,7 +29,8 @@ Application( @@
         use_docker: bool = False,
         migrate: bool = False,
         stop_container: bool = False,
-        remove_container: bool = False
+        remove_container: bool = False,
+        log_statements: bool = False
     )
     ```
@@ Expand Down Expand Up / @@ -60,6 +61,9 @@ Application( @@
     - `remove_container` (`bool`, optional):
       If `True`, removes the Docker container running the Polypheny instance after stopping. Defaults to `False`.
+    - `remove_container` (`bool`, optional):
+      If `True`, all data- or schema modifying statements executed by this application are logged. Defaults to `False`.
     ## Examples
     ```python
     from polynom.application import Application
@@ Expand Down @@

docs/config.md

-Original file line number
+Diff line change
@@ Expand Up @@
     - `DEFAULT_PASS`:
       Password used for default authentication. Default: `''` (empty string)
+    - `STATEMENT_LOG_FILE_NAME`:
+      Name of the statement log file. Default: `statements.log`
     ### Derived Configuration
     - `CHANGE_LOG_IDENTIFIER`:
@@ Expand Down @@

docs/dump.md

-Original file line number
+Diff line change
@@ -0,0 +1,62 @@
+    ---
+    layout: page
+    title: "Dump"
+    toc: true
+    docs_area: "PolyNOM"
+    tags: backup, dump, load, import
+    lang: en
+    ---
+    # Dump
+    The database state of a PolyNOM application can be persisted into a multi-language query (MLQ) file. While this procedure is explained in detail in the application documentation, this page takes a closer look at the MLQ file generated.
+    ## Structure
+    Every MLQ file is split into two sections. The first section is the header containing meta-information about the file. The second section contains the statements in one or more query languages. MLQ is a format extending traditional SQL dumps. MLQ focuses on reverse compatibility with traditional SQL dumps in cases where the only query language used in the dump is SQL. Therefore, all extensions are wrapped in comments to be ignored by traditional systems.
+    ## Header
+    The header of an MLQ file is a multiline comment containing three key-value pairs prefixed with an `@` symbol.
+    - **@format_version**: Denotes the format used for the MLQ dump. As of now, the corresponding value is always `1`. If the MLQ format is extended in the future, this number will be increased to enable differentiation and proper parsing.
+    - **@app_uuid**: The application UUID of the application from which the dump originates. This ensures that dumps are only loaded by the appropriate application.
+    - **@snapshot**: A JSON string representing a list of all schemas defined by the application from which the dump originated. This can be used to easily check for schema changes. It is used on load to verify that the schema to be created by the dump actually matches the expectations of the application loading the dump.
+    ## Statements
+    The header is followed by a series of statements in one or more query languages. Each line begins with a comment of the form:
+    ```sql
+    /*language@namespace*/
+    ```
+    - `language` specifies the query language used for the following statement.
+    - `namespace` specifies in which namespace the statement should be executed.
+    ## Example
+    Below an example of a short application dump as an MLQ is shown:
+    ```sql
+    /*
+    @format_version: 1
+    @app_uuid: a8817239-9bae-4961-a619-1e9ef5575eff
+    @snapshot: {"version": "20250902T084338", "schemas": [{"entity_name": "User", "namespace_name": "polynom_entities", "data_model": "RELATIONAL", "fields": [{"name": "_entry_id", "db_name": "_entry_id", "type": "VarChar", "previous_name": null, "nullable": false, "unique": true, "default": null, "is_primary_key": true, "is_foreign_key": false}, {"name": "username", "db_name": "username", "type": "VarChar", "previous_name": "username2", "nullable": false, "unique": true, "default": null, "is_primary_key": false, "is_foreign_key": false}, {"name": "email", "db_name": "email", "type": "VarChar", "previous_name": null, "nullable": false, "unique": true, "default": null, "is_primary_key": false, "is_foreign_key": false}, {"name": "first_name", "db_name": "first_name", "type": "VarChar", "previous_name": null, "nullable": true, "unique": false, "default": null, "is_primary_key": false, "is_foreign_key": false}, {"name": "last_name", "db_name": "last_name", "type": "VarChar", "previous_name": null, "nullable": true, "unique": false, "default": null, "is_primary_key": false, "is_foreign_key": false}, {"name": "active", "db_name": "active", "type": "Boolean", "previous_name": null, "nullable": true, "unique": false, "default": null, "is_primary_key": false, "is_foreign_key": false}, {"name": "is_admin", "db_name": "is_admin", "type": "Boolean", "previous_name": null, "nullable": true, "unique": false, "default": null, "is_primary_key": false, "is_foreign_key": false}]}, {"entity_name": "Bike", "namespace_name": "polynom_entities", "data_model": "RELATIONAL", "fields": [{"name": "_entry_id", "db_name": "_entry_id", "type": "VarChar", "previous_name": null, "nullable": false, "unique": true, "default": null, "is_primary_key": true, "is_foreign_key": false}, {"name": "brand", "db_name": "brand", "type": "VarChar", "previous_name": null, "nullable": false, "unique": false, "default": null, "is_primary_key": false, "is_foreign_key": false}, {"name": "model", "db_name": "model", "type": "VarChar", "previous_name": null, "nullable": false, "unique": false, "default": null, "is_primary_key": false, "is_foreign_key": false}, {"name": "owner_id", "db_name": "owner_id", "type": "VarChar", "previous_name": null, "nullable": false, "unique": null, "default": false, "is_primary_key": false, "is_foreign_key": true, "references_namespace": "polynom_entities", "references_entity": "User", "references_field": "_entry_id"}]}]}
+    */
+    /*sql@None*/ CREATE RELATIONAL NAMESPACE IF NOT EXISTS "polynom_internal"
+    /*sql@None*/ CREATE RELATIONAL NAMESPACE IF NOT EXISTS "polynom_entities"
+    /*sql@polynom_entities*/ CREATE TABLE IF NOT EXISTS "polynom_entities"."User" ("_entry_id" VARCHAR(36) NOT NULL, "username" VARCHAR(80) NOT NULL, "email" VARCHAR(80) NOT NULL, "first_name" VARCHAR(30), "last_name" VARCHAR(30), "active" BOOLEAN, "is_admin" BOOLEAN, PRIMARY KEY (_entry_id), UNIQUE ("_entry_id"), UNIQUE ("username"), UNIQUE ("email"));
+    /*sql@polynom_entities*/ INSERT INTO "polynom_entities"."User" (_entry_id, username, email, first_name, last_name, active, is_admin) VALUES ('93a8779e-3c40-4baf-a7a5-27765effd6f8', 'testuser', 'u1@demo.ch', 'max', 'muster', TRUE, FALSE)
+    /*sql@polynom_entities*/ INSERT INTO "polynom_entities"."User" (_entry_id, username, email, first_name, last_name, active, is_admin) VALUES ('1c87088a-7a23-47e5-be1d-6891b097138f', 'testuser2', 'u2@demo.ch', 'mira', 'muster', FALSE, TRUE)
+    /*sql@polynom_entities*/ INSERT INTO "polynom_entities"."User" (_entry_id, username, email, first_name, last_name, active, is_admin) VALUES ('0a6a071a-8fc3-470d-bec1-bb9adf7a0015', 'testuser3', 'u3@demo.ch', 'miraculix', 'musterin', FALSE, TRUE)
+    /*sql@polynom_entities*/ INSERT INTO "polynom_entities"."User" (_entry_id, username, email, first_name, last_name, active, is_admin) VALUES ('a09435c6-6037-4d8e-aba0-8d2d67063945', 'testuser4', 'u4@demo.ch', 'maxine', 'meier', TRUE, FALSE)
+    /*sql@polynom_entities*/ INSERT INTO "polynom_entities"."User" (_entry_id, username, email, first_name, last_name, active, is_admin) VALUES ('d63a89b6-f339-4e5e-955a-4d36ad5edb37', 'testuser5', 'u5@demo.ch', 'mia', 'müller', FALSE, FALSE)
+    /*sql@polynom_entities*/ CREATE TABLE IF NOT EXISTS "polynom_entities"."Bike" ("_entry_id" VARCHAR(36) NOT NULL, "brand" VARCHAR(50) NOT NULL, "model" VARCHAR(50) NOT NULL, "owner_id" VARCHAR(36) NOT NULL DEFAULT 'False', FOREIGN KEY ("owner_id") REFERENCES "polynom_entities"."User"("_entry_id"), PRIMARY KEY (_entry_id), UNIQUE ("_entry_id"));
+    /*sql@polynom_entities*/ INSERT INTO "polynom_entities"."Bike" (_entry_id, brand, model, owner_id) VALUES ('d48d23ac-a308-45f0-9250-86fdde8d58dd', 'Trek', 'Marlin 7', '93a8779e-3c40-4baf-a7a5-27765effd6f8')
+    /*sql@polynom_entities*/ INSERT INTO "polynom_entities"."Bike" (_entry_id, brand, model, owner_id) VALUES ('ba13bb9a-40cc-437a-8b66-8a8b00bb5f6d', 'Specialized', 'Rockhopper', '93a8779e-3c40-4baf-a7a5-27765effd6f8')
+    /*sql@polynom_entities*/ INSERT INTO "polynom_entities"."Bike" (_entry_id, brand, model, owner_id) VALUES ('040ecfde-db16-4a3f-bb6a-4e93331b0925', 'Cannondale', 'Trail 8', '0a6a071a-8fc3-470d-bec1-bb9adf7a0015')
+    /*sql@polynom_entities*/ INSERT INTO "polynom_entities"."Bike" (_entry_id, brand, model, owner_id) VALUES ('7c36a17b-deb8-45e5-91da-0c33943504ec', 'Giant', 'Talon 3', 'a09435c6-6037-4d8e-aba0-8d2d67063945')
+    ```

docs/models.md

-Original file line number
+Diff line change
@@ Expand Up @@
     - **Python type**: `bytes`
     - **Polytype**: `File`
+    ## Flex Model
+    The `Flex Model` is a special model class to be used if the schema or the model class of an entry is only known at runtime and thus a static implementation is not possible.
+    This situation might arise when working with data from data models that do not use fixed schemas such as the document model. A flex model instance can be created from a schema using the static `from_schema(cls, schema)` method provided by the `Flex Model` class. This instance will then have all fields specified in the specified schema. Further a kwargs constructor is provided that takes values for all specified fields. `Flex Model` instances can be added to sessions like regular models.
+    Another usecase for a `Flex Model` arises if the schema is known but the model class is not. This might be the case if entries are retireved from origins not under the control of PolyNOM.
+    WARNING: When developing, one should always first attempt to achieve the desired functionality without flex models. Flex models should only be used when it is not possible to know the schema or the model.

docs/query.md

-Original file line number
+Diff line change
@@ -0,0 +1,143 @@
+    ---
+    layout: page
+    title: "Query"
+    toc: true
+    docs_area: "PolyNOM"
+    tags: query, filter, search, manipulation
+    lang: en
+    ---
+    ## Query
+    This page discusses the retrieval of data. This involves three classes. Namely those are the `Session`, `Model` and the `Query` class. Each retrieval triggers queries on the underlying polypheny instance and must thus take place as part of a `Session`. To avoid the need for user written statements, the `Query` class provides a variety of methods for filtering as well as calculations. Each `Model` provides a corresponding instance of the `Query` class by its `query` method:
+    ### `query(session: Session)`
+    Returns a `Query` instance that can be used to query entries of the type of the Model class.
+    - `session` (`Session`): The session under as part of which to execute the query.
+    ## Filter Methods
+    Filter methods are the first type of method provided by a `Query` instance. Filter methods allow to define restriction to filter the entries to be returned. Filter methods always return a `Query` instance allowing them to be chained to bulid more copmlex filters.
+    ### `filter_by(**kwargs) → Query`
+    Adds simple equality filters to the query based on model fields.
+    - `**kwargs`: Key-value pairs where keys are model attribute names and values are the values to filter by. Their format is `key=value` (e.g. last_name='meyer').
+    Returns the updated `Query` instance to allow the chaining of query methods.
+    ---
+    #### `filter(*expressions) → Query`
+    Adds complex filter expressions to the query. Each expression must be a tuple of `(operator, field, value)`.
+    - `expressions`: Tuples specifying conditions.
+    - `operator`: A string SQL operator (e.g., `"="`, `">"`, `"LIKE"`).
+    - `field`: The name of the model field to compare.
+    - `value`: The value to compare against.
+    Returns the updated `Query` instance to allow the chaining of query methods.
+    ---
+    ### `distinct() → Query`
+    Marks the query to return only distinct results.
+    Returns the updated `Query` instance to allow the chaining of query methods.
+    ---
+    ### `limit(n: int) → Query`
+    Limits the number of results returned by the query.
+    - `n`: Maximum number of entries to return.
+    Returns the updated `Query` instance to allow the chaining of query methods.
+    ---
+    #### `order_by(field_name: str) → Query`
+    Specifies a field to order the query results by.
+    - `field_name`: The name of the model field to sort by.
+    Raises `ValueError` if the field does not exist.
+    Returns the updated `Query` instance to allow the chaining of query methods.
+    ---
+    ### `join(related_model: Type[BaseModel], on: Optional[str] = None) → Query`
+    Adds a JOIN to include related models in the query.
+    - `related_model`: The model class to join.
+    - `on` (optional): Custom join condition as a SQL string. If omitted, the system attempts to infer the join based on foreign keys.
+    Returns the updated `Query` instance to allow the chaining of query methods.
+    ---
+    ### `options(*opts) → Query`
+    Applies query options, such as eager-loading instructions.
+    - `*opts`: Option objects to apply. Currently supports `JoinedLoad` for eager loading of related models.
+    Returns the updated `Query` instance to allow the chaining of query methods.
+    ## Result Methods
+    Result methods are the secon type of method provided by a `Query` instance. Result methods trigger the execution of the `Query` instance and return a method specific result.
+    ### `all() → List[BaseModel]`
+    Executes the query and returns all matching model instances.
+    Returns a list of model instances. Related children are attached according to any configured eager loads.
+    ---
+    ### `first() → Optional[BaseModel]`
+    Fetches the first row matching the query, if any.
+    Returns a single model instance if a row exists; otherwise, `None`.
+    ---
+    ### `exists() → bool`
+    Checks whether any records exist matching the current query filters.
+    Returns `True` if at least one matching row exists; otherwise, `False`.
+    ---
+    #### `count() → int`
+    Counts the number of entries matching the current query filters.
+    Returns an integer representing the number of matching entries.
+    ---
+    ### `delete() → int`
+    Deletes all model instances matching the current query.
+    Returns the number of models deleted.
+    ---
+    ### `update(values: Dict[str, Any]) → int`
+    Updates all models matching the current query with the specified field values.
+    - `values`: Dictionary mapping model field names to new values.
+    Returns the number of models updated.
+    Raises `AttributeError` if a field in `values` does not exist on the model.
+    ---
+    ### `get(pk_value: Any) → Optional[BaseModel]`
+    Fetches a single entry by its primary key.
+    - `pk_value`: The value of the primary key to search for.
+    Returns the model instance if found; otherwise, `None`.
+    Raises `ValueError` if the model does not have a `PrimaryKeyField`.
+    ---

docs/session.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -88,7 +88,7 @@ Discards all changes made as part of this session. This invalidates all tracked
  
    ---

    #### `_execute(language, statement, parameters=None, namespace=None, fetch=True)`

    #### `_execute(language: str, statement: str, parameters=None, namespace: str=None, fetch: bool=True)`

    Executes a statement using the session's internal cursor. This method supports both DDL and DML operations across different query languages (e.g., SQL, Cypher, MQL).

    @@ -99,6 +99,16 @@ Executes a statement using the session's internal cursor. This method supports b
  
    - `fetch` (optional, default=`True`): If `True`, the result of the query is returned if present. If set to `False` no results are retrieved independent of the query type.

    ---

    WARNING: This method is considered deprecated and will be removed in the future.

    #### `_execute(statement: Statement, fetch=True)`

    Executes a statement object using the session's internal cursor. This method is the replacement for the previous, deprecated method of the same name.

    - `statement`: The statement object to execute.

    - `fetch` (optional, default=`True`): If `True`, the result of the query is returned if present. If set to `False` no results are retrieved independent of the query type.

    ---

    #### `get_session_state() → _SessionState`

    Returns the current state of the session.

docs/statement.md

-Original file line number
+Diff line change
@@ -0,0 +1,54 @@
+    ---
+    layout: page
+    title: "Statement"
+    toc: true
+    docs_area: "PolyNOM"
+    tags: query, statement, language
+    lang: en
+    ---
+    ## Statement
+    The `Statement` class represents an individual statement to be executed as part of a session. It thereby combines a query language, a statement in that language, optional parameters and an optional namespace to execute the expression in. Statements enable users to write and execute queries directly beyond the methods provided by the `Query` object.
+    ## Initialization Parameters
+    ```python
+    Statement(
+        language: str,
+        statement: str,
+        values: Tuple[Any, ...] = None,
+        namespace: str = None
+    )
+    ```
+    - `language` (`str`, required):
+      The name of the query language of the provided statement (e.g., `'sql'`, `'cypher'`, `'mongo'`).
+    - `statement`:
+      The statement string to be executed. This must be in the query language specified useing the `language` parameter.
+    - `values` (`str`, optional):
+      If the statement string provided contains placeholders, their values must be specified here. Values are assigned to placeholders from left to right.
+      If no placeholders are present no values must be set.
+    - `namespace` (`str`, optional):
+      The name of the namespace in which the statement should be executed. If not specified the default namespace configured in the PolyNOM config is used.
+    ## Examples
+    ```python
+    from polynom.application import Application
+    from polynom.session import Session
+    from polynom.statement import Statement
+    APP_UUID = 'a8817239-9bae-4961-a619-1e9ef5575eff'
+    with Application(APP_UUID, ('localhost', 20590)) as app:
+        with Session(app) as session:
+            expensive_brands_statement = Statement(
+                'sql',
+                'SELECT DISTINCT brand FROM bikes WHERE price > ?;',
+                (6000,),
+                'cycling'
+            )
+            expensive_brands = session._execute(expensive_brands_statement, fetch=True)
+    ```

docs/statement_log.md

-Original file line number
+Diff line change
@@ -0,0 +1,34 @@
+    ---
+    layout: page
+    title: "Statement Log"
+    toc: true
+    docs_area: "PolyNOM"
+    tags: backup, dump, file, log
+    lang: en
+    ---
+    # Statement Log
+    Every PolyNOM application maintains a statement log. This log records all committed, data-modifying statements executed by the application. The base name of the log file can be configured using the PolyNOM configuration. The base name is extended by the UUID of the application. A new file is created once the current one reaches a size of 1 GB. The new file's name is further extended with a timestamp of its creation.
+    **WARNING:** Statement log files are kept indefinitely. It is the user's responsibility to manage and dispose of old logs that are no longer needed.
+    ## File Name
+    If the base name in the configuration is set to `statements.log`, the first file created might be named: `statements_a8817239-9bae-4961-a619-1e9ef5575eff.log`. Here, the UUID of the application is `a8817239-9bae-4961-a619-1e9ef5575eff`. Once this file reaches 1 GB, a new file is created with a timestamp appended to the name: `statements_a8817239-9bae-4961-a619-1e9ef5575eff.20250902-100915.log`.
+    ## Structure
+    The statement log contains one statement per line in one or more query languages. Similar to MLQ files used for application dumps, each line begins with comments storing metadata for the statement:
+. **Timestamp comment** – indicates when the entry was created.
+. **Language and namespace comment** – specifies the query language and the namespace in which the statement was executed.
+    Example:
+    ```sql
+    /*2025-09-02 08:44:06,008*/ /*sql@polynom_entities*/ DELETE FROM "polynom_entities"."User" WHERE _entry_id = '19b69a99-bc5a-4e10-b90e-c276f56b440f'
+    ```
+    ## Rollbacks and Reads
+    The log only contains statements from committed sessions. Statements from sessions that were rolled back, either manually or automatically, are omitted. Only statements that modify schema or data are recorded. Statements that do not modify any data are omitted.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Statement Logging #8

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!