Changing runs logs placements from the SQL to an object storage

[sidoruka]

Background
At the moment, Cloud Pipeline stores runs' logs in the SQL database which is not the most optimal solution for the text blobs.

Approach
Let's consider switching this to the object storage (s3/gs/az) and keeping a link to the log file in the DB only.

[SilinPavel]

Run log migration to object storage

Summary

Pipeline run logs are normally stored in PostgreSQL (pipeline_run_log). This feature copies those logs to a configured system data storage (e.g. S3-compatible bucket) when a run is cleaned up, records the storage
location on the run, removes the corresponding rows from the database, and serves logs from storage when the run already has a storage path. If migration runs again while logs_storage_path is already set (e.g. new
lines were written to the DB after a first pass), the implementation loads existing task log files from storage and merges them with the new DB content before writing back.

Why

• Reduce DB size and cost for long-lived deployments with heavy logging.
• Keep a durable copy of logs in object storage aligned with the rest of the platform’s storage model.
• Support incremental / repeat migration when additional logs appear in the DB after an initial migration (merge path).

Configuration

Configuration is a system preference (JSON) storage.system.run.logs.config :

The target data storage is resolved as:

1. storageName from RunLogStorageConfig, or if blank
2. the global system storage name from storage.system.storage.name (DATA_STORAGE_SYSTEM_DATA_STORAGE_NAME).

Example RunLogStorageConfig:

  {
    "enabled": true,
    "pathPrefix": "logs/runs/",
    "storageName": "my-system-logs-storage"
  }

enabled - Must be true or migration is disabled (RunLogStorageManager.isRunLogMigrationConfigured()).
pathPrefix - Folder prefix inside the storage (default in code is logs/runs/). Trailing slashes are normalized when building paths.
storageName - Optional. Name (or id) of an existing data storage registered in Cloud Pipeline. If omitted, the default system storage from storage.system.storage.name is used.

Requirements for migration to run:
• enabled: true
• Non-blank pathPrefix
• A resolvable storage (storageName or default system storage)