PINS Redaction System

This repository defines the automated redaction that is available to PINS services.

Project structure

├── redaction-system/                         // The redaction-system folder can be in a directory of your choice
│    ├── redactor/                            // Where the redaction tool is defined
│    │   ├── core/                            // Where the core functionality of the tool is defined
│    │   │   ├── config/                      // Where config files for redaction are defined
│    │   │   ├── io/                          // Where functionality for reading/writing files is defined
│    │   │   ├── util/                        // Where misc utility functionality is defined
│    │   │   ├── redaction/                   // Where functionality for redaction processes is defined
│    │   │   │   ├── redactor.py              // Where scripts for redacting pieces of data are defined. *Files are composed of data
│    │   │   │   ├── file_processor.py        // Module for redacting files 
│    │   │   │   ├── config_processor.py      // Module for processing config files 
│    │   │   │   ├── main.py                  // Script for executing the redaction process via terminal for a file
│    │   │   │   ├── config.py                // Redaction config classes
│    │   │   │   ├── result.py                // Redaction result classes
│    │   │   │   ├── exceptions.py            // Custom exceptions 
│    │   ├── test/                            // Tests
│    │   │   ├── unit_test/                   // Unit tests
│    │   │   ├── integration_test/            // Integration tests
│    │   │   ├── e2e_test/                    // End-to-end tests
│    │   │   ├── resources/                   // Resources for tests
│    ├── pipelines/
│    │   ├── jobs/                            // Where pipeline jobs are defined
│    │   ├── steps/                           // Where pipeline steps are defined
│    │   └── scripts/                         // Utility python/bash scripts are defined here
│    ├── infrastructure/
│    │   └── environments/                    // Utility python/bash scripts are defined here

Local setup

1. Create a .env file in the project's root directory
   • Set the content of the env file based on the Environment variables section of the readme
2. Install Python 3.13
3. Create a virtual environment
4. Install the requirements using python3 -m pip install -r redactor/requirements.txt
5. You may need to run the below command to set up your Python environment
   1. export PYTHONPATH="${PYTHONPATH}:/./"

Running python files locally

• Run specific Python files using python3 path/to/file/to/run.py
  • NOTE: Currently the redaction process can be run using redactor/core/main.py , more info cam be found within this file

Running the function app

• Install the core functions tools https://learn.microsoft.com/en-us/azure/azure-functions/functions-run-local?tabs=macos%2Cisolated-process%2Cnode-v4%2Cpython-v2%2Chttp-trigger%2Ccontainer-apps&pivots=programming-language-python
• run func start
• You can then connect to the function via http requests, or via the Azure portal

Environmment variables

Below are the environment variables used by the project

Variable	Description
OPENAI_ENDPOINT	The Open AI host. For example: "https://myazurefoundryresource.openai.azure.com/"

Testing

Prerequisites

1. Be connected to the PINS VPN (required for cloud-backed E2E/perf and Azure Storage access).
2. Be logged into Azure CLI (az login) and using the expected subscription.
3. Ensure .env in repo root contains E2E values (E2E_STORAGE_ACCOUNT, E2E_CONTAINER_NAME, etc.).
4. Install local runtime tools: Azure Functions Core Tools (func) and Azurite (azurite).

Run tests

1. Start local services:
   • make run
2. In another terminal, run E2E:
   • make e2e
3. Run perf tests:
   • make perf

Optional perf tuning:

• PERF_TOTAL=20 PERF_CONCURRENCY=5 PERF_TIMEOUT_S=1200 make perf