Skip to content

Working Add Data - Integrate assign entities #94

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pooleycodes merged 12 commits into from
Feb 3, 2026
Merged

Working Add Data - Integrate assign entities #94

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
554d759
add data now uses pipeline transfrom and assign entries to detect new…
pooleycodes Jan 23, 2026
b122140
test update and file clean up
pooleycodes Jan 25, 2026
be46a4f
lint
pooleycodes Jan 25, 2026
05167cf
integration assertions update and temp coverage change dev only
pooleycodes Jan 25, 2026
0eac83e
test workflow publish to use branch name
pooleycodes Jan 26, 2026
0e0215b
rename to main tag in publish
pooleycodes Jan 26, 2026
7e9e48a
all pipeline files in use now
pooleycodes Jan 26, 2026
7307c55
use start-date, plugin and licence
pooleycodes Jan 27, 2026
bd283b9
tidying up pipline and workflow, rerun pipeline on new entity creation
pooleycodes Feb 2, 2026
8a3a624
Updatae debug scripts
pooleycodes Feb 2, 2026
62ec1c9
lint
pooleycodes Feb 3, 2026
195aad1
move tests around to show new workflow
pooleycodes Feb 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 24 additions & 2 deletions .vscode/launch.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,33 @@
"python": "${workspaceFolder}/request-processor/.venv/bin/python",
"configurations": [
{
"name": "Debug Request Processor - Manual Task Trigger",
"name": "Debug Request Processor - Manual Check URL Task Trigger",
"type": "python",
"request": "launch",
"python": "${workspaceFolder}/request-processor/.venv/bin/python",
"program": "${workspaceFolder}/scripts/debug_trigger.py",
"program": "${workspaceFolder}/scripts/debug_trigger_checkurl.py",
"console": "integratedTerminal",
"justMyCode": false,
"env": {
"PYTHONPATH": "${workspaceFolder}:${workspaceFolder}/request-processor:${workspaceFolder}/request-processor/.venv/src/digital-land",
"DATABASE_URL": "postgresql://postgres:password@localhost:54320/request_database",
"CELERY_BROKER_URL": "redis://localhost:6379/0",
"AWS_ENDPOINT_URL": "http://localhost:4566",
"AWS_DEFAULT_REGION": "eu-west-2",
"AWS_ACCESS_KEY_ID": "example",
"AWS_SECRET_ACCESS_KEY": "example",
"REQUEST_FILES_BUCKET_NAME": "dluhc-data-platform-request-files-local",
"SENTRY_ENABLED": "false"
},
"args": [],
"cwd": "${workspaceFolder}/request-processor"
},
{
"name": "Debug Request Processor - Manual Add Data Task",
"type": "python",
"request": "launch",
"python": "${workspaceFolder}/request-processor/.venv/bin/python",
"program": "${workspaceFolder}/scripts/debug_trigger_add_data.py",
"console": "integratedTerminal",
"justMyCode": false,
"env": {
Expand Down
3 changes: 2 additions & 1 deletion request-processor/makerules/makerules.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ endif
config:
# local copy of organsiation datapackage
@mkdir -p $(CACHE_DIR)
curl -qfs "https://raw.githubusercontent.com/digital-land/organisation-dataset/main/collection/organisation.csv" > $(CACHE_DIR)organisation.csv
curl -qfs "https://files.planning.data.gov.uk/organisation-collection/dataset/organisation.csv" > $(CACHE_DIR)organisation.csv


init:: config
2 changes: 1 addition & 1 deletion request-processor/makerules/python.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,4 @@ coverage-unit:
pytest --random-order --cov=src tests/unit/

coverage-integration:
pytest --random-order --cov=src --cov-append --cov-fail-under=85 tests/integration/
pytest --random-order --cov=src --cov-append --cov-fail-under=80 tests/integration/
2 changes: 1 addition & 1 deletion request-processor/src/application/configurations/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

source_url = "https://raw.githubusercontent.com/digital-land/"
DATASTORE_URL = os.getenv("DATASTORE_URL", "https://files.planning.data.gov.uk/")
CONFIG_URL = f"{DATASTORE_URL}config/"
CONFIG_URL = f"{source_url}config/refs/heads/main/"


class Directories:
Expand Down
Loading
Loading