453 csv upload in new design by gazdagergo · Pull Request #88 · sortitionfoundation/opendlp

gazdagergo · 2026-03-22T16:16:25Z

ticket: 453

backend/src/opendlp/entrypoints/blueprints/backoffice.py

foobacca

I know it's WIP but I thought I'd have a quick look.

backend/pyproject.toml

backend/src/opendlp/adapters/sql_repository.py

backend/templates/backoffice/assembly_data.html

Add admin-only developer tool page at /backoffice/dev/service-docs that provides interactive documentation for CSV upload service functions. Disabled in production for security. Features: - Documents import_respondents_from_csv, import_targets_from_csv, get_or_create_csv_config, and update_csv_config services - Shows code references, parameters, return types, and error cases - Interactive "Try It" sections to execute services directly - Sample data loading for testing - Tab navigation with URL query parameter sync - Adds x-cloak CSS rule for Alpine.js CSP build compatibility Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Replace nested x-model paths with flat properties (CSP build limitation) - Update execute/reset/sample loader methods to use flat properties - Add X-CSRFToken header to fetch request for CSRF protection - Add x-cloak to loading spinners to prevent flash Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Create /backoffice/dev/patterns route for interactive pattern docs - Add patterns.html with dropdown pattern examples (URL select, inline state) - Include live examples, code snippets, CSP notes, and implementation index - Update AGENTS.md with Alpine.js CSP constraints reference Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Use & separator when URL already has query params, otherwise use ? - Fixes URL like ?tab=dropdown?demo=option (invalid) becoming ?tab=dropdown&demo=option - Add server-side selected attributes to patterns page dropdown Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Create /backoffice/dev route with cards linking to dev tools - Links to Service Docs, Frontend Patterns, and Component Showcase - Admin-only, disabled in production - Update breadcrumbs in service_docs and patterns to link to dev dashboard Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

UI Component: - Add file_input macro to components/input.html - Supports label, hint, error, accept, required, disabled - Tailwind styling with file: pseudo-class for button Pattern Documentation: - Add File Upload tab to frontend patterns page - Document architecture (UI -> Flask route -> service layer) - Live demo with client-side file validation and preview - Template and Flask route code examples - Implementation index linking to existing code Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add file_input to input component showcase - Show basic, required with hint, error, and disabled states - Update description and code example to include file_input Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

The DjHTML linter reformatted the usage examples in a way that broke Jinja comment parsing, causing the examples to be executed as actual code. Simplified to plain text examples matching the pattern used by other macros. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add shared assembly_tabs macro for consistent tab rendering - Create Targets page showing gsheet config info or CSV placeholder - Create Respondents page showing gsheet config info or CSV placeholder - Add routes: /assembly/<id>/targets and /assembly/<id>/respondents - Update all assembly templates to use shared tabs macro - Tabs are always visible but disabled until data source is configured - For gsheet source: tabs show info about configured sheet tab names - Add BDD tests for tab visibility and enabled/disabled states - Add docs/agent/453-csv-upload.md for feature documentation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add service functions: get_csv_upload_status, delete_targets_for_assembly, delete_respondents_for_assembly - Add routes for uploading/deleting targets and respondents CSV files - Update assembly_data.html with CSV upload panel showing Target and People upload sections with proper state management - Lock data source selector when CSV data is uploaded - Enable Targets/Respondents tabs based on uploaded data - Add BDD tests for CSV upload scenarios - Include sample CSV files for testing Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add Selection tab to /backoffice/dev/service-docs with documentation for: - check_db_selection_data() - start_db_select_task() - get_selection_run_status() - generate_selection_csvs() - cancel_task() - check_and_update_task_health() - get_respondent_attribute_columns() - translate_run_report_to_html() - Update 453-csv-upload.md with sortition service layer functions - Add "selection" to valid tabs in service_docs route Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Revert accidental downgrade and update to latest version. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Selection tab is now disabled when no data source is configured. Updated test scenarios to include gsheet configuration where needed. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

By specifying Callable args and return type

Replace one-at-a-time session.delete() loops with single DELETE WHERE statements in both TargetCategory and Respondent repositories. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

rather than a dict - makes the code easier to read and reason about

… service functions Cover delete_targets_for_assembly and delete_respondents_for_assembly with service-level tests (permissions, not-found, isolation, happy path) and add repository-level tests for respondent bulk deletion. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Remove the restriction that required targets to be uploaded before people. Both CSV files can now be uploaded in any order. Update BDD test to match the new behavior. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

backend/src/opendlp/entrypoints/blueprints/backoffice.py

Use COVERAGE_PROCESS_START mechanism instead of wrapping commands with `coverage run`. Coverage's installed .pth file auto-instruments Python subprocesses when the env var is set, and pytest-cov automatically combines the resulting data files during its finalization. - Replace _build_coverage_command() with _add_coverage_env() that sets COVERAGE_PROCESS_START and COVERAGE_FILE per subprocess - Use --pool=solo for celery when collecting coverage to avoid prefork workers fighting over the data file - Use start_new_session + os.killpg for clean process group termination - Remove manual coverage combine step from justfile (pytest-cov handles it) Also change the justfile to use spaces instead of tabs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Documents blueprint and service layer architecture with visual diagrams showing dependencies, route mappings, and recommendations for potential code organization improvements. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Move /backoffice/dev/* routes to a separate dev.py blueprint for cleaner separation of developer tools from production code. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Allow users to specify which column contains unique identifiers when uploading respondents CSV. If left blank, the first column is used. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

backend/src/opendlp/entrypoints/blueprints/backoffice.py

backend/src/opendlp/adapters/sql_repository.py

foobacca · 2026-03-26T22:22:21Z

backend/src/opendlp/entrypoints/flask_app.py

    app.register_blueprint(profile_bp)
    app.register_blueprint(health_bp)
    app.register_blueprint(backoffice_bp, url_prefix="/backoffice")
+    app.register_blueprint(dev_bp, url_prefix="/backoffice")


I was thinking we could do something like:

Suggested change

app.register_blueprint(dev_bp, url_prefix="/backoffice")

if not app.config.is_production():

app.register_blueprint(dev_bp, url_prefix="/backoffice")

That way, if we are in production, the blueprint does not exist at all. This means we can drop all the if config.is_production(): abort(404) checks in blueprints/dev.py

2478cd1 introduces it

* main: regenerate translations following justfile fix for pybabel args fix linting with auto reformat Apply suggested fix to backend/tests/fakes.py from Copilot Autofix Apply suggested fix to backend/tests/fakes.py from Copilot Autofix Apply suggested fix to backend/justfile from Copilot Autofix Apply suggested fix to backend/justfile from Copilot Autofix Apply suggested fix to backend/justfile from Copilot Autofix trigger flask reload (if running) when rebuilding css upgrade requests for security fix (and other pkgs) Have .well-known/change-password always go the change password form test: add contract tests for all repositories, fix 2 fake bugs test: add contract test pattern for PasswordResetTokenRepository edits to well known files feat: add well-known URL endpoints for robots.txt, security.txt, and change-password build(deps): Bump picomatch in /backend fix: auto-confirm email for users created via CLI commands add agent plan for invite email improvements feat: persist invite email address on UserInvite domain model Bump sass from 1.97.3 to 1.98.0 in /backend

Instead of checking is_production() in each route handler, only register the dev blueprint when not in production. This means the routes don't exist at all in production. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Visual overview of E2E and BDD test coverage for backoffice routes and services, with clear indicators for covered/uncovered code and test types (happy path, validation, permission, etc.). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

github-advanced-security bot found potential problems Mar 22, 2026

View reviewed changes

backend/src/opendlp/entrypoints/blueprints/backoffice.py Fixed Show fixed Hide fixed

gazdagergo force-pushed the 453-csv-upload branch from 1b64e7e to 06dabc1 Compare March 23, 2026 05:20

foobacca reviewed Mar 23, 2026

View reviewed changes

backend/pyproject.toml Outdated Show resolved Hide resolved

backend/src/opendlp/adapters/sql_repository.py Outdated Show resolved Hide resolved

backend/templates/backoffice/assembly_data.html Outdated Show resolved Hide resolved

foobacca marked this pull request as ready for review March 24, 2026 11:07

gazdagergo and others added 13 commits March 24, 2026 13:27

Add file_input to component showcase

a1f27b4

- Add file_input to input component showcase - Show basic, required with hint, error, and disabled states - Update description and code example to include file_input Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Upgrade sortition-algorithms to 0.12.3

6590ad7

Revert accidental downgrade and update to latest version. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Fix BDD tests for Selection tab requiring gsheet configuration

a58d859

Selection tab is now disabled when no data source is configured. Updated test scenarios to include gsheet configuration where needed. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

gazdagergo force-pushed the 453-csv-upload branch from dcfaf6c to a58d859 Compare March 24, 2026 12:33

foobacca and others added 7 commits March 24, 2026 14:44

Fix type error

7cc1d2e

By specifying Callable args and return type

re-upgrade pyasn1 for security issue

284ae56

re-apply changes to pyproject.toml that were overwritten on this branch

3ad3da0

refactor: use bulk DELETE for delete_all_for_assembly repository methods

288ae9e

Replace one-at-a-time session.delete() loops with single DELETE WHERE statements in both TargetCategory and Respondent repositories. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Use a small dataclass for CSVUploadStatus

9a9b9a2

rather than a dict - makes the code easier to read and reason about

Allow independent CSV uploads for targets and people

fa08f3f

Remove the restriction that required targets to be uploaded before people. Both CSV files can now be uploaded in any order. Update BDD test to match the new behavior. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

gazdagergo force-pushed the 453-csv-upload branch from 9ffd883 to fa08f3f Compare March 25, 2026 08:57

github-code-quality bot found potential problems Mar 25, 2026

View reviewed changes

backend/src/opendlp/entrypoints/blueprints/backoffice.py Fixed Show fixed Hide fixed

github code quality suggested fix

9229f23

gazdagergo force-pushed the 453-csv-upload branch from a940589 to 9229f23 Compare March 25, 2026 09:16

gazdagergo changed the title ~~WIP: 453 csv upload in new design~~ 453 csv upload in new design Mar 25, 2026

gazdagergo requested a review from foobacca March 25, 2026 12:17

foobacca and others added 5 commits March 25, 2026 12:27

Update template wording for target category addition

c1aeee2

refactor: extract dev routes from backoffice blueprint

5254338

Move /backoffice/dev/* routes to a separate dev.py blueprint for cleaner separation of developer tools from production code. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

feat: add id_column input to respondents CSV upload

11a56ec

Allow users to specify which column contains unique identifiers when uploading respondents CSV. If left blank, the first column is used. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

foobacca reviewed Mar 26, 2026

View reviewed changes

foobacca and others added 3 commits March 26, 2026 22:26

refactor: only register dev blueprint in non-production

2478cd1

Instead of checking is_production() in each route handler, only register the dev blueprint when not in production. This means the routes don't exist at all in production. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: add backoffice test coverage report

7d88ba0

Visual overview of E2E and BDD test coverage for backoffice routes and services, with clear indicators for covered/uncovered code and test types (happy path, validation, permission, etc.). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

453 csv upload in new design#88

453 csv upload in new design#88
gazdagergo wants to merge 29 commits intomainfrom
453-csv-upload

gazdagergo commented Mar 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

foobacca left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

foobacca Mar 26, 2026

Uh oh!

gazdagergo Mar 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	app.register_blueprint(dev_bp, url_prefix="/backoffice")
	if not app.config.is_production():
	app.register_blueprint(dev_bp, url_prefix="/backoffice")

Conversation

gazdagergo commented Mar 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

foobacca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

foobacca Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

gazdagergo Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gazdagergo commented Mar 22, 2026 •

edited

Loading