Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
800bf76
Add rudimentary archival test
lukasjuhrich Dec 22, 2021
7ad673b
Extract timedelta from archivable membership filter
lukasjuhrich Feb 21, 2022
611b948
reintroduce `cast` to silence warning
lukasjuhrich Feb 21, 2022
6e764c7
Add test for longer delta in archivable query
lukasjuhrich Feb 21, 2022
bbbb218
Add `archive_users` stub and test
lukasjuhrich Feb 21, 2022
bf34fd1
Extract last membership select from archive query
lukasjuhrich Aug 4, 2022
f3732ec
Switch membership selection to `from … select` style
lukasjuhrich Aug 4, 2022
093ea24
Decouple archivable users select statement from execution
lukasjuhrich Aug 4, 2022
e5dfc24
Switch archival query to `from … select` style
lukasjuhrich Aug 4, 2022
2748035
Add comment explaining window args
lukasjuhrich Aug 4, 2022
11ff9e6
Switch archivable member selection from delta to year-based cutoff
lukasjuhrich Aug 4, 2022
bea667f
Improve archivability test performance
lukasjuhrich Aug 4, 2022
118690b
Replace `add_columns` by `with_only_columns` and fix type information
lukasjuhrich Jun 17, 2025
deb5823
Implement scrubbing routine for `mail`s
lukasjuhrich Jun 17, 2025
2a5375c
Remove `archive_users`
lukasjuhrich Jun 17, 2025
87b9f0b
Implement mail scrubbing in batches of 100
lukasjuhrich Jun 17, 2025
adfcb57
Add type hints for `message` in `deferred_gettext`
lukasjuhrich Jun 24, 2025
7aee819
Add reference to privacy policy to scrubbing functions
lukasjuhrich Jun 24, 2025
c7a5cbd
Add stubs to other scrubbers with references to privacy policy
lukasjuhrich Jun 24, 2025
5cc31d0
Fix typo
FestplattenSchnitzel Jun 27, 2025
7584510
Make number of years required for archivability a parameter
FestplattenSchnitzel Jun 27, 2025
72c8513
Update `psycopg2-binary`, `factory-boy` and add `pytest-freezegun`
lukasjuhrich Jul 11, 2025
4c91fea
model: add `ScrubLog`
lukasjuhrich Jul 11, 2025
6aecfa0
include `model.scrubbing` in `model._all`
lukasjuhrich Jul 11, 2025
3f132d0
upgrade pytest to v8.4.1
lukasjuhrich Jul 12, 2025
6be611a
Upgrade uv to v0.7.11
lukasjuhrich Jul 12, 2025
5f06b99
Add a ScrubLogEntry when scrubbing mail address and test scrubbing
lukasjuhrich Jul 29, 2025
ea07a25
Fix tests
lukasjuhrich Jul 29, 2025
5432d09
Add select statements for other scrubbers
lukasjuhrich Jul 29, 2025
05cc6e9
Remove annoying daker args
lukasjuhrich Jul 29, 2025
161afdc
Add stub implementation and test for bulk mail scrubbing
lukasjuhrich Jul 29, 2025
3c241bb
Add bulk mail scrubbing without user log entries
lukasjuhrich Jul 29, 2025
f4c268b
scrublog: Add server_default and fix migration
lukasjuhrich Sep 19, 2025
b6ec15f
scrublog: fix indices and nullability in new migration
lukasjuhrich Sep 19, 2025
9d8ee82
Fix usage of _WindowArgs type hints
lukasjuhrich Sep 19, 2025
1442cae
Use `type[]` instead of `typing.Type[]`
lukasjuhrich Sep 19, 2025
01a3553
scrublog: Use server default for execution time
lukasjuhrich Sep 19, 2025
f478809
scrubbing: Allow customization of user filter in bulk mail scrubber
lukasjuhrich Sep 19, 2025
a4af703
Create common `identity` function in `helpers.functional`
lukasjuhrich Sep 19, 2025
6571b25
Add users with to-be-deleted name, address statements
FestplattenSchnitzel Oct 31, 2025
c437074
First attempt of a bulk deletion query
lukasjuhrich Sep 19, 2025
44c1f2b
Add row_number()s to intermediate CTEs so we can properly join
lukasjuhrich Oct 31, 2025
bd94f03
re-lock using `uv` with `--upgrade`
lukasjuhrich Oct 31, 2025
a6cfc9e
Replace buggy scrublog migration migrations by proper one
lukasjuhrich Oct 31, 2025
0983164
Fix mail deletion query and flash message
FestplattenSchnitzel Nov 6, 2025
e981772
Extend bulk scrubbing test to log entries
lukasjuhrich Nov 29, 2025
d66e053
Insert log entries for each removed mail, not just one per scrub
lukasjuhrich Nov 29, 2025
fe96209
Fix docstring of get_archivable_users
lukasjuhrich Nov 29, 2025
2a9958f
Add type annotations to queries
lukasjuhrich Nov 29, 2025
e4d9ce2
Fix type annotation at membership query
lukasjuhrich Nov 29, 2025
b6da980
Use `fetchmany` instead of `partitions` in archivable mails preview
lukasjuhrich Nov 29, 2025
c588814
Remove obsolete TODOs
lukasjuhrich Nov 29, 2025
3dae241
Merge develop and archival
lukasjuhrich Feb 1, 2026
9911428
Update uv to 0.9.28
lukasjuhrich Feb 1, 2026
571e61d
Re-Lock
lukasjuhrich Feb 1, 2026
1871aa5
Make semgrep happy
lukasjuhrich Nov 29, 2025
f35aef0
Clean up imports
lukasjuhrich Nov 29, 2025
f344934
Add missing type annotation
lukasjuhrich Nov 29, 2025
741abe5
Remove unused import from web.blueprints.user
lukasjuhrich Nov 29, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/docker-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ jobs:
- name: set up `uv`
uses: astral-sh/setup-uv@v5
with:
version: "0.8.18"
version: "0.9.28"
enable-cache: true
cache-dependency-glob: "uv.lock"
- name: install pip dependencies with uv
Expand Down
3 changes: 1 addition & 2 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ repos:
rev: 1.7.0
hooks:
- id: darker
args: ["--diff", "--check"]
- repo: https://github.com/asottile/pyupgrade
rev: v3.17.0
hooks:
Expand All @@ -16,7 +15,7 @@ repos:
hooks:
- id: ruff
- repo: https://github.com/astral-sh/uv-pre-commit
rev: '0.8.18'
rev: '0.9.28'
hooks:
- id: uv-lock
- repo: https://github.com/semgrep/pre-commit
Expand Down
2 changes: 1 addition & 1 deletion docker/base.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ ARG GID=1000
ENV LANG=C.UTF-8 DEBIAN_FRONTEND=noninteractive

COPY etc/apt /etc/apt
COPY --from=ghcr.io/astral-sh/uv:0.6.2 /uv /uvx /usr/local/bin/
COPY --from=ghcr.io/astral-sh/uv:0.9.28 /uv /uvx /usr/local/bin/

# Install Debian packages
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
Expand Down
4 changes: 4 additions & 0 deletions pycroft/helpers/functional.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,7 @@ def map_collecting_errors[
except error_type as e:
errors.append(e)
return results, errors


def identity[T](x: T) -> T:
return x
2 changes: 1 addition & 1 deletion pycroft/helpers/i18n/deferred.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
from .message import SimpleMessage, NumericalMessage


def deferred_gettext(message) -> SimpleMessage:
def deferred_gettext(message: str) -> SimpleMessage:
return SimpleMessage(message)


Expand Down
6 changes: 3 additions & 3 deletions pycroft/helpers/i18n/message.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ def from_json(cls, json_string: str) -> Message:
return m

def __init__(self, domain: str | None = None):
self.domain = domain
self.domain: str | None = domain
self.args: typing.Iterable[Serializable] = ()
self.kwargs: dict[str, Serializable] = {}

Expand Down Expand Up @@ -131,9 +131,9 @@ def _gettext(self):
class SimpleMessage(Message):
__slots__ = ("message",)

def __init__(self, message, domain=None):
def __init__(self, message: str, domain: str | None = None):
super().__init__(domain)
self.message = message
self.message: str = message

@t.override
def _base_dict(self):
Expand Down
5 changes: 1 addition & 4 deletions pycroft/helpers/i18n/serde.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,11 @@
from datetime import date, datetime, time, timedelta
from decimal import Decimal

from ..functional import identity
from .types import Interval, NegativeInfinity, PositiveInfinity, Bound, Money
from .utils import qualified_typename


def identity(x):
return x


def deserialize_money(v):
try:
return Money(Decimal(v[0]), v[1])
Expand Down
2 changes: 1 addition & 1 deletion pycroft/lib/logging.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
def _create_log_entry[
TLogEntry: LogEntry
](
class_: t.Type[TLogEntry],
class_: type[TLogEntry],
message: str,
author: User,
created_at: datetime | None = None,
Expand Down
49 changes: 48 additions & 1 deletion pycroft/lib/membership.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,16 @@
management.

"""
from __future__ import annotations
import typing as t

from sqlalchemy import and_, func, distinct, Result
from sqlalchemy import and_, func, distinct, Result, nulls_last
from sqlalchemy.future import select
from sqlalchemy.orm import aliased, Session
from sqlalchemy.sql import Select
from sqlalchemy.sql._typing import _TypedColumnClauseArgument, _ByArgument

from pycroft import Config
from pycroft.helpers import utc
from pycroft.helpers.i18n import deferred_gettext
from pycroft.helpers.interval import UnboundedInterval, IntervalSet, Interval, closedopen
Expand Down Expand Up @@ -239,3 +243,46 @@ def change_membership_active_during(
.to_json()
)
log_user_event(message, processor, membership.user)


def select_user_and_last_mem() -> Select[tuple[int, int, DateTimeTz]]:
"""Select users with their last membership of a user in the ``member`` group.

:returns: a select statement with columns ``user_id``, ``mem_id``, ``mem_end``.
"""
mem_ends_at = func.upper(Membership.active_during)
# see FunctionElement.over for documentation on `partition_by`, `order_by`
# ideally, sqlalchemy would support named windows;
# instead, we have to re-use the arguments.
window_args: _WindowArgs = {
"partition_by": User.id,
"order_by": nulls_last(mem_ends_at),
}
return (
select()
.select_from(User)
.distinct()
.join(Membership)
.join(Config, Config.member_group_id == Membership.group_id)
.with_only_columns(
User.id.label("user_id"),
t.cast(
_TypedColumnClauseArgument[int],
func.last_value(Membership.id)
.over(**window_args, rows=(None, None))
.label("mem_id"),
),
t.cast(
_TypedColumnClauseArgument[str],
func.last_value(mem_ends_at)
.over(**window_args, rows=(None, None))
.label("mem_end"),
),
)
)


class _WindowArgs(t.TypedDict):
partition_by: _ByArgument
order_by: _ByArgument

Loading
Loading