Skip to content

Backport 8407 #220

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JVickery-TBS merged 6 commits into from
Jan 26, 2026
Merged

Backport 8407 #220

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
99d460e
Merge pull request #8421 from ckan/8407-package-update-fasterer
amercader Oct 23, 2024
bb35525
feat(schema): allow metadata created;
JVickery-TBS Jan 19, 2026
381c790
feat(misc): changelog;
JVickery-TBS Jan 19, 2026
8f734cc
feat(logic): package extras dict save;
JVickery-TBS Jan 19, 2026
a54f798
feat(logic): resource metadata modified;
JVickery-TBS Jan 19, 2026
1fb215a
feat(logic): package metadata created update schema;
JVickery-TBS Jan 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changes/220.backport.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Backported [#8421](https://github.com/ckan/ckan/pull/8421) for faster metadata updates. Comes with the `metadata_modified` and `metadata_created` schemas, allowing for site package migrations with correct metadata for these fields.
15 changes: 15 additions & 0 deletions changes/8407.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
Faster dataset metadata updates by detecting changes and only updating resource
metadata when dataset fields defined by IDatasetForm.resource_validation_dependencies
have changed (default: none)

Now activities are created and metadata_modified updated only if there is a real change.

metadata_modified may now be set by sysadmins which is useful for harvesting or mirroring.

The allow_partial_update context parameter has been removed, now normal API
users may call package_update without passing resources. In this case the existing
resources will remain untouched instead of being deleted.

package_update and actions that call it now report whether there was a real change by
adding the package id to a new changed_entities context value or changed_entities
envelope value for API calls.
29 changes: 20 additions & 9 deletions ckan/lib/dictization/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
from __future__ import annotations

import datetime
from typing import Any, Callable, Iterable
from typing import Any, Callable, Iterable, Literal

import sqlalchemy
from sqlalchemy import Table
Expand Down Expand Up @@ -109,13 +109,21 @@ def get_unique_constraints(table: Table, context: Context) -> list[list[str]]:
return list_of_constraints


def table_dict_save(table_dict: dict[str, Any],
ModelClass: Any,
context: Context,
extra_attrs: Iterable[str] = ()) -> Any:
def table_dict_save(
table_dict: dict[str, Any],
ModelClass: Any,
context: Context,
extra_attrs: Iterable[str] = ()
) -> tuple[Any, Literal['create', 'update', None]]:
'''Given a dict and a model class, update or create a sqlalchemy object.
This will use an existing object if "id" is supplied OR if any unique
constraints are met. e.g supplying just a tag name will get out that tag obj.
constraints are met. e.g supplying just a tag name will get out that tag
obj.

Returns (obj, change) where change is:
- 'create' if this is a new object
- 'update' if any fields were changed or extra_attrs passed
- None if no change for an existing object
'''
session = context["session"]

Expand All @@ -128,7 +136,8 @@ def table_dict_save(table_dict: dict[str, Any],
if id:
obj = session.query(ModelClass).get(id)

if not obj:
new = not obj
if new:
unique_constraints = get_unique_constraints(table, context)
for constraint in unique_constraints:
params = dict((key, table_dict.get(key)) for key in constraint)
Expand All @@ -144,11 +153,13 @@ def table_dict_save(table_dict: dict[str, Any],
if not obj:
obj = ModelClass()

obj.from_dict(table_dict)
changed, _skipped = obj.from_dict(table_dict)
for a in extra_attrs:
if a in table_dict:
setattr(obj, a, table_dict[a])

session.add(obj)

return obj
return (
obj, 'create' if new else 'update' if changed or extra_attrs else None
)
Loading