Skip to content

Refactor _grants_collect_schemas macro to accept schema_names and is_… #23

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jonhopper-dataengineers wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Refactor _grants_collect_schemas macro to accept schema_names and is_… #23

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3a4f751
Refactor _grants_collect_schemas macro to accept schema_names and is_…
jonhopper-dataengineers Mar 23, 2026
b243bf4
Update CHANGELOG for v0.3.11: Refactor _grants_collect_schemas macro …
jonhopper-dataengineers Mar 23, 2026
1ebb7f5
Update CHANGELOG for v0.3.11: Add internal stages cloning option to d…
jonhopper-dataengineers Mar 23, 2026
7fcdc28
Fix formatting in CHANGELOG and update database_clone macro to correc…
jonhopper-dataengineers Mar 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
# Data Engineers Snowflake DataOps Utils Project Changelog
This file contains the changelog for the Data Engineers Snowflake DataOps Utils project, detailing updates, fixes, and enhancements made to the project over time.

## v0.3.11 - 2026-03-24 - External Share Permissions & Internal Stage Cloning

- refactored the _grants_collect_schemas macro to accept schema_names and is_exclude_list parameters, and updated related grant macros for consistency and improved functionality.
- added new macro `grant_external_share_read` to apply `select` permissions on all tables and views for specified schemas
- modified the macro `database_clone` to optionally allow for the cloning of internal stages

## v0.3.10.6 - 2026-03-19 - Clean up Semantic Views/Agents

* modified macro `clean_objects` to cater for removing Semantic Views and Snowflake Agents
Expand Down
2 changes: 1 addition & 1 deletion dbt_project.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'dbt_dataengineers_utils'
version: '0.3.10'
version: '0.3.11'
config-version: 2

require-dbt-version: [">=1.8.0", "<2.0.0"]
Expand Down
2 changes: 2 additions & 0 deletions macros/database/database.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ macros:
description: Destination database name, i.e. the new database to be created
- name: new_owner_role
description: "[Optional] The new owner role of the newly created object"
- name: include_internal_stages
description: "[Optional] When true, includes internal stages in the clone. Defaults to false."

- name: schema_clone
description: Clone (zero-copy) a source schema into a destination schema; database defaults to target if not supplied.
Expand Down
4 changes: 2 additions & 2 deletions macros/database/database_clone.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
{% macro database_clone(source_database, destination_database, new_owner_role='', comment='') %}
{% macro database_clone(source_database, destination_database, new_owner_role='', comment='', include_internal_stages=false) %}

{% if source_database and destination_database %}

{{ (log("Cloning existing database " ~ source_database ~ " into database " ~ destination_database, info=True)) }}

{% call statement('clone_database', fetch_result=True, auto_begin=False) -%}
create or replace database {{ destination_database }} clone {{ source_database }} comment = '{{ comment }}';
create or replace database {{ destination_database }} clone {{ source_database }}{% if include_internal_stages %} include internal stages{% endif %} comment = '{{ comment }}';
{%- endcall %}

{%- set result = load_result('clone_database') -%}
Expand Down
22 changes: 15 additions & 7 deletions macros/grants/_helpers.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,22 +7,30 @@ Potential future consolidation ideas (kept as comments to avoid interface change
These ideas intentionally deferred to keep current refactor incremental.
#}

{% macro _grants_collect_schemas(exclude_schemas) %}
{% macro _grants_collect_schemas(schema_names, is_exclude_list=true) %}
{% set include_schemas = [] %}
{% if exclude_schemas is not iterable %}
{% set exclude_schemas = [] %}
{% if schema_names is not iterable %}
{% set schema_names = [] %}
{% endif %}
{% if "INFORMATION_SCHEMA" not in exclude_schemas %}
{% do exclude_schemas.append("INFORMATION_SCHEMA") %}
{% if "INFORMATION_SCHEMA" not in schema_names and is_exclude_list %}
{% do schema_names.append("INFORMATION_SCHEMA") %}
{% endif %}
{% set query %}
show schemas in database {{ target.database }};
Comment on lines +10 to 19
Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai bot Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Callers and helper both manage INFORMATION_SCHEMA in exclude mode, leading to duplicated responsibility.

The new is_exclude_list flag makes _grants_collect_schemas append INFORMATION_SCHEMA for exclude lists, but some callers (e.g. grant_* macros) still append it before calling _grants_collect_schemas(..., is_exclude_list=true). Even though double-insertion is avoided, the responsibility for excluding INFORMATION_SCHEMA is now split. It would be clearer to handle this in a single place (either only in _grants_collect_schemas or only in the callers).

Suggested implementation:

These ideas intentionally deferred to keep current refactor incremental.
#}

-- NOTE:
-- Responsibility for excluding INFORMATION_SCHEMA is centralized here.
-- Callers MUST NOT add "INFORMATION_SCHEMA" to schema_names themselves when
-- using exclude-mode; instead, pass only the user-specified schemas and
-- rely on this helper to append INFORMATION_SCHEMA when is_exclude_list is true.
{% macro _grants_collect_schemas(schema_names, is_exclude_list=true) %}
    {% set include_schemas = [] %}
    {% if schema_names is not iterable %}
        {% set schema_names = [] %}
    {% endif %}
    {% if is_exclude_list and "INFORMATION_SCHEMA" not in schema_names %}
        {% do schema_names.append("INFORMATION_SCHEMA") %}
    {% endif %}
    {% set query %}

To fully implement the suggestion and avoid duplicated responsibility:

  1. Find all callers of _grants_collect_schemas (for example, in grant_* macros in the grants-related SQL macro files).
  2. In exclude-mode callers (where is_exclude_list is true or omitted to use the default), remove any manual appending of "INFORMATION_SCHEMA" to the schema list before calling _grants_collect_schemas.
    • e.g. change patterns like:
      • {% set exclude_schemas = exclude_schemas + ["INFORMATION_SCHEMA"] %}
      • or {% do exclude_schemas.append("INFORMATION_SCHEMA") %}
        before _grants_collect_schemas(exclude_schemas, is_exclude_list=true)
        so that "INFORMATION_SCHEMA" is no longer added explicitly.
  3. Ensure include-mode callers (where is_exclude_list is false) do not rely on this helper to manage "INFORMATION_SCHEMA"; they should pass the exact include list they want.

{% endset %}
{% set results = run_query(query) %}
{% if execute and results %}
{% for row in results %}
{% if row.name not in exclude_schemas and row.name not in include_schemas %}
{% do include_schemas.append(row.name) %}
{% if row.name not in include_schemas %}
{% if is_exclude_list %}
{% if row.name not in schema_names %}
{% do include_schemas.append(row.name) %}
{% endif %}
{% else %}
{% if row.name in schema_names %}
{% do include_schemas.append(row.name) %}
{% endif %}
{% endif %}
{% endif %}
{% endfor %}
{% endif %}
Expand Down
57 changes: 57 additions & 0 deletions macros/grants/grant_external_share_read.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
{% macro grant_internal_share_read(share_name, include_schemas, dry_run=false) %}
Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai bot Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): Macro name and file name are mismatched and appear to duplicate an existing macro, which is likely to override the original implementation.

This file is named grant_external_share_read.sql but defines grant_internal_share_read, which already exists in macros/grants/grant_internal_share_read.sql. In dbt the last definition wins, so this will silently override the original. Please either rename this macro (e.g. to grant_external_share_read) or consolidate the logic to avoid unexpected overrides and ambiguity about which implementation is used.

{% if flags.WHICH in ['run', 'run-operation'] %}
{% if execute %}
{% set database = target.database %}
{% if include_schemas is none %}
{% set include_schemas = [] %}
{% elif include_schemas is string %}
{% set include_schemas = [include_schemas] %}
{% elif include_schemas is not iterable %}
{% set include_schemas = [] %}
{% endif %}
{% if include_schemas | length == 0 %}
{% do log("No schemas to grant for share: " ~ share_name, info=True) %}
{% do return(none) %}
{% endif %}
{% set schemas = dbt_dataengineers_utils._grants_collect_schemas(include_schemas, is_exclude_list=false) %}
{% do log("Granting SELECT on all tables and views in all schemas for share: " ~ share_name ~ " (dry_run=" ~ dry_run ~ ")", info=True) %}
{% for schema in schemas %}
{% set grant_usage %}
grant usage on schema {{ database }}.{{ schema }} to share {{ share_name }};
{% endset %}
{% if dry_run %}
{% do log(grant_usage, info=True) %}
{% else %}
{% do run_query(grant_usage) %}
{% endif %}
{# Grant SELECT on all tables in schema #}
{% set grant_tables_sql %}
grant select on all tables in schema {{ database }}.{{ schema }} to share {{ share_name }};
{% endset %}
{% if dry_run %}
{% do log(grant_tables_sql, info=True) %}
{% else %}
{% do run_query(grant_tables_sql) %}
{% endif %}

{# Grant SELECT on views individually #}
{% set views_query %}
show views in schema {{ database }}.{{ schema }};
{% endset %}
{% set views_result = run_query(views_query) %}
{% if execute and views_result is not none %}
{% for row in views_result %}
{% set grant_view_sql %}
grant select on view {{ database }}.{{ schema }}.{{ row.name }} to share {{ share_name }};
{% endset %}
{% if dry_run %}
{% do log(grant_view_sql, info=True) %}
{% else %}
{% do run_query(grant_view_sql) %}
{% endif %}
{% endfor %}
{% endif %}
{% endfor %}
{% endif %}
{% endif %}
{% endmacro %}
2 changes: 1 addition & 1 deletion macros/grants/grant_internal_share_read.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
{% elif exclude_schemas is not iterable %}
{% set exclude_schemas = [] %}
{% endif %}
{% set schemas = dbt_dataengineers_utils._grants_collect_schemas(exclude_schemas) %}
{% set schemas = dbt_dataengineers_utils._grants_collect_schemas(exclude_schemas, is_exclude_list=true) %}
{% do log("Granting SELECT on all tables and views in all schemas for share: " ~ share_name ~ " (dry_run=" ~ dry_run ~ ")", info=True) %}
{% for schema in schemas %}
{% set grant_usage %}
Expand Down
2 changes: 1 addition & 1 deletion macros/grants/grant_procedure_usage.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
{% if flags.WHICH not in ['run','run-operation'] %}{% do log('grant_schema_procedure_usage: skip (context)', info=True) %}{% do return(none) %}{% endif %}
{% set dry_run = var('grants_dry_run', false) %}
{% if 'INFORMATION_SCHEMA' not in exclude_schemas %}{% do exclude_schemas.append('INFORMATION_SCHEMA') %}{% endif %}
{% set include_schemas = dbt_dataengineers_utils._grants_collect_schemas(exclude_schemas) %}
{% set include_schemas = dbt_dataengineers_utils._grants_collect_schemas(exclude_schemas, is_exclude_list=true) %}
{% if include_schemas | length == 0 %}{% do log('grant_schema_procedure_usage: no schemas to process', info=True) %}{% do return(none) %}{% endif %}
{% do log('grant_schema_procedure_usage: processing ' ~ (include_schemas | length) ~ ' schemas for roles: ' ~ (grant_roles | join(', ')), info=True) %}
{% do dbt_dataengineers_utils.grant_schema_procedure_usage_specific(include_schemas, grant_roles, true, dry_run) %}
Expand Down
2 changes: 1 addition & 1 deletion macros/grants/grant_schema_monitor.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
{% if flags.WHICH not in ['run','run-operation'] %}{% do log('grant_schema_monitor: skip (context)', info=True) %}{% do return(none) %}{% endif %}
{% set dry_run = var('grants_dry_run', false) %}
{% if 'INFORMATION_SCHEMA' not in exclude_schemas %}{% do exclude_schemas.append('INFORMATION_SCHEMA') %}{% endif %}
{% set include_schemas = dbt_dataengineers_utils._grants_collect_schemas(exclude_schemas) %}
{% set include_schemas = dbt_dataengineers_utils._grants_collect_schemas(exclude_schemas, is_exclude_list=true) %}
{% if include_schemas | length == 0 %}{% do log('grant_schema_monitor: no schemas to process', info=True) %}{% do return(none) %}{% endif %}
{% do log('grant_schema_monitor: processing ' ~ (include_schemas | length) ~ ' schemas for roles: ' ~ (grant_roles | join(', ')), info=True) %}
{% do dbt_dataengineers_utils.grant_schema_monitor_specific(include_schemas, grant_roles, true, dry_run) %}
Expand Down
2 changes: 1 addition & 1 deletion macros/grants/grant_schema_operate.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
{% if flags.WHICH not in ['run','run-operation'] %}{% do log('grant_schema_operate: skip (context)', info=True) %}{% do return(none) %}{% endif %}
{% set dry_run = var('grants_dry_run', false) %}
{% if 'INFORMATION_SCHEMA' not in exclude_schemas %}{% do exclude_schemas.append('INFORMATION_SCHEMA') %}{% endif %}
{% set include_schemas = dbt_dataengineers_utils._grants_collect_schemas(exclude_schemas) %}
{% set include_schemas = dbt_dataengineers_utils._grants_collect_schemas(exclude_schemas, is_exclude_list=true) %}
{% if include_schemas | length == 0 %}{% do log('grant_schema_operate: no schemas to process', info=True) %}{% do return(none) %}{% endif %}
{% do log('grant_schema_operate: processing ' ~ (include_schemas | length) ~ ' schemas for roles: ' ~ (grant_roles | join(', ')), info=True) %}
{% do dbt_dataengineers_utils.grant_schema_operate_specific(include_schemas, grant_roles, true, dry_run) %}
Expand Down
2 changes: 1 addition & 1 deletion macros/grants/grant_schema_ownership.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
{% do exclude_schemas.append('INFORMATION_SCHEMA') %}
{% endif %}

{% set include_schemas = dbt_dataengineers_utils._grants_collect_schemas(exclude_schemas) %}
{% set include_schemas = dbt_dataengineers_utils._grants_collect_schemas(exclude_schemas, is_exclude_list=true) %}
{% if include_schemas | length == 0 %}
{% do log('No schemas eligible for ownership processing in ' ~ target.database, info=True) %}
{% do return(none) %}
Expand Down
2 changes: 1 addition & 1 deletion macros/grants/grant_schema_read.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
{% if 'INFORMATION_SCHEMA' not in exclude_schemas %}
{% do exclude_schemas.append('INFORMATION_SCHEMA') %}
{% endif %}
{% set include_schemas = dbt_dataengineers_utils._grants_collect_schemas(exclude_schemas) %}
{% set include_schemas = dbt_dataengineers_utils._grants_collect_schemas(exclude_schemas, is_exclude_list=true) %}
{% if include_schemas | length == 0 %}
{% do log('grant_schema_read: no schemas to process', info=True) %}
{% do return(none) %}
Expand Down
2 changes: 1 addition & 1 deletion macros/grants/grant_semantic_views_privileges.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
{% if 'INFORMATION_SCHEMA' not in exclude_schemas %}
{% do exclude_schemas.append('INFORMATION_SCHEMA') %}
{% endif %}
{% set include_schemas = dbt_dataengineers_utils._grants_collect_schemas(exclude_schemas) %}
{% set include_schemas = dbt_dataengineers_utils._grants_collect_schemas(exclude_schemas, is_exclude_list=true) %}
{% if include_schemas | length == 0 %}
{% do log('grant_semantic_views_privileges: no schemas to process', info=True) %}
{% do return(none) %}
Expand Down