Skip to content

Theodw 2917 part 1 convert curated managed tables to external tables in curated layer#2436

Closed
anushaveerapalli68 wants to merge 70 commits intomainfrom
THEODW-2917-Part-1-Convert-Curated-Managed-tables-to-External-tables-in-Curated-Layer
Closed

Theodw 2917 part 1 convert curated managed tables to external tables in curated layer#2436
anushaveerapalli68 wants to merge 70 commits intomainfrom
THEODW-2917-Part-1-Convert-Curated-Managed-tables-to-External-tables-in-Curated-Layer

Conversation

@anushaveerapalli68
Copy link
Copy Markdown
Contributor

JIRA Ticket Reference:https://pins-ds.atlassian.net/browse/THEODW-2917

  • Updated curated‑layer notebooks to preserve external table behaviour during overwrite operations.
  • Retrieved the existing table path using DESCRIBE EXTENDED <table_name>.
  • Extracted the Location value and stored it as table_path.
  • Added .option("path", table_path) to the DataFrame write operation.
  • Ensures Spark writes back to the existing external directory instead of creating a managed table.
  • Prevents curated tables from being unintentionally recreated as managed tables.
  • Curated tables now consistently remain external with their original storage paths retained.

Comment thread workspace/notebook/appeal_document.json Outdated
Comment thread workspace/notebook/appeal_document.json Outdated
Comment thread workspace/notebook/appeal_event.json Outdated
Comment thread workspace/notebook/appeal_event.json
Comment thread workspace/notebook/appeal_event.json Outdated
Comment thread workspace/notebook/appeal_representation.json Outdated
Comment thread workspace/notebook/appeal_service_user.json Outdated
Comment thread workspace/notebook/examination_timetable.json Outdated
Comment thread workspace/notebook/legacy_folder_data.json Outdated
Comment thread workspace/notebook/legacy_folder_data.json Outdated
Copy link
Copy Markdown
Contributor

@roytalari roytalari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@anushaveerapalli68 anushaveerapalli68 deleted the THEODW-2917-Part-1-Convert-Curated-Managed-tables-to-External-tables-in-Curated-Layer branch April 7, 2026 12:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants