Terraform — Azure Databricks with Unity Catalog & ADLS Gen2

This Terraform project provisions an end-to-end Azure Databricks workspace with Unity Catalog backed by ADLS Gen2 external storage.

Architecture

┌─────────────────────────────────────────────────────────────┐
│  Azure Resource Group                                       │
│                                                             │
│  ┌───────────────────┐     Storage Blob Data Contributor    │
│  │  ADLS Gen2         │◄────────────────────────────────┐   │
│  │  (HNS enabled)     │                                 │   │
│  │  └─ Container      │                                 │   │
│  └───────────────────┘                                  │   │
│                                                         │   │
│  ┌───────────────────┐    System-Assigned Identity       │   │
│  │  Access Connector  │─────────────────────────────────┘   │
│  └────────┬──────────┘                                      │
│           │                                                 │
│  ┌────────▼──────────┐                                      │
│  │  Databricks        │                                     │
│  │  Workspace         │                                     │
│  │  (Premium SKU)     │                                     │
│  │                    │     ┌──────────────────────────┐    │
│  │  Unity Catalog:    │────►│  Existing Metastore      │    │
│  │  ├─ Storage Cred   │     └──────────────────────────┘    │
│  │  ├─ External Loc   │                                     │
│  │  └─ Catalog        │                                     │
│  └───────────────────┘                                      │
└─────────────────────────────────────────────────────────────┘

Resources Created

Resource	Type	Description
Resource Group	azurerm_resource_group	Container for all Azure resources
Storage Account	azurerm_storage_account	ADLS Gen2 with hierarchical namespace enabled
Storage Container	azurerm_storage_container	Private container for Unity Catalog data
Access Connector	azurerm_databricks_access_connector	Managed identity for Databricks-to-ADLS access
Role Assignment	azurerm_role_assignment	Grants Storage Blob Data Contributor to the access connector
Databricks Workspace	azurerm_databricks_workspace	Premium-tier workspace
Metastore Assignment	databricks_metastore_assignment	Attaches workspace to an existing Unity Catalog metastore
Storage Credential	databricks_storage_credential	Unity Catalog credential backed by the access connector
External Location	databricks_external_location	Maps the ADLS container as a Unity Catalog external location
Catalog	databricks_catalog	Unity Catalog catalog with storage root at the external location

Prerequisites

• Terraform >= 1.5.0
• An Azure subscription with permissions to create resources
• An existing Databricks Unity Catalog metastore (you'll need its ID)
• Azure CLI authenticated (az login) or a service principal with environment variables:
  • ARM_CLIENT_ID
  • ARM_CLIENT_SECRET
  • ARM_TENANT_ID
  • ARM_SUBSCRIPTION_ID

File Structure

.
├── providers.tf              # Provider configuration (azurerm, databricks)
├── variables.tf              # Input variable definitions
├── main.tf                   # ADLS Gen2, access connector, role assignment
├── databricks.tf             # Databricks workspace & metastore assignment
├── unity_catalog.tf          # Storage credential, external location, catalog
├── outputs.tf                # Output values
├── terraform.tfvars.example  # Example variable values
└── README.md

Quick Start

1. Configure Variables

cp terraform.tfvars.example terraform.tfvars

Edit terraform.tfvars with your values:

region                    = "eastus"
resource_group_name       = "rg-field-eng-databricks"
storage_account_name      = "fieldengadls2026"          # must be globally unique
container_name            = "unity-catalog-data"
databricks_workspace_name = "dbw-field-eng"
access_connector_name     = "dbac-field-eng"
databricks_sku            = "premium"
metastore_id              = "<your-existing-metastore-id>"
catalog_name              = "field_eng_catalog"

tags = {
  environment = "field-engineering"
  managed_by  = "terraform"
}

2. Initialize and Apply

# Download providers
terraform init

# Preview changes
terraform plan

# Apply
terraform apply

3. Verify

After a successful apply, Terraform will output key resource identifiers:

terraform output

Input Variables

Name	Type	Required	Default	Description
region	string	no	eastus	Azure region for all resources
metastore_id	string	yes	—	Existing Unity Catalog metastore ID
resource_group_name	string	yes	—	Resource group name
storage_account_name	string	yes	—	Globally unique storage account name (3-24 lowercase alphanumeric)
container_name	string	no	unity-catalog-data	Storage container name
databricks_workspace_name	string	yes	—	Databricks workspace name
access_connector_name	string	yes	—	Databricks access connector name
catalog_name	string	no	field_eng_catalog	Unity Catalog catalog name
databricks_sku	string	no	premium	Workspace SKU (standard, premium, or trial)
tags	map(string)	no	see variables.tf	Tags applied to all resources

Outputs

Name	Description
resource_group_name	Name of the created resource group
storage_account_name	ADLS Gen2 storage account name
storage_account_id	ADLS Gen2 storage account resource ID
storage_container_url	ABFSS URL of the storage container
access_connector_id	Resource ID of the access connector
access_connector_principal_id	Managed identity principal ID
databricks_workspace_url	Databricks workspace URL
databricks_workspace_id	Databricks workspace ID
storage_credential_name	Unity Catalog storage credential name
external_location_name	Unity Catalog external location name
catalog_name	Unity Catalog catalog name

Cleanup

To destroy all resources created by this project:

terraform destroy

Note: Destroying the catalog will remove all schemas and tables within it. The metastore itself is not affected since it is externally managed.