Advanced storage mapping configuration view

[GregorShear]

Data Plane Selection & Storage Mappings UI Redesign

Problem

Customers lack visibility and control over storage mappings configuration:

• No UI option for Azure Blob Storage (despite backend support)
• Cannot create subprefix storage mappings (e.g., tenant/eu/) without contacting support
• Cannot select which data planes are associated with a storage mapping
• No way to see or set which data plane is the default
• Array ordering determines the default, which isn't communicated to users

Solution

Replace the current admin view with a clearer interface showing the relationship between storage mappings and data planes.

Mockup: https://estuary-ui.vercel.app/data-planes

Key Features

• Hierarchical Storage Mapping View — Display mappings grouped by tenant prefix with provider, bucket, region, and status.
• Storage Mapping Configuration Modal — Edit mapping details, select default data plane from dropdown, guide users toward data planes in this preferred order: private > same region > everything else
• Verify fragment store authorization — Help users add dataplane service creds to their gs/s3 accounts and test connection right from the UI. Block until connections are verified.

Requirements

Continue development in gql api to support this feature:

Mutations

• createStorageMapping - prevent creating new storage mappings that would impact existing specs
• updateStorageMapping - still need to understand how to handle republishing specs (and potential failures).

Queries

• prefixes - consider extending prefix query to return a list of all prefixes/subprefixes, and associated specs - would need something like this for autocomplete prefix fields in the UI, and letting the user know they can't create a new storage mapping because of existing specs

References

• PRD: Data Plane Selection & Storage Mappings Mk2

[GregorShear]

Some notes from Slack DMs:

• Omit “Mappings” term - To end users it is confusing as mapping usually implies a key -> value, or source -> target relation, which is very much abstracted away here. How about “Collection Storage(s)“.
• Call attention to which specific regions are mismatched - “Storage mapping region us-east-1 does not match the default data plane us-west-1 for this bucket.” Also I’d move this warning to where the action to fix it is located, so under the “default data plane” option. Finally, let's say “To avoid egress fees” vs “For the lowest latency”

In the “New Mapping” modal:

• Our own “Prefix” and “Storage prefix” for the buckets in the same context can be very overloaded. Maybe we further disambiguate by calling our prefixes as “Estuary Prefix”
• There are multiple mentions of “Flow” when it should be just “Estuary”

In the "edit storage mapping" modal:

• omit redundant “Name” field because the title of the modal is already the name
• “Provider” is not consistent with “Cloud Provider” from the “New Mapping” modal
• “Tenant Prefix” is not consistent with the “Prefix” from the “New Mapping” modal
• “Available Data Planes” is not consistent with the “Data Planes” from the “New Mapping” modal

[GregorShear]

Reviewed this dialog during UI/UX call today, general agreement this is a good direction

Also decided that post-edit republishing can happen (and fail) silently for now with a "touch publish" for each affected spec under the storage mapping. I'm going to get more detail from Phil on this...