From f833154217ded184e4c9bb9942366806e48b147a Mon Sep 17 00:00:00 2001 From: Cory O'Daniel Date: Wed, 14 Jan 2026 14:23:19 -0800 Subject: [PATCH 1/4] integration docs --- sidebars.js | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sidebars.js b/sidebars.js index 1288281..372caf3 100644 --- a/sidebars.js +++ b/sidebars.js @@ -52,6 +52,11 @@ module.exports = { label: "Guides", items: [{ type: "autogenerated", dirName: "guides" }], }, + { + type: "category", + label: "Integrations", + items: [{ type: "autogenerated", dirName: "integrations" }], + }, { type: "category", label: "JSON Schema Cheat Sheet", From f7106af2d6c23330ff700a8efc03fb10af602f34 Mon Sep 17 00:00:00 2001 From: Cory O'Daniel Date: Wed, 14 Jan 2026 14:23:46 -0800 Subject: [PATCH 2/4] integration docs --- docs/integrations/00-overview.md | 31 ++++ .../01-aws-cost-and-usage-reports.md | 158 ++++++++++++++++++ .../02-azure-cost-management-exports.md | 151 +++++++++++++++++ 3 files changed, 340 insertions(+) create mode 100644 docs/integrations/00-overview.md create mode 100644 docs/integrations/01-aws-cost-and-usage-reports.md create mode 100644 docs/integrations/02-azure-cost-management-exports.md diff --git a/docs/integrations/00-overview.md b/docs/integrations/00-overview.md new file mode 100644 index 0000000..1a9d5d0 --- /dev/null +++ b/docs/integrations/00-overview.md @@ -0,0 +1,31 @@ +# Integrations + +Massdriver integrations connect your cloud accounts to enable cost tracking, resource monitoring, and other platform features. + +## Available Integrations + +### Cost Management + +| Integration | Cloud Provider | Description | +|-------------|----------------|-------------| +| [AWS Cost and Usage Reports](./aws-cost-and-usage-reports) | AWS | Collect detailed billing data from AWS using Cost and Usage Reports | +| [Azure Cost Management Exports](./azure-cost-management-exports) | Azure | Collect cost data from Azure using Cost Management Exports | + +## How Integrations Work + +1. **Provision Cloud Resources** - Run the OpenTofu module provided for each integration to create the necessary cloud resources (storage, reports, IAM roles) +2. **Configure Integration** - Provide the outputs from OpenTofu to Massdriver via the API or UI +3. **Enable Integration** - Massdriver validates access to your resources and begins collecting data +4. **Automated Collection** - Data is collected automatically on a daily schedule + +## Prerequisites + +All integrations require: + +- An active Massdriver organization +- Access to your cloud provider account with permissions to create resources +- [OpenTofu](https://opentofu.org/) or Terraform installed locally + +## OpenTofu Modules + +Each integration has an OpenTofu module available in the [Massdriver Integrations repository](https://github.com/massdriver-cloud/integrations). These modules create all necessary cloud resources with minimal permissions following the principle of least privilege. diff --git a/docs/integrations/01-aws-cost-and-usage-reports.md b/docs/integrations/01-aws-cost-and-usage-reports.md new file mode 100644 index 0000000..347a19d --- /dev/null +++ b/docs/integrations/01-aws-cost-and-usage-reports.md @@ -0,0 +1,158 @@ +# AWS Cost and Usage Reports + +The AWS Cost and Usage Reports integration enables Massdriver to collect detailed billing data from your AWS account, allowing you to track costs by package and resource. + +## Prerequisites + +- AWS account with billing access +- [OpenTofu](https://opentofu.org/) or Terraform installed +- Permissions to create IAM roles, S3 buckets, and CUR reports + +## Setup + +### Step 1: Clone the Integration Module + +```bash +git clone https://github.com/massdriver-cloud/integrations.git +cd integrations/aws-cost-and-usage-reports +``` + +### Step 2: Configure Variables + +Create a `terraform.tfvars` file: + +```hcl +massdriver_aws_account_id = "YOUR_MASSDRIVER_ACCOUNT_ID" +``` + +Contact Massdriver support for your `massdriver_aws_account_id` value. + +### Step 3: Apply the Module + +```bash +tofu init +tofu plan +tofu apply +``` + +### Step 4: Retrieve Outputs + +After applying, retrieve the configuration values: + +```bash +tofu output -json massdriver_integration_config +``` + +This outputs: + +```json +{ + "iam_role_arn": "arn:aws:iam::123456789012:role/massdriver-cur-reader", + "external_id": "abc123-def456-...", + "bucket_name": "massdriver-costs-a1b2c3d4" +} +``` + +### Step 5: Configure Massdriver + +Provide the following values when configuring the integration in Massdriver: + +| Field | Description | Source | +|-------|-------------|--------| +| IAM Role ARN | The role Massdriver assumes to read reports | `iam_role_arn` output | +| External ID | Security token for role assumption | `external_id` output | +| S3 Bucket Name | Where CUR reports are stored | `bucket_name` output | + +## Resources Created + +The OpenTofu module creates: + +| Resource | Name | Purpose | +|----------|------|---------| +| S3 Bucket | `massdriver-costs-{hash}` | Stores Cost and Usage Reports | +| S3 Bucket Policy | - | Allows AWS Billing to write reports | +| CUR Report | `massdriver-costs` | Daily cost report with resource-level details | +| IAM Role | `massdriver-cur-reader` | Cross-account role for Massdriver | +| IAM Policy | `massdriver-cur-reader-policy` | Minimal S3 read + tagging permissions | + +## IAM Permissions + +The IAM role grants Massdriver these minimal permissions: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": ["s3:ListBucket"], + "Resource": "arn:aws:s3:::massdriver-costs-*" + }, + { + "Effect": "Allow", + "Action": ["s3:GetObject"], + "Resource": "arn:aws:s3:::massdriver-costs-*/*" + }, + { + "Effect": "Allow", + "Action": ["tag:GetResources"], + "Resource": "*" + } + ] +} +``` + +## Report Configuration + +The CUR report is configured with: + +- **Time Granularity**: Daily +- **Format**: CSV (text/csv) +- **Compression**: ZIP +- **Additional Schema Elements**: RESOURCES (resource-level details) +- **Report Versioning**: OVERWRITE_REPORT + +:::note +Cost and Usage Reports can only be created in `us-east-1`. The S3 bucket is also created in this region. +::: + +## Data Collection + +Once enabled, Massdriver: + +1. Assumes the IAM role using the external ID +2. Lists the S3 bucket for available reports +3. Downloads and parses the latest report +4. Aggregates costs by `md-package` tag +5. Stores daily and monthly cost data + +Data is collected every 24 hours. + +## Troubleshooting + +### Enable fails with "access_denied" + +The IAM role trust policy may not include the correct Massdriver AWS account ID. Verify the `massdriver_aws_account_id` variable and re-apply. + +### Enable fails with "bucket_not_found" + +The S3 bucket doesn't exist or the IAM role doesn't have `s3:ListBucket` permission. Verify the OpenTofu module was applied successfully. + +### No cost data appears + +- Verify resources have the `md-package` tag applied +- CUR reports take up to 24 hours to generate initially +- Check that the report is being written to the S3 bucket + +## Cleanup + +To remove the integration resources: + +```bash +cd integrations/aws-cost-and-usage-reports +tofu destroy +``` + +:::warning +This will delete the S3 bucket and all stored reports. Cost data already collected by Massdriver will be retained. +::: diff --git a/docs/integrations/02-azure-cost-management-exports.md b/docs/integrations/02-azure-cost-management-exports.md new file mode 100644 index 0000000..2e5614b --- /dev/null +++ b/docs/integrations/02-azure-cost-management-exports.md @@ -0,0 +1,151 @@ +# Azure Cost Management Exports + +The Azure Cost Management Exports integration enables Massdriver to collect detailed billing data from your Azure subscription, allowing you to track costs by package and resource. + +## Prerequisites + +- Azure subscription with Cost Management access +- [OpenTofu](https://opentofu.org/) or Terraform installed +- Permissions to create resource groups, storage accounts, and service principals + +## Setup + +### Step 1: Clone the Integration Module + +```bash +git clone https://github.com/massdriver-cloud/integrations.git +cd integrations/azure-cost-management-exports +``` + +### Step 2: Authenticate with Azure + +```bash +az login +az account set --subscription "YOUR_SUBSCRIPTION_ID" +``` + +### Step 3: Apply the Module + +```bash +tofu init +tofu plan +tofu apply +``` + +### Step 4: Retrieve Outputs + +After applying, retrieve the configuration values: + +```bash +tofu output -json massdriver_integration_config +``` + +This outputs: + +```json +{ + "tenant_id": "abc123-...", + "subscription_id": "def456-...", + "client_id": "ghi789-...", + "client_secret": "***", + "storage_account_name": "mdcostsa1b2c3d4", + "container_name": "massdriver-costs-a1b2c3d4" +} +``` + +### Step 5: Configure Massdriver + +Provide the following values when configuring the integration in Massdriver: + +| Field | Description | Source | +|-------|-------------|--------| +| Tenant ID | Your Azure AD tenant | `tenant_id` output | +| Subscription ID | Azure subscription to monitor | `subscription_id` output | +| Client ID | Service principal application ID | `client_id` output | +| Client Secret | Service principal secret | `client_secret` output | +| Storage Account Name | Where cost exports are stored | `storage_account_name` output | +| Container Name | Blob container for exports | `container_name` output | + +## Resources Created + +The OpenTofu module creates: + +| Resource | Name | Purpose | +|----------|------|---------| +| Resource Group | `massdriver-costs-{hash}` | Contains cost management resources | +| Storage Account | `mdcosts{hash}` | Stores cost export files | +| Blob Container | `massdriver-costs-{hash}` | Container for export CSV files | +| Cost Export | `massdriver-costs` | Daily cost export schedule | +| Azure AD Application | `massdriver-cost-reader` | App registration for Massdriver | +| Service Principal | - | Identity for Massdriver access | +| Role Assignment | Storage Blob Data Reader | Read access to cost exports | + +## Service Principal Permissions + +The service principal is granted minimal read-only access: + +- **Role**: `Storage Blob Data Reader` +- **Scope**: The storage account containing cost exports + +This follows the principle of least privilege - Massdriver can only read the exported cost data. + +## Export Configuration + +The cost export is configured with: + +- **Type**: ActualCost (not forecasted) +- **Timeframe**: MonthToDate +- **Granularity**: Daily +- **Format**: CSV +- **Schedule**: Daily recurrence + +:::note +All resources are created in the `eastus` region. Cost exports include all resources across all regions in the subscription. +::: + +## Data Collection + +Once enabled, Massdriver: + +1. Authenticates using the service principal credentials +2. Lists blobs in the cost export container +3. Downloads the latest export CSV +4. Parses and aggregates costs by `md-package` tag +5. Stores daily and monthly cost data + +Data is collected every 24 hours. + +## Troubleshooting + +### Enable fails with "access_denied" + +The service principal may not have the correct role assignment. Verify the Storage Blob Data Reader role is assigned to the storage account. + +### Enable fails with "container_not_found" + +The blob container doesn't exist. Verify the OpenTofu module was applied successfully and the container name matches. + +### No cost data appears + +- Verify resources have the `md-package` tag applied +- Cost exports take up to 24 hours to generate initially +- Check the Azure portal to confirm exports are being written + +### Authentication errors + +- Verify the client secret hasn't expired +- Confirm the service principal is in the correct tenant +- Check that the subscription ID matches + +## Cleanup + +To remove the integration resources: + +```bash +cd integrations/azure-cost-management-exports +tofu destroy +``` + +:::warning +This will delete the storage account and all stored exports. Cost data already collected by Massdriver will be retained. +::: From 967b71019a927ca7d3629c2e41c7c3dc651fcfa6 Mon Sep 17 00:00:00 2001 From: Cory O'Daniel Date: Wed, 14 Jan 2026 14:34:43 -0800 Subject: [PATCH 3/4] opentofu --- docs/integrations/01-aws-cost-and-usage-reports.md | 4 ++-- docs/integrations/02-azure-cost-management-exports.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/integrations/01-aws-cost-and-usage-reports.md b/docs/integrations/01-aws-cost-and-usage-reports.md index 347a19d..8a9b59b 100644 --- a/docs/integrations/01-aws-cost-and-usage-reports.md +++ b/docs/integrations/01-aws-cost-and-usage-reports.md @@ -5,7 +5,7 @@ The AWS Cost and Usage Reports integration enables Massdriver to collect detaile ## Prerequisites - AWS account with billing access -- [OpenTofu](https://opentofu.org/) or Terraform installed +- [OpenTofu](https://opentofu.org/) installed - Permissions to create IAM roles, S3 buckets, and CUR reports ## Setup @@ -19,7 +19,7 @@ cd integrations/aws-cost-and-usage-reports ### Step 2: Configure Variables -Create a `terraform.tfvars` file: +Create a `tofu.tfvars` file: ```hcl massdriver_aws_account_id = "YOUR_MASSDRIVER_ACCOUNT_ID" diff --git a/docs/integrations/02-azure-cost-management-exports.md b/docs/integrations/02-azure-cost-management-exports.md index 2e5614b..9033bfa 100644 --- a/docs/integrations/02-azure-cost-management-exports.md +++ b/docs/integrations/02-azure-cost-management-exports.md @@ -5,7 +5,7 @@ The Azure Cost Management Exports integration enables Massdriver to collect deta ## Prerequisites - Azure subscription with Cost Management access -- [OpenTofu](https://opentofu.org/) or Terraform installed +- [OpenTofu](https://opentofu.org/) installed - Permissions to create resource groups, storage accounts, and service principals ## Setup From 52643ebaec669dcb46c6f18508c88350e0078bbe Mon Sep 17 00:00:00 2001 From: Cory O'Daniel Date: Wed, 14 Jan 2026 15:15:18 -0800 Subject: [PATCH 4/4] aws keys --- .../01-aws-cost-and-usage-reports.md | 44 +++++++++---------- 1 file changed, 20 insertions(+), 24 deletions(-) diff --git a/docs/integrations/01-aws-cost-and-usage-reports.md b/docs/integrations/01-aws-cost-and-usage-reports.md index 8a9b59b..96c239b 100644 --- a/docs/integrations/01-aws-cost-and-usage-reports.md +++ b/docs/integrations/01-aws-cost-and-usage-reports.md @@ -6,7 +6,7 @@ The AWS Cost and Usage Reports integration enables Massdriver to collect detaile - AWS account with billing access - [OpenTofu](https://opentofu.org/) installed -- Permissions to create IAM roles, S3 buckets, and CUR reports +- Permissions to create IAM users, S3 buckets, and CUR reports ## Setup @@ -17,17 +17,7 @@ git clone https://github.com/massdriver-cloud/integrations.git cd integrations/aws-cost-and-usage-reports ``` -### Step 2: Configure Variables - -Create a `tofu.tfvars` file: - -```hcl -massdriver_aws_account_id = "YOUR_MASSDRIVER_ACCOUNT_ID" -``` - -Contact Massdriver support for your `massdriver_aws_account_id` value. - -### Step 3: Apply the Module +### Step 2: Apply the Module ```bash tofu init @@ -35,7 +25,7 @@ tofu plan tofu apply ``` -### Step 4: Retrieve Outputs +### Step 3: Retrieve Outputs After applying, retrieve the configuration values: @@ -47,20 +37,20 @@ This outputs: ```json { - "iam_role_arn": "arn:aws:iam::123456789012:role/massdriver-cur-reader", - "external_id": "abc123-def456-...", + "access_key_id": "AKIAIOSFODNN7EXAMPLE", + "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", "bucket_name": "massdriver-costs-a1b2c3d4" } ``` -### Step 5: Configure Massdriver +### Step 4: Configure Massdriver Provide the following values when configuring the integration in Massdriver: | Field | Description | Source | |-------|-------------|--------| -| IAM Role ARN | The role Massdriver assumes to read reports | `iam_role_arn` output | -| External ID | Security token for role assumption | `external_id` output | +| Access Key ID | IAM user access key | `access_key_id` output | +| Secret Access Key | IAM user secret key | `secret_access_key` output | | S3 Bucket Name | Where CUR reports are stored | `bucket_name` output | ## Resources Created @@ -72,17 +62,23 @@ The OpenTofu module creates: | S3 Bucket | `massdriver-costs-{hash}` | Stores Cost and Usage Reports | | S3 Bucket Policy | - | Allows AWS Billing to write reports | | CUR Report | `massdriver-costs` | Daily cost report with resource-level details | -| IAM Role | `massdriver-cur-reader` | Cross-account role for Massdriver | -| IAM Policy | `massdriver-cur-reader-policy` | Minimal S3 read + tagging permissions | +| IAM User | `massdriver-costs` | Dedicated user for Massdriver access | +| IAM Policy | `massdriver-costs-policy` | Minimal S3 read + tagging permissions | +| Access Key | - | Credentials for the IAM user | ## IAM Permissions -The IAM role grants Massdriver these minimal permissions: +The IAM user has these minimal permissions: ```json { "Version": "2012-10-17", "Statement": [ + { + "Effect": "Allow", + "Action": ["s3:HeadBucket"], + "Resource": "*" + }, { "Effect": "Allow", "Action": ["s3:ListBucket"], @@ -120,7 +116,7 @@ Cost and Usage Reports can only be created in `us-east-1`. The S3 bucket is also Once enabled, Massdriver: -1. Assumes the IAM role using the external ID +1. Authenticates using the IAM user credentials 2. Lists the S3 bucket for available reports 3. Downloads and parses the latest report 4. Aggregates costs by `md-package` tag @@ -132,11 +128,11 @@ Data is collected every 24 hours. ### Enable fails with "access_denied" -The IAM role trust policy may not include the correct Massdriver AWS account ID. Verify the `massdriver_aws_account_id` variable and re-apply. +The IAM user may not have the required permissions. Verify the OpenTofu module was applied successfully and the policy is attached. ### Enable fails with "bucket_not_found" -The S3 bucket doesn't exist or the IAM role doesn't have `s3:ListBucket` permission. Verify the OpenTofu module was applied successfully. +The S3 bucket doesn't exist or the IAM user doesn't have `s3:HeadBucket` permission. Verify the OpenTofu module was applied successfully. ### No cost data appears