Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions docs/integrations/00-overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Integrations

Massdriver integrations connect your cloud accounts to enable cost tracking, resource monitoring, and other platform features.

## Available Integrations

### Cost Management

| Integration | Cloud Provider | Description |
|-------------|----------------|-------------|
| [AWS Cost and Usage Reports](./aws-cost-and-usage-reports) | AWS | Collect detailed billing data from AWS using Cost and Usage Reports |
| [Azure Cost Management Exports](./azure-cost-management-exports) | Azure | Collect cost data from Azure using Cost Management Exports |

## How Integrations Work

1. **Provision Cloud Resources** - Run the OpenTofu module provided for each integration to create the necessary cloud resources (storage, reports, IAM roles)
2. **Configure Integration** - Provide the outputs from OpenTofu to Massdriver via the API or UI
3. **Enable Integration** - Massdriver validates access to your resources and begins collecting data
4. **Automated Collection** - Data is collected automatically on a daily schedule

## Prerequisites

All integrations require:

- An active Massdriver organization
- Access to your cloud provider account with permissions to create resources
- [OpenTofu](https://opentofu.org/) or Terraform installed locally

## OpenTofu Modules

Each integration has an OpenTofu module available in the [Massdriver Integrations repository](https://github.com/massdriver-cloud/integrations). These modules create all necessary cloud resources with minimal permissions following the principle of least privilege.
154 changes: 154 additions & 0 deletions docs/integrations/01-aws-cost-and-usage-reports.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
# AWS Cost and Usage Reports

The AWS Cost and Usage Reports integration enables Massdriver to collect detailed billing data from your AWS account, allowing you to track costs by package and resource.

## Prerequisites

- AWS account with billing access
- [OpenTofu](https://opentofu.org/) installed
- Permissions to create IAM users, S3 buckets, and CUR reports

## Setup

### Step 1: Clone the Integration Module

```bash
git clone https://github.com/massdriver-cloud/integrations.git
cd integrations/aws-cost-and-usage-reports
```

### Step 2: Apply the Module

```bash
tofu init
tofu plan
tofu apply
```

### Step 3: Retrieve Outputs

After applying, retrieve the configuration values:

```bash
tofu output -json massdriver_integration_config
```

This outputs:

```json
{
"access_key_id": "AKIAIOSFODNN7EXAMPLE",
"secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"bucket_name": "massdriver-costs-a1b2c3d4"
}
```

### Step 4: Configure Massdriver

Provide the following values when configuring the integration in Massdriver:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: @ChristensenJoe Id love to have the integration UI's URL here, helpful for what to do after running the integration module. any idea what the paths in the browser look like?


| Field | Description | Source |
|-------|-------------|--------|
| Access Key ID | IAM user access key | `access_key_id` output |
| Secret Access Key | IAM user secret key | `secret_access_key` output |
| S3 Bucket Name | Where CUR reports are stored | `bucket_name` output |

## Resources Created

The OpenTofu module creates:

| Resource | Name | Purpose |
|----------|------|---------|
| S3 Bucket | `massdriver-costs-{hash}` | Stores Cost and Usage Reports |
| S3 Bucket Policy | - | Allows AWS Billing to write reports |
| CUR Report | `massdriver-costs` | Daily cost report with resource-level details |
| IAM User | `massdriver-costs` | Dedicated user for Massdriver access |
| IAM Policy | `massdriver-costs-policy` | Minimal S3 read + tagging permissions |
| Access Key | - | Credentials for the IAM user |

## IAM Permissions

The IAM user has these minimal permissions:

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:HeadBucket"],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": "arn:aws:s3:::massdriver-costs-*"
},
{
"Effect": "Allow",
"Action": ["s3:GetObject"],
"Resource": "arn:aws:s3:::massdriver-costs-*/*"
},
{
"Effect": "Allow",
"Action": ["tag:GetResources"],
"Resource": "*"
}
]
}
```

## Report Configuration

The CUR report is configured with:

- **Time Granularity**: Daily
- **Format**: CSV (text/csv)
- **Compression**: ZIP
- **Additional Schema Elements**: RESOURCES (resource-level details)
- **Report Versioning**: OVERWRITE_REPORT

:::note
Cost and Usage Reports can only be created in `us-east-1`. The S3 bucket is also created in this region.
:::

## Data Collection

Once enabled, Massdriver:

1. Authenticates using the IAM user credentials
2. Lists the S3 bucket for available reports
3. Downloads and parses the latest report
4. Aggregates costs by `md-package` tag
5. Stores daily and monthly cost data

Data is collected every 24 hours.

## Troubleshooting

### Enable fails with "access_denied"

The IAM user may not have the required permissions. Verify the OpenTofu module was applied successfully and the policy is attached.

### Enable fails with "bucket_not_found"

The S3 bucket doesn't exist or the IAM user doesn't have `s3:HeadBucket` permission. Verify the OpenTofu module was applied successfully.

### No cost data appears

- Verify resources have the `md-package` tag applied
- CUR reports take up to 24 hours to generate initially
- Check that the report is being written to the S3 bucket

## Cleanup

To remove the integration resources:

```bash
cd integrations/aws-cost-and-usage-reports
tofu destroy
```

:::warning
This will delete the S3 bucket and all stored reports. Cost data already collected by Massdriver will be retained.
:::
151 changes: 151 additions & 0 deletions docs/integrations/02-azure-cost-management-exports.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
# Azure Cost Management Exports

The Azure Cost Management Exports integration enables Massdriver to collect detailed billing data from your Azure subscription, allowing you to track costs by package and resource.

## Prerequisites

- Azure subscription with Cost Management access
- [OpenTofu](https://opentofu.org/) installed
- Permissions to create resource groups, storage accounts, and service principals

## Setup

### Step 1: Clone the Integration Module

```bash
git clone https://github.com/massdriver-cloud/integrations.git
cd integrations/azure-cost-management-exports
```

### Step 2: Authenticate with Azure

```bash
az login
az account set --subscription "YOUR_SUBSCRIPTION_ID"
```

### Step 3: Apply the Module

```bash
tofu init
tofu plan
tofu apply
```

### Step 4: Retrieve Outputs

After applying, retrieve the configuration values:

```bash
tofu output -json massdriver_integration_config
```

This outputs:

```json
{
"tenant_id": "abc123-...",
"subscription_id": "def456-...",
"client_id": "ghi789-...",
"client_secret": "***",
"storage_account_name": "mdcostsa1b2c3d4",
"container_name": "massdriver-costs-a1b2c3d4"
}
```

### Step 5: Configure Massdriver

Provide the following values when configuring the integration in Massdriver:

| Field | Description | Source |
|-------|-------------|--------|
| Tenant ID | Your Azure AD tenant | `tenant_id` output |
| Subscription ID | Azure subscription to monitor | `subscription_id` output |
| Client ID | Service principal application ID | `client_id` output |
| Client Secret | Service principal secret | `client_secret` output |
| Storage Account Name | Where cost exports are stored | `storage_account_name` output |
| Container Name | Blob container for exports | `container_name` output |

## Resources Created

The OpenTofu module creates:

| Resource | Name | Purpose |
|----------|------|---------|
| Resource Group | `massdriver-costs-{hash}` | Contains cost management resources |
| Storage Account | `mdcosts{hash}` | Stores cost export files |
| Blob Container | `massdriver-costs-{hash}` | Container for export CSV files |
| Cost Export | `massdriver-costs` | Daily cost export schedule |
| Azure AD Application | `massdriver-cost-reader` | App registration for Massdriver |
| Service Principal | - | Identity for Massdriver access |
| Role Assignment | Storage Blob Data Reader | Read access to cost exports |

## Service Principal Permissions

The service principal is granted minimal read-only access:

- **Role**: `Storage Blob Data Reader`
- **Scope**: The storage account containing cost exports

This follows the principle of least privilege - Massdriver can only read the exported cost data.

## Export Configuration

The cost export is configured with:

- **Type**: ActualCost (not forecasted)
- **Timeframe**: MonthToDate
- **Granularity**: Daily
- **Format**: CSV
- **Schedule**: Daily recurrence

:::note
All resources are created in the `eastus` region. Cost exports include all resources across all regions in the subscription.
:::

## Data Collection

Once enabled, Massdriver:

1. Authenticates using the service principal credentials
2. Lists blobs in the cost export container
3. Downloads the latest export CSV
4. Parses and aggregates costs by `md-package` tag
5. Stores daily and monthly cost data

Data is collected every 24 hours.

## Troubleshooting

### Enable fails with "access_denied"

The service principal may not have the correct role assignment. Verify the Storage Blob Data Reader role is assigned to the storage account.

### Enable fails with "container_not_found"

The blob container doesn't exist. Verify the OpenTofu module was applied successfully and the container name matches.

### No cost data appears

- Verify resources have the `md-package` tag applied
- Cost exports take up to 24 hours to generate initially
- Check the Azure portal to confirm exports are being written

### Authentication errors

- Verify the client secret hasn't expired
- Confirm the service principal is in the correct tenant
- Check that the subscription ID matches

## Cleanup

To remove the integration resources:

```bash
cd integrations/azure-cost-management-exports
tofu destroy
```

:::warning
This will delete the storage account and all stored exports. Cost data already collected by Massdriver will be retained.
:::
5 changes: 5 additions & 0 deletions sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,11 @@ module.exports = {
label: "Guides",
items: [{ type: "autogenerated", dirName: "guides" }],
},
{
type: "category",
label: "Integrations",
items: [{ type: "autogenerated", dirName: "integrations" }],
},
{
type: "category",
label: "JSON Schema Cheat Sheet",
Expand Down