-
Notifications
You must be signed in to change notification settings - Fork 3
integration docs #160
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
coryodaniel
wants to merge
4
commits into
main
Choose a base branch
from
integrations
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+341
−0
Open
integration docs #160
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| # Integrations | ||
|
|
||
| Massdriver integrations connect your cloud accounts to enable cost tracking, resource monitoring, and other platform features. | ||
|
|
||
| ## Available Integrations | ||
|
|
||
| ### Cost Management | ||
|
|
||
| | Integration | Cloud Provider | Description | | ||
| |-------------|----------------|-------------| | ||
| | [AWS Cost and Usage Reports](./aws-cost-and-usage-reports) | AWS | Collect detailed billing data from AWS using Cost and Usage Reports | | ||
| | [Azure Cost Management Exports](./azure-cost-management-exports) | Azure | Collect cost data from Azure using Cost Management Exports | | ||
|
|
||
| ## How Integrations Work | ||
|
|
||
| 1. **Provision Cloud Resources** - Run the OpenTofu module provided for each integration to create the necessary cloud resources (storage, reports, IAM roles) | ||
| 2. **Configure Integration** - Provide the outputs from OpenTofu to Massdriver via the API or UI | ||
| 3. **Enable Integration** - Massdriver validates access to your resources and begins collecting data | ||
| 4. **Automated Collection** - Data is collected automatically on a daily schedule | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| All integrations require: | ||
|
|
||
| - An active Massdriver organization | ||
| - Access to your cloud provider account with permissions to create resources | ||
| - [OpenTofu](https://opentofu.org/) or Terraform installed locally | ||
|
|
||
| ## OpenTofu Modules | ||
|
|
||
| Each integration has an OpenTofu module available in the [Massdriver Integrations repository](https://github.com/massdriver-cloud/integrations). These modules create all necessary cloud resources with minimal permissions following the principle of least privilege. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,154 @@ | ||
| # AWS Cost and Usage Reports | ||
|
|
||
| The AWS Cost and Usage Reports integration enables Massdriver to collect detailed billing data from your AWS account, allowing you to track costs by package and resource. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - AWS account with billing access | ||
| - [OpenTofu](https://opentofu.org/) installed | ||
| - Permissions to create IAM users, S3 buckets, and CUR reports | ||
|
|
||
| ## Setup | ||
|
|
||
| ### Step 1: Clone the Integration Module | ||
|
|
||
| ```bash | ||
| git clone https://github.com/massdriver-cloud/integrations.git | ||
| cd integrations/aws-cost-and-usage-reports | ||
| ``` | ||
|
|
||
| ### Step 2: Apply the Module | ||
|
|
||
| ```bash | ||
| tofu init | ||
| tofu plan | ||
| tofu apply | ||
| ``` | ||
|
|
||
| ### Step 3: Retrieve Outputs | ||
|
|
||
| After applying, retrieve the configuration values: | ||
|
|
||
| ```bash | ||
| tofu output -json massdriver_integration_config | ||
| ``` | ||
|
|
||
| This outputs: | ||
|
|
||
| ```json | ||
| { | ||
| "access_key_id": "AKIAIOSFODNN7EXAMPLE", | ||
| "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", | ||
| "bucket_name": "massdriver-costs-a1b2c3d4" | ||
| } | ||
| ``` | ||
|
|
||
| ### Step 4: Configure Massdriver | ||
|
|
||
| Provide the following values when configuring the integration in Massdriver: | ||
|
|
||
| | Field | Description | Source | | ||
| |-------|-------------|--------| | ||
| | Access Key ID | IAM user access key | `access_key_id` output | | ||
| | Secret Access Key | IAM user secret key | `secret_access_key` output | | ||
| | S3 Bucket Name | Where CUR reports are stored | `bucket_name` output | | ||
|
|
||
| ## Resources Created | ||
|
|
||
| The OpenTofu module creates: | ||
|
|
||
| | Resource | Name | Purpose | | ||
| |----------|------|---------| | ||
| | S3 Bucket | `massdriver-costs-{hash}` | Stores Cost and Usage Reports | | ||
| | S3 Bucket Policy | - | Allows AWS Billing to write reports | | ||
| | CUR Report | `massdriver-costs` | Daily cost report with resource-level details | | ||
| | IAM User | `massdriver-costs` | Dedicated user for Massdriver access | | ||
| | IAM Policy | `massdriver-costs-policy` | Minimal S3 read + tagging permissions | | ||
| | Access Key | - | Credentials for the IAM user | | ||
|
|
||
| ## IAM Permissions | ||
|
|
||
| The IAM user has these minimal permissions: | ||
|
|
||
| ```json | ||
| { | ||
| "Version": "2012-10-17", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": ["s3:HeadBucket"], | ||
| "Resource": "*" | ||
| }, | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": ["s3:ListBucket"], | ||
| "Resource": "arn:aws:s3:::massdriver-costs-*" | ||
| }, | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": ["s3:GetObject"], | ||
| "Resource": "arn:aws:s3:::massdriver-costs-*/*" | ||
| }, | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": ["tag:GetResources"], | ||
| "Resource": "*" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| ## Report Configuration | ||
|
|
||
| The CUR report is configured with: | ||
|
|
||
| - **Time Granularity**: Daily | ||
| - **Format**: CSV (text/csv) | ||
| - **Compression**: ZIP | ||
| - **Additional Schema Elements**: RESOURCES (resource-level details) | ||
| - **Report Versioning**: OVERWRITE_REPORT | ||
|
|
||
| :::note | ||
| Cost and Usage Reports can only be created in `us-east-1`. The S3 bucket is also created in this region. | ||
| ::: | ||
|
|
||
| ## Data Collection | ||
|
|
||
| Once enabled, Massdriver: | ||
|
|
||
| 1. Authenticates using the IAM user credentials | ||
| 2. Lists the S3 bucket for available reports | ||
| 3. Downloads and parses the latest report | ||
| 4. Aggregates costs by `md-package` tag | ||
| 5. Stores daily and monthly cost data | ||
|
|
||
| Data is collected every 24 hours. | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| ### Enable fails with "access_denied" | ||
|
|
||
| The IAM user may not have the required permissions. Verify the OpenTofu module was applied successfully and the policy is attached. | ||
|
|
||
| ### Enable fails with "bucket_not_found" | ||
|
|
||
| The S3 bucket doesn't exist or the IAM user doesn't have `s3:HeadBucket` permission. Verify the OpenTofu module was applied successfully. | ||
|
|
||
| ### No cost data appears | ||
|
|
||
| - Verify resources have the `md-package` tag applied | ||
| - CUR reports take up to 24 hours to generate initially | ||
| - Check that the report is being written to the S3 bucket | ||
|
|
||
| ## Cleanup | ||
|
|
||
| To remove the integration resources: | ||
|
|
||
| ```bash | ||
| cd integrations/aws-cost-and-usage-reports | ||
| tofu destroy | ||
| ``` | ||
|
|
||
| :::warning | ||
| This will delete the S3 bucket and all stored reports. Cost data already collected by Massdriver will be retained. | ||
| ::: | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,151 @@ | ||
| # Azure Cost Management Exports | ||
|
|
||
| The Azure Cost Management Exports integration enables Massdriver to collect detailed billing data from your Azure subscription, allowing you to track costs by package and resource. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Azure subscription with Cost Management access | ||
| - [OpenTofu](https://opentofu.org/) installed | ||
| - Permissions to create resource groups, storage accounts, and service principals | ||
|
|
||
| ## Setup | ||
|
|
||
| ### Step 1: Clone the Integration Module | ||
|
|
||
| ```bash | ||
| git clone https://github.com/massdriver-cloud/integrations.git | ||
| cd integrations/azure-cost-management-exports | ||
| ``` | ||
|
|
||
| ### Step 2: Authenticate with Azure | ||
|
|
||
| ```bash | ||
| az login | ||
| az account set --subscription "YOUR_SUBSCRIPTION_ID" | ||
| ``` | ||
|
|
||
| ### Step 3: Apply the Module | ||
|
|
||
| ```bash | ||
| tofu init | ||
| tofu plan | ||
| tofu apply | ||
| ``` | ||
|
|
||
| ### Step 4: Retrieve Outputs | ||
|
|
||
| After applying, retrieve the configuration values: | ||
|
|
||
| ```bash | ||
| tofu output -json massdriver_integration_config | ||
| ``` | ||
|
|
||
| This outputs: | ||
|
|
||
| ```json | ||
| { | ||
| "tenant_id": "abc123-...", | ||
| "subscription_id": "def456-...", | ||
| "client_id": "ghi789-...", | ||
| "client_secret": "***", | ||
| "storage_account_name": "mdcostsa1b2c3d4", | ||
| "container_name": "massdriver-costs-a1b2c3d4" | ||
| } | ||
| ``` | ||
|
|
||
| ### Step 5: Configure Massdriver | ||
|
|
||
| Provide the following values when configuring the integration in Massdriver: | ||
|
|
||
| | Field | Description | Source | | ||
| |-------|-------------|--------| | ||
| | Tenant ID | Your Azure AD tenant | `tenant_id` output | | ||
| | Subscription ID | Azure subscription to monitor | `subscription_id` output | | ||
| | Client ID | Service principal application ID | `client_id` output | | ||
| | Client Secret | Service principal secret | `client_secret` output | | ||
| | Storage Account Name | Where cost exports are stored | `storage_account_name` output | | ||
| | Container Name | Blob container for exports | `container_name` output | | ||
|
|
||
| ## Resources Created | ||
|
|
||
| The OpenTofu module creates: | ||
|
|
||
| | Resource | Name | Purpose | | ||
| |----------|------|---------| | ||
| | Resource Group | `massdriver-costs-{hash}` | Contains cost management resources | | ||
| | Storage Account | `mdcosts{hash}` | Stores cost export files | | ||
| | Blob Container | `massdriver-costs-{hash}` | Container for export CSV files | | ||
| | Cost Export | `massdriver-costs` | Daily cost export schedule | | ||
| | Azure AD Application | `massdriver-cost-reader` | App registration for Massdriver | | ||
| | Service Principal | - | Identity for Massdriver access | | ||
| | Role Assignment | Storage Blob Data Reader | Read access to cost exports | | ||
|
|
||
| ## Service Principal Permissions | ||
|
|
||
| The service principal is granted minimal read-only access: | ||
|
|
||
| - **Role**: `Storage Blob Data Reader` | ||
| - **Scope**: The storage account containing cost exports | ||
|
|
||
| This follows the principle of least privilege - Massdriver can only read the exported cost data. | ||
|
|
||
| ## Export Configuration | ||
|
|
||
| The cost export is configured with: | ||
|
|
||
| - **Type**: ActualCost (not forecasted) | ||
| - **Timeframe**: MonthToDate | ||
| - **Granularity**: Daily | ||
| - **Format**: CSV | ||
| - **Schedule**: Daily recurrence | ||
|
|
||
| :::note | ||
| All resources are created in the `eastus` region. Cost exports include all resources across all regions in the subscription. | ||
| ::: | ||
|
|
||
| ## Data Collection | ||
|
|
||
| Once enabled, Massdriver: | ||
|
|
||
| 1. Authenticates using the service principal credentials | ||
| 2. Lists blobs in the cost export container | ||
| 3. Downloads the latest export CSV | ||
| 4. Parses and aggregates costs by `md-package` tag | ||
| 5. Stores daily and monthly cost data | ||
|
|
||
| Data is collected every 24 hours. | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| ### Enable fails with "access_denied" | ||
|
|
||
| The service principal may not have the correct role assignment. Verify the Storage Blob Data Reader role is assigned to the storage account. | ||
|
|
||
| ### Enable fails with "container_not_found" | ||
|
|
||
| The blob container doesn't exist. Verify the OpenTofu module was applied successfully and the container name matches. | ||
|
|
||
| ### No cost data appears | ||
|
|
||
| - Verify resources have the `md-package` tag applied | ||
| - Cost exports take up to 24 hours to generate initially | ||
| - Check the Azure portal to confirm exports are being written | ||
|
|
||
| ### Authentication errors | ||
|
|
||
| - Verify the client secret hasn't expired | ||
| - Confirm the service principal is in the correct tenant | ||
| - Check that the subscription ID matches | ||
|
|
||
| ## Cleanup | ||
|
|
||
| To remove the integration resources: | ||
|
|
||
| ```bash | ||
| cd integrations/azure-cost-management-exports | ||
| tofu destroy | ||
| ``` | ||
|
|
||
| :::warning | ||
| This will delete the storage account and all stored exports. Cost data already collected by Massdriver will be retained. | ||
| ::: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO: @ChristensenJoe Id love to have the integration UI's URL here, helpful for what to do after running the integration module. any idea what the paths in the browser look like?