Cribl-Microsoft Integration

Automation toolkit for integrating Cribl Stream with Microsoft Azure services. This repository provides PowerShell automation and testing environments for data collection, infrastructure provisioning, and configuration management.

Repository Contents

Core Automation Tools

DCR-Automation

PowerShell automation for Azure Data Collection Rules

• Automated DCR creation for 50+ native Azure tables and custom tables
• Automatic Cribl Stream destination configuration export
• Supports Direct DCRs (30-char limit) and DCE-based DCRs (64-char limit)
• Interactive menu interface with non-interactive CI/CD mode
• Name abbreviation intelligence for Azure limits
• Quick Start Guide

DCR-Templates

Pre-built ARM templates for manual deployment

• 100+ ready-to-deploy ARM templates for Sentinel native tables
• DCE and non-DCE configurations
• SecurityEvent, CommonSecurityLog, DeviceEvents, ASim tables, and more

Testing Environments

Azure Flow Log Lab

Lab environment for Azure Flow Log testing and development

• VNet with dual-level flow logging (vNet-level + subnet-level)
• VPN Gateway for site-to-site VPN connectivity
• Test VM deployment with auto-shutdown schedules
• Automatic Cribl collector configuration generation

Additional Resources

Lookups

Lookup tables and enrichment data for Cribl Stream

• Static lookup tables for data enrichment
• Dynamic lookups with Active Directory integration (Python-based LDAP)
• Reference data for log processing workflows

Quick Start

DCR Automation (Azure Log Analytics)

cd Azure/CustomDeploymentTemplates/DCR-Automation
.\Run-DCRAutomation.ps1

See DCR-Automation Quick Start

Manual Template Deployment

Browse templates in Azure/CustomDeploymentTemplates/DCR-Templates/

Azure Flow Log Lab

cd Azure/Labs/AzureFlowLogLab
.\Run-AzureFlowLogLab.ps1

Prerequisites

• Azure subscription with Log Analytics workspace
• PowerShell 5.1+ with Azure modules (Az.Accounts, Az.Resources, Az.OperationalInsights)
• Cribl Stream instance (4.14+ for Direct DCRs)
• Azure AD app registration (for Cribl authentication)

Key Features

• Automated DCR Creation: PowerShell scripts to simplify DCR deployments
• Cribl Integration: Auto-generates Cribl Stream source and destination configurations
• Template Library: Pre-built ARM templates for common scenarios
• Multi-Mode Support: Direct DCRs (simple) or DCE-based (advanced routing)
• Testing Environments: Lab environments for development and testing
• Menu-Driven Interfaces: Interactive menus with non-interactive CI/CD modes
• Configuration-Driven: JSON-based configuration files separate from code

Architecture Patterns

All major components follow consistent design patterns:

1. Interactive Menu Pattern: Run-*.ps1 main entry points with menu interfaces
2. Configuration-Driven Design: Separate azure-parameters.json and operation-parameters.json files
3. Template-Based Generation: Automated generation of ARM templates and Cribl configurations
4. Modular Script Design: Helper functions and reusable components
5. Documentation-First Approach: Comprehensive README, Quick Start, and Architecture guides

Documentation

Automation Guides

• DCR-Automation README - Detailed DCR automation

Configuration Guides

• Cribl Destinations Guide - Cribl configuration details
• Custom Tables Guide - Creating custom table schemas
• Active Directory Lookups Guide - Dynamic AD lookup integration

Knowledge Articles

• Azure Monitor Migration - Migration guidance
• Private Link Configuration - Detailed Private Link setup
• O365 App Registration - Office 365 app setup

Project Documentation

• CLAUDE.md - Comprehensive project guidance and architecture (for AI assistants)
• PROJECT_REVIEW_2025-10-27.md - Detailed project review

Technology Stack

PowerShell: 5.1+

• Az.Accounts, Az.Resources, Az.OperationalInsights, Az.EventHub, Az.Monitor

Infrastructure-as-Code:

• ARM Templates (Azure)

Cloud Platform:

• Microsoft Azure

Integrations:

• Cribl Stream 4.14+
• Microsoft Sentinel
• Azure Log Analytics
• Azure Event Hub
• Azure Data Explorer
• Active Directory (LDAP)

Security Considerations

• Azure AD credentials managed via app registrations
• Role-based access control (RBAC) for all cloud resources
• Secure credential storage recommendations in documentation
• Never commit real credentials to version control
• Support for Cribl secrets management

Git Workflow

The main branch is protected - all changes must come through pull requests.

Branch Naming Convention

• feature/ - New features or enhancements
• fix/ - Bug fixes
• docs/ - Documentation updates
• refactor/ - Code refactoring
• test/ - Test additions or updates

Commit Message Guidelines

• Use present tense verbs ("Add" not "Added")
• Keep first line under 50 characters
• Be descriptive and specific

Contributing

See CONTRIBUTORS.md for contribution guidelines.

Before submitting changes:

1. Test thoroughly in lab/development environments first
2. Test with both Direct and DCE-based configurations (for Azure DCR changes)
3. Verify configuration files are valid JSON
4. Update documentation for new features
5. Follow existing architecture patterns

License

MIT License - see LICENSE file.

Support

• Documentation: Start with Quick Start guides for each component
• Issues: Create an issue in the repository
• Questions: Check component-specific README files for troubleshooting sections

---

Getting Started?

• For Azure Log Analytics ingestion: DCR-Automation Quick Start
• For testing and development: Azure Flow Log Lab