Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/configuration/devices/aws/amazon-s3.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

## Synopsis

Amazon S3 device processes files from Amazon S3 buckets using SQS event notifications. This pull-type device consumes S3 event messages from an SQS queue, downloads referenced objects from S3, and processes them through DataStream pipelines. The device supports multiple file formats including JSON, JSONL, Parquet, and compressed archives.
Amazon S3 device processes files from Amazon S3 buckets using SQS event notifications. The device consumes S3 event messages from an SQS queue, downloads referenced objects from S3, and processes them through DataStream pipelines. The device supports multiple file formats including JSON, JSONL, Parquet, and compressed archives.

## Schema

Expand Down Expand Up @@ -143,7 +143,7 @@ When using cross-account role assumption (`role_arn`), the calling identity also
When accessing S3 buckets in another AWS account, configure `role_arn` and optionally use temporary credentials. The assumed role must have the S3 and SQS permissions above. The target role's trust policy must allow assumption from the source account, with optional `ExternalId` condition for Security Lake scenarios.
:::

The Amazon S3 device operates as an event-driven pull-type data source that processes S3 objects based on SQS notifications. The device continuously polls an SQS queue for S3 event messages, downloads the referenced objects, and processes their contents through the telemetry pipeline.
The Amazon S3 device processes S3 objects based on SQS notifications. The device continuously polls an SQS queue for S3 event messages, downloads the referenced objects, and processes their contents through the telemetry pipeline.

**Event Processing Flow**: The device receives S3 event notifications from SQS containing bucket name and object key information. For each ObjectCreated event (Put, Post, Copy, CompleteMultipartUpload), the device downloads the S3 object and processes it according to its file type. After successful processing, the SQS message is deleted to prevent reprocessing.

Expand Down
2 changes: 1 addition & 1 deletion docs/configuration/devices/aws/amazon-security-lake.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ When using IAM credentials or role assumption, the following permissions are req
Security Lake typically requires cross-account access via `role_arn` with `external_id`. The calling identity needs `sts:AssumeRole` on the target role. The target role's trust policy must allow assumption from the source account with the configured `ExternalId` condition. The assumed role must have the S3 and SQS permissions above attached to it.
:::

The Amazon Security Lake device implements a pull-type consumer pattern that integrates with Amazon Security Lake's S3-backed architecture. Security Lake stores normalized security data in OCSF format as Parquet files, and publishes S3 ObjectCreated events to an SQS queue. The device polls this queue, downloads referenced Parquet files, and ingests OCSF events into DataStream.
The Amazon Security Lake device integrates with Amazon Security Lake's S3-backed architecture. Security Lake stores normalized security data in OCSF format as Parquet files, and publishes S3 ObjectCreated events to an SQS queue. The device polls this queue, downloads referenced Parquet files, and ingests OCSF events into DataStream.

**OCSF Schema Validation**: When enabled, the device validates each Parquet record against OCSF schema requirements. Invalid records generate warnings but do not halt file processing. Disable validation for performance-critical scenarios or when processing pre-validated data.

Expand Down
4 changes: 2 additions & 2 deletions docs/configuration/devices/azure/azure-blob-storage.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

## Synopsis

Azure Blob Storage device reads and processes files from Azure storage containers. This pull-type device connects to Azure Blob Storage containers to retrieve files in various formats (JSON, JSONL, Parquet) and processes them through DataStream pipelines. The device supports both connection string and service principal authentication methods.
Azure Blob Storage device reads and processes files from Azure storage containers. The device connects to Azure Blob Storage containers to retrieve files in various formats (JSON, JSONL, Parquet) and processes them through DataStream pipelines. The device supports both connection string and service principal authentication methods.

## Schema

Expand Down Expand Up @@ -86,7 +86,7 @@ When using connection string authentication, Azure RBAC roles are not applicable
The device validates connectivity at startup by reading blob service properties and queue metadata. The recommended roles above may not fully cover these validation calls. If startup validation fails, either use a custom role with the exact data actions or assign `Storage Queue Data Contributor` instead of `Storage Queue Data Message Processor` for broader queue access.
:::

The Azure Blob Storage device operates as a pull-type data source that periodically scans Azure storage containers for new files. The device supports multiple file formats and provides flexible authentication options for enterprise environments.
The Azure Blob Storage device periodically scans Azure storage containers for new files. The device supports multiple file formats and provides flexible authentication options for enterprise environments.

**File Format Processing**: The device automatically detects and processes files based on the configured format. JSON files are parsed as individual objects, JSONL files process each line as a separate record, and Parquet files are read using columnar processing for efficient large-data handling.

Expand Down
6 changes: 0 additions & 6 deletions docs/configuration/devices/azure/azure-monitor.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -61,12 +61,6 @@ All collection types share a single set of credentials.

Each device instance runs alerts, logs, and metrics collection concurrently via separate goroutines within a single collection cycle. After all three goroutines complete, the collector waits for `event_frequency` seconds before starting the next cycle. Each collection type maintains its own checkpoint keyed by device ID and type, so a failure in one type does not affect the others' progress.

See the individual definition pages for field references, RBAC requirements, and examples:

- [Alerts](./alerts)
- [Logs](./logs)
- [Metrics](./metrics)

## Examples

### Basic
Expand Down
4 changes: 2 additions & 2 deletions docs/configuration/devices/azure/event-hubs.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,8 @@ EventHubs requires checkpoint storage. Choose one method:
|Field|Required|Default|Description|
|---|---|---|---|
|`tls.status`|N|`false`|Enable TLS encryption|
|`tls.cert_name`|N*||TLS certificate file path (required if TLS enabled)|
|`tls.key_name`|N*||TLS private key file path (required if TLS enabled)|
|`tls.cert_name`|N*||TLS certificate file name (required if TLS enabled)|
|`tls.key_name`|N*||TLS private key file name (required if TLS enabled)|

\* = Conditionally required (only when `tls.status: true`)

Expand Down
4 changes: 2 additions & 2 deletions docs/configuration/devices/azure/microsoft-graph-api.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -109,8 +109,8 @@ Two strategies control incremental data retrieval. If both are configured, `time
|Field|Required|Default|Description|
|---|---|---|---|
|`tls.status`|N|`false`|Enable mutual TLS|
|`tls.cert_name`|N*|`client_cert.pem`|Client certificate file path|
|`tls.key_name`|N*|`client_key.pem`|Client private key file path|
|`tls.cert_name`|N*|`client_cert.pem`|Client certificate file name|
|`tls.key_name`|N*|`client_key.pem`|Client private key file name|
|`tls.insecure_skip_verify`|N|`false`|Skip server certificate verification|

\* = Conditionally required (only when `tls.status: true`)
Expand Down
10 changes: 8 additions & 2 deletions docs/configuration/devices/management.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,15 @@ The Devices dashboard is where you manage all configured <Topic id="devices-over

### Dashboard Interface

The overview page displays all available device types as cards organized by category.
The overview page displays all available device types as cards organized by category:

The interface provides a **Search devices** field to filter by name. A card count displays "Viewing X devices" or "No devices found".
- **Protocols** — Syslog, HTTP, UDP, TCP, eStreamer
- **Operating Systems** — Windows, Linux (and coming soon: macOS, Solaris, AIX, BSD variants)
- **Microsoft Azure** — Azure Blob Storage, Azure Event Hubs, Microsoft Sentinel
- **Amazon Web Services** — Amazon S3, Amazon Security Lake
- **Other** — All other device types

A sidebar filter panel allows selection by category. The interface provides a **Search devices** field to filter by name. A card count displays "Viewing X devices" or "No devices found".

Each device type displays as a card showing **Icon**, **Title**, **Description**, **Enabled Count**, **Disabled Count**, and optionally a **Coming Soon Tag** for unavailable types. Clicking a card navigates to that device type's management page.

Expand Down
4 changes: 2 additions & 2 deletions docs/configuration/devices/mq/kafka.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,8 @@ The following fields are used to define the device:
|Field|Required|Default|Description|
|---|---|---|---|
|`tls.status`|N|`false`|Enable TLS encryption|
|`tls.cert_name`|Y||TLS certificate file path (required if TLS enabled)|
|`tls.key_name`|Y||TLS private key file path (required if TLS enabled)|
|`tls.cert_name`|Y||TLS certificate file name (required if TLS enabled)|
|`tls.key_name`|Y||TLS private key file name (required if TLS enabled)|

:::note
The TLS certificate and key files must be placed in the service root directory.
Expand Down
4 changes: 2 additions & 2 deletions docs/configuration/devices/mq/nats.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,8 @@ The following fields are used to define the device:
|Field|Required|Default|Description|
|---|---|---|---|
|`tls.status`|N|`false`|Enable TLS encryption|
|`tls.cert_name`|Y||TLS certificate file path (required if TLS enabled)|
|`tls.key_name`|Y||TLS private key file path (required if TLS enabled)|
|`tls.cert_name`|Y||TLS certificate file name (required if TLS enabled)|
|`tls.key_name`|Y||TLS private key file name (required if TLS enabled)|

:::note
The TLS certificate and key files must be placed in the service root directory.
Expand Down
4 changes: 2 additions & 2 deletions docs/configuration/devices/mq/rabbitmq.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,8 @@ The following fields are used to define the device:
|Field|Required|Default|Description|
|---|---|---|---|
|`tls.status`|N|`false`|Enable TLS encryption|
|`tls.cert_name`|Y||TLS certificate file path (required if TLS enabled)|
|`tls.key_name`|Y||TLS private key file path (required if TLS enabled)|
|`tls.cert_name`|Y||TLS certificate file name (required if TLS enabled)|
|`tls.key_name`|Y||TLS private key file name (required if TLS enabled)|

:::note
The TLS certificate and key files must be placed in the service root directory.
Expand Down
4 changes: 2 additions & 2 deletions docs/configuration/devices/mq/redis.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,8 @@ The following fields are used to define the device:
|Field|Required|Default|Description|
|---|---|---|---|
|`tls.status`|N|`false`|Enable TLS encryption|
|`tls.cert_name`|Y||TLS certificate file path (required if TLS enabled)|
|`tls.key_name`|Y||TLS private key file path (required if TLS enabled)|
|`tls.cert_name`|Y||TLS certificate file name (required if TLS enabled)|
|`tls.key_name`|Y||TLS private key file name (required if TLS enabled)|

:::note
The TLS certificate and key files must be placed in the service root directory.
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration/devices/os/agents.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ DataStream supports two deployment approaches for Windows and Linux endpoints. B

### Architecture

Agents operate as pull-type devices that collect logs locally and push them to the Director:
Agents collect logs locally on the endpoint and forward them to the Director:

<Image id="agents-architecture" maxWidth="600px" />

Expand Down
4 changes: 2 additions & 2 deletions docs/configuration/devices/other/wec.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -93,8 +93,8 @@ If `address` is `0.0.0.0`, set `hostname` explicitly to a client-resolvable FQDN
|Field|Required|Default|Description|
|---|---|---|---|
|`tls.status`|N|`false`|Enable TLS encryption|
|`tls.cert_name`|N*||TLS certificate file path (required if TLS enabled)|
|`tls.key_name`|N*||TLS private key file path (required if TLS enabled)|
|`tls.cert_name`|N*||TLS certificate file name (required if TLS enabled)|
|`tls.key_name`|N*||TLS private key file name (required if TLS enabled)|
|`tls.client_auth`|N|`0`|TLS client authentication mode (maps to `tls.ClientAuthType`)|
|`tls.client_ca_cert_file`|N||Path to PEM-encoded CA certificate used to verify client certificates|

Expand Down
40 changes: 20 additions & 20 deletions docs/configuration/devices/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -83,44 +83,44 @@ Preprocessing pipelines attached to devices execute **sequentially** in the orde

The system supports the following device types:

* **Network Protocol** - These devices listen for incoming network connections:
* **HTTP**: Accepts JSON data via HTTP/HTTPS POST requests with authentication options
* **Protocols** — Network listeners and flow collectors:
* **Syslog**: Specialized for syslog format messages with RFC compliance
* **TCP**: Receives messages over TCP connections with framing and TLS support
* **UDP**: Collects datagram-based messages with high throughput capabilities
* **Syslog**: Specialized for syslog format messages with RFC compliance
* **HTTP**: Accepts JSON data via HTTP/HTTPS POST requests with authentication options
* **eStreamer**: Connects to Cisco eStreamer servers
* **SNMP Trap**: Receives SNMP trap notifications
* **SMTP**: Receives email messages for log processing

* **Flow Monitoring** - These devices collect network flow data:
* **NetFlow**: Cisco NetFlow v5/v9 network traffic analysis
* **sFlow**: sFlow sampling-based network monitoring
* **IPFix**: IP Flow Information Export (IETF standard)
* **TFTP**: Receives files via Trivial File Transfer Protocol

* **Cloud Integration** - These devices connect to cloud services:
* **Amazon S3**: Processes files from Amazon S3 buckets using SQS event notifications
* **Amazon Security Lake**: Consumes OCSF Parquet files from Amazon Security Lake via SQS notifications
* **Azure Blob Storage**: Pulls data from Azure Blob containers
* **Azure Monitor**: Collects logs from Azure Log Analytics workspaces
* **Microsoft Azure** — Azure cloud service integrations:
* **Azure Blob Storage**: Reads and processes files from Azure storage containers
* **Azure Monitor**: Collects alerts, logs, and metrics from Azure Monitor
* **Event Hubs**: Consumes events from Azure Event Hubs
* **Microsoft Graph API**: Polls Microsoft Graph API for audit logs, security events, identity protection, and reports
* **Microsoft Sentinel**: Pulls security data from Microsoft Sentinel
* **Microsoft Sentinel**: Collects security data from Microsoft Sentinel

* **Amazon Web Services** — AWS cloud service integrations:
* **Amazon S3**: Processes files from Amazon S3 buckets using SQS event notifications
* **Amazon Security Lake**: Consumes OCSF Parquet files from Amazon Security Lake via SQS notifications

* **Message Queue** - These devices consume from messaging platforms:
* **Message Queues** — Messaging platform consumers:
* **Kafka**: Consumes from Apache Kafka topics
* **NATS**: Subscribes to NATS messaging subjects
* **RabbitMQ**: Consumes from RabbitMQ queues
* **Redis**: Subscribes to Redis pub/sub channels

* **Security Integration** - These devices integrate with security products:
* **eStreamer**: Connects to Cisco eStreamer servers
* **Proofpoint**: Consumes Proofpoint On Demand log stream via WebSocket
* **SNMP Trap**: Receives SNMP trap notifications

* **System Integration** - These devices interact with operating systems:
* **Operating Systems** — Agent-based system monitoring:
* **Agents**: VirtualMetric Agent deployment and management
* **Windows**: Collects Windows events via Agent
* **Linux**: Collects Linux logs and metrics via Agent

* **File Transfer** - These devices receive files:
* **TFTP**: Receives files via Trivial File Transfer Protocol
* **Other** — Specialized integrations:
* **Proofpoint On Demand**: Consumes Proofpoint log stream via WebSocket
* **WEC**: Windows Event Collector server using WS-Management protocol

## Use Cases

Expand Down
4 changes: 2 additions & 2 deletions docs/configuration/devices/protocols/estreamer.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,8 @@ The following fields are used to define the device:
|Field|Required|Default|Description|
|---|---|---|---|
|`tls.status`|Y|`true`|Enable TLS encryption (always required)|
|`tls.cert_name`|Y||Client certificate file path|
|`tls.key_name`|Y||Client private key file path|
|`tls.cert_name`|Y||Client certificate file name|
|`tls.key_name`|Y||Client private key file name|
|`tls.non_secure`|N|`false`|Allow less secure TLS versions|

:::note
Expand Down
4 changes: 2 additions & 2 deletions docs/configuration/devices/protocols/http.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -97,8 +97,8 @@ The following fields are used to define the device:
|Field|Required|Default|Description|
|---|---|---|---|
|`tls.status`|N|`false`|Enable TLS encryption|
|`tls.cert_name`|Y||TLS certificate file path (required if TLS enabled)|
|`tls.key_name`|Y||TLS private key file path (required if TLS enabled)|
|`tls.cert_name`|Y||TLS certificate file name (required if TLS enabled)|
|`tls.key_name`|Y||TLS private key file name (required if TLS enabled)|

:::note
TLS certificate and key files must be placed in the service root directory.
Expand Down
4 changes: 2 additions & 2 deletions docs/configuration/devices/protocols/smtp.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,8 @@ The following fields are used to define the device:
|Field|Required|Default|Description|
|---|---|---|---|
|`tls.status`|N|`false`|Enable TLS encryption|
|`tls.cert_name`|Y||TLS certificate file path (required if TLS enabled)|
|`tls.key_name`|Y||TLS private key file path (required if TLS enabled)|
|`tls.cert_name`|Y||TLS certificate file name (required if TLS enabled)|
|`tls.key_name`|Y||TLS private key file name (required if TLS enabled)|

:::note
The TLS certificate and key files must be placed in the service root directory.
Expand Down
4 changes: 2 additions & 2 deletions docs/configuration/devices/protocols/syslog.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,8 @@ The following are only applicable when `protocol` is set to `tcp`.
|Field|Required|Default|Description|
|---|---|---|---|
|`tls.status`|N|`false`|Enable TLS encryption|
|`tls.cert_name`|Y||TLS certificate file path (required if TLS enabled)|
|`tls.key_name`|Y||TLS private key file path (required if TLS enabled)|
|`tls.cert_name`|Y||TLS certificate file name (required if TLS enabled)|
|`tls.key_name`|Y||TLS private key file name (required if TLS enabled)|

:::note
The TLS certificate and key files must be placed in the service root directory.
Expand Down
4 changes: 2 additions & 2 deletions docs/configuration/devices/protocols/tcp.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,8 @@ When using delimiter framing, ensure that the `line_delimiter` matches the clien
|Field|Required|Default|Description|
|---|---|---|---|
|`tls.status`|N|`false`|Enable TLS encryption|
|`tls.cert_name`|Y||TLS certificate file path (required if TLS enabled)|
|`tls.key_name`|Y||TLS private key file path (required if TLS enabled)|
|`tls.cert_name`|Y||TLS certificate file name (required if TLS enabled)|
|`tls.key_name`|Y||TLS private key file name (required if TLS enabled)|

:::note
The TLS certificate and key files must be placed in the service root directory.
Expand Down