From f65e2ce29b54845df921e0621a245ceaec93acdf Mon Sep 17 00:00:00 2001 From: Koray Erkan Date: Mon, 2 Mar 2026 12:14:56 +0300 Subject: [PATCH] Post-WEC documentation fixes for device categorization Remove dead links to deleted Azure Monitor subpages, replace stale push/pull device terminology with behavioral descriptions, align overview Device Types with subdirectory structure, list UI categories in management dashboard section, and correct TLS field descriptions from "file path" to "file name" across all 12 device files. Co-Authored-By: Claude Opus 4.6 --- docs/configuration/devices/aws/amazon-s3.mdx | 4 +- .../devices/aws/amazon-security-lake.mdx | 2 +- .../devices/azure/azure-blob-storage.mdx | 4 +- .../devices/azure/azure-monitor.mdx | 6 --- .../devices/azure/event-hubs.mdx | 4 +- .../devices/azure/microsoft-graph-api.mdx | 4 +- docs/configuration/devices/management.mdx | 10 ++++- docs/configuration/devices/mq/kafka.mdx | 4 +- docs/configuration/devices/mq/nats.mdx | 4 +- docs/configuration/devices/mq/rabbitmq.mdx | 4 +- docs/configuration/devices/mq/redis.mdx | 4 +- docs/configuration/devices/os/agents.mdx | 2 +- docs/configuration/devices/other/wec.mdx | 4 +- docs/configuration/devices/overview.mdx | 40 +++++++++---------- .../devices/protocols/estreamer.mdx | 4 +- docs/configuration/devices/protocols/http.mdx | 4 +- docs/configuration/devices/protocols/smtp.mdx | 4 +- .../devices/protocols/syslog.mdx | 4 +- docs/configuration/devices/protocols/tcp.mdx | 4 +- 19 files changed, 58 insertions(+), 58 deletions(-) diff --git a/docs/configuration/devices/aws/amazon-s3.mdx b/docs/configuration/devices/aws/amazon-s3.mdx index a0243ccf..1e285556 100644 --- a/docs/configuration/devices/aws/amazon-s3.mdx +++ b/docs/configuration/devices/aws/amazon-s3.mdx @@ -4,7 +4,7 @@ ## Synopsis -Amazon S3 device processes files from Amazon S3 buckets using SQS event notifications. This pull-type device consumes S3 event messages from an SQS queue, downloads referenced objects from S3, and processes them through DataStream pipelines. The device supports multiple file formats including JSON, JSONL, Parquet, and compressed archives. +Amazon S3 device processes files from Amazon S3 buckets using SQS event notifications. The device consumes S3 event messages from an SQS queue, downloads referenced objects from S3, and processes them through DataStream pipelines. The device supports multiple file formats including JSON, JSONL, Parquet, and compressed archives. ## Schema @@ -143,7 +143,7 @@ When using cross-account role assumption (`role_arn`), the calling identity also When accessing S3 buckets in another AWS account, configure `role_arn` and optionally use temporary credentials. The assumed role must have the S3 and SQS permissions above. The target role's trust policy must allow assumption from the source account, with optional `ExternalId` condition for Security Lake scenarios. ::: -The Amazon S3 device operates as an event-driven pull-type data source that processes S3 objects based on SQS notifications. The device continuously polls an SQS queue for S3 event messages, downloads the referenced objects, and processes their contents through the telemetry pipeline. +The Amazon S3 device processes S3 objects based on SQS notifications. The device continuously polls an SQS queue for S3 event messages, downloads the referenced objects, and processes their contents through the telemetry pipeline. **Event Processing Flow**: The device receives S3 event notifications from SQS containing bucket name and object key information. For each ObjectCreated event (Put, Post, Copy, CompleteMultipartUpload), the device downloads the S3 object and processes it according to its file type. After successful processing, the SQS message is deleted to prevent reprocessing. diff --git a/docs/configuration/devices/aws/amazon-security-lake.mdx b/docs/configuration/devices/aws/amazon-security-lake.mdx index b2582ec6..2d8a443a 100644 --- a/docs/configuration/devices/aws/amazon-security-lake.mdx +++ b/docs/configuration/devices/aws/amazon-security-lake.mdx @@ -163,7 +163,7 @@ When using IAM credentials or role assumption, the following permissions are req Security Lake typically requires cross-account access via `role_arn` with `external_id`. The calling identity needs `sts:AssumeRole` on the target role. The target role's trust policy must allow assumption from the source account with the configured `ExternalId` condition. The assumed role must have the S3 and SQS permissions above attached to it. ::: -The Amazon Security Lake device implements a pull-type consumer pattern that integrates with Amazon Security Lake's S3-backed architecture. Security Lake stores normalized security data in OCSF format as Parquet files, and publishes S3 ObjectCreated events to an SQS queue. The device polls this queue, downloads referenced Parquet files, and ingests OCSF events into DataStream. +The Amazon Security Lake device integrates with Amazon Security Lake's S3-backed architecture. Security Lake stores normalized security data in OCSF format as Parquet files, and publishes S3 ObjectCreated events to an SQS queue. The device polls this queue, downloads referenced Parquet files, and ingests OCSF events into DataStream. **OCSF Schema Validation**: When enabled, the device validates each Parquet record against OCSF schema requirements. Invalid records generate warnings but do not halt file processing. Disable validation for performance-critical scenarios or when processing pre-validated data. diff --git a/docs/configuration/devices/azure/azure-blob-storage.mdx b/docs/configuration/devices/azure/azure-blob-storage.mdx index f734cb9e..e5b512b3 100644 --- a/docs/configuration/devices/azure/azure-blob-storage.mdx +++ b/docs/configuration/devices/azure/azure-blob-storage.mdx @@ -4,7 +4,7 @@ ## Synopsis -Azure Blob Storage device reads and processes files from Azure storage containers. This pull-type device connects to Azure Blob Storage containers to retrieve files in various formats (JSON, JSONL, Parquet) and processes them through DataStream pipelines. The device supports both connection string and service principal authentication methods. +Azure Blob Storage device reads and processes files from Azure storage containers. The device connects to Azure Blob Storage containers to retrieve files in various formats (JSON, JSONL, Parquet) and processes them through DataStream pipelines. The device supports both connection string and service principal authentication methods. ## Schema @@ -86,7 +86,7 @@ When using connection string authentication, Azure RBAC roles are not applicable The device validates connectivity at startup by reading blob service properties and queue metadata. The recommended roles above may not fully cover these validation calls. If startup validation fails, either use a custom role with the exact data actions or assign `Storage Queue Data Contributor` instead of `Storage Queue Data Message Processor` for broader queue access. ::: -The Azure Blob Storage device operates as a pull-type data source that periodically scans Azure storage containers for new files. The device supports multiple file formats and provides flexible authentication options for enterprise environments. +The Azure Blob Storage device periodically scans Azure storage containers for new files. The device supports multiple file formats and provides flexible authentication options for enterprise environments. **File Format Processing**: The device automatically detects and processes files based on the configured format. JSON files are parsed as individual objects, JSONL files process each line as a separate record, and Parquet files are read using columnar processing for efficient large-data handling. diff --git a/docs/configuration/devices/azure/azure-monitor.mdx b/docs/configuration/devices/azure/azure-monitor.mdx index f30fbb6f..12fcdfce 100644 --- a/docs/configuration/devices/azure/azure-monitor.mdx +++ b/docs/configuration/devices/azure/azure-monitor.mdx @@ -61,12 +61,6 @@ All collection types share a single set of credentials. Each device instance runs alerts, logs, and metrics collection concurrently via separate goroutines within a single collection cycle. After all three goroutines complete, the collector waits for `event_frequency` seconds before starting the next cycle. Each collection type maintains its own checkpoint keyed by device ID and type, so a failure in one type does not affect the others' progress. -See the individual definition pages for field references, RBAC requirements, and examples: - -- [Alerts](./alerts) -- [Logs](./logs) -- [Metrics](./metrics) - ## Examples ### Basic diff --git a/docs/configuration/devices/azure/event-hubs.mdx b/docs/configuration/devices/azure/event-hubs.mdx index 3308b0f6..ea1df7ef 100644 --- a/docs/configuration/devices/azure/event-hubs.mdx +++ b/docs/configuration/devices/azure/event-hubs.mdx @@ -98,8 +98,8 @@ EventHubs requires checkpoint storage. Choose one method: |Field|Required|Default|Description| |---|---|---|---| |`tls.status`|N|`false`|Enable TLS encryption| -|`tls.cert_name`|N*||TLS certificate file path (required if TLS enabled)| -|`tls.key_name`|N*||TLS private key file path (required if TLS enabled)| +|`tls.cert_name`|N*||TLS certificate file name (required if TLS enabled)| +|`tls.key_name`|N*||TLS private key file name (required if TLS enabled)| \* = Conditionally required (only when `tls.status: true`) diff --git a/docs/configuration/devices/azure/microsoft-graph-api.mdx b/docs/configuration/devices/azure/microsoft-graph-api.mdx index 27c94ef5..181c2480 100644 --- a/docs/configuration/devices/azure/microsoft-graph-api.mdx +++ b/docs/configuration/devices/azure/microsoft-graph-api.mdx @@ -109,8 +109,8 @@ Two strategies control incremental data retrieval. If both are configured, `time |Field|Required|Default|Description| |---|---|---|---| |`tls.status`|N|`false`|Enable mutual TLS| -|`tls.cert_name`|N*|`client_cert.pem`|Client certificate file path| -|`tls.key_name`|N*|`client_key.pem`|Client private key file path| +|`tls.cert_name`|N*|`client_cert.pem`|Client certificate file name| +|`tls.key_name`|N*|`client_key.pem`|Client private key file name| |`tls.insecure_skip_verify`|N|`false`|Skip server certificate verification| \* = Conditionally required (only when `tls.status: true`) diff --git a/docs/configuration/devices/management.mdx b/docs/configuration/devices/management.mdx index dc32b1d0..d3c4b684 100644 --- a/docs/configuration/devices/management.mdx +++ b/docs/configuration/devices/management.mdx @@ -12,9 +12,15 @@ The Devices dashboard is where you manage all configured diff --git a/docs/configuration/devices/other/wec.mdx b/docs/configuration/devices/other/wec.mdx index 626ef6b1..d03d08c2 100644 --- a/docs/configuration/devices/other/wec.mdx +++ b/docs/configuration/devices/other/wec.mdx @@ -93,8 +93,8 @@ If `address` is `0.0.0.0`, set `hostname` explicitly to a client-resolvable FQDN |Field|Required|Default|Description| |---|---|---|---| |`tls.status`|N|`false`|Enable TLS encryption| -|`tls.cert_name`|N*||TLS certificate file path (required if TLS enabled)| -|`tls.key_name`|N*||TLS private key file path (required if TLS enabled)| +|`tls.cert_name`|N*||TLS certificate file name (required if TLS enabled)| +|`tls.key_name`|N*||TLS private key file name (required if TLS enabled)| |`tls.client_auth`|N|`0`|TLS client authentication mode (maps to `tls.ClientAuthType`)| |`tls.client_ca_cert_file`|N||Path to PEM-encoded CA certificate used to verify client certificates| diff --git a/docs/configuration/devices/overview.mdx b/docs/configuration/devices/overview.mdx index f64ba51f..dbb9a4ee 100644 --- a/docs/configuration/devices/overview.mdx +++ b/docs/configuration/devices/overview.mdx @@ -83,44 +83,44 @@ Preprocessing pipelines attached to devices execute **sequentially** in the orde The system supports the following device types: -* **Network Protocol** - These devices listen for incoming network connections: - * **HTTP**: Accepts JSON data via HTTP/HTTPS POST requests with authentication options +* **Protocols** — Network listeners and flow collectors: + * **Syslog**: Specialized for syslog format messages with RFC compliance * **TCP**: Receives messages over TCP connections with framing and TLS support * **UDP**: Collects datagram-based messages with high throughput capabilities - * **Syslog**: Specialized for syslog format messages with RFC compliance + * **HTTP**: Accepts JSON data via HTTP/HTTPS POST requests with authentication options + * **eStreamer**: Connects to Cisco eStreamer servers + * **SNMP Trap**: Receives SNMP trap notifications * **SMTP**: Receives email messages for log processing - -* **Flow Monitoring** - These devices collect network flow data: * **NetFlow**: Cisco NetFlow v5/v9 network traffic analysis * **sFlow**: sFlow sampling-based network monitoring * **IPFix**: IP Flow Information Export (IETF standard) + * **TFTP**: Receives files via Trivial File Transfer Protocol -* **Cloud Integration** - These devices connect to cloud services: - * **Amazon S3**: Processes files from Amazon S3 buckets using SQS event notifications - * **Amazon Security Lake**: Consumes OCSF Parquet files from Amazon Security Lake via SQS notifications - * **Azure Blob Storage**: Pulls data from Azure Blob containers - * **Azure Monitor**: Collects logs from Azure Log Analytics workspaces +* **Microsoft Azure** — Azure cloud service integrations: + * **Azure Blob Storage**: Reads and processes files from Azure storage containers + * **Azure Monitor**: Collects alerts, logs, and metrics from Azure Monitor * **Event Hubs**: Consumes events from Azure Event Hubs * **Microsoft Graph API**: Polls Microsoft Graph API for audit logs, security events, identity protection, and reports - * **Microsoft Sentinel**: Pulls security data from Microsoft Sentinel + * **Microsoft Sentinel**: Collects security data from Microsoft Sentinel + +* **Amazon Web Services** — AWS cloud service integrations: + * **Amazon S3**: Processes files from Amazon S3 buckets using SQS event notifications + * **Amazon Security Lake**: Consumes OCSF Parquet files from Amazon Security Lake via SQS notifications -* **Message Queue** - These devices consume from messaging platforms: +* **Message Queues** — Messaging platform consumers: * **Kafka**: Consumes from Apache Kafka topics * **NATS**: Subscribes to NATS messaging subjects * **RabbitMQ**: Consumes from RabbitMQ queues * **Redis**: Subscribes to Redis pub/sub channels -* **Security Integration** - These devices integrate with security products: - * **eStreamer**: Connects to Cisco eStreamer servers - * **Proofpoint**: Consumes Proofpoint On Demand log stream via WebSocket - * **SNMP Trap**: Receives SNMP trap notifications - -* **System Integration** - These devices interact with operating systems: +* **Operating Systems** — Agent-based system monitoring: + * **Agents**: VirtualMetric Agent deployment and management * **Windows**: Collects Windows events via Agent * **Linux**: Collects Linux logs and metrics via Agent -* **File Transfer** - These devices receive files: - * **TFTP**: Receives files via Trivial File Transfer Protocol +* **Other** — Specialized integrations: + * **Proofpoint On Demand**: Consumes Proofpoint log stream via WebSocket + * **WEC**: Windows Event Collector server using WS-Management protocol ## Use Cases diff --git a/docs/configuration/devices/protocols/estreamer.mdx b/docs/configuration/devices/protocols/estreamer.mdx index 92d3055c..6b2ad5ab 100644 --- a/docs/configuration/devices/protocols/estreamer.mdx +++ b/docs/configuration/devices/protocols/estreamer.mdx @@ -65,8 +65,8 @@ The following fields are used to define the device: |Field|Required|Default|Description| |---|---|---|---| |`tls.status`|Y|`true`|Enable TLS encryption (always required)| -|`tls.cert_name`|Y||Client certificate file path| -|`tls.key_name`|Y||Client private key file path| +|`tls.cert_name`|Y||Client certificate file name| +|`tls.key_name`|Y||Client private key file name| |`tls.non_secure`|N|`false`|Allow less secure TLS versions| :::note diff --git a/docs/configuration/devices/protocols/http.mdx b/docs/configuration/devices/protocols/http.mdx index f21341ef..58ec5990 100644 --- a/docs/configuration/devices/protocols/http.mdx +++ b/docs/configuration/devices/protocols/http.mdx @@ -97,8 +97,8 @@ The following fields are used to define the device: |Field|Required|Default|Description| |---|---|---|---| |`tls.status`|N|`false`|Enable TLS encryption| -|`tls.cert_name`|Y||TLS certificate file path (required if TLS enabled)| -|`tls.key_name`|Y||TLS private key file path (required if TLS enabled)| +|`tls.cert_name`|Y||TLS certificate file name (required if TLS enabled)| +|`tls.key_name`|Y||TLS private key file name (required if TLS enabled)| :::note TLS certificate and key files must be placed in the service root directory. diff --git a/docs/configuration/devices/protocols/smtp.mdx b/docs/configuration/devices/protocols/smtp.mdx index 180a753a..6fed622f 100644 --- a/docs/configuration/devices/protocols/smtp.mdx +++ b/docs/configuration/devices/protocols/smtp.mdx @@ -60,8 +60,8 @@ The following fields are used to define the device: |Field|Required|Default|Description| |---|---|---|---| |`tls.status`|N|`false`|Enable TLS encryption| -|`tls.cert_name`|Y||TLS certificate file path (required if TLS enabled)| -|`tls.key_name`|Y||TLS private key file path (required if TLS enabled)| +|`tls.cert_name`|Y||TLS certificate file name (required if TLS enabled)| +|`tls.key_name`|Y||TLS private key file name (required if TLS enabled)| :::note The TLS certificate and key files must be placed in the service root directory. diff --git a/docs/configuration/devices/protocols/syslog.mdx b/docs/configuration/devices/protocols/syslog.mdx index 0d540b77..a74e6eef 100644 --- a/docs/configuration/devices/protocols/syslog.mdx +++ b/docs/configuration/devices/protocols/syslog.mdx @@ -82,8 +82,8 @@ The following are only applicable when `protocol` is set to `tcp`. |Field|Required|Default|Description| |---|---|---|---| |`tls.status`|N|`false`|Enable TLS encryption| -|`tls.cert_name`|Y||TLS certificate file path (required if TLS enabled)| -|`tls.key_name`|Y||TLS private key file path (required if TLS enabled)| +|`tls.cert_name`|Y||TLS certificate file name (required if TLS enabled)| +|`tls.key_name`|Y||TLS private key file name (required if TLS enabled)| :::note The TLS certificate and key files must be placed in the service root directory. diff --git a/docs/configuration/devices/protocols/tcp.mdx b/docs/configuration/devices/protocols/tcp.mdx index fb1051a4..225113dd 100644 --- a/docs/configuration/devices/protocols/tcp.mdx +++ b/docs/configuration/devices/protocols/tcp.mdx @@ -76,8 +76,8 @@ When using delimiter framing, ensure that the `line_delimiter` matches the clien |Field|Required|Default|Description| |---|---|---|---| |`tls.status`|N|`false`|Enable TLS encryption| -|`tls.cert_name`|Y||TLS certificate file path (required if TLS enabled)| -|`tls.key_name`|Y||TLS private key file path (required if TLS enabled)| +|`tls.cert_name`|Y||TLS certificate file name (required if TLS enabled)| +|`tls.key_name`|Y||TLS private key file name (required if TLS enabled)| :::note The TLS certificate and key files must be placed in the service root directory.