Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 22 additions & 6 deletions docs/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,19 @@ Specification access to [this](https://github.com/kaasops/vector-operator/blob/m
The `VectorPipeline` is a namespace-scoped CRD.
The `VectorPipeline` CRD defines Sources, Transforms and Sinks rules for Vector.
All `VectorPipelines`, with validated configuration file, added to Vector configuration file.
Depending on the types of sources, it can take on the role of either an agent or an aggregator.
The pipeline role is determined automatically based on source types:

**Agent role** (routed to Vector DaemonSet):
- Only [kubernetes_logs](https://vector.dev/docs/reference/configuration/sources/kubernetes_logs/) source type is allowed
- Collects logs only from the namespace where VectorPipeline is defined
- `extra_namespace_label_selector` is auto-configured to the pipeline's namespace; setting a different namespace will result in an error

**Aggregator role** (routed to VectorAggregator in the same namespace):
- Supports aggregator source types: `kafka`, `http_server`, `amqp`, `socket`, `redis`, `nats`, etc.
- No namespace restrictions on data sources

## Restrictions
- For source available only [kubernetes_logs](https://vector.dev/docs/reference/configuration/sources/kubernetes_logs/) type
- For source field `extra_namespace_label_selector` cannot be installed. The operator control this field and sets the namespace there, where VectorPipeline is defined.
- All source types must belong to the same role.
- All sources in a pipeline must belong to the same role. Mixing agent and aggregator source types is not allowed.

## Specification
Specification access to [this](https://github.com/kaasops/vector-operator/blob/main/docs/specification.md#vectorpipelinespec-clustervectorpipelinespec) page
Expand All @@ -45,9 +52,18 @@ Specification access to [this](https://github.com/kaasops/vector-operator/blob/m
The `ClusterVectorPipeline` is a cluster-scoped CRD.
The `ClusterVectorPipeline` CRD defines Sources, Transforms and Sinks rules for Vector.
All `ClusterVectorPipelines`, with validated configuration file, added to Vector configuration file.
Depending on the types of sources, it can take on the role of either an agent or an aggregator.
The pipeline role is determined automatically based on source types:

**Agent role** (routed to Vector DaemonSet):
- Supports all agent source types: `kubernetes_logs`, `file`, `journald`, `host_metrics`, `docker_logs`, etc.
- Can collect logs from any namespace
- No restrictions on `extra_namespace_label_selector`

ClusterVectorPipelines works like VectorPipeline, but without restrictions.
**Aggregator role** (routed to ClusterVectorAggregator):
- Supports aggregator source types: `kafka`, `http_server`, `amqp`, `socket`, `redis`, `nats`, etc.

## Restrictions
- All sources in a pipeline must belong to the same role. Mixing agent and aggregator source types is not allowed.

## Specification
Specification access to [this](https://github.com/kaasops/vector-operator/blob/main/docs/specification.md#vectorpipelinespec-clustervectorpipelinespec) page
Expand Down
6 changes: 3 additions & 3 deletions docs/journald-logs.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Secure credential

If you want collect service journald logs from node you can use example.
If you want to collect service journald logs from node you can use example.

> Type `journald` in source block work only in ClusterVectorPipeline. In VectorPipeline can use only `kubernetes_logs` type
> Type `journald` is an agent source type that requires node-level access. In VectorPipeline with an agent role, only `kubernetes_logs` is allowed. Use ClusterVectorPipeline for `journald` sources.

> If you want collect journald logs, needs to use vector-agent container with journalctl. `timberio/vector:0.48.0-debian` - for example
> If you want to collect journald logs, needs to use vector-agent container with journalctl. `timberio/vector:0.48.0-debian` - for example


```yaml
Expand Down
2 changes: 1 addition & 1 deletion docs/logs-from-file.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

If you want collect logs from file (like k8s-audit logs) you can use example.

> Type `file` in source block work only in ClusterVectorPipeline. In VectorPipeline can use only `kubernetes_logs` type
> Type `file` is an agent source type that requires node-level access. In VectorPipeline with agent role, only `kubernetes_logs` is allowed. Use ClusterVectorPipeline for `file` sources.


```yaml
Expand Down
10 changes: 9 additions & 1 deletion docs/quick-start.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,9 @@ git clone https://github.com/kaasops/vector-operator.git
cd vector-operator
kubectl apply -f config/crd/bases/observability.kaasops.io_vectors.yaml
kubectl apply -f config/crd/bases/observability.kaasops.io_vectorpipelines.yaml
kubectl apply -f config/crd/bases/observability.kaasops.io_clustervectorpipelines.yaml
kubectl apply -f config/crd/bases/observability.kaasops.io_clustervectorpipelines.yaml
kubectl apply -f config/crd/bases/observability.kaasops.io_vectoraggregators.yaml
kubectl apply -f config/crd/bases/observability.kaasops.io_clustervectoraggregators.yaml
```

## Start Vector Operator
Expand Down Expand Up @@ -177,6 +179,8 @@ rules:
- clustervectorpipelines
- vectorpipelines
- vectors
- vectoraggregators
- clustervectoraggregators
verbs:
- create
- delete
Expand All @@ -191,6 +195,8 @@ rules:
- clustervectorpipelines/status
- vectorpipelines/status
- vectors/status
- vectoraggregators/status
- clustervectoraggregators/status
verbs:
- get
- patch
Expand Down Expand Up @@ -394,4 +400,6 @@ kubectl delete clusterrolebinding vector-operator
kubectl delete -f config/crd/bases/observability.kaasops.io_vectors.yaml
kubectl delete -f config/crd/bases/observability.kaasops.io_vectorpipelines.yaml
kubectl delete -f config/crd/bases/observability.kaasops.io_clustervectorpipelines.yaml
kubectl delete -f config/crd/bases/observability.kaasops.io_vectoraggregators.yaml
kubectl delete -f config/crd/bases/observability.kaasops.io_clustervectoraggregators.yaml
```