Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions docs/blog/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,51 @@

# Blog

## Try out the first experience: Use Loggie and VictoriaLogs to quickly build a new generation of logging system

<aside style="font-size: .7rem" markdown>
<span>:octicons-person-16: [__ethfoo__] &nbsp;</span>
<span>
:octicons-calendar-24: 2023-6-28 &nbsp;
:octicons-clock-24: 20 min read
</span>
</aside>

Basic moments:

- Pain points of current log storage
- Advantages of VictoriaLogs
- Use Loggie to collect logs into VictoriaLogs

[:octicons-arrow-right-24: __Continue reading__](./victorialogs/loggie_victorialogs.md)
[__ethfoo__]: https://github.com/ethfoo
[网易基于Loggie的云原生大规模日志架构演进]: (./victorialogs/loggie_victorialogs.md)

## The evolution of NetEase’s cloud-native large-scale log architecture based on Loggie

<aside style="font-size: .7rem" markdown>
<span>:octicons-person-16: [__ethfoo__] &nbsp;</span>
<span>
:octicons-calendar-24: 2022-12-17 &nbsp;
:octicons-clock-24: 15 min read
</span>
</aside>

Sharing from the 2022 Top 100 Global Software Case Study Summit.

Basic moments:

- Initial exploration of the log scene
- Large-scale migration to cloud native
- Unified logging platform using Loggie as the core
- In-depth logging architecture best practices

[:octicons-arrow-right-24: __Continue reading__](https://github.com/loggie-io/asserts/blob/main/ppt/%E7%BD%91%E6%98%93%E5%9F%BA%E4%BA%8ELoggie%E7%9A%84%E4%BA%91%E5%8E%9F%E7%94%9F%E5%A4%A7%E8%A7%84%E6%A8%A1%E6%97%A5%E5%BF%97%E6%9E%B6%E6%9E%84%E6%BC%94%E8%BF%9B.pdf)
[__ethfoo__]: https://github.com/ethfoo
[网易基于Loggie的云原生大规模日志架构演进]: (https://github.com/loggie-io/asserts/blob/main/ppt/%E7%BD%91%E6%98%93%E5%9F%BA%E4%BA%8ELoggie%E7%9A%84%E4%BA%91%E5%8E%9F%E7%94%9F%E5%A4%A7%E8%A7%84%E6%A8%A1%E6%97%A5%E5%BF%97%E6%9E%B6%E6%9E%84%E6%BC%94%E8%BF%9B.pdf)

---

## Quickly Build a Scalable Cloud-Native Logging Architecture with Loggie

<aside style="font-size: .7rem" markdown>
Expand Down
Binary file added docs/blog/victorialogs/img/benchmark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/blog/victorialogs/img/terminal1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/blog/victorialogs/img/terminal2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/blog/victorialogs/img/ui.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
311 changes: 311 additions & 0 deletions docs/blog/victorialogs/loggie_victorialogs.md

Large diffs are not rendered by default.

67 changes: 67 additions & 0 deletions docs/developer-guide/build.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Compile and build

## Persistence engine selection

Loggie provides badger engines in addition to sqlite in v1.5 and later versions.

For the configuration of the two persistence engines, sqlite and badger, please refer to [here](../reference/global/db.md).

Why introduce the badger engine? The problems with using sqlite and CGO are:

- SQLite has requirements for the c lib library version of the environment. In the host deployment scenario, there are many machines with lower versions that may not be able to run.
- Avoid some construction pitfalls introduced by CGO, such as building multi-architecture (amd64/arm64) images

## Container image build

By default, Loggie will build images for all release branches and main branches and push them to [dockerhub](https://hub.docker.com/r/loggieio/loggie/). However, pulling the image from dockerhub may have problems such as current limitation, so it is recommended that you build the image yourself or re-push it to your own image warehouse.

Two Dockerfiles are provided under the Loggie project:

- Dockerfile: The default dockerfile uses sqlite, so it will also be built using CGO.
- Dockerfile.badger: You can use the `-f` parameter during docker build to specify the use of this dockerfile, which uses badger as the persistence engine, pure Golang, and no CGO.

In addition to using docker build, you can also use make to build:

```bash
make docker-build REPO=<YourRepoHost>/loggie TAG=v1.x.x
make docker-push REPO=<YourRepoHost>/loggie TAG=v1.x.x
```

Or build and push multi-architecture images directly:

```bash
make docker-multi-arch REPO=<YourRepoHost>/loggie TAG=v1.x.x
```

Note:

- The TAG here does not need to be passed. By default, it will be automatically generated based on the tag or branch in Git.

### Multiple Architectures

All Dockerfiles in the Loggie project support multi-architecture construction. The Loggie image on dockerhub is multi-architecture of amd64 and arm64, and the local architecture will be automatically recognized when pulling.

However, if you want to modify the tag and re-push it to your own warehouse, please do not directly docker pull & docker tag & docker push. This will only push the image of your local architecture, causing the multi-architecture image in the warehouse to become invalid.
If you want to push multi-architecture images to your own warehouse, you can use some other open source tools, such as [regctl](https://github.com/regclient/regclient/blob/main/docs/regctl.md), which can be used locally Use a command like `regctl image copy loggieio/loggie:xxx <YourRepoImage>` to push.

If you connect your own build process or tools, please use `make docker-multi-arch` or `docker buildx` to build multi-architecture images.

## Binary build

Binaries are available for host deployment scenarios.

- sqlite:

```bash
make build
```

- badger:

```bash
make build-in-badger
```

If cross compilation is required, please add GOOS and GOARCH.

If the local build using `make build` fails, you can modify the extra_flags extra build parameters in the makefile. Please try to remove `-extldflags "-static"`.
13 changes: 5 additions & 8 deletions docs/getting-started/install/kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Make sure you have kubectl and helm executable locally.
### Download helm-chart

```bash
VERSION=v1.3.0
VERSION=v1.5.0
helm pull https://github.com/loggie-io/installation/releases/download/${VERSION}/loggie-${VERSION}.tgz && tar xvzf loggie-${VERSION}.tgz
```
Please replace `<VERSION>` above with the specific version number such as v1.3.0, which can be found [release tag](https://github.com/loggie-io/loggie/tags).
Expand Down Expand Up @@ -250,7 +250,7 @@ servicePorts:

For the initial deployment, we specify that the deployment is under the `loggie` namespace, and let helm automatically create the namespace.
```bash
helm install loggie ./ -nloggie --create-namespace
helm install loggie ./ -n loggie --create-namespace
```

If `loggie` namespace has been created in your environment, you can ignore `-nloggie` and `--create-namespace`. Of course, you can use your own namespace.
Expand All @@ -261,11 +261,10 @@ If `loggie` namespace has been created in your environment, you can ignore `-nlo
```
If you have a similar problem during helm install, it means that your Kubernetes version is too low and does not support the apiextensions.k8s.io/v1 version CRD. Loggie temporarily retains the CRD of the v1beta1 version, please delete the v1beta1 version in the charts, `rm loggie/crds/crds.yaml` and reinstall it.


### Check deployment status
After execution, use the helm command to check the deployment status:
```
helm list -nloggie
helm list -n loggie
```
Result should be like:
```
Expand All @@ -275,15 +274,13 @@ loggie loggie 1 2021-11-30 18:06:16.976334232 +0800 CST deployed loggi

At the same time, you can also use the kubectl command to check whether the Pod has been created.
```
kubectl -nloggie get po
kubectl -n loggie get po
```
Result should be like:
```
loggie-sxxwh 1/1 Running 0 5m21s 10.244.0.5 kind-control-plane <none> <none>
```



## Deploy Loggie Aggregator

Deploying Aggregator is basically the same as deploying Agent. In Helm chart we provide `aggregator config`. Modify as `enabled: true`.
Expand Down Expand Up @@ -312,7 +309,7 @@ At the same time, you can add content in values.yaml according to the cases:

Command reference:
```
helm install loggie-aggregator ./ -nloggie-aggregator --create-namespace
helm install loggie-aggregator ./ -n loggie-aggregator --create-namespace
```

!!! note
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started/install/node.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ The current release only contains binary executables generated by GOOS=linux GOA
## Download Binary

```
VERSION=v1.3.0
VERSION=v1.5.0
mkdir /opt/loggie && curl https://github.com/loggie-io/loggie/releases/download/${VERSION}/loggie-linux-amd64 -o /opt/loggie/loggie && chmod +x /opt/loggie/loggie
```

Expand Down
6 changes: 3 additions & 3 deletions docs/getting-started/quick-start/kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,22 +23,22 @@ You can choose:
#### Download the chart and deploy it

```bash
VERSION=v1.3.0
VERSION=v1.5.0
helm pull https://github.com/loggie-io/installation/releases/download/${VERSION}/loggie-${VERSION}.tgz && tar xvzf loggie-${VERSION}.tgz
```
Try to modify values.yaml in it. Please replace the `<VERSION>` above with the specific version number.

Deploy:

```bash
helm install loggie ./loggie -nloggie --create-namespace
helm install loggie ./loggie -n loggie --create-namespace
```

You can also:
#### Deploy directly:

```bash
helm install loggie -nloggie --create-namespace https://github.com/loggie-io/installation/releases/download/${VERSION}/loggie-${VERSION}.tgz
helm install loggie -n loggie --create-namespace https://github.com/loggie-io/installation/releases/download/${VERSION}/loggie-${VERSION}.tgz
```
Please replace the `<VERSION>` above with the specific version number.

Expand Down
5 changes: 2 additions & 3 deletions docs/getting-started/quick-start/node.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ We will demonstrate the simplest scenario of collecting host log files.
### 1. Download the Executable File
Please find a Linux server host and download the Loggie binary executable file.
```shell
VERSION=v1.3.0
VERSION=v1.5.0
curl -LJ https://github.com/loggie-io/loggie/releases/download/${VERSION}/loggie-linux-amd64 -o loggie
```

Expand Down Expand Up @@ -54,6 +54,5 @@ After adding the above two configuration files on the host, we can start Loggie.
./loggie -config.system=./loggie.yml -config.pipeline=./pipelines.yml -log.jsonFormat=false
```

Fill the file paths of logie.yml and pipelines.yml in CMD arguments.

Fill the file paths of `loggie.yml` and pipelines.yml in CMD arguments.
Normal startup log shows that Loggie has started to work normally. Contents of files matching `/var/log/*.log` will be printed to standard output.
16 changes: 0 additions & 16 deletions docs/getting-started/roadmap/roadmap-2023.md

This file was deleted.

44 changes: 44 additions & 0 deletions docs/reference/apis/config.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Pipeline config configure related interfaces

## /api/v1/reload/config

### URL

```bash
GET /api/v1/reload/config
```

### describe

View the contents of the Pipeline configuration file on the Loggie disk.
Therefore, every time this interface is called, it will be read from the configuration file specified in the Loggie startup parameter `-config.pipeline`.
However, this configuration file is not necessarily the configuration currently read in the Loggie memory, because there are situations such as reload interval and configuration file format errors.

### Request parameters

none

### return

Pipeline configuration file text content.

## /api/v1/controller/pipelines

### URL

```bash
GET /api/v1/controller/pipelines
```

### describe

View the configuration file read in Loggie's memory, which is also the running Pipeline configuration.
The configuration content returned by this interface is consistent with that in the [help](help.md#apiv1help) interface.

### Request parameters

none

### return

Pipeline configuration file text content.
Loading