FlowG is a log management platform that lets you ingest, transform, and query logs using a visual pipeline builder. It handles structured logs without requiring predefined schemas and relies on BadgerDB as its storage backend.

For more information, consult the documentation.

You wish to contribute? Please read our guidelines first.

:sparkles: Key Features

• Pipelines: Build and manage log processing pipelines visually, with support for transformations with VRL scripts and conditional routing to dedicated streams
• Dynamic Logs: Manage logs with varying structures without needing predefined schemas
• Interoperability: Configuration and log ingestion can entirely be done via the REST API
• Observability: Query and visualize logs in real-time in the integrated Web interface

:question: Why should I use this?

FlowG's primary goal is to make log refinement and routing as easy as possible. As such, it relies on React Flow to help you build such pipelines with as little code as possible. For the actual code part, we use the Vector Remap Language, which gives a solid base for log refinement.

It aims to replace tools like Logstash by integrating the feature right in the solution.

It also leverages BadgerDB which is a battle tested Key/Value database, with the right feature set to easily support indexing dynamically structured logs, as well as log compression.

:zap: Benchmark

Run with (see Build section bellow):

export BENCHMARK_ITERATIONS="100000"
cargo install oha
task docker:build test:bench

System:

• CPU: 13th Gen Intel(R) Core(TM) i7-13700K
• RAM: 64GB

Result:

Summary:
  Success rate: 100.00%
  Total:        5702.4524 ms
  Slowest:      150.2867 ms
  Fastest:      0.1578 ms
  Average:      2.8478 ms
  Requests/sec: 17536.3148

  Total data:   3.53 MiB
  Size/request: 37 B
  Size/sec:     633.64 KiB

:building_construction: Build

Requirements:

• Go 1.22+
• C Toolchain
• Rust and Cargo (edition 2021)
• NodeJS
• gettext (for msgcat)

go install github.com/go-task/task/v3/cmd/task@latest
task build:all

Then, start the server with:

./bin/flowg-server

Now, you can access:

• the WebUI at http://localhost:5080
• the API documentation at http://localhost:5080/api/docs
• the Syslog Server at udp://localhost:5514
• the Management API at http://localhost:9113:
  • /health: health check, always return 200 OK
  • /metrics: Prometheus Exporter

A default user root (password: root) and a default pipeline are bootsrapped if no configuration exists during startup.

To build the Docker image linksociety/flowg:latest locally:

task docker:build

:rocket: Deploy

Using Docker:

docker run \
  -p 5080:5080/tcp \
  -p 9113:9113/tcp \
  -p 5514:5514/udp \
  -v flowg-data:/data \
  linksociety/flowg:latest

:warning: EXPERIMENTAL :warning: Using Kubernetes (and Helm):

helm install flowg ./k8s/charts/flowg -n flowg-system --create-namespace

:memo: License

This software is released under the terms of the MIT License