go-streams

go-streams provides a lightweight and efficient stream processing framework for Go. Its concise DSL allows for easy definition of declarative data pipelines using composable sources, flows, and sinks.

Wiki
In computing, a pipeline, also known as a data pipeline, is a set of data processing elements connected in series, where the output of one element is the input of the next one. The elements of a pipeline are often executed in parallel or in time-sliced fashion. Some amount of buffer storage is often inserted between elements.

Overview

The core module has no external dependencies and provides three key components for constructing stream processing pipelines:

• Source: The entry point of a pipeline, emitting data into the stream. (One open output)
• Flow: A processing unit, transforming data as it moves through the pipeline. (One open input, one open output)
• Sink: The termination point of a pipeline, consuming processed data and often acting as a subscriber. (One open input)

Flows

The flow package provides a collection of Flow implementations for common stream processing operations. These building blocks can be used to transform and manipulate data within pipelines.

• Map: Transforms each element in the stream.
• FlatMap: Transforms each element into a stream of slices of zero or more elements.
• Filter: Selects elements from the stream based on a condition.
• Fold: Combines elements of the stream with the last folded value and emits the new value. Requires an initial value.
• Reduce: Combines elements of the stream with the last reduced value and emits the new value. Does not require an initial value.
• PassThrough: Passes elements through unchanged.
• Split¹: Divides the stream into two streams based on a boolean predicate.
• FanOut¹: Duplicates the stream to multiple outputs for parallel processing.
• RoundRobin¹: Distributes elements evenly across multiple outputs.
• Merge¹: Combines multiple streams into a single stream.
• ZipWith¹: Combines elements from multiple streams using a function.
• Flatten¹: Flattens a stream of slices of elements into a stream of elements.
• Batch: Breaks a stream of elements into batches based on size or timing.
• Throttler: Limits the rate at which elements are processed.
• AdaptiveThrottler: Limits the rate at which elements are processed based on the current system resource utilization (CPU and memory usage).
• SlidingWindow: Creates overlapping windows of elements.
• TumblingWindow: Creates non-overlapping, fixed-size windows of elements.
• SessionWindow: Creates windows based on periods of activity and inactivity.
• Keyed: Groups elements by key for parallel processing of related data.

¹ Utility Flows

Connectors

Standard Source and Sink implementations are located in the extension package.

• Go channel inbound and outbound connector
• File inbound and outbound connector
• Standard Go io.Reader Source and io.Writer Sink connectors
• os.Stdout and Discard Sink connectors (useful for development and debugging)

The following connectors are available as separate modules and have their own dependencies.

• Aerospike
• Apache Kafka
• Apache Pulsar
• AWS (S3)
• Azure (Blob Storage)
• GCP (Storage)
• NATS
• Redis
• WebSocket

Usage Examples

See the examples directory for practical code samples demonstrating how to build complete stream processing pipelines, covering various use cases and integration scenarios.

License

Licensed under the MIT License.