← Explore by area/Scala data processing and streaming

Big Data SQL & Processing Engines

20 repos

Distributed SQL engines and data processing frameworks built on JVM languages, designed for large-scale analytics and data warehousing. These systems emphasize query optimization, data lake integration, and polyglot support—enabling SQL interfaces over distributed compute clusters. The cluster includes foundational engines like Apache Spark alongside specialized tools for metadata management, query federation, and interactive analytics, with particular strength in Scala and Java implementations.

Java · 8

Scala · 8

TypeScript · 2

C++ · 1

Rust · 1

scala ·111,095

java ·110,961

big-data ·79,432

sql ·76,217

python ·70,308

spark ·58,746

jdbc ·49,756

kafka ·44,309

r ·43,981

streaming ·33,675

Related clusters

Scala reactive streaming and Akka ecosystem27 Cluster 43919327 Scala and JVM Streaming Systems2 SQL databases and query tools36 Java DevOps and Agent Frameworks27 Java/Kotlin Data Integration & Persistence23

apache/spark

Apache Spark - A unified analytics engine for large-scale data processing

Scala

43,981

40,719 commits

WeBankFinTech/Linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and…

Java

3,411

4,142 commits

delta-io/delta

An open-source storage framework that enables building a Lakehouse architecture with compute…

Scala

8,990

5,429 commits

apache/kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data…

Scala

2,364

4,545 commits

enso-org/enso

Enso Analytics is a self-service data prep and analysis platform designed for data teams.

Java

7,442

6,442 commits

yahoo/maha

A framework for rapid reporting API development; with out of the box support for high cardinality…

Scala

134

2,696 commits

gatling/gatling

Modern Load Testing as Code

Scala

6,956

7,141 commits

intel-hadoop/HiBench

HiBench is a big data benchmark suite.

Java

1,484

964 commits

AutoMQ/automq

Diskless Kafka® on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds.…

Java

10,612

12,787 commits

airbnb/aerosolve

A machine learning package built for humans.

Scala

4,808

587 commits

neo4j/neo4j

Graphs for Everyone

Java

17,190

70,747 commits

clyphy/Human-AI

No description

TypeScript

1 commits

xai-org/x-algorithm

Algorithm powering the For You feed on X

Rust

32,738

0 commits

galax-io/gatling-kafka-plugin

Gatling plugin for Apache Kafka — produce, request-reply, and consume load testing with Avro,…

Scala

179 commits

apache/flink

Apache Flink

Java

26,327

30,378 commits

twitter/the-algorithm

Source code for the X Recommendation Algorithm

Scala

73,881

6 commits

apache/kafka

Apache Kafka - A distributed event streaming platform

Java

33,675

15,169 commits

GoogleCloudDataproc/spark-bigquery-connector

BigQuery data source for Apache Spark: Read data from BigQuery into DataFrames, write DataFrames…

Java

425

903 commits

apache/polaris-tools

Apache Polaris Tools, additional tooling for Apache Polaris

TypeScript

179 commits

chenhongge/RobustTrees

[ICML 2019, 20 min long talk] Robust Decision Trees Against Adversarial Examples

C++

11 commits