.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
See the code
.NET for Apache Spark provides high-performance APIs for using Apache Spark from C# and F#. Use DataFrames and Spark SQL to work with structured data, and Structured Streaming to process streaming data.
Microsoft.Spark targets .NET Standard 2.0 and 2.1. Running an application also requires a compatible .NET Worker, Apache Spark runtime, and Scala bridge. The Worker runs on Windows, Linux, and macOS using .NET 8, or on Windows using .NET Framework.
Deployment guides are available for Azure HDInsight Spark, Amazon EMR Spark, and Databricks on AWS and Azure. Check the target platform's runtime and Worker compatibility before following a guide; these guides do not validate every managed-service runtime.
For background on the proposal for upstream .NET bindings, see SPIP: .NET bindings for Apache Spark.
The next release removes Apache Spark 2.x support and retains the Spark 3.0 through 3.5 runtime lines. See the migration guide before upgrading an existing application.
The getting-started examples use Spark 3.5.3 with Scala 2.12. Current source targets Spark 4.0.0–4.0.4 on Windows and Linux with .NET 8, the Scala 2.13 bridge, and JDK 17. Spark 4 support is pending release; use matching source-built or candidate packages, not the published v2.3.1 package. This does not imply full API or deployment parity with Spark 3.x.
For Spark 4.0 semi-structured data, see Variant schemas and Spark SQL; native .NET Variant values and Variant UDFs are not supported.
The published v2.3.1 release supports Spark 2.4 (except 2.4.2) and Spark 3.0 through 3.5, not Spark 4.0. See the release compatibility notes for that release; they do not describe the current source tree.
.NET for Apache Spark releases are available here and NuGet packages are available here.
The 2.4.0-rc1 preparation notes describe the candidate scope and known limitations; release artifacts are still pending validation and publication.
These instructions will show you how to run a .NET for Apache Spark app using .NET 8.
Follow the platform-specific instructions to install prerequisites, build the bridge and .NET components, and run the tests.
| Platform | Instructions |
|---|---|
| Windows | .NET 8 · .NET Framework 4.8 |
| Ubuntu | .NET 8 |
Browse the sample index for DataFrame and Structured Streaming examples. See Building from Source for setup instructions and the benchmark guide for running TPC-H queries.
| Scenario | Description | Code |
|---|---|---|
| DataFrames and Spark SQL | Basic queries and transformations | C# · F# |
| Streaming word count | Count words from a network stream | C# · F# |
| Windowed word count | Aggregate streaming data over time windows | C# · F# |
| Kafka word count | Process streaming data from Apache Kafka | C# · F# |
| TPC-H queries | Express benchmark queries using DataFrame APIs or SQL | DataFrame APIs · SQL |
End-to-end test status for 
; updates may be delayed.
| Spark | Windows | Linux |
|---|---|---|
| 3.0.0 |  |  |
| 3.0.1 |  |  |
| 3.0.2 |  |  |
| 3.1.1 |  |  |
| 3.1.2 |  |  |
| 3.2.1 |  |  |
| 3.2.2 |  |  |
| 3.2.3 |  |  |
| 3.3.0 |  |  |
| 3.3.1 |  |  |
| 3.3.2 |  |  |
| 3.3.3 |  |  |
| 3.3.4 |  |  |
| 3.4.0 |  |  |
| 3.4.1 |  |  |
| 3.4.2 |  |  |
| 3.4.3 |  |  |
| 3.4.4 |  |  |
| 3.5.0 |  |  |
| 3.5.1 |  |  |
| 3.5.2 |  |  |
| 3.5.3 |  |  |
| 4.0.0 |  |  |
| 4.0.1 |  |  |
| 4.0.2 |  |  |
| 4.0.3 |  |  |
| 4.0.4 |  |  |
We welcome contributions! Please review our contribution guide.
This project would not have been possible without the outstanding work from the following communities:
The .NET for Apache Spark team encourages contributions, both issues and PRs. The first step is finding an existing issue you want to contribute to or, if you cannot find any, opening an issue.
.NET for Apache Spark is an open source project under the .NET Foundation and does not come with Microsoft Support unless otherwise noted by the specific product. For issues with or questions about .NET for Apache Spark, please create an issue.
The .NET for Apache Spark project is part of the .NET Foundation.
This project has adopted the code of conduct defined by the Contributor Covenant to clarify expected behavior in our community. For more information, see the .NET Foundation Code of Conduct.
.NET for Apache Spark is licensed under the MIT license.
Overall
Maintained
Code review
Latest release
Releases this year
604 followers · starred Apr 2019
208 followers · starred Apr 2019
2,035 followers · starred Apr 2019
101 followers · starred Apr 2019
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
See the code
.NET for Apache Spark provides high-performance APIs for using Apache Spark from C# and F#. Use DataFrames and Spark SQL to work with structured data, and Structured Streaming to process streaming data.
Microsoft.Spark targets .NET Standard 2.0 and 2.1. Running an application also requires a compatible .NET Worker, Apache Spark runtime, and Scala bridge. The Worker runs on Windows, Linux, and macOS using .NET 8, or on Windows using .NET Framework.
Deployment guides are available for Azure HDInsight Spark, Amazon EMR Spark, and Databricks on AWS and Azure. Check the target platform's runtime and Worker compatibility before following a guide; these guides do not validate every managed-service runtime.
For background on the proposal for upstream .NET bindings, see SPIP: .NET bindings for Apache Spark.
The next release removes Apache Spark 2.x support and retains the Spark 3.0 through 3.5 runtime lines. See the migration guide before upgrading an existing application.
The getting-started examples use Spark 3.5.3 with Scala 2.12. Current source targets Spark 4.0.0–4.0.4 on Windows and Linux with .NET 8, the Scala 2.13 bridge, and JDK 17. Spark 4 support is pending release; use matching source-built or candidate packages, not the published v2.3.1 package. This does not imply full API or deployment parity with Spark 3.x.
For Spark 4.0 semi-structured data, see Variant schemas and Spark SQL; native .NET Variant values and Variant UDFs are not supported.
The published v2.3.1 release supports Spark 2.4 (except 2.4.2) and Spark 3.0 through 3.5, not Spark 4.0. See the release compatibility notes for that release; they do not describe the current source tree.
.NET for Apache Spark releases are available here and NuGet packages are available here.
The 2.4.0-rc1 preparation notes describe the candidate scope and known limitations; release artifacts are still pending validation and publication.
These instructions will show you how to run a .NET for Apache Spark app using .NET 8.
Follow the platform-specific instructions to install prerequisites, build the bridge and .NET components, and run the tests.
| Platform | Instructions |
|---|---|
| Windows | .NET 8 · .NET Framework 4.8 |
| Ubuntu | .NET 8 |
Browse the sample index for DataFrame and Structured Streaming examples. See Building from Source for setup instructions and the benchmark guide for running TPC-H queries.
| Scenario | Description | Code |
|---|---|---|
| DataFrames and Spark SQL | Basic queries and transformations | C# · F# |
| Streaming word count | Count words from a network stream | C# · F# |
| Windowed word count | Aggregate streaming data over time windows | C# · F# |
| Kafka word count | Process streaming data from Apache Kafka | C# · F# |
| TPC-H queries | Express benchmark queries using DataFrame APIs or SQL | DataFrame APIs · SQL |
End-to-end test status for ; updates may be delayed.
| Spark | Windows | Linux |
|---|---|---|
| 3.0.0 | | |
| 3.0.1 | | |
| 3.0.2 | | |
| 3.1.1 | | |
| 3.1.2 | | |
| 3.2.1 | | |
| 3.2.2 | | |
| 3.2.3 | | |
| 3.3.0 | | |
| 3.3.1 | | |
| 3.3.2 | | |
| 3.3.3 | | |
| 3.3.4 | | |
| 3.4.0 | | |
| 3.4.1 | | |
| 3.4.2 | | |
| 3.4.3 | | |
| 3.4.4 | | |
| 3.5.0 | | |
| 3.5.1 | | |
| 3.5.2 | | |
| 3.5.3 | | |
| 4.0.0 | | |
| 4.0.1 | | |
| 4.0.2 | | |
| 4.0.3 | | |
| 4.0.4 | | |
We welcome contributions! Please review our contribution guide.
This project would not have been possible without the outstanding work from the following communities:
The .NET for Apache Spark team encourages contributions, both issues and PRs. The first step is finding an existing issue you want to contribute to or, if you cannot find any, opening an issue.
.NET for Apache Spark is an open source project under the .NET Foundation and does not come with Microsoft Support unless otherwise noted by the specific product. For issues with or questions about .NET for Apache Spark, please create an issue.
The .NET for Apache Spark project is part of the .NET Foundation.
This project has adopted the code of conduct defined by the Contributor Covenant to clarify expected behavior in our community. For more information, see the .NET Foundation Code of Conduct.
.NET for Apache Spark is licensed under the MIT license.
Overall
Maintained
Code review
Latest release
Releases this year
604 followers · starred Apr 2019
208 followers · starred Apr 2019
2,035 followers · starred Apr 2019
101 followers · starred Apr 2019
