Apache DataFusion Comet is a high-performance accelerator for Apache Spark, built on top of the powerful
Apache DataFusion query engine. Comet is designed to significantly enhance the
performance of Apache Spark workloads while leveraging commodity hardware and seamlessly integrating with the
Spark ecosystem without requiring any code changes.
Comet delivers a performance speedup for many queries, enabling faster data processing and shorter time-to-insights.
The following chart shows the time it takes to run the 22 TPC-H queries against 100 GB of data in Parquet format
using a single executor with 8 cores. See the Comet Benchmarking Guide
for details of the environment used for these benchmarks.
When using Comet, the overall run time is reduced from 649 seconds to 440 seconds, a 1.5x speedup.
Running the same queries with DataFusion standalone (without Spark) using the same number of cores results in a 3.9x
speedup compared to Spark.
Comet is not yet achieving full DataFusion speeds in all cases, but with future work we aim to provide a 2x-4x speedup
for many use cases.
Here is a breakdown showing relative performance of Spark, Comet, and DataFusion for each TPC-H query.
The following chart shows how much Comet currently accelerates each query from the benchmark. Performance optimization
is an ongoing task, and we welcome contributions from the community to help achieve even greater speedups in the future.
These benchmarks can be reproduced in any environment using the documentation in the
Comet Benchmarking Guide. We encourage
you to run your own benchmarks.
Comet leverages commodity hardware, eliminating the need for costly hardware upgrades or
specialized hardware accelerators, such as GPUs or FGPA. By maximizing the utilization of commodity hardware, Comet
ensures cost-effectiveness and scalability for your Spark deployments.
Comet aims for 100% compatibility with all supported versions of Apache Spark, allowing you to integrate Comet into
your existing Spark deployments and workflows seamlessly. With no code changes required, you can immediately harness
the benefits of Comet’s acceleration capabilities without disrupting your Spark applications.
Comet tightly integrates with the core Apache DataFusion project, leveraging its powerful execution engine. With
seamless interoperability between Comet and DataFusion, you can achieve optimal performance and efficiency in your
Spark workloads.
Comet boasts a vibrant and active community of developers, contributors, and users dedicated to advancing the
capabilities of Apache DataFusion and accelerating the performance of Apache Spark.
To get started with Apache DataFusion Comet, follow the
installation instructions. Join the
DataFusion Slack and Discord channels to connect
with other users, ask questions, and share your experiences with Comet.
We welcome contributions from the community to help improve and enhance Apache DataFusion Comet. Whether it’s fixing
bugs, adding new features, writing documentation, or optimizing performance, your contributions are invaluable in
shaping the future of Comet. Check out our
contributor guide to get started.
Apache DataFusion Comet is licensed under the Apache License 2.0. See the LICENSE.txt file for details.
We would like to express our gratitude to the Apache DataFusion community for their support and contributions to
Comet. Together, we’re building a faster, more efficient future for big data processing with Apache Spark.
