Introducing Apache Beam
The Unified Apache Beam Model
The easiest way to do batch and streaming data processing. Write once, run anywhere data processing for mission-critical production workloads.
Learn moreHow Does It Work?
Data Sourcing
Beam reads your data from a diverse set of supported sources, no matter if it’s on-prem or in the cloud.
Data Processing
Beam executes your business logic for both batch and streaming use cases.
Data Writing
Beam writes the results of your data processing logic to the most popular data sinks in the industry.
Apache Beam Features
Unified
A simplified, single programming model for both batch and streaming use cases for every member of your data and application teams.
Extensible
Apache Beam is extensible, with projects such as TensorFlow Extended and Apache Hop built on top of Apache Beam.
Portable
Execute pipelines on multiple execution environments (runners), providing flexibility and avoiding lock-in.
Open Source
Open, community-based development and support to help evolve your application and meet the needs of your specific use cases.
Try Beam Playground
Beam Playground is an interactive environment to try out Beam transforms and examples without having to install Apache Beam in your environment. You can try the Apache Beam examples at Beam Playground (Beta).
Apache Beam has future-proofed Credit Karma’s data and ML platform for scalability and efficiency, enabling MLOps with unified pipelines, processing 5-10 TB daily at 5K events per second, and managing 20K+ ML features.
Apache Beam is a central component to Intuit’s Stream Processing Platform, which has driven 3x faster time-to-production for authoring a stream processing pipeline.
Apache Beam enabled real-time ML streaming feature generation and model execution playing a pivotal role in optimizing Lyft’s Marketplace ML predictions, processing ~4mil events per minute to generate ~100 features.
Seznam, a Czech search engine, has been an early contributor and adopter of Apache Beam, and they migrated several petabyte-scale workloads to Apache Beam pipelines.
Palo Alto Networks, Inc. is a global cybersecurity leader that uses Apache Beam to process ~10 millions of security log events per second for their real-time streaming infrastructure.
Apache Beam provides Ricardo, a leading Swiss second hand marketplace, with a scalable and reliable data processing framework that supports fundamental business scenarios and enables real-time and ML data processing.
Apache Hop, an open-source data orchestration platform, uses Apache Beam to “design once, run anywhere” and creates a value-add for Apache Beam users by enabling visual pipeline development and lifecycle management.
