Common pipeline patterns

Pipeline patterns demonstrate common Beam use cases. Pipeline patterns are based on real-world Beam deployments. Each pattern has a description, examples, and a solution or psuedocode.

File processing patterns - Patterns for reading from and writing to files

• Processing files as they arrive
• Accessing filenames

Side input patterns - Patterns for processing supplementary data

• Slowly updating global window side inputs

Pipeline option patterns - Patterns for configuring pipelines

• Retroactively logging runtime parameters

Custom I/O patterns - Patterns for pipeline I/O

• Choosing between built-in and custom connectors

Custom window patterns - Patterns for windowing functions

• Using data to dynamically set session window gaps

BigQuery patterns - Patterns for BigQueryIO

• Google BigQuery patterns

AI Platform integration patterns - Patterns for using Google Cloud AI Platform transforms

• Analysing the structure and meaning of text
• Getting predictions

Schema patterns - Patterns for using Schemas

• Using Joins

BQML integration patterns - Patterns on integrating BigQuery ML into your Beam pipeline

• Model export and on-worker serving using BQML and TFX_BSL

Cross-language patterns - Patterns for creating cross-language pipelines

• Cross-language patterns

State & timers patterns - Patterns for using state & timers

• Grouping elements for efficient external service calls

Contributing a pattern

To contribute a new pipeline pattern, create an issue with the pipeline-patterns label and add details to the issue description. See Get started contributing for more information.

What’s next

• Try an end-to-end example
• Execute your pipeline on a runner