Custom I/O patterns

This page describes common patterns in pipelines with custom I/O connectors. Custom I/O connectors connect pipelines to databases that aren’t supported by Beam’s built-in I/O transforms.

Choosing between built-in and custom connectors

Built-in I/O connectors are tested and hardened, so use them whenever possible. Only use custom I/O connectors when:

• No built-in options exist
• Your pipeline pulls in a small subset of source data

For instance, use a custom I/O connector to enrich pipeline elements with a small subset of source data. If you’re processing a sales order and adding information to each purchase, you can use a custom I/O connector to pull the small subset of data into your pipeline (instead of processing the entire source).

Beam distributes work across many threads, so custom I/O connectors can increase your data source’s load average. You can reduce the load with the startstart and finishfinish bundle annotations.