I/O Connectors

Apache Beam I/O connectors provide read and write transforms for the most popular data storage systems so that Beam users can benefit from native optimised connectivity. With the available I/Os, Apache Beam pipelines can read and write data from and to an external storage type in a unified and distributed way.

I/O connectors denoted via X-language have been made available using the Apache Beam multi-language pipelines framework.

I/O connectors denoted as Supported via Managed API can be accessed using the simplified managed I/O APIs for Java and Python.

Built-in I/O Connectors

This table provides a consolidated, at-a-glance overview of the available built-in I/O connectors.

Connector Name	Source Supported	Sink Supported	Java	Python	Go	Typescript	Batch Supported	Streaming Supported	Supported via Managed API
FileIO	✔	✔	✔ native	✔ native	✔ native	Not available	✔	✔	✘
AvroIO	✔	✔	✔ native	✔ native	✔ native	✔ via X-language	✔	✔	✘
TextIO (metrics)	✔	✔	✔ native	✔ native	✔ native	✔ via X-language	✔	✔	✘
TFRecordIO	✔	✔	✔ native	✔ native	Not available	Not available	✔	✘	✘
XmlIO	✔	✔	✔ native	Not available	Not available	Not available	✔	✘	✘
TikaIO	✔	✘	✔ native	Not available	Not available	Not available	✔	✔	✘
ParquetIO (guide)	✔	✔	✔ native	✔ native	✔ native	✔ via X-language	✔	✘	✘
ThriftIO	✔	✔	✔ native	Not available	Not available	Not available	✔	✘	✘
HadoopFileSystem	✔	✔	✔ native	✔ native	Not available	✔via X-language	✔	✘	✘
GcsFileSystem (metrics)	✔	✔	✔ native	✔ native	✔ native	✔via X-language	✔	✘	✘
LocalFileSystem	✔	✔	✔ native	✔ native	✔ native	✔via X-language	✔	✘	✘
S3FileSystem	✔	✔	✔ native	✔ native	Not available	✔via X-language	✔	✘	✘
In-memory	✘	✔	✘	✘	✔ native	✘	✔	✘	✘
KinesisIO	✔	✔	✔ native	✔ via X-language	Not available	Not available	✔	✔	✘
AmqpIO	✔	✔	✔ native	Not available	Not available	Not available	✔	✔	✘
KafkaIO	✔	✔	✔ native	✔ via X-language	✔ via X-language	✔ via X-language	✔	✔	✔
PubSubIO	✔	✔	✔ native	✔ native	✔ native	✔ via X-language	✔	✔	✘
JmsIO	✔	✔	✔ native	Not available	Not available	Not available	✔	✔	✘
MqttIO	✔	✔	✔ native	Not available	Not available	Not available	✔	✔	✘
RabbitMqIO	✔	✔	✔ native	Not available	Not available	Not available	✔	✔	✘
SqsIO	✔	✔	✔ native	Not available	Not available	Not available	✔	✔	✘
SnsIO	✘	✔	✔ native	Not available	Not available	Not available	✔	✘	✘
CassandraIO	✔	✔	✔ native	Not available	Not available	Not available	✔	✘	✘
HadoopFormatIO (guide)	✔	✔	✔ native	Not available	Not available	Not available	✔	✔	✘
HBaseIO	✔	✔	✔ native	Not available	Not available	Not available	✔	✘	✘
HCatalogIO (guide)	✔	✔	✔ native	Not available	Not available	Not available	✔	✔	✘
KuduIO	✔	✔	✔ native	Not available	Not available	Not available	✔	✘	✘
SolrIO	✔	✔	✔ native	Not available	Not available	Not available	✔	✔	✘
ElasticsearchIO	✔	✔	✔ native	Not available	Not available	Not available	✔	✔	✘
BigQueryIO (guide) (metrics)	✔	✔	✔ native	✔ native	✔ native ✔ via X-language	✔ via X-language	✔	✔	✔
BigTableIO (metrics)	✔	✔	✔ native	✔ native (sink) ✔ via X-language	✔ native (sink) ✔ via X-language	Not available	✔	✔	✘
DatastoreIO	✔	✔	✔ native	✔ native	✔ native	Not available	✔	✔	✘
SnowflakeIO (guide)	✔	✔	✔ native	✔ via X-language	Not available	Not available	✔	✘	✘
SpannerIO	✔	✔	✔ native	✔ via X-language	✔ native	Not available	✔	✔	✘
JdbcIO	✔	✔	✔ native	✔ via X-language	✔ via X-language	Not available	✔	✘	✘
DebeziumIO	✔	✘	✔ native	✔ via X-language	✔ via X-language	Not available	✔	✔	✘
MongoDbIO	✔	✔	✔ native	✔ native	✔ native	Not available	✔	✘	✘
MongoDbGridFSIO	✔	✔	✔ native	Not available	Not available	Not available	✔	✘	✘
RedisIO	✔	✔	✔ native	Not available	Not available	Not available	✔	✘	✘
DynamoDBIO	✔	✔	✔ native	Not available	Not available	Not available	✔	✔	✘
ClickHouseIO	✘	✔	✔ native	Not available	Not available	Not available	✔	✘	✘
DatabaseIO	✔	✔	✘	✘	✔ native	Not available	✔	✘	✘
GenerateSequence	✔	✘	✔ native	Not available	Not available	Not available	✔	✔	✘
SplunkIO	✘	✔	✔ native	Not available	Not available	Not available	✔	✔	✘
FhirIO	✔	✔	✔ native	Not available	✔ native	Not available	✔	✔	✘
HL7v2IO	✔	✔	✔ native	Not available	Not available	Not available	✔	✔	✘
DicomIO	✔	✔	✔ native	✔ native	Not available	Not available	✔	✔	✘
FlinkStreaming ImpulseSource	✔	✘	Not available	✔ native	Not available	Not available	✔	✔	✘
Firestore IO	✔	✔	✔ native	Not available	Not available	Not available	✔	✘	✘
Neo4j	✔	✔	✔ native	Not available	Not available	Not available	✔	✘	✘
Pub/Sub Lite	✔	✔	✔ native	✔ via X-language	Not available	✔ via X-language	✔	✔	✘
InfluxDB	✔	✔	✔ native	Not available	Not available	Not available	✔	✔	✘
SparkReceiverIO (guide)	✔	✘	✔ native	Not available	Not available	Not available	✘	✔	✘
CdapIO (guide)	✔	✔	✔ native	Not available	Not available	Not available	✔	✔	✘
SingleStoreDB (guide)	✔	✔	✔ native	Not available	Not available	Not available	✔	✘	✘
GoogleAdsIO	✔	✔	✔ native	Not available	Not available	Not available	✔	✔	✘
Web APIs (guide)	✔	✔	✔ native	✔ native	Not available	Not available	✔	✔	✘
Iceberg (Managed I/O)	✔	✔	✔ native	✔ via X-language	Not available	Not available	✔	✔	✔

Other I/O Connectors for Apache Beam

Connector Name	Source Supported	Sink Supported	Java	Python	Go	Typescript	Batch Supported	Streaming Supported
Solace	✔	✘	✔ native	Not available	Not available	Not available	✔	✔
SAP Hana to Google BigQuery	✔	✔	✔ native	Not available	Not available	Not available	✔	✘
MySQL	✔	✔	Not available	✔ native	Not available	Not available	✔	✘
TrepWsIO	✔	✘	✔ native	Not available	Not available	Not available	✔	✔
KineticaDB	✔	✔	✔ native	Not available	Not available	Not available	✔	✘
Cognite Data Fusion	✔	✔	✔ native	Not available	Not available	Not available	✔	✔
Pyodbc	✔	✔	Not available	✔ native	Not available	Not available	✔	✘
Go Connect	✔	✔	✘	✘	✔ native	Not available	✔	✔
Tinybird	✘	✔	Not available	✔ native	Not available	Not available	✔	✔
Cloud SQL	✔	✘	Not available	✔ native	Not available	Not available	✔	✘
Cloud Bigtable (HBase based)	✔	✔	✔ native	Not available	Not available	Not available	✔	✘
Beam PyIO (Collection of Python IO connectors)	✔	✔	Not available	✔ native	Not available	Not available	✔	✔

Last updated on 2025/05/15

Have you found everything you were looking for?

Was it all useful and clear? Is there anything that you would like to change? Let us know!