apache_beam.transforms.xlang.io module

Cross-language transforms in this module can be imported from the apache_beam.io package.

class apache_beam.transforms.xlang.io.DatadogWrite(api_key, url, batch_count=None, error_handling=None, max_buffer_size=None, min_batch_count=None, parallelism=None, expansion_service=None)[source]

Bases: ExternalTransform

Parameters:

  • api_key – (str) The Datadog API key.

  • url – (str) The Datadog API URL.

  • batch_count – (int32) The number of events to batch together for each write.

  • error_handling – (Row(output=<class ‘str’>)) Specifies how to handle errors.

  • max_buffer_size – (int64) The maximum buffer size in bytes.

  • min_batch_count – (int32) The minimum number of events to batch together for each write.

  • parallelism – (int32) The degree of parallelism for writing.

identifier: str = 'beam:schematransform:org.apache.beam:datadog_write:v1'
class apache_beam.transforms.xlang.io.GenerateSequence(start, end=None, rate=None, expansion_service=None)[source]

Bases: ExternalTransform

Outputs a PCollection of Beam Rows, each containing a single INT64 number called “value”. The count is produced from the given “start” value and either up to the given “end” or until 2^63 - 1. To produce an unbounded PCollection, simply do not specify an “end” value. Unbounded sequences can specify a “rate” for output elements. In all cases, the sequence of numbers is generated in parallel, so there is no inherent ordering between the generated values

Parameters:

  • start – (int64) The minimum number to generate (inclusive).

  • end – (int64) The maximum number to generate (exclusive). Will be an unbounded sequence if left unspecified.

  • rate – (Row(elements=<class ‘int64’>, seconds=typing.Optional[int64])) Specifies the rate to generate a given number of elements per a given number of seconds. Applicable only to unbounded sequences.

identifier: str = 'beam:schematransform:org.apache.beam:generate_sequence:v1'
class apache_beam.transforms.xlang.io.MongodbWrite(collection, database, uri, batch_size=None, error_handling=None, expansion_service=None)[source]

Bases: ExternalTransform

Parameters:

  • collection – (str) The MongoDB collection to write to.

  • database – (str) The MongoDB database to write to.

  • uri – (str) The connection URI for the MongoDB server.

  • batch_size – (int64) The number of documents to include in each batch write.

  • error_handling – (Row(output=<class ‘str’>)) This option specifies whether and where to output unwritable rows. Note: Error handling is currently limited to data conversion failures before sending to the MongoDB driver, as the underlying MongoDbIO does not yet support dead-letter queues for write failures.

identifier: str = 'beam:schematransform:org.apache.beam:mongodb_write:v1'
class apache_beam.transforms.xlang.io.TfrecordRead(compression, file_pattern, validate, error_handling=None, expansion_service=None)[source]

Bases: ExternalTransform

Parameters:

  • compression – (str) Decompression type to use when reading input files.

  • file_pattern – (str) Filename or file pattern used to find input files.

  • validate – (boolean) Validate file pattern.

  • error_handling – (Row(output=<class ‘str’>)) This option specifies whether and where to output unwritable rows.

identifier: str = 'beam:schematransform:org.apache.beam:tfrecord_read:v1'
class apache_beam.transforms.xlang.io.TfrecordWrite(compression, num_shards, output_prefix, error_handling=None, filename_suffix=None, max_num_writers_per_bundle=None, no_spilling=None, shard_template=None, expansion_service=None)[source]

Bases: ExternalTransform

Parameters:

  • compression – (str) Option to indicate the output sink’s compression type. Default is NONE.

  • num_shards – (int32) The number of shards to use, or 0 for automatic.

  • output_prefix – (str) The directory to which files will be written.

  • error_handling – (Row(output=<class ‘str’>)) This option specifies whether and where to output unwritable rows.

  • filename_suffix – (str) The suffix of each file written, combined with prefix and shardTemplate.

  • max_num_writers_per_bundle – (int32) Maximum number of writers created in a bundle before spilling to shuffle.

  • no_spilling – (boolean) Whether to skip the spilling of data caused by having maxNumWritersPerBundle.

  • shard_template – (str) The shard template of each file written, combined with prefix and suffix.

identifier: str = 'beam:schematransform:org.apache.beam:tfrecord_write:v1'
class apache_beam.transforms.xlang.io.ReadFromMqtt(connection_configuration, max_num_records=None, max_read_time_seconds=None, expansion_service=None)[source]

Bases: ExternalTransform

Reads messages from an MQTT broker and outputs each payload as a single bytes field.

By default the read is unbounded (streaming): it keeps consuming messages from the subscribed topic until the pipeline is stopped. Setting maxNumRecords and/or maxReadTimeSeconds bounds the read, producing a bounded (batch) PCollection.

Note: streaming reads require a runner that supports portable streaming (e.g. Prism, Flink, or Dataflow). The legacy local Python DirectRunner does not execute portable streaming cross-language reads.

Parameters:

  • connection_configuration – (Row(client_id=typing.Optional[str], password=typing.Optional[str], server_uri=<class ‘str’>, topic=typing.Optional[str], username=typing.Optional[str])) Configuration options to set up the MQTT connection.

  • max_num_records – (int64) The max number of records to receive. Setting this will result in a bounded PCollection.

  • max_read_time_seconds – (int64) The maximum time for this source to read messages. Setting this will result in a bounded PCollection.

identifier: str = 'beam:schematransform:org.apache.beam:mqtt_read:v1'
class apache_beam.transforms.xlang.io.WriteToMqtt(connection_configuration, retained=None, expansion_service=None)[source]

Bases: ExternalTransform

Publishes messages to an MQTT broker. Expects an input PCollection of rows with a single bytes field, each of which is published as one MQTT message.

Works with both bounded (batch) and unbounded (streaming) input PCollections.

Parameters:

  • connection_configuration – (Row(client_id=typing.Optional[str], password=typing.Optional[str], server_uri=<class ‘str’>, topic=typing.Optional[str], username=typing.Optional[str])) Configuration options to set up the MQTT connection.

  • retained – (boolean) Whether or not the publish message should be retained by the messaging engine. When a subscriber connects, it gets the latest retained message. Defaults to False, which will clear the retained message from the server.

identifier: str = 'beam:schematransform:org.apache.beam:mqtt_write:v1'