Package org.apache.beam.sdk.io.arrowflight

Class ArrowFlightIO

java.lang.Object
org.apache.beam.sdk.io.arrowflight.ArrowFlightIO
public class ArrowFlightIO extends Object
IO to read and write data using Apache Arrow Flight.

Arrow Flight is an RPC framework for transferring Arrow-formatted data over gRPC.

Reading from an Arrow Flight server

read() returns a bounded PCollection of Row elements. Each row is converted from Arrow record batches using the existing ArrowConversion extension. 


 PCollection<Row> rows = pipeline.apply(
     ArrowFlightIO.read()
         .withHost("localhost")
         .withPort(47470)
         .withCommand("SELECT * FROM my_table"));

Writing to an Arrow Flight server

write() accepts a PCollection of Row elements and streams them to a Flight server using doPut. 


 rows.apply(
     ArrowFlightIO.write()
         .withHost("localhost")
         .withPort(47470)
         .withDescriptor("my_table")
         .withBatchSize(1024));

Java runtime configuration

On Java 17 or later, Arrow memory access requires --add-opens=java.base/java.nio=ALL-UNNAMED on JVMs executing this connector. Beam SDK containers enable it by default. For other environments, add it to the local or runner JVM and pass --JdkAddOpenModules=java.base/java.nio=ALL-UNNAMED to configure SDK harness JVMs. See the Arrow Java installation guide.