public class TextIO extends java.lang.Object
PTransforms for reading and writing text files.
Pipeline p = ...; // A simple Read of a local file (only runs locally): PCollection<String> lines = p.apply(TextIO.read().from("/local/path/to/file.txt"));
By default, all input is put into the global window before writing. If per-window writes are
desired - for example, when using a streaming runner -
TextIO.Write.withWindowedWrites() will cause windowing and triggering to be
preserved. When producing windowed writes, the number of output shards must be set explicitly
TextIO.Write.withNumShards(int); some runners may set this for you to a
runner-chosen value, so you may need not set it yourself. A
FileBasedSink.FilenamePolicy can also be
set in case you need better control over naming files created by unique windows.
DefaultFilenamePolicy policy for producing unique filenames might not be appropriate
for your use case.
Any existing files with the same names as generated output files will be overwritten.
// A simple Write to a local file (only runs locally): PCollection<String> lines = ...; lines.apply(TextIO.write().to("/path/to/file.txt")); // Same as above, only with Gzip compression: PCollection<String> lines = ...; lines.apply(TextIO.write().to("/path/to/file.txt")); .withSuffix(".txt") .withWritableByteChannelFactory(FileBasedSink.CompressionType.GZIP));
|Modifier and Type||Class and Description|
Possible text file compression types.