public abstract static class AvroIO.Read<T> extends PTransform<PBegin,PCollection<T>>
AvroIO.read(java.lang.Class<T>) and AvroIO.readGenericRecords(org.apache.avro.Schema).name| Constructor and Description |
|---|
Read() |
| Modifier and Type | Method and Description |
|---|---|
PCollection<T> |
expand(PBegin input)
Override this method to specify how this
PTransform should be expanded
on the given InputT. |
AvroIO.Read<T> |
from(java.lang.String filepattern)
Like
from(ValueProvider). |
AvroIO.Read<T> |
from(ValueProvider<java.lang.String> filepattern)
Reads from the given filename or filepattern.
|
void |
populateDisplayData(DisplayData.Builder builder)
Register display data for the given transform or component.
|
AvroIO.Read<T> |
watchForNewFiles(Duration pollInterval,
Watch.Growth.TerminationCondition<java.lang.String,?> terminationCondition)
Continuously watches for new files matching the filepattern, polling it at the given
interval, until the given termination condition is reached.
|
AvroIO.Read<T> |
withEmptyMatchTreatment(EmptyMatchTreatment treatment)
Configures whether or not a filepattern matching no files is allowed.
|
AvroIO.Read<T> |
withHintMatchesManyFiles()
Hints that the filepattern specified in
from(String) matches a very large number of
files. |
AvroIO.Read<T> |
withMatchConfiguration(FileIO.MatchConfiguration matchConfiguration)
Sets the
FileIO.MatchConfiguration. |
getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, toString, validatepublic AvroIO.Read<T> from(ValueProvider<java.lang.String> filepattern)
If it is known that the filepattern will match a very large number of files (at least tens
of thousands), use withHintMatchesManyFiles() for better performance and scalability.
public AvroIO.Read<T> from(java.lang.String filepattern)
from(ValueProvider).public AvroIO.Read<T> withMatchConfiguration(FileIO.MatchConfiguration matchConfiguration)
FileIO.MatchConfiguration.public AvroIO.Read<T> withEmptyMatchTreatment(EmptyMatchTreatment treatment)
@Experimental(value=SPLITTABLE_DO_FN) public AvroIO.Read<T> watchForNewFiles(Duration pollInterval, Watch.Growth.TerminationCondition<java.lang.String,?> terminationCondition)
PCollection
is unbounded.
This works only in runners supporting Experimental.Kind.SPLITTABLE_DO_FN.
public AvroIO.Read<T> withHintMatchesManyFiles()
from(String) matches a very large number of
files.
This hint may cause a runner to execute the transform differently, in a way that improves performance for this case, but it may worsen performance if the filepattern matches only a small number of files (e.g., in a runner that supports dynamic work rebalancing, it will happen less efficiently within individual files).
public PCollection<T> expand(PBegin input)
PTransformPTransform should be expanded
on the given InputT.
NOTE: This method should not be called directly. Instead apply the
PTransform should be applied to the InputT using the apply
method.
Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).
expand in class PTransform<PBegin,PCollection<T>>public void populateDisplayData(DisplayData.Builder builder)
PTransformpopulateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect
display data via DisplayData.from(HasDisplayData). Implementations may call
super.populateDisplayData(builder) in order to register display data in the current
namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use
the namespace of the subcomponent.
By default, does not register any display data. Implementors may override this method to provide their own display data.
populateDisplayData in interface HasDisplayDatapopulateDisplayData in class PTransform<PBegin,PCollection<T>>builder - The builder to populate with display data.HasDisplayData