Util/helper functions used in apache_beam.ml.inference.
WatchFilePattern(file_pattern, interval=360, stop_timestamp=Timestamp(9223372036854.775000))[source]¶
Watches a directory for updates to files matching a given file pattern.
- file_pattern – The file path to read from as a local file path or a
gs://path. The path can contain glob characters (
[...]sets). interval: Interval at which to check for files matching file_pattern in seconds.
- stop_timestamp – Timestamp after which no more files will be checked.
- Any previously used filenames cannot be reused. If a file is added
- or updated to a previously used filename, this transform will ignore that update. To trigger a model update, always upload a file with unique name.
- Initially, before the pipeline startup time, WatchFilePattern expects
- at least one file present that matches the file_pattern.
- This transform is supported in streaming mode since
- MatchContinuously produces an unbounded source. Running in batch mode can lead to undesired results or result in pipeline being stuck.
- file_pattern – The file path to read from as a local file path or a GCS