Class AddFiles

All Implemented Interfaces:
Serializable, HasDisplayData

public class AddFiles extends PTransform<PCollection<String>,PCollectionRowTuple>
Registers existing Parquet, ORC or Avro files in an Iceberg table without rewriting them: each path becomes a DataFile with partition metadata and column stats, batched into manifests and committed as snapshots.

AddFiles is offered as a Managed transform. Its input is a PCollection of Rows with a single string field, the file's path, and its options are listed in AddFilesSchemaTransformProvider.Configuration:


 Map<String, Object> config = Map.of(
         "table", "db.sales",
         "catalog_properties", Map.of("type", "rest", "uri", catalogUri),
         "error_handling", Map.of("output", "errors"));

 PCollectionRowTuple result =
     paths.apply(Managed.write(Managed.ICEBERG_ADD_FILES).withConfig(config));
 

Outputs: snapshots (one row per commit), errors (one row per file that could not be registered: file, error), and dry_run_report when a dry run is configured. Through Managed, error rows are only output when error_handling is set, under the name it gives.

Schema evolution. With schema evolution options set (SchemaEvolutionConfig), a pre-pass reads every Parquet footer, classifies the change each distinct file schema needs on the table (add a column, relax a required column, promote a type), commits the allowed changes in one transaction, and only then registers the files. Manifest entries are immutable, so this ordering is what guarantees that every registered file carries stats for every column it has. Files whose schema needs a change that is not allowed, or that conflicts with the table or with another file, are incompatible: by default the pipeline fails before committing anything, or routes them to errors (see SchemaEvolutionConfig.IncompatibleSchemaHandling). The per-file checks read Parquet footers: an ORC or Avro file, or a pinned column the footer has no null count for, cannot be verified and goes to errors unless SchemaEvolutionConfig.UnverifiableFileHandling.ACCEPT registers it on trust. When the table does not exist, the pre-pass creates it from the union of the file schemas; if no readable Parquet schema can seed it, nothing is created and every file goes to errors. Schema evolution currently requires bounded input; unbounded input with options set is rejected at construction.


 Map<String, Object> config = Map.of(
         "table", "db.sales",
         "catalog_properties", catalogProperties,
         "schema_evolution_options", List.of("ALLOW_FIELD_ADDITION", "ALLOW_TYPE_PROMOTION"),
         "required_columns", List.of("id"));
 paths.apply(Managed.write(Managed.ICEBERG_ADD_FILES).withConfig(config));
 

Without options the table schema is never changed and files register as-is: columns the table does not have get no stats and are not readable, and a nested column the table does not know can make that file, and any scan that includes it, fail in an Iceberg reader.

See Also:
  • Constructor Details

  • Method Details

    • withPathPattern

      public AddFiles withPathPattern(String pathPattern)
      A rule that routes each file to a destination table, read from the file's own path: a glob with {name} placeholders, whose captures fill in the table identifier as a template. So gs://lake/export/{tbl}/** with table sales.{tbl} routes a file under export/orders/ to sales.orders.

      Everything that can be settled without the input is settled here: a mistyped pattern, a template naming something the pattern never captures, or an option this cannot be combined with is one exception at submit time, not one error row per file.

    • expand

      public PCollectionRowTuple expand(PCollection<String> input)
      Description copied from class: PTransform
      Override this method to specify how this PTransform should be expanded on the given InputT.

      NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

      Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

      Specified by:
      expand in class PTransform<PCollection<String>,PCollectionRowTuple>
    • inferFormat

      public static org.apache.iceberg.FileFormat inferFormat(String path)
      Attempt to infer file format from extension.