Package org.apache.beam.io.debezium

Interface OffsetRetainer

All Superinterfaces:
Serializable
All Known Implementing Classes:
FileSystemOffsetRetainer
public interface OffsetRetainer extends Serializable
Strategy interface for persisting and restoring Debezium connector offsets across pipeline restarts.

When configured via DebeziumIO.Read.withOffsetRetainer(OffsetRetainer), the pipeline behaves as follows:

  1. On startup, loadOffset() is called once. If a non-null offset is returned, the Debezium connector resumes from that position; otherwise it starts from the beginning of the change stream.
  2. After each successful task.commit(), saveOffset(Map) is called with the latest committed offset.

A ready-to-use filesystem-based implementation is provided by FileSystemOffsetRetainer, which supports any Beam-compatible filesystem (local, GCS, S3, etc.)

Implementations must be Serializable because they are embedded inside DebeziumIO.Read, which is a PTransform that gets serialized and shipped to workers.

  • Method Summary

    Modifier and Type
    Method
    Description
    @Nullable Map<String,Object>
    loadOffset()
    Returns the most recently saved offset, or null if no offset has been saved yet.
    void
    saveOffset(Map<String,Object> offset)
    Persists the given offset so it can be recovered after a pipeline restart.

  • Method Details

    • loadOffset

      @Nullable Map<String,Object> loadOffset()
      Returns the most recently saved offset, or null if no offset has been saved yet.

      A null return causes the connector to start from the beginning of the change stream. Implementations should handle transient I/O errors gracefully and return null on failure rather than propagating an exception.

    • saveOffset

      void saveOffset(Map<String,Object> offset)
      Persists the given offset so it can be recovered after a pipeline restart.

      Called after each successful task.commit() with the latest committed offset. Implementations should swallow transient errors rather than throwing, so that a failed save does not terminate the pipeline.

      Parameters:
      offset - The current connector offset, as returned by SourceRecord.sourceOffset().