apache_beam.transforms.userstate module¶

User-facing interfaces for the Beam State and Timer APIs.

class apache_beam.transforms.userstate.StateSpec(name: str, coder: apache_beam.coders.coders.Coder)[source]¶

Bases: object

Specification for a user DoFn state cell.

to_runner_api(context: PipelineContext) → org.apache.beam.model.pipeline.v1.beam_runner_api_pb2.StateSpec[source]¶

class apache_beam.transforms.userstate.ReadModifyWriteStateSpec(name: str, coder: apache_beam.coders.coders.Coder)[source]¶

Bases: apache_beam.transforms.userstate.StateSpec

Specification for a user DoFn value state cell.

to_runner_api(context: PipelineContext) → org.apache.beam.model.pipeline.v1.beam_runner_api_pb2.StateSpec[source]¶

class apache_beam.transforms.userstate.BagStateSpec(name: str, coder: apache_beam.coders.coders.Coder)[source]¶

Bases: apache_beam.transforms.userstate.StateSpec

Specification for a user DoFn bag state cell.

to_runner_api(context: PipelineContext) → org.apache.beam.model.pipeline.v1.beam_runner_api_pb2.StateSpec[source]¶

class apache_beam.transforms.userstate.SetStateSpec(name: str, coder: apache_beam.coders.coders.Coder)[source]¶

Bases: apache_beam.transforms.userstate.StateSpec

Specification for a user DoFn Set State cell

to_runner_api(context: PipelineContext) → org.apache.beam.model.pipeline.v1.beam_runner_api_pb2.StateSpec[source]¶

class apache_beam.transforms.userstate.CombiningValueStateSpec(name: str, coder: Optional[apache_beam.coders.coders.Coder] = None, combine_fn: Any = None)[source]¶

Bases: apache_beam.transforms.userstate.StateSpec

Specification for a user DoFn combining value state cell.

Initialize the specification for CombiningValue state.

CombiningValueStateSpec(name, combine_fn) -> Coder-inferred combining value: state spec.
CombiningValueStateSpec(name, coder, combine_fn) -> Combining value state: spec with coder and combine_fn specified.

Parameters:	name (str) – The name by which the state is identified. coder (Coder) – Coder specifying how to encode the values to be combined. May be inferred. combine_fn (`CombineFn` or `callable`) – Function specifying how to combine the values passed to state.

to_runner_api(context: PipelineContext) → org.apache.beam.model.pipeline.v1.beam_runner_api_pb2.StateSpec[source]¶

class apache_beam.transforms.userstate.Timer(user_key, dynamic_timer_tag, windows, clear_bit, fire_timestamp, hold_timestamp, paneinfo)¶

Bases: tuple

Create new instance of Timer(user_key, dynamic_timer_tag, windows, clear_bit, fire_timestamp, hold_timestamp, paneinfo)

clear_bit¶: Alias for field number 3

dynamic_timer_tag¶: Alias for field number 1

fire_timestamp¶: Alias for field number 4

hold_timestamp¶: Alias for field number 5

paneinfo¶: Alias for field number 6

user_key¶: Alias for field number 0

windows¶: Alias for field number 2

class apache_beam.transforms.userstate.TimerSpec(name: str, time_domain: str)[source]¶

Bases: object

Specification for a user stateful DoFn timer.

prefix = 'ts-'¶

to_runner_api(context: PipelineContext, key_coder: apache_beam.coders.coders.Coder, window_coder: apache_beam.coders.coders.Coder) → org.apache.beam.model.pipeline.v1.beam_runner_api_pb2.TimerFamilySpec[source]¶

apache_beam.transforms.userstate.on_timer(timer_spec: apache_beam.transforms.userstate.TimerSpec) → Callable[[CallableT], CallableT][source]¶

Decorator for timer firing DoFn method.

This decorator allows a user to specify an on_timer processing method in a stateful DoFn. Sample usage:

class MyDoFn(DoFn):
  TIMER_SPEC = TimerSpec('timer', TimeDomain.WATERMARK)

  @on_timer(TIMER_SPEC)
  def my_timer_expiry_callback(self):
    logging.info('Timer expired!')

apache_beam.transforms.userstate.get_dofn_specs(dofn: DoFn) → Tuple[Set[apache_beam.transforms.userstate.StateSpec], Set[apache_beam.transforms.userstate.TimerSpec]][source]¶

Gets the state and timer specs for a DoFn, if any.

Parameters:	dofn (apache_beam.transforms.core.DoFn) – The DoFn instance to introspect for timer and state specs.

apache_beam.transforms.userstate.is_stateful_dofn(dofn: DoFn) → bool[source]¶: Determines whether a given DoFn is a stateful DoFn.

apache_beam.transforms.userstate.validate_stateful_dofn(dofn: DoFn) → None[source]¶: Validates the proper specification of a stateful DoFn.

class apache_beam.transforms.userstate.BaseTimer[source]¶

Bases: object

clear(dynamic_timer_tag: str = '') → None[source]¶

set(timestamp: apache_beam.utils.timestamp.Timestamp, dynamic_timer_tag: str = '') → None[source]¶

class apache_beam.transforms.userstate.RuntimeTimer[source]¶

Bases: apache_beam.transforms.userstate.BaseTimer

Timer interface object passed to user code.

clear(dynamic_timer_tag: str = '') → None[source]¶

set(timestamp: apache_beam.utils.timestamp.Timestamp, dynamic_timer_tag: str = '') → None[source]¶

class apache_beam.transforms.userstate.RuntimeState[source]¶

Bases: object

State interface object passed to user code.

prefetch() → None[source]¶

finalize() → None[source]¶

class apache_beam.transforms.userstate.ReadModifyWriteRuntimeState[source]¶

Bases: apache_beam.transforms.userstate.RuntimeState

read() → Any[source]¶

write(value: Any) → None[source]¶

clear() → None[source]¶

commit() → None[source]¶

class apache_beam.transforms.userstate.AccumulatingRuntimeState[source]¶

Bases: apache_beam.transforms.userstate.RuntimeState

read() → Iterable[Any][source]¶

add(value: Any) → None[source]¶

clear() → None[source]¶

commit() → None[source]¶

class apache_beam.transforms.userstate.BagRuntimeState[source]¶

Bases: apache_beam.transforms.userstate.AccumulatingRuntimeState

Bag state interface object passed to user code.

class apache_beam.transforms.userstate.SetRuntimeState[source]¶

Bases: apache_beam.transforms.userstate.AccumulatingRuntimeState

Set state interface object passed to user code.

class apache_beam.transforms.userstate.CombiningValueRuntimeState[source]¶

Bases: apache_beam.transforms.userstate.AccumulatingRuntimeState

Combining value state interface object passed to user code.

class apache_beam.transforms.userstate.UserStateContext[source]¶

Bases: object

Wrapper allowing user state and timers to be accessed by a DoFnInvoker.

get_timer(timer_spec: apache_beam.transforms.userstate.TimerSpec, key: Any, window: windowed_value.BoundedWindow, timestamp: apache_beam.utils.timestamp.Timestamp, pane: apache_beam.utils.windowed_value.PaneInfo) → apache_beam.transforms.userstate.BaseTimer[source]¶

get_state(state_spec: apache_beam.transforms.userstate.StateSpec, key: Any, window: windowed_value.BoundedWindow) → apache_beam.transforms.userstate.RuntimeState[source]¶

commit() → None[source]¶