Package org.apache.beam.sdk.ml.inference.remote

Interface BaseModelHandler<ParamT extends BaseModelParameters,InputT extends BaseInput,OutputT extends BaseResponse>

All Known Implementing Classes:
OpenAIModelHandler
public interface BaseModelHandler<ParamT extends BaseModelParameters,InputT extends BaseInput,OutputT extends BaseResponse>
Interface for model-specific handlers that perform remote inference operations.

Implementations of this interface encapsulate all logic for communicating with a specific remote inference service. Each handler is responsible for:

  • Initializing and managing client connections
  • Converting Beam inputs to service-specific request formats
  • Making inference API calls
  • Converting service responses to Beam output types
  • Handling errors and retries if applicable

Lifecycle

Handler instances follow this lifecycle:

  1. Instantiation via no-argument constructor
  2. createClient(ParamT) called with parameters during setup
  3. request(java.util.List<InputT>) called for each batch of inputs

Handlers typically contain non-serializable client objects. Mark client fields as transient and initialize them in createClient(ParamT)

Batching Considerations

The request(java.util.List<InputT>) method receives a list of inputs. Implementations should:

  • Batch inputs efficiently if the service supports batch inference
  • Return results in the same order as inputs
  • Maintain input-output correspondence in PredictionResult

  • Method Details

    • createClient

      void createClient(ParamT parameters)
      Initializes the remote model client with the provided parameters.

    • request

      Iterable<PredictionResult<InputT,OutputT>> request(List<InputT> input)
      Performs inference on a batch of inputs and returns the results.