apache_beam.ml.inference.openai_inference module
A ModelHandler for OpenAI models using the OpenAI Python SDK.
This module provides an integration between Apache Beam’s RunInference transform and OpenAI’s API, enabling batch inference and embeddings in Beam pipelines.
Example usage:
import apache_beam as beam
from apache_beam.ml.inference.base import RunInference
from apache_beam.ml.inference.openai_inference import (
OpenAIModelHandler,
chat_completion_from_string,
)
# Basic text generation with chat completions
model_handler = OpenAIModelHandler(
model_name='gpt-4o-mini',
api_key='your-api-key',
request_fn=chat_completion_from_string,
)
# With system prompt and structured output passed via inference_args
inference_args = {
'system': 'You are a helpful assistant that responds concisely.',
'response_format': {
'type': 'json_schema',
'json_schema': {
'name': 'answer_response',
'schema': {
'type': 'object',
'properties': {
'answer': {'type': 'string'},
'confidence': {'type': 'number'},
},
'required': ['answer', 'confidence'],
'additionalProperties': False,
},
'strict': True,
},
},
}
with beam.Pipeline() as p:
results = (
p
| beam.Create(['What is Apache Beam?', 'Explain MapReduce.'])
| RunInference(model_handler, inference_args=inference_args)
)
- class apache_beam.ml.inference.openai_inference.OpenAIModelHandler(model_name: str, request_fn: Callable[[str, Sequence[Any], OpenAI, dict[str, Any]], Any], api_key: str | None = None, *, organization: str | None = None, project: str | None = None, base_url: str | None = None, client_args: dict[str, Any] | None = None, min_batch_size: int | None = None, max_batch_size: int | None = None, max_batch_duration_secs: int | None = None, max_batch_weight: int | None = None, element_size_fn: Callable[[Any], int] | None = None, batch_length_fn: Callable[[Any], int] | None = None, batch_bucket_boundaries: list[int] | None = None, **kwargs)[source]
Bases:
RemoteModelHandler[Any,PredictionResult,OpenAI]Implementation of the ModelHandler interface for OpenAI models.
NOTE: This API and its implementation are under development and do not provide backward compatibility guarantees.
This handler connects to the OpenAI API using the OpenAI Python SDK to run inference using models such as GPT-4o, GPT-4o-mini, or embedding models. It supports chat completions from string prompts or multi-turn conversations (with each batch’s per-item requests dispatched concurrently), natively batched embeddings, system prompts, structured outputs, and custom OpenAI-compatible endpoints via base_url.
A single thread-safe
OpenAIclient is created per worker process and shared across bundle-processing threads, so pooled HTTP connections are reused across batches.- Parameters:
model_name – the OpenAI model to send requests to (e.g. ‘gpt-4o’, ‘gpt-4o-mini’, ‘text-embedding-3-small’).
request_fn – the function to use to send requests. Should take the model name, batch, client, and inference_args and return the responses from OpenAI. Built-in options are chat_completion_from_string, chat_completion_from_conversation, and embedding_from_string.
api_key – the OpenAI API key. If not provided, the client will look for the OPENAI_API_KEY environment variable.
organization – optional OpenAI organization ID. If not provided, the client will look for the OPENAI_ORG_ID environment variable.
project – optional OpenAI project ID. If not provided, the client will look for the OPENAI_PROJECT_ID environment variable.
base_url – optional base URL for the API requests. This can be used to target OpenAI-compatible servers (such as vLLM, Ollama, or Azure OpenAI endpoints).
client_args – optional dictionary of additional keyword arguments passed when instantiating the OpenAI client (e.g. timeout, default_headers).
min_batch_size – optional. the minimum batch size to use when batching inputs.
max_batch_size – optional. the maximum batch size to use when batching inputs.
max_batch_duration_secs – optional. the maximum amount of time to buffer a batch before emitting; used in streaming contexts.
max_batch_weight – the maximum total weight of a batch. Cannot be combined with batch_length_fn or batch_bucket_boundaries.
element_size_fn – a function that returns the size (weight) of an element.
batch_length_fn – a callable that returns the length of an element for length-aware batching. Cannot be used together with max_batch_weight.
batch_bucket_boundaries – a sorted list of positive boundary values for length-aware batching buckets. Cannot be used together with max_batch_weight.
**kwargs – optional arguments passed to RemoteModelHandler (e.g. num_retries, throttle_delay_secs, rate_limiter).
- create_client() OpenAI[source]
Creates the OpenAI client used to send requests.
Sets
max_retries=0by default so that Apache Beam’sRemoteModelHandlerretry and client-side throttling mechanics are the sole retry layer. If api_key, organization, project, base_url, or client_args were provided at construction time, they are passed to the OpenAI constructor. Otherwise, the client falls back to standard environment variables (OPENAI_API_KEY, OPENAI_ORG_ID, etc.).
- request(batch: Sequence[Any], model: OpenAI, inference_args: dict[str, Any] | None = None) Iterable[PredictionResult][source]
Sends a prediction request to the OpenAI API.
- Parameters:
batch – a sequence of inputs to be passed to the request function.
model – an OpenAI client instance.
inference_args – additional arguments to send as part of the prediction request (e.g. temperature, max_tokens, system, response_format).
- Returns:
An iterable of PredictionResults.
- apache_beam.ml.inference.openai_inference.chat_completion_from_string(model_name: str, batch: Sequence[str], client: OpenAI, inference_args: dict[str, Any]) list[Any][source]
Request function that sends string prompts to OpenAI’s Chat Completions API.
The Chat Completions endpoint accepts a single conversation per HTTP request, so each string in the batch is sent as a separate user message request, dispatched concurrently. If a ‘system’ parameter is provided in inference_args, a system message is prepended. The results are returned as a list of ChatCompletion response objects matching the batch order.
- Parameters:
model_name – the OpenAI model to use (e.g. ‘gpt-4o’, ‘gpt-4o-mini’).
batch – the string prompts to send to OpenAI.
client – the OpenAI client instance.
inference_args – additional arguments passed to the chat.completions.create call (e.g. ‘temperature’, ‘max_tokens’, ‘response_format’, ‘system’).
- apache_beam.ml.inference.openai_inference.chat_completion_from_conversation(model_name: str, batch: Sequence[list[dict[str, Any]]], client: OpenAI, inference_args: dict[str, Any]) list[Any][source]
Request function that sends multi-turn conversations to OpenAI.
Each element in the batch is a list of message dicts (e.g. with ‘role’ and ‘content’ keys), representing a multi-turn conversation. Conversations are sent as separate requests, dispatched concurrently. If a ‘system’ parameter is provided in inference_args, a system message is prepended.
- Parameters:
model_name – the OpenAI model to use.
batch – a sequence of conversations (each a list of message dicts).
client – the OpenAI client instance.
inference_args – additional arguments passed to the chat.completions.create call.
- apache_beam.ml.inference.openai_inference.embedding_from_string(model_name: str, batch: Sequence[str], client: OpenAI, inference_args: dict[str, Any]) list[Any][source]
Request function that sends string inputs to OpenAI’s Embeddings API.
The Embeddings endpoint natively accepts a list of inputs, so the entire batch is sent in a single request to
client.embeddings.create. The returned embeddings are sorted by their index to guarantee ordering matches the batch.- Parameters:
model_name – the OpenAI embedding model to use (e.g. ‘text-embedding-3-small’).
batch – the string inputs to embed.
client – the OpenAI client instance.
inference_args – additional arguments passed to the embeddings.create call (e.g. ‘dimensions’, ‘encoding_format’).
- Returns:
A list of Embedding objects matching the batch order.