Go SDK Roadmap
The Go SDK is currently experimental. As the first purely portable Beam SDK, the Go SDK is constrained by the status of the Beam Portabillity Framework and the existence of portable runners.
With that in mind present work on the Go SDK has the following goals:
The Go SDK should not block users from writing the pipelines they need to run.
Since the Go SDK has no official runner support at this time, this gives us the chance to work on the performance of the SDK harness. When portable runners are ready, the Go SDK should be ready as well.
- Reducing framework overhead on the SDK Harness side BEAM-4726
- Tooling to enable and verify pipeline performance BEAM-3612
- Measure the performance of the SDK at scale.
Integrating with the Go ecosystem
The authors of the Go SDK intended to keep the parts of the language we love, and intend for user pipelines to be as close to ordinary Go as possible, minimizing framework boiler plate code.
Implement Portability features
The following are dependant on the features being implemented in the Portability Framework.
- Splitable DoFns (SDFs) for Scalable IO BEAM-3301
- This is the primary blocker for the Go SDK to scale to ingest large numbers of elements in both batch and streaming.
- Triggers and Timers BEAM-3304
- Advanced WindowFns
Without SDFs, IOs are always constrained to work on a single machine prior to a sharding operation (like CoGroupByKey), which makes scalable IOs difficult to impossible to write.
Otherwise, improving examples and documentation for devs and users alike is ongoing. Contributions are welcome. Please contact the dev list for assistance in finding a place to help out.