Go SDK Roadmap
The Go SDK is currently experimental. As the first purely portable Beam SDK, the Go SDK is constrained by the status of the Beam Portability Framework and the existence of portable runners.
April 2020 Update This year we hope to move the SDK out of experimental at least for Batch usage.
To do so, there are a few blocking changes:
- Support Interoperability with other SDKs
- Scalable IOs
- Portability Proto stability
- Go Ecosystem integration improvements
Interoperability while not a blocking feature in itself, requires Beam Schema support, which the SDK could then use as it’s default coder. Batch Splittable DoFns are nearly ready for playing with on Flink, and the Python Direct Runner, answering the scalable batch IO question. There’s much work getting the portability protos to a stable baseline. This will allow runner and SDK independance, so they don’t need to be updated in lockstep. Finally, the Go SDK should adopt Go Modules as it’s versioning solution, and officially “catch up” with the current beam version.
The Go SDK has the following goals for the next few months:
Integrating with the Go ecosystem
The authors of the Go SDK intended to keep the parts of the language we love, and intend for user pipelines to be as close to ordinary Go as possible, minimizing framework boiler plate code.
Implement Portability features
The following are dependant on the features being implemented in the Portability Framework.
- Splitable DoFns (SDFs) for Scalable IO BEAM-3301
- This is the primary blocker for the Go SDK to scale to ingest large numbers of elements in both batch and streaming.
- Triggers and Timers BEAM-3304
- Advanced WindowFns
Without SDFs, IOs are always constrained to work on a single machine prior to a sharding operation (like CoGroupByKey), which makes scalable IOs difficult to impossible to write.
Otherwise, improving examples and documentation for devs and users alike is ongoing. Contributions are welcome. Please contact the dev list for assistance in finding a place to help out.
- JIRA: sdk-go
- Contact: Robert Burke (@lostluck) Email - Please also cc the email@example.com list. I strongly prefer public discussion of Go SDK matters.
Last updated on 2020/10/26
Have you found everything you were looking for?
Was it all useful and clear? Is there anything that you would like to change? Let us know!