Go SDK Roadmap

The Go SDK is currently experimental. As the first purely portable Beam SDK, the Go SDK is constrained by the status of the Beam Portability Framework and the existence of portable runners.

April 2020 Update This year we hope to move the SDK out of experimental at least for Batch usage.

To do so, there are a few blocking changes:

Interoperability while not a blocking feature in itself, requires Beam Schema support, which the SDK could then use as it’s default coder. Batch Splittable DoFns are nearly ready for playing with on Flink, and the Python Direct Runner, answering the scalable batch IO question. There’s much work getting the portability protos to a stable baseline. This will allow runner and SDK independance, so they don’t need to be updated in lockstep. Finally, the Go SDK should adopt Go Modules as it’s versioning solution, and officially “catch up” with the current beam version.

The Go SDK has the following goals for the next few months:

Usability

Integrating with the Go ecosystem

The authors of the Go SDK intended to keep the parts of the language we love, and intend for user pipelines to be as close to ordinary Go as possible, minimizing framework boiler plate code.

Implement Portability features

The following are dependant on the features being implemented in the Portability Framework.

Without SDFs, IOs are always constrained to work on a single machine prior to a sharding operation (like CoGroupByKey), which makes scalable IOs difficult to impossible to write.

Otherwise, improving examples and documentation for devs and users alike is ongoing. Contributions are welcome. Please contact the dev list for assistance in finding a place to help out.