@Experimental(value=SCHEMAS) public class Group extends java.lang.Object
PCollections.
When used without a combiner, this transforms simply acts as a GroupByKey but without
the need for the user to explicitly extract the keys. For example, consider the following input
type:
@DefaultSchema(JavaFieldSchema.class)
public class UserPurchase {
public String userId;
public String country;
public long cost;
public double transactionDuration;
}
PCollection<UserPurchase> purchases = readUserPurchases();
You can group all purchases by user and country as follows:
@DefaultSchema(JavaFieldSchema.class)
PCollection<Row> byUser = purchases.apply(Group.byFieldNames("userId', "country"));
However often an aggregation of some form is desired. The builder methods inside the Group class allows building up separate aggregations for every field (or set of fields) on the input schema, and generating an output schema based on these aggregations. For example:
PCollection<Row> aggregated = purchases
.apply(Group.byFieldNames("userId', "country")
.aggregateField("cost", Sum.ofLongs(), "total_cost")
.aggregateField("cost", Top.<Long>largestLongsFn(10), "top_purchases")
.aggregateField("cost", ApproximateQuantilesCombineFn.create(21),
Field.of("transactionDurations", FieldType.array(FieldType.INT64)));
The result will be a new row schema containing the fields total_cost, top_purchases, and transactionDurations, containing the sum of all purchases costs (for that user and country), the top ten purchases, and a histogram of transaction durations. The schema will also contain a key field, which will be a row containing userId and country.
Note that usually the field type can be automatically inferred from the Combine.CombineFn
passed in. However sometimes it cannot be inferred, due to Java type erasure, in which case a
Schema.Field object containing the field type must be passed in. This is currently the case for
ApproximateQuantilesCombineFn in the above example.
| Modifier and Type | Class and Description |
|---|---|
static class |
Group.AggregateCombiner<InputT>
a
PTransform that does a combine using an aggregation built up by calls to
aggregateField and aggregateFields. |
static class |
Group.ByFields<InputT>
a
PTransform that groups schema elements based on the given fields. |
static class |
Group.CombineFieldsByFields<InputT>
a
PTransform that does a per-key combine using an aggregation built up by calls to
aggregateField and aggregateFields. |
static class |
Group.CombineFieldsGlobally<InputT>
a
PTransform that does a global combine using an aggregation built up by calls to
aggregateField and aggregateFields. |
static class |
Group.CombineGlobally<InputT,OutputT>
a
PTransform that does a global combine using a provider Combine.CombineFn. |
static class |
Group.Global<InputT>
A
PTransform for doing global aggregations on schema PCollections. |
| Constructor and Description |
|---|
Group() |
| Modifier and Type | Method and Description |
|---|---|
static <T> Group.ByFields<T> |
byFieldAccessDescriptor(FieldAccessDescriptor fieldAccess)
Returns a transform that groups all elements in the input
PCollection keyed by the
fields specified. |
static <T> Group.ByFields<T> |
byFieldIds(java.lang.Integer... fieldIds)
Returns a transform that groups all elements in the input
PCollection keyed by the list
of fields specified. |
static <T> Group.ByFields<T> |
byFieldIds(java.lang.Iterable<java.lang.Integer> fieldIds)
Same as
byFieldIds(Integer...). |
static <T> Group.ByFields<T> |
byFieldNames(java.lang.Iterable<java.lang.String> fieldNames)
Same as
byFieldNames(String...). |
static <T> Group.ByFields<T> |
byFieldNames(java.lang.String... fieldNames)
Returns a transform that groups all elements in the input
PCollection keyed by the list
of fields specified. |
static <T> Group.Global<T> |
globally()
Returns a transform that groups all elements in the input
PCollection. |
public static <T> Group.Global<T> globally()
PCollection. The returned
transform contains further builder methods to control how the grouping is done.public static <T> Group.ByFields<T> byFieldNames(java.lang.String... fieldNames)
PCollection keyed by the list
of fields specified. The output of this transform will be a KV keyed by a Row
containing the specified extracted fields. The returned transform contains further builder
methods to control how the grouping is done.public static <T> Group.ByFields<T> byFieldNames(java.lang.Iterable<java.lang.String> fieldNames)
byFieldNames(String...).public static <T> Group.ByFields<T> byFieldIds(java.lang.Integer... fieldIds)
PCollection keyed by the list
of fields specified. The output of this transform will have a key field of type Row
containing the specified extracted fields. It will also have a value field of type Row
containing the specified extracted fields. The returned transform contains further builder
methods to control how the grouping is done.public static <T> Group.ByFields<T> byFieldIds(java.lang.Iterable<java.lang.Integer> fieldIds)
byFieldIds(Integer...).public static <T> Group.ByFields<T> byFieldAccessDescriptor(FieldAccessDescriptor fieldAccess)
PCollection keyed by the
fields specified. The output of this transform will have a key field of type Row
containing the specified extracted fields. It will also have a value field of type Row
containing the specified extracted fields. The returned transform contains further builder
methods to control how the grouping is done.