org.apache.beam.sdk.extensions.ml

Class DLPReidentifyText

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<KV<java.lang.String,java.lang.String>>,PCollection<KV<java.lang.String,com.google.privacy.dlp.v2.ReidentifyContentResponse>>>

org.apache.beam.sdk.extensions.ml.DLPReidentifyText

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

@Experimental
public abstract class DLPReidentifyText
extends PTransform<PCollection<KV<java.lang.String,java.lang.String>>,PCollection<KV<java.lang.String,com.google.privacy.dlp.v2.ReidentifyContentResponse>>>

A PTransform connecting to Cloud DLP (https://cloud.google.com/dlp/docs/libraries) and inspecting text for identifying data according to provided settings.

The transform supports both delimited columnar input data and unstructured input.

If the headerColumns property is set and a sideinput with headers is added to the PTransform, delimiter also should be set, else the results will be incorrect. If headerColumns is neither set nor passed as sideinput, input is assumed to be unstructured.

Batch size defines how big are batches sent to DLP at once in bytes.

The transform consumes KV of Strings (assumed to be filename as key and contents as value) and outputs KV of String (eg. filename) and ReidentifyContentResponse, which will contain Table of results for the user to consume.

Batch size defines how big are batches sent to DLP at once in bytes.

Either reidentifyTemplateName String or reidentifyConfig DeidentifyConfig need to be set. inspectConfig InspectConfig and inspectTemplateName String are optional.

Batch size defines how big are batches sent to DLP at once in bytes.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DLPReidentifyText.Builder

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.Integer	DLP_PAYLOAD_LIMIT_BYTES

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name

Constructor Summary

Constructors

Constructor and Description
DLPReidentifyText()

Method Summary

Modifier and Type	Method and Description
PCollection<KV<java.lang.String,com.google.privacy.dlp.v2.ReidentifyContentResponse>>	expand(PCollection<KV<java.lang.String,java.lang.String>> input) The transform converts the contents of input PCollection into Table.Rows and then calls Cloud DLP service to perform the reidentification according to provided settings.
abstract java.lang.Integer	getBatchSizeBytes()
abstract @Nullable java.lang.String	getColumnDelimiter()
abstract @Nullable PCollectionView<java.util.List<java.lang.String>>	getHeaderColumns()
abstract @Nullable com.google.privacy.dlp.v2.InspectConfig	getInspectConfig()
abstract @Nullable java.lang.String	getInspectTemplateName()
abstract java.lang.String	getProjectId()
abstract @Nullable com.google.privacy.dlp.v2.DeidentifyConfig	getReidentifyConfig()
abstract @Nullable java.lang.String	getReidentifyTemplateName()
static DLPReidentifyText.Builder	newBuilder()

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, populateDisplayData, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DLP_PAYLOAD_LIMIT_BYTES

public static final java.lang.Integer DLP_PAYLOAD_LIMIT_BYTES

Constructor Detail

DLPReidentifyText

public DLPReidentifyText()

Method Detail

getInspectTemplateName

public abstract @Nullable java.lang.String getInspectTemplateName()

Returns:

Template name for data inspection.

getReidentifyTemplateName

public abstract @Nullable java.lang.String getReidentifyTemplateName()

Returns:

Template name for data reidentification.

getInspectConfig

public abstract @Nullable com.google.privacy.dlp.v2.InspectConfig getInspectConfig()

Returns:

Configuration object for data inspection. If present, supersedes the template settings.

getReidentifyConfig

public abstract @Nullable com.google.privacy.dlp.v2.DeidentifyConfig getReidentifyConfig()

Returns:

Configuration object for reidentification. If present, supersedes the template.

getColumnDelimiter

public abstract @Nullable java.lang.String getColumnDelimiter()

Returns:

Delimiter to be used when splitting values from input strings into columns.

getHeaderColumns

public abstract @Nullable PCollectionView<java.util.List<java.lang.String>> getHeaderColumns()

Returns:

List of column names if the input KV value is a delimited row.

getBatchSizeBytes

public abstract java.lang.Integer getBatchSizeBytes()

Returns:

Size of input elements batch to be sent to Cloud DLP service in one request.

getProjectId

public abstract java.lang.String getProjectId()

Returns:

ID of Google Cloud project to be used when deidentifying data.

newBuilder

public static DLPReidentifyText.Builder newBuilder()

expand

public PCollection<KV<java.lang.String,com.google.privacy.dlp.v2.ReidentifyContentResponse>> expand(PCollection<KV<java.lang.String,java.lang.String>> input)

The transform converts the contents of input PCollection into Table.Rows and then calls Cloud DLP service to perform the reidentification according to provided settings.

Specified by:

expand in class PTransform<PCollection<KV<java.lang.String,java.lang.String>>,PCollection<KV<java.lang.String,com.google.privacy.dlp.v2.ReidentifyContentResponse>>>

Parameters:

input - input PCollection

Returns:

PCollection after transformations