apache_beam.dataframe.frames module¶

Analogs for pandas.DataFrame and pandas.Series: DeferredDataFrame and DeferredSeries.

These classes are effectively wrappers around a schema-aware PCollection that provide a set of operations compatible with the pandas API.

Note that we aim for the Beam DataFrame API to be completely compatible with the pandas API, but there are some features that are currently unimplemented for various reasons. Pay particular attention to the ‘Differences from pandas’ section for each operation to understand where we diverge.

class apache_beam.dataframe.frames.DeferredSeries(expr)[source]¶

Bases: apache_beam.dataframe.frames.DeferredDataFrameOrSeries

name¶

Return the name of the Series.

The name of a Series becomes its index or column name if it is used to form a DataFrame. It is also used whenever displaying the Series using the interpreter.

Returns:	The name of the DeferredSeries, also the column name if part of a DeferredDataFrame.
Return type:	label (hashable object)

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.rename: Sets the DeferredSeries name when given a scalar input.
Index.name: Corresponding Index property.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

The Series name can be set initially when calling the constructor.

>>> s = pd.Series([1, 2, 3], dtype=np.int64, name='Numbers')
>>> s
0    1
1    2
2    3
Name: Numbers, dtype: int64
>>> s.name = "Integers"
>>> s
0    1
1    2
2    3
Name: Integers, dtype: int64

The name of a Series within a DataFrame is its column name.

>>> df = pd.DataFrame([[1, 2], [3, 4], [5, 6]],
...                   columns=["Odd Numbers", "Even Numbers"])
>>> df
   Odd Numbers  Even Numbers
0            1             2
1            3             4
2            5             6
>>> df["Even Numbers"].name
'Even Numbers'

dtype¶

Return the dtype object of the underlying data.

Differences from pandas

This operation has no known divergences from the pandas API.

dtypes¶

Return the dtype object of the underlying data.

Differences from pandas

This operation has no known divergences from the pandas API.

keys()[source]¶

Return alias for index.

Returns:	Index of the DeferredSeries.
Return type:	Index

Differences from pandas

This operation has no known divergences from the pandas API.

T(**kwargs)¶

Return the transpose, which is by definition self.

Differences from pandas

This operation has no known divergences from the pandas API.

append(to_append, ignore_index, verify_integrity, **kwargs)[source]¶

Concatenate two or more Series.

Parameters:	to_append (DeferredSeries or list/tuple of DeferredSeries) – DeferredSeries to append with self. ignore_index (bool, default False) – If True, the resulting axis will be labeled 0, 1, …, n - 1. verify_integrity (bool, default False) – If True, raise Exception on creating index with duplicates.
Returns:	Concatenated DeferredSeries.
Return type:	DeferredSeries

Differences from pandas

ignore_index=True is not supported, because it requires generating an order-sensitive index.

See also

concat(): General function to concatenate DeferredDataFrame or DeferredSeries objects.

Notes

Iteratively appending to a DeferredSeries can be more computationally intensive than a single concatenate. A better solution is to append values to a list and then concatenate the list with the original DeferredSeries all at once.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> s1 = pd.Series([1, 2, 3])
>>> s2 = pd.Series([4, 5, 6])
>>> s3 = pd.Series([4, 5, 6], index=[3, 4, 5])
>>> s1.append(s2)
0    1
1    2
2    3
0    4
1    5
2    6
dtype: int64

>>> s1.append(s3)
0    1
1    2
2    3
3    4
4    5
5    6
dtype: int64

With `ignore_index` set to True:

>>> s1.append(s2, ignore_index=True)
0    1
1    2
2    3
3    4
4    5
5    6
dtype: int64

With `verify_integrity` set to True:

>>> s1.append(s2, verify_integrity=True)
Traceback (most recent call last):
...
ValueError: Indexes have overlapping values: [0, 1, 2]

align(other, join, axis, level, method, **kwargs)[source]¶

Align two objects on their axes with the specified join method.

Join method is specified for each axis Index.

Parameters:	other (DeferredDataFrame or DeferredSeries) – join ({'outer', 'inner', 'left', 'right'}, default 'outer') – axis (allowed axis of the other object, default None) – Align on index (0), columns (1), or both (None). level (int or level name, default None) – Broadcast across a level, matching Index values on the passed MultiIndex level. copy (bool, default True) – Always returns new objects. If copy=False and no reindexing is required then original objects are returned. fill_value (scalar, default np.NaN) – Value to use for missing values. Defaults to NaN, but can be any “compatible” value. method ({'backfill', 'bfill', 'pad', 'ffill', None}, default None) – Method to use for filling holes in reindexed DeferredSeries: pad / ffill: propagate last valid observation forward to next valid. backfill / bfill: use NEXT valid observation to fill gap. limit (int, default None) – If method is specified, this is the maximum number of consecutive NaN values to forward/backward fill. In other words, if there is a gap with more than this number of consecutive NaNs, it will only be partially filled. If method is not specified, this is the maximum number of entries along the entire axis where NaNs will be filled. Must be greater than 0 if not None. fill_axis ({0 or 'index'}, default 0) – Filling axis, method and limit. broadcast_axis ({0 or 'index'}, default None) – Broadcast values along this axis, if aligning two objects of different dimensions.
Returns:	(left, right) – Aligned objects.
Return type:	(DeferredSeries, type of other)

Differences from pandas

Aligning per-level is not yet supported. Only the default, level=None, is allowed.

Filling NaN values via method is not supported, because it is sensitive to the order of the data (see https://s.apache.org/dataframe-order-sensitive-operations). Only the default, method=None, is allowed.

array¶

pandas.Series.array is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

ravel(**kwargs)¶

pandas.Series.ravel is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

rename(**kwargs)¶

Alter Series index labels or name.

Function / dict values must be unique (1-to-1). Labels not contained in a dict / Series will be left as-is. Extra labels listed don’t throw an error.

Alternatively, change Series.name with a scalar value.

See the user guide for more.

Parameters:	axis ({0 or "index"}) – Unused. Accepted for compatibility with DeferredDataFrame method only. index (scalar, hashable sequence, dict-like or function, optional) – Functions or dict-like are transformations to apply to the index. Scalar or hashable sequence-like will alter the `DeferredSeries.name` attribute. **kwargs – Additional keyword arguments passed to the function. Only the “inplace” keyword is used.
Returns:	DeferredSeries with index labels or name altered or None if `inplace=True`.
Return type:	DeferredSeries or None

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.rename(): Corresponding DeferredDataFrame method.
DeferredSeries.rename_axis(): Set the name of the axis.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series([1, 2, 3])
>>> s
0    1
1    2
2    3
dtype: int64
>>> s.rename("my_name")  # scalar, changes Series.name
0    1
1    2
2    3
Name: my_name, dtype: int64
>>> s.rename(lambda x: x ** 2)  # function, changes labels
0    1
1    2
4    3
dtype: int64
>>> s.rename({1: 3, 2: 5})  # mapping, changes labels
0    1
3    2
5    3
dtype: int64

between(**kwargs)¶

Return boolean Series equivalent to left <= series <= right.

This function returns a boolean vector containing True wherever the corresponding Series element is between the boundary values left and right. NA values are treated as False.

Parameters:	left (scalar or list-like) – Left boundary. right (scalar or list-like) – Right boundary. inclusive (bool, default True) – Include boundaries.
Returns:	DeferredSeries representing whether each element is between left and right (inclusive).
Return type:	DeferredSeries

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.gt(): Greater than of series and other.
DeferredSeries.lt(): Less than of series and other.

Notes

This function is equivalent to (left <= ser) & (ser <= right)

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series([2, 0, 4, 8, np.nan])

Boundary values are included by default:

>>> s.between(1, 4)
0     True
1    False
2     True
3    False
4    False
dtype: bool

With `inclusive` set to ``False`` boundary values are excluded:

>>> s.between(1, 4, inclusive=False)
0     True
1    False
2    False
3    False
4    False
dtype: bool

`left` and `right` can be any scalar value:

>>> s = pd.Series(['Alice', 'Bob', 'Carol', 'Eve'])
>>> s.between('Anna', 'Daniel')
0    False
1     True
2     True
3    False
dtype: bool

add_suffix(**kwargs)¶

Suffix labels with string suffix.

For Series, the row labels are suffixed. For DataFrame, the column labels are suffixed.

Parameters:	suffix (str) – The string to add after each label.
Returns:	New DeferredSeries or DeferredDataFrame with updated labels.
Return type:	DeferredSeries or DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.add_prefix(): Prefix row labels with string prefix.
DeferredDataFrame.add_prefix(): Prefix column labels with string prefix.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series([1, 2, 3, 4])
>>> s
0    1
1    2
2    3
3    4
dtype: int64

>>> s.add_suffix('_item')
0_item    1
1_item    2
2_item    3
3_item    4
dtype: int64

>>> df = pd.DataFrame({'A': [1, 2, 3, 4], 'B': [3, 4, 5, 6]})
>>> df
   A  B
0  1  3
1  2  4
2  3  5
3  4  6

>>> df.add_suffix('_col')
     A_col  B_col
0       1       3
1       2       4
2       3       5
3       4       6

add_prefix(**kwargs)¶

Prefix labels with string prefix.

For Series, the row labels are prefixed. For DataFrame, the column labels are prefixed.

Parameters:	prefix (str) – The string to add before each label.
Returns:	New DeferredSeries or DeferredDataFrame with updated labels.
Return type:	DeferredSeries or DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.add_suffix(): Suffix row labels with string suffix.
DeferredDataFrame.add_suffix(): Suffix column labels with string suffix.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series([1, 2, 3, 4])
>>> s
0    1
1    2
2    3
3    4
dtype: int64

>>> s.add_prefix('item_')
item_0    1
item_1    2
item_2    3
item_3    4
dtype: int64

>>> df = pd.DataFrame({'A': [1, 2, 3, 4], 'B': [3, 4, 5, 6]})
>>> df
   A  B
0  1  3
1  2  4
2  3  5
3  4  6

>>> df.add_prefix('col_')
     col_A  col_B
0       1       3
1       2       4
2       3       5
3       4       6

dot(other)[source]¶

Compute the matrix multiplication between the DataFrame and other.

This method computes the matrix product between the DataFrame and the values of an other Series, DataFrame or a numpy array.

It can also be called using self @ other in Python >= 3.5.

Parameters:	other (DeferredSeries, DeferredDataFrame or array-like) – The other object to compute the matrix product with.
Returns:	If other is a DeferredSeries, return the matrix product between self and other as a DeferredSeries. If other is a DeferredDataFrame or a numpy.array, return the matrix product of self and other in a DeferredDataFrame of a np.array.
Return type:	DeferredSeries or DeferredDataFrame

Differences from pandas

other must be a DeferredDataFrame or DeferredSeries instance. Computing the dot product with an array-like is not supported because it is order-sensitive.

See also

DeferredSeries.dot(): Similar method for DeferredSeries.

Notes

The dimensions of DeferredDataFrame and other must be compatible in order to compute the matrix multiplication. In addition, the column names of DeferredDataFrame and the index of other must contain the same values, as they will be aligned prior to the multiplication.

The dot method for DeferredSeries computes the inner product, instead of the matrix product here.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

Here we multiply a DataFrame with a Series.

>>> df = pd.DataFrame([[0, 1, -2, -1], [1, 1, 1, 1]])
>>> s = pd.Series([1, 1, 2, 1])
>>> df.dot(s)
0    -4
1     5
dtype: int64

Here we multiply a DataFrame with another DataFrame.

>>> other = pd.DataFrame([[0, 1], [1, 2], [-1, -1], [2, 0]])
>>> df.dot(other)
    0   1
0   1   4
1   2   2

Note that the dot method give the same result as @

>>> df @ other
    0   1
0   1   4
1   2   2

The dot method works also if other is an np.array.

>>> arr = np.array([[0, 1], [1, 2], [-1, -1], [2, 0]])
>>> df.dot(arr)
    0   1
0   1   4
1   2   2

Note how shuffling of the objects does not change the result.

>>> s2 = s.reindex([1, 0, 2, 3])
>>> df.dot(s2)
0    -4
1     5
dtype: int64

std(*args, **kwargs)[source]¶

Return sample standard deviation over requested axis.

Normalized by N-1 by default. This can be changed using the ddof argument

Parameters:	axis ({index (0)}) – skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar. ddof (int, default 1) – Delta Degrees of Freedom. The divisor used in calculations is N - ddof, where N represents the number of elements. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries.
Returns:
Return type:	scalar or DeferredSeries (if level specified)

Differences from pandas

This operation has no known divergences from the pandas API.

Notes

To have the same behaviour as numpy.std, use ddof=0 (instead of the default ddof=1)

var(axis, skipna, level, ddof, **kwargs)[source]¶

Return unbiased variance over requested axis.

Normalized by N-1 by default. This can be changed using the ddof argument

Parameters:	axis ({index (0)}) – skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar. ddof (int, default 1) – Delta Degrees of Freedom. The divisor used in calculations is N - ddof, where N represents the number of elements. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries.
Returns:
Return type:	scalar or DeferredSeries (if level specified)

Differences from pandas

Per-level aggregation is not yet supported (BEAM-11777). Only the default, level=None, is allowed.

Notes

To have the same behaviour as numpy.std, use ddof=0 (instead of the default ddof=1)

corr(other, method, min_periods)[source]¶

Compute correlation with other Series, excluding missing values.

Parameters:	other (DeferredSeries) – DeferredSeries with which to compute the correlation. method ({'pearson', 'kendall', 'spearman'} or callable) – Method used to compute correlation: pearson : Standard correlation coefficient kendall : Kendall Tau correlation coefficient spearman : Spearman rank correlation callable: Callable with input two 1d ndarrays and returning a float. New in version 0.24.0: Note that the returned matrix from corr will have 1 along the diagonals and will be symmetric regardless of the callable’s behavior. min_periods (int, optional) – Minimum number of observations needed to have a valid result.
Returns:	Correlation with other.
Return type:	float

Differences from pandas

Only method='pearson' is currently parallelizable.

See also

DeferredDataFrame.corr(): Compute pairwise correlation between columns.
DeferredDataFrame.corrwith(): Compute pairwise correlation with another DeferredDataFrame or DeferredSeries.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> def histogram_intersection(a, b):
...     v = np.minimum(a, b).sum().round(decimals=1)
...     return v
>>> s1 = pd.Series([.2, .0, .6, .2])
>>> s2 = pd.Series([.3, .6, .0, .1])
>>> s1.corr(s2, method=histogram_intersection)
0.3

cov(other, min_periods, ddof)[source]¶

Compute covariance with Series, excluding missing values.

Parameters:	other (DeferredSeries) – DeferredSeries with which to compute the covariance. min_periods (int, optional) – Minimum number of observations needed to have a valid result. ddof (int, default 1) – Delta degrees of freedom. The divisor used in calculations is `N - ddof`, where `N` represents the number of elements. New in version 1.1.0.
Returns:	Covariance between DeferredSeries and other normalized by N-1 (unbiased estimator).
Return type:	float

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.cov(): Compute pairwise covariance of columns.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s1 = pd.Series([0.90010907, 0.13484424, 0.62036035])
>>> s2 = pd.Series([0.12528585, 0.26962463, 0.51111198])
>>> s1.cov(s2)
-0.01685762652715874

dropna(**kwargs)[source]¶

Return a new Series with missing values removed.

See the User Guide for more on which values are considered missing, and how to work with missing data.

Parameters:	axis ({0 or 'index'}, default 0) – There is only one axis to drop values from. inplace (bool, default False) – If True, do operation inplace and return None. how (str, optional) – Not in use. Kept for compatibility.
Returns:	DeferredSeries with NA entries dropped from it or None if `inplace=True`.
Return type:	DeferredSeries or None

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.isna(): Indicate missing values.
DeferredSeries.notna(): Indicate existing (non-missing) values.
DeferredSeries.fillna(): Replace missing values.
DeferredDataFrame.dropna(): Drop rows or columns which contain NA values.
Index.dropna(): Drop missing indices.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> ser = pd.Series([1., 2., np.nan])
>>> ser
0    1.0
1    2.0
2    NaN
dtype: float64

Drop NA values from a Series.

>>> ser.dropna()
0    1.0
1    2.0
dtype: float64

Keep the Series with valid entries in the same variable.

>>> ser.dropna(inplace=True)
>>> ser
0    1.0
1    2.0
dtype: float64

Empty strings are not considered NA values. ``None`` is considered an
NA value.

>>> ser = pd.Series([np.NaN, 2, pd.NaT, '', None, 'I stay'])
>>> ser
0       NaN
1         2
2       NaT
3
4      None
5    I stay
dtype: object
>>> ser.dropna()
1         2
3
5    I stay
dtype: object

isnull(**kwargs)¶

Detect missing values.

Return a boolean same-sized object indicating if the values are NA. NA values, such as None or numpy.NaN, gets mapped to True values. Everything else gets mapped to False values. Characters such as empty strings '' or numpy.inf are not considered NA values (unless you set pandas.options.mode.use_inf_as_na = True).

Returns:	Mask of bool values for each element in DeferredSeries that indicates whether an element is an NA value.
Return type:	DeferredSeries

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.isnull(): Alias of isna.
DeferredSeries.notna(): Boolean inverse of isna.
DeferredSeries.dropna(): Omit axes labels with missing values.
isna(): Top-level isna.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Show which entries in a DataFrame are NA.

>>> df = pd.DataFrame(dict(age=[5, 6, np.NaN],
...                    born=[pd.NaT, pd.Timestamp('1939-05-27'),
...                          pd.Timestamp('1940-04-25')],
...                    name=['Alfred', 'Batman', ''],
...                    toy=[None, 'Batmobile', 'Joker']))
>>> df
   age       born    name        toy
0  5.0        NaT  Alfred       None
1  6.0 1939-05-27  Batman  Batmobile
2  NaN 1940-04-25              Joker

>>> df.isna()
     age   born   name    toy
0  False   True  False   True
1  False  False  False  False
2   True  False  False  False

Show which entries in a Series are NA.

>>> ser = pd.Series([5, 6, np.NaN])
>>> ser
0    5.0
1    6.0
2    NaN
dtype: float64

>>> ser.isna()
0    False
1    False
2     True
dtype: bool

isna(**kwargs)¶

Detect missing values.

Return a boolean same-sized object indicating if the values are NA. NA values, such as None or numpy.NaN, gets mapped to True values. Everything else gets mapped to False values. Characters such as empty strings '' or numpy.inf are not considered NA values (unless you set pandas.options.mode.use_inf_as_na = True).

Returns:	Mask of bool values for each element in DeferredSeries that indicates whether an element is an NA value.
Return type:	DeferredSeries

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.isnull(): Alias of isna.
DeferredSeries.notna(): Boolean inverse of isna.
DeferredSeries.dropna(): Omit axes labels with missing values.
isna(): Top-level isna.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Show which entries in a DataFrame are NA.

>>> df = pd.DataFrame(dict(age=[5, 6, np.NaN],
...                    born=[pd.NaT, pd.Timestamp('1939-05-27'),
...                          pd.Timestamp('1940-04-25')],
...                    name=['Alfred', 'Batman', ''],
...                    toy=[None, 'Batmobile', 'Joker']))
>>> df
   age       born    name        toy
0  5.0        NaT  Alfred       None
1  6.0 1939-05-27  Batman  Batmobile
2  NaN 1940-04-25              Joker

>>> df.isna()
     age   born   name    toy
0  False   True  False   True
1  False  False  False  False
2   True  False  False  False

Show which entries in a Series are NA.

>>> ser = pd.Series([5, 6, np.NaN])
>>> ser
0    5.0
1    6.0
2    NaN
dtype: float64

>>> ser.isna()
0    False
1    False
2     True
dtype: bool

notnull(**kwargs)¶

Detect existing (non-missing) values.

Return a boolean same-sized object indicating if the values are not NA. Non-missing values get mapped to True. Characters such as empty strings '' or numpy.inf are not considered NA values (unless you set pandas.options.mode.use_inf_as_na = True). NA values, such as None or numpy.NaN, get mapped to False values.

Returns:	Mask of bool values for each element in DeferredSeries that indicates whether an element is not an NA value.
Return type:	DeferredSeries

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.notnull(): Alias of notna.
DeferredSeries.isna(): Boolean inverse of notna.
DeferredSeries.dropna(): Omit axes labels with missing values.
notna(): Top-level notna.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Show which entries in a DataFrame are not NA.

>>> df = pd.DataFrame(dict(age=[5, 6, np.NaN],
...                    born=[pd.NaT, pd.Timestamp('1939-05-27'),
...                          pd.Timestamp('1940-04-25')],
...                    name=['Alfred', 'Batman', ''],
...                    toy=[None, 'Batmobile', 'Joker']))
>>> df
   age       born    name        toy
0  5.0        NaT  Alfred       None
1  6.0 1939-05-27  Batman  Batmobile
2  NaN 1940-04-25              Joker

>>> df.notna()
     age   born  name    toy
0   True  False  True  False
1   True   True  True   True
2  False   True  True   True

Show which entries in a Series are not NA.

>>> ser = pd.Series([5, 6, np.NaN])
>>> ser
0    5.0
1    6.0
2    NaN
dtype: float64

>>> ser.notna()
0     True
1     True
2    False
dtype: bool

notna(**kwargs)¶

Detect existing (non-missing) values.

Return a boolean same-sized object indicating if the values are not NA. Non-missing values get mapped to True. Characters such as empty strings '' or numpy.inf are not considered NA values (unless you set pandas.options.mode.use_inf_as_na = True). NA values, such as None or numpy.NaN, get mapped to False values.

Returns:	Mask of bool values for each element in DeferredSeries that indicates whether an element is not an NA value.
Return type:	DeferredSeries

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.notnull(): Alias of notna.
DeferredSeries.isna(): Boolean inverse of notna.
DeferredSeries.dropna(): Omit axes labels with missing values.
notna(): Top-level notna.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Show which entries in a DataFrame are not NA.

>>> df = pd.DataFrame(dict(age=[5, 6, np.NaN],
...                    born=[pd.NaT, pd.Timestamp('1939-05-27'),
...                          pd.Timestamp('1940-04-25')],
...                    name=['Alfred', 'Batman', ''],
...                    toy=[None, 'Batmobile', 'Joker']))
>>> df
   age       born    name        toy
0  5.0        NaT  Alfred       None
1  6.0 1939-05-27  Batman  Batmobile
2  NaN 1940-04-25              Joker

>>> df.notna()
     age   born  name    toy
0   True  False  True  False
1   True   True  True   True
2  False   True  True   True

Show which entries in a Series are not NA.

>>> ser = pd.Series([5, 6, np.NaN])
>>> ser
0    5.0
1    6.0
2    NaN
dtype: float64

>>> ser.notna()
0     True
1     True
2    False
dtype: bool

items(**kwargs)¶

pandas.Series.items is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

iteritems(**kwargs)¶

pandas.Series.iteritems is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

tolist(**kwargs)¶

pandas.Series.tolist is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

to_numpy(**kwargs)¶

pandas.Series.to_numpy is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

to_string(**kwargs)¶

pandas.Series.to_string is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

duplicated(keep)[source]¶

Indicate duplicate Series values.

Duplicated values are indicated as True values in the resulting Series. Either all duplicates, all except the first or all except the last occurrence of duplicates can be indicated.

Parameters:

keep ({'first', 'last', False}, default 'first') –

Method to handle dropping duplicates:

’first’ : Mark duplicates as True except for the first occurrence.
’last’ : Mark duplicates as True except for the last occurrence.
False : Mark all duplicates as True.

Returns: DeferredSeries indicating whether each value has occurred in the preceding values.

Return type: DeferredSeries

Differences from pandas

Only keep=False and keep="any" are supported. Other values of keep make this an order-sensitive operation. Note keep="any" is a Beam-specific option that guarantees only one duplicate will be kept, but unlike "first" and "last" it makes no guarantees about _which_ duplicate element is kept.

See also

Index.duplicated(): Equivalent method on pandas.Index.
DeferredDataFrame.duplicated(): Equivalent method on pandas.DeferredDataFrame.
DeferredSeries.drop_duplicates(): Remove duplicate values from DeferredSeries.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

By default, for each set of duplicated values, the first occurrence is
set on False and all others on True:

>>> animals = pd.Series(['lama', 'cow', 'lama', 'beetle', 'lama'])
>>> animals.duplicated()
0    False
1    False
2     True
3    False
4     True
dtype: bool

which is equivalent to

>>> animals.duplicated(keep='first')
0    False
1    False
2     True
3    False
4     True
dtype: bool

By using 'last', the last occurrence of each set of duplicated values
is set on False and all others on True:

>>> animals.duplicated(keep='last')
0     True
1    False
2     True
3    False
4    False
dtype: bool

By setting keep on ``False``, all duplicates are True:

>>> animals.duplicated(keep=False)
0     True
1    False
2     True
3    False
4     True
dtype: bool

drop_duplicates(keep)[source]¶

Return Series with duplicate values removed.

Parameters:	keep ({‘first’, ‘last’, `False`}, default ‘first’) – Method to handle dropping duplicates: ’first’ : Drop duplicates except for the first occurrence. ’last’ : Drop duplicates except for the last occurrence. `False` : Drop all duplicates. inplace (bool, default `False`) – If `True`, performs operation inplace and returns None.
Returns:	DeferredSeries with duplicates dropped or None if `inplace=True`.
Return type:	DeferredSeries or None

Differences from pandas

Only keep=False and keep="any" are supported. Other values of keep make this an order-sensitive operation. Note keep="any" is a Beam-specific option that guarantees only one duplicate will be kept, but unlike "first" and "last" it makes no guarantees about _which_ duplicate element is kept.

See also

Index.drop_duplicates(): Equivalent method on Index.
DeferredDataFrame.drop_duplicates(): Equivalent method on DeferredDataFrame.
DeferredSeries.duplicated(): Related method on DeferredSeries, indicating duplicate DeferredSeries values.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

Generate a Series with duplicated entries.

>>> s = pd.Series(['lama', 'cow', 'lama', 'beetle', 'lama', 'hippo'],
...               name='animal')
>>> s
0      lama
1       cow
2      lama
3    beetle
4      lama
5     hippo
Name: animal, dtype: object

With the 'keep' parameter, the selection behaviour of duplicated values
can be changed. The value 'first' keeps the first occurrence for each
set of duplicated entries. The default value of keep is 'first'.

>>> s.drop_duplicates()
0      lama
1       cow
3    beetle
5     hippo
Name: animal, dtype: object

The value 'last' for parameter 'keep' keeps the last occurrence for
each set of duplicated entries.

>>> s.drop_duplicates(keep='last')
1       cow
3    beetle
4      lama
5     hippo
Name: animal, dtype: object

The value ``False`` for parameter 'keep' discards all sets of
duplicated entries. Setting the value of 'inplace' to ``True`` performs
the operation inplace and returns ``None``.

>>> s.drop_duplicates(keep=False, inplace=True)
>>> s
1       cow
3    beetle
5     hippo
Name: animal, dtype: object

aggregate(func, axis, *args, **kwargs)[source]¶

Aggregate using one or more operations over the specified axis.

Parameters:

func (function, str, list or dict) –
Function to use for aggregating the data. If a function, must either work when passed a DeferredSeries or when passed to DeferredSeries.apply.

Accepted combinations are:
- function
- string function name
- list of functions and/or function names, e.g. [np.sum, 'mean']
- dict of axis labels -> functions, function names or list of such.
axis ({0 or 'index'}) – Parameter needed for compatibility with DeferredDataFrame.
*args – Positional arguments to pass to func.
**kwargs – Keyword arguments to pass to func.

Returns:

The return can be:

scalar : when DeferredSeries.agg is called with single function
DeferredSeries : when DeferredDataFrame.agg is called with a single function
DeferredDataFrame : when DeferredDataFrame.agg is called with several functions

Return scalar, DeferredSeries or DeferredDataFrame.

Return type:

scalar, DeferredSeries or DeferredDataFrame

Differences from pandas

Some aggregation methods cannot be parallelized, and computing them will require collecting all data on a single machine.

See also

DeferredSeries.apply(): Invoke function on a DeferredSeries.
DeferredSeries.transform(): Transform function producing a DeferredSeries with like indexes.

Notes

agg is an alias for aggregate. Use the alias.

A passed user-defined-function will be passed a DeferredSeries for evaluation.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> s = pd.Series([1, 2, 3, 4])
>>> s
0    1
1    2
2    3
3    4
dtype: int64

>>> s.agg('min')
1

>>> s.agg(['min', 'max'])
min   1
max   4
dtype: int64

agg(func, axis, *args, **kwargs)¶

Aggregate using one or more operations over the specified axis.

Parameters:

func (function, str, list or dict) –
Function to use for aggregating the data. If a function, must either work when passed a DeferredSeries or when passed to DeferredSeries.apply.

Accepted combinations are:
- function
- string function name
- list of functions and/or function names, e.g. [np.sum, 'mean']
- dict of axis labels -> functions, function names or list of such.
axis ({0 or 'index'}) – Parameter needed for compatibility with DeferredDataFrame.
*args – Positional arguments to pass to func.
**kwargs – Keyword arguments to pass to func.

Returns:

The return can be:

scalar : when DeferredSeries.agg is called with single function
DeferredSeries : when DeferredDataFrame.agg is called with a single function
DeferredDataFrame : when DeferredDataFrame.agg is called with several functions

Return scalar, DeferredSeries or DeferredDataFrame.

Return type:

scalar, DeferredSeries or DeferredDataFrame

Differences from pandas

Some aggregation methods cannot be parallelized, and computing them will require collecting all data on a single machine.

See also

DeferredSeries.apply(): Invoke function on a DeferredSeries.
DeferredSeries.transform(): Transform function producing a DeferredSeries with like indexes.

Notes

agg is an alias for aggregate. Use the alias.

A passed user-defined-function will be passed a DeferredSeries for evaluation.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> s = pd.Series([1, 2, 3, 4])
>>> s
0    1
1    2
2    3
3    4
dtype: int64

>>> s.agg('min')
1

>>> s.agg(['min', 'max'])
min   1
max   4
dtype: int64

axes¶

Return a list of the row axis labels.

Differences from pandas

This operation has no known divergences from the pandas API.

clip(**kwargs)¶

Trim values at input threshold(s).

Assigns values outside boundary to boundary values. Thresholds can be singular values or array like, and in the latter case the clipping is performed element-wise in the specified axis.

Parameters:	lower (float or array_like, default None) – Minimum threshold value. All values below this threshold will be set to it. upper (float or array_like, default None) – Maximum threshold value. All values above this threshold will be set to it. axis (int or str axis name, optional) – Align object with lower and upper along the given axis. inplace (bool, default False) – Whether to perform the operation in place on the data. *kwargs (args,) – Additional keywords have no effect but might be accepted for compatibility with numpy.
Returns:	Same type as calling object with the values outside the clip boundaries replaced or None if `inplace=True`.
Return type:	DeferredSeries or DeferredDataFrame or None

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.clip(): Trim values at input threshold in series.
DeferredDataFrame.clip(): Trim values at input threshold in dataframe.
numpy.clip(): Clip (limit) the values in an array.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> data = {'col_0': [9, -3, 0, -1, 5], 'col_1': [-2, -7, 6, 8, -5]}
>>> df = pd.DataFrame(data)
>>> df
   col_0  col_1
0      9     -2
1     -3     -7
2      0      6
3     -1      8
4      5     -5

Clips per column using lower and upper thresholds:

>>> df.clip(-4, 6)
   col_0  col_1
0      6     -2
1     -3     -4
2      0      6
3     -1      6
4      5     -4

Clips using specific lower and upper thresholds per column element:

>>> t = pd.Series([2, -4, -1, 6, 3])
>>> t
0    2
1   -4
2   -1
3    6
4    3
dtype: int64

>>> df.clip(t, t + 4, axis=0)
   col_0  col_1
0      6      2
1     -3     -4
2      0      3
3      6      8
4      5      3

all(*args, **kwargs)¶

Return whether all elements are True, potentially over an axis.

Returns True unless there at least one element within a series or along a Dataframe axis that is False or equivalent (e.g. zero or empty).

Parameters:	axis ({0 or 'index', 1 or 'columns', None}, default 0) – Indicate which axis or axes should be reduced. 0 / ‘index’ : reduce the index, return a DeferredSeries whose index is the original column labels. 1 / ‘columns’ : reduce the columns, return a DeferredSeries whose index is the original index. None : reduce all axes, return a scalar. bool_only (bool, default None) – Include only boolean columns. If None, will attempt to use everything, then use only boolean data. Not implemented for DeferredSeries. skipna (bool, default True) – Exclude NA/null values. If the entire row/column is NA and skipna is True, then the result will be True, as for an empty row/column. If skipna is False, then NA are treated as True, because these are not equal to zero. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar. *kwargs (any, default None*) – Additional keywords have no effect but might be accepted for compatibility with NumPy.
Returns:	If level is specified, then, DeferredSeries is returned; otherwise, scalar is returned.
Return type:	scalar or DeferredSeries

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.all(): Return True if all elements are True.
DeferredDataFrame.any(): Return True if one (or more) elements are True.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

**Series**

>>> pd.Series([True, True]).all()
True
>>> pd.Series([True, False]).all()
False
>>> pd.Series([]).all()
True
>>> pd.Series([np.nan]).all()
True
>>> pd.Series([np.nan]).all(skipna=False)
True

**DataFrames**

Create a dataframe from a dictionary.

>>> df = pd.DataFrame({'col1': [True, True], 'col2': [True, False]})
>>> df
   col1   col2
0  True   True
1  True  False

Default behaviour checks if column-wise values all return True.

>>> df.all()
col1     True
col2    False
dtype: bool

Specify ``axis='columns'`` to check if row-wise values all return True.

>>> df.all(axis='columns')
0     True
1    False
dtype: bool

Or ``axis=None`` for whether every value is True.

>>> df.all(axis=None)
False

any(*args, **kwargs)¶

Return whether any element is True, potentially over an axis.

Returns False unless there is at least one element within a series or along a Dataframe axis that is True or equivalent (e.g. non-zero or non-empty).

Parameters:	axis ({0 or 'index', 1 or 'columns', None}, default 0) – Indicate which axis or axes should be reduced. 0 / ‘index’ : reduce the index, return a DeferredSeries whose index is the original column labels. 1 / ‘columns’ : reduce the columns, return a DeferredSeries whose index is the original index. None : reduce all axes, return a scalar. bool_only (bool, default None) – Include only boolean columns. If None, will attempt to use everything, then use only boolean data. Not implemented for DeferredSeries. skipna (bool, default True) – Exclude NA/null values. If the entire row/column is NA and skipna is True, then the result will be False, as for an empty row/column. If skipna is False, then NA are treated as True, because these are not equal to zero. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar. *kwargs (any, default None*) – Additional keywords have no effect but might be accepted for compatibility with NumPy.
Returns:	If level is specified, then, DeferredSeries is returned; otherwise, scalar is returned.
Return type:	scalar or DeferredSeries

Differences from pandas

This operation has no known divergences from the pandas API.

See also

numpy.any(): Numpy version of this method.
DeferredSeries.any(): Return whether any element is True.
DeferredSeries.all(): Return whether all elements are True.
DeferredDataFrame.any(): Return whether any element is True over requested axis.
DeferredDataFrame.all(): Return whether all elements are True over requested axis.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

**Series**

For Series input, the output is a scalar indicating whether any element
is True.

>>> pd.Series([False, False]).any()
False
>>> pd.Series([True, False]).any()
True
>>> pd.Series([]).any()
False
>>> pd.Series([np.nan]).any()
False
>>> pd.Series([np.nan]).any(skipna=False)
True

**DataFrame**

Whether each column contains at least one True element (the default).

>>> df = pd.DataFrame({"A": [1, 2], "B": [0, 2], "C": [0, 0]})
>>> df
   A  B  C
0  1  0  0
1  2  2  0

>>> df.any()
A     True
B     True
C    False
dtype: bool

Aggregating over the columns.

>>> df = pd.DataFrame({"A": [True, False], "B": [1, 2]})
>>> df
       A  B
0   True  1
1  False  2

>>> df.any(axis='columns')
0    True
1    True
dtype: bool

>>> df = pd.DataFrame({"A": [True, False], "B": [1, 0]})
>>> df
       A  B
0   True  1
1  False  0

>>> df.any(axis='columns')
0    True
1    False
dtype: bool

Aggregating over the entire DataFrame with ``axis=None``.

>>> df.any(axis=None)
True

`any` for an empty DataFrame is an empty Series.

>>> pd.DataFrame([]).any()
Series([], dtype: bool)

count(*args, **kwargs)¶

Return number of non-NA/null observations in the Series.

Parameters:	level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a smaller DeferredSeries.
Returns:	Number of non-null values in the DeferredSeries.
Return type:	int or DeferredSeries (if level specified)

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.count(): Count non-NA cells for each column or row.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series([0.0, 1.0, np.nan])
>>> s.count()
2

describe(*args, **kwargs)¶

Generate descriptive statistics.

Descriptive statistics include those that summarize the central tendency, dispersion and shape of a dataset’s distribution, excluding NaN values.

Analyzes both numeric and object series, as well as DataFrame column sets of mixed data types. The output will vary depending on what is provided. Refer to the notes below for more detail.

Parameters:	percentiles (list-like of numbers, optional) – The percentiles to include in the output. All should fall between 0 and 1. The default is `[.25, .5, .75]`, which returns the 25th, 50th, and 75th percentiles. include ('all', list-like of dtypes or None (default), optional) – A white list of data types to include in the result. Ignored for `DeferredSeries`. Here are the options: ’all’ : All columns of the input will be included in the output. A list-like of dtypes : Limits the results to the provided data types. To limit the result to numeric types submit `numpy.number`. To limit it instead to object columns submit the `numpy.object` data type. Strings can also be used in the style of `select_dtypes` (e.g. `df.describe(include=['O'])`). To select pandas categorical columns, use `'category'` None (default) : The result will include all numeric columns. exclude (list-like of dtypes or None (default), optional,) – A black list of data types to omit from the result. Ignored for `DeferredSeries`. Here are the options: A list-like of dtypes : Excludes the provided data types from the result. To exclude numeric types submit `numpy.number`. To exclude object columns submit the data type `numpy.object`. Strings can also be used in the style of `select_dtypes` (e.g. `df.describe(include=['O'])`). To exclude pandas categorical columns, use `'category'` None (default) : The result will exclude nothing. datetime_is_numeric (bool, default False) – Whether to treat datetime dtypes as numeric. This affects statistics calculated for the column. For DeferredDataFrame input, this also controls whether datetime columns are included by default. New in version 1.1.0.
Returns:	Summary statistics of the DeferredSeries or Dataframe provided.
Return type:	DeferredSeries or DeferredDataFrame

Differences from pandas

describe cannot currently be parallelized. It will require collecting all data on a single node.

See also

DeferredDataFrame.count(): Count number of non-NA/null observations.
DeferredDataFrame.max(): Maximum of the values in the object.
DeferredDataFrame.min(): Minimum of the values in the object.
DeferredDataFrame.mean(): Mean of the values.
DeferredDataFrame.std(): Standard deviation of the observations.
DeferredDataFrame.select_dtypes(): Subset of a DeferredDataFrame including/excluding columns based on their dtype.

Notes

For numeric data, the result’s index will include count, mean, std, min, max as well as lower, 50 and upper percentiles. By default the lower percentile is 25 and the upper percentile is 75. The 50 percentile is the same as the median.

For object data (e.g. strings or timestamps), the result’s index will include count, unique, top, and freq. The top is the most common value. The freq is the most common value’s frequency. Timestamps also include the first and last items.

If multiple object values have the highest count, then the count and top results will be arbitrarily chosen from among those with the highest count.

For mixed data types provided via a DeferredDataFrame, the default is to return only an analysis of numeric columns. If the dataframe consists only of object and categorical data without any numeric columns, the default is to return an analysis of both the object and categorical columns. If include='all' is provided as an option, the result will include a union of attributes of each type.

The include and exclude parameters can be used to limit which columns in a DeferredDataFrame are analyzed for the output. The parameters are ignored when analyzing a DeferredSeries.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

Describing a numeric ``Series``.

>>> s = pd.Series([1, 2, 3])
>>> s.describe()
count    3.0
mean     2.0
std      1.0
min      1.0
25%      1.5
50%      2.0
75%      2.5
max      3.0
dtype: float64

Describing a categorical ``Series``.

>>> s = pd.Series(['a', 'a', 'b', 'c'])
>>> s.describe()
count     4
unique    3
top       a
freq      2
dtype: object

Describing a timestamp ``Series``.

>>> s = pd.Series([
...   np.datetime64("2000-01-01"),
...   np.datetime64("2010-01-01"),
...   np.datetime64("2010-01-01")
... ])
>>> s.describe(datetime_is_numeric=True)
count                      3
mean     2006-09-01 08:00:00
min      2000-01-01 00:00:00
25%      2004-12-31 12:00:00
50%      2010-01-01 00:00:00
75%      2010-01-01 00:00:00
max      2010-01-01 00:00:00
dtype: object

Describing a ``DataFrame``. By default only numeric fields
are returned.

>>> df = pd.DataFrame({'categorical': pd.Categorical(['d','e','f']),
...                    'numeric': [1, 2, 3],
...                    'object': ['a', 'b', 'c']
...                   })
>>> df.describe()
       numeric
count      3.0
mean       2.0
std        1.0
min        1.0
25%        1.5
50%        2.0
75%        2.5
max        3.0

Describing all columns of a ``DataFrame`` regardless of data type.

>>> df.describe(include='all')  
       categorical  numeric object
count            3      3.0      3
unique           3      NaN      3
top              f      NaN      a
freq             1      NaN      1
mean           NaN      2.0    NaN
std            NaN      1.0    NaN
min            NaN      1.0    NaN
25%            NaN      1.5    NaN
50%            NaN      2.0    NaN
75%            NaN      2.5    NaN
max            NaN      3.0    NaN

Describing a column from a ``DataFrame`` by accessing it as
an attribute.

>>> df.numeric.describe()
count    3.0
mean     2.0
std      1.0
min      1.0
25%      1.5
50%      2.0
75%      2.5
max      3.0
Name: numeric, dtype: float64

Including only numeric columns in a ``DataFrame`` description.

>>> df.describe(include=[np.number])
       numeric
count      3.0
mean       2.0
std        1.0
min        1.0
25%        1.5
50%        2.0
75%        2.5
max        3.0

Including only string columns in a ``DataFrame`` description.

>>> df.describe(include=[object])  
       object
count       3
unique      3
top         a
freq        1

Including only categorical columns from a ``DataFrame`` description.

>>> df.describe(include=['category'])
       categorical
count            3
unique           3
top              d
freq             1

Excluding numeric columns from a ``DataFrame`` description.

>>> df.describe(exclude=[np.number])  
       categorical object
count            3      3
unique           3      3
top              f      a
freq             1      1

Excluding object columns from a ``DataFrame`` description.

>>> df.describe(exclude=[object])  
       categorical  numeric
count            3      3.0
unique           3      NaN
top              f      NaN
freq             1      NaN
mean           NaN      2.0
std            NaN      1.0
min            NaN      1.0
25%            NaN      1.5
50%            NaN      2.0
75%            NaN      2.5
max            NaN      3.0

min(*args, **kwargs)¶

Return the minimum of the values over the requested axis.

If you want the index of the minimum, use idxmin. This isthe equivalent of the numpy.ndarray method argmin.

Parameters:	axis ({index (0)}) – Axis for the function to be applied on. skipna (bool, default True) – Exclude NA/null values when computing the result. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries. **kwargs – Additional keyword arguments to be passed to the function.
Returns:
Return type:	scalar or DeferredSeries (if level specified)

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.sum(): Return the sum.
DeferredSeries.min(): Return the minimum.
DeferredSeries.max(): Return the maximum.
DeferredSeries.idxmin(): Return the index of the minimum.
DeferredSeries.idxmax(): Return the index of the maximum.
DeferredDataFrame.sum(): Return the sum over the requested axis.
DeferredDataFrame.min(): Return the minimum over the requested axis.
DeferredDataFrame.max(): Return the maximum over the requested axis.
DeferredDataFrame.idxmin(): Return the index of the minimum over the requested axis.
DeferredDataFrame.idxmax(): Return the index of the maximum over the requested axis.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> idx = pd.MultiIndex.from_arrays([
...     ['warm', 'warm', 'cold', 'cold'],
...     ['dog', 'falcon', 'fish', 'spider']],
...     names=['blooded', 'animal'])
>>> s = pd.Series([4, 2, 0, 8], name='legs', index=idx)
>>> s
blooded  animal
warm     dog       4
         falcon    2
cold     fish      0
         spider    8
Name: legs, dtype: int64

>>> s.min()
0

Min using level names, as well as indices.

>>> s.min(level='blooded')
blooded
warm    2
cold    0
Name: legs, dtype: int64

>>> s.min(level=0)
blooded
warm    2
cold    0
Name: legs, dtype: int64

max(*args, **kwargs)¶

Return the maximum of the values over the requested axis.

If you want the index of the maximum, use idxmax. This isthe equivalent of the numpy.ndarray method argmax.

Parameters:	axis ({index (0)}) – Axis for the function to be applied on. skipna (bool, default True) – Exclude NA/null values when computing the result. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries. **kwargs – Additional keyword arguments to be passed to the function.
Returns:
Return type:	scalar or DeferredSeries (if level specified)

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.sum(): Return the sum.
DeferredSeries.min(): Return the minimum.
DeferredSeries.max(): Return the maximum.
DeferredSeries.idxmin(): Return the index of the minimum.
DeferredSeries.idxmax(): Return the index of the maximum.
DeferredDataFrame.sum(): Return the sum over the requested axis.
DeferredDataFrame.min(): Return the minimum over the requested axis.
DeferredDataFrame.max(): Return the maximum over the requested axis.
DeferredDataFrame.idxmin(): Return the index of the minimum over the requested axis.
DeferredDataFrame.idxmax(): Return the index of the maximum over the requested axis.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> idx = pd.MultiIndex.from_arrays([
...     ['warm', 'warm', 'cold', 'cold'],
...     ['dog', 'falcon', 'fish', 'spider']],
...     names=['blooded', 'animal'])
>>> s = pd.Series([4, 2, 0, 8], name='legs', index=idx)
>>> s
blooded  animal
warm     dog       4
         falcon    2
cold     fish      0
         spider    8
Name: legs, dtype: int64

>>> s.max()
8

Max using level names, as well as indices.

>>> s.max(level='blooded')
blooded
warm    4
cold    8
Name: legs, dtype: int64

>>> s.max(level=0)
blooded
warm    4
cold    8
Name: legs, dtype: int64

prod(*args, **kwargs)¶

Return the product of the values over the requested axis.

Parameters:	axis ({index (0)}) – Axis for the function to be applied on. skipna (bool, default True) – Exclude NA/null values when computing the result. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries. min_count (int, default 0) – The required number of valid values to perform the operation. If fewer than `min_count` non-NA values are present the result will be NA. **kwargs – Additional keyword arguments to be passed to the function.
Returns:
Return type:	scalar or DeferredSeries (if level specified)

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.sum(): Return the sum.
DeferredSeries.min(): Return the minimum.
DeferredSeries.max(): Return the maximum.
DeferredSeries.idxmin(): Return the index of the minimum.
DeferredSeries.idxmax(): Return the index of the maximum.
DeferredDataFrame.sum(): Return the sum over the requested axis.
DeferredDataFrame.min(): Return the minimum over the requested axis.
DeferredDataFrame.max(): Return the maximum over the requested axis.
DeferredDataFrame.idxmin(): Return the index of the minimum over the requested axis.
DeferredDataFrame.idxmax(): Return the index of the maximum over the requested axis.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

By default, the product of an empty or all-NA Series is ``1``

>>> pd.Series([]).prod()
1.0

This can be controlled with the ``min_count`` parameter

>>> pd.Series([]).prod(min_count=1)
nan

Thanks to the ``skipna`` parameter, ``min_count`` handles all-NA and
empty series identically.

>>> pd.Series([np.nan]).prod()
1.0

>>> pd.Series([np.nan]).prod(min_count=1)
nan

product(*args, **kwargs)¶

Return the product of the values over the requested axis.

Parameters:	axis ({index (0)}) – Axis for the function to be applied on. skipna (bool, default True) – Exclude NA/null values when computing the result. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries. min_count (int, default 0) – The required number of valid values to perform the operation. If fewer than `min_count` non-NA values are present the result will be NA. **kwargs – Additional keyword arguments to be passed to the function.
Returns:
Return type:	scalar or DeferredSeries (if level specified)

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.sum(): Return the sum.
DeferredSeries.min(): Return the minimum.
DeferredSeries.max(): Return the maximum.
DeferredSeries.idxmin(): Return the index of the minimum.
DeferredSeries.idxmax(): Return the index of the maximum.
DeferredDataFrame.sum(): Return the sum over the requested axis.
DeferredDataFrame.min(): Return the minimum over the requested axis.
DeferredDataFrame.max(): Return the maximum over the requested axis.
DeferredDataFrame.idxmin(): Return the index of the minimum over the requested axis.
DeferredDataFrame.idxmax(): Return the index of the maximum over the requested axis.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

By default, the product of an empty or all-NA Series is ``1``

>>> pd.Series([]).prod()
1.0

This can be controlled with the ``min_count`` parameter

>>> pd.Series([]).prod(min_count=1)
nan

Thanks to the ``skipna`` parameter, ``min_count`` handles all-NA and
empty series identically.

>>> pd.Series([np.nan]).prod()
1.0

>>> pd.Series([np.nan]).prod(min_count=1)
nan

sum(*args, **kwargs)¶

Return the sum of the values over the requested axis.

This is equivalent to the method numpy.sum.

Parameters:	axis ({index (0)}) – Axis for the function to be applied on. skipna (bool, default True) – Exclude NA/null values when computing the result. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries. min_count (int, default 0) – The required number of valid values to perform the operation. If fewer than `min_count` non-NA values are present the result will be NA. **kwargs – Additional keyword arguments to be passed to the function.
Returns:
Return type:	scalar or DeferredSeries (if level specified)

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.sum(): Return the sum.
DeferredSeries.min(): Return the minimum.
DeferredSeries.max(): Return the maximum.
DeferredSeries.idxmin(): Return the index of the minimum.
DeferredSeries.idxmax(): Return the index of the maximum.
DeferredDataFrame.sum(): Return the sum over the requested axis.
DeferredDataFrame.min(): Return the minimum over the requested axis.
DeferredDataFrame.max(): Return the maximum over the requested axis.
DeferredDataFrame.idxmin(): Return the index of the minimum over the requested axis.
DeferredDataFrame.idxmax(): Return the index of the maximum over the requested axis.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> idx = pd.MultiIndex.from_arrays([
...     ['warm', 'warm', 'cold', 'cold'],
...     ['dog', 'falcon', 'fish', 'spider']],
...     names=['blooded', 'animal'])
>>> s = pd.Series([4, 2, 0, 8], name='legs', index=idx)
>>> s
blooded  animal
warm     dog       4
         falcon    2
cold     fish      0
         spider    8
Name: legs, dtype: int64

>>> s.sum()
14

Sum using level names, as well as indices.

>>> s.sum(level='blooded')
blooded
warm    6
cold    8
Name: legs, dtype: int64

>>> s.sum(level=0)
blooded
warm    6
cold    8
Name: legs, dtype: int64

By default, the sum of an empty or all-NA Series is ``0``.

>>> pd.Series([]).sum()  # min_count=0 is the default
0.0

This can be controlled with the ``min_count`` parameter. For example, if
you'd like the sum of an empty series to be NaN, pass ``min_count=1``.

>>> pd.Series([]).sum(min_count=1)
nan

Thanks to the ``skipna`` parameter, ``min_count`` handles all-NA and
empty series identically.

>>> pd.Series([np.nan]).sum()
0.0

>>> pd.Series([np.nan]).sum(min_count=1)
nan

mean(*args, **kwargs)¶

Return the mean of the values over the requested axis.

Parameters:	axis ({index (0)}) – Axis for the function to be applied on. skipna (bool, default True) – Exclude NA/null values when computing the result. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries. **kwargs – Additional keyword arguments to be passed to the function.
Returns:
Return type:	scalar or DeferredSeries (if level specified)

Differences from pandas

mean cannot currently be parallelized. It will require collecting all data on a single node.

median(*args, **kwargs)¶

Return the median of the values over the requested axis.

Parameters:	axis ({index (0)}) – Axis for the function to be applied on. skipna (bool, default True) – Exclude NA/null values when computing the result. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries. **kwargs – Additional keyword arguments to be passed to the function.
Returns:
Return type:	scalar or DeferredSeries (if level specified)

Differences from pandas

median cannot currently be parallelized. It will require collecting all data on a single node.

argmax(**kwargs)¶

pandas.Series.argmax is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

argmin(**kwargs)¶

pandas.Series.argmin is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

cummax(**kwargs)¶

pandas.Series.cummax is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

cummin(**kwargs)¶

pandas.Series.cummin is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

cumprod(**kwargs)¶

pandas.Series.cumprod is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

cumsum(**kwargs)¶

pandas.Series.cumsum is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

diff(**kwargs)¶

pandas.Series.diff is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

first(**kwargs)¶

pandas.Series.first is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

head(**kwargs)¶

pandas.Series.head is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

interpolate(**kwargs)¶

pandas.Series.interpolate is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

last(**kwargs)¶

pandas.Series.last is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

searchsorted(**kwargs)¶

pandas.Series.searchsorted is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

shift(**kwargs)¶

pandas.Series.shift is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

tail(**kwargs)¶

pandas.Series.tail is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

filter(**kwargs)¶

Subset the dataframe rows or columns according to the specified index labels.

Note that this routine does not filter a dataframe on its contents. The filter is applied to the labels of the index.

Parameters:	items (list-like) – Keep labels from axis which are in items. like (str) – Keep labels from axis for which “like in label == True”. regex (str (regular expression)) – Keep labels from axis for which re.search(regex, label) == True. axis ({0 or ‘index’, 1 or ‘columns’, None}, default None) – The axis to filter on, expressed either as an index (int) or axis name (str). By default this is the info axis, ‘index’ for DeferredSeries, ‘columns’ for DeferredDataFrame.
Returns:
Return type:	same type as input object

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.loc(): Access a group of rows and columns by label(s) or a boolean array.

Notes

The items, like, and regex parameters are enforced to be mutually exclusive.

axis defaults to the info axis that is used when indexing with [].

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame(np.array(([1, 2, 3], [4, 5, 6])),
...                   index=['mouse', 'rabbit'],
...                   columns=['one', 'two', 'three'])
>>> df
        one  two  three
mouse     1    2      3
rabbit    4    5      6

>>> # select columns by name
>>> df.filter(items=['one', 'three'])
         one  three
mouse     1      3
rabbit    4      6

>>> # select columns by regular expression
>>> df.filter(regex='e$', axis=1)
         one  three
mouse     1      3
rabbit    4      6

>>> # select rows containing 'bbi'
>>> df.filter(like='bbi', axis=0)
         one  two  three
rabbit    4    5      6

memory_usage(**kwargs)¶

pandas.Series.memory_usage is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

nlargest(keep, **kwargs)[source]¶

Return the largest n elements.

Parameters:	n (int, default 5) – Return this many descending sorted values. keep ({'first', 'last', 'all'}, default 'first') – When there are duplicate values that cannot all fit in a DeferredSeries of n elements: `first` : return the first n occurrences in order of appearance. `last` : return the last n occurrences in reverse order of appearance. `all` : keep all occurrences. This can result in a DeferredSeries of size larger than n.
Returns:	The n largest values in the DeferredSeries, sorted in decreasing order.
Return type:	DeferredSeries

Differences from pandas

Only keep=False and keep="any" are supported. Other values of keep make this an order-sensitive operation. Note keep="any" is a Beam-specific option that guarantees only one duplicate will be kept, but unlike "first" and "last" it makes no guarantees about _which_ duplicate element is kept.

See also

DeferredSeries.nsmallest(): Get the n smallest elements.
DeferredSeries.sort_values(): Sort DeferredSeries by values.
DeferredSeries.head(): Return the first n rows.

Notes

Faster than .sort_values(ascending=False).head(n) for small n relative to the size of the DeferredSeries object.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> countries_population = {"Italy": 59000000, "France": 65000000,
...                         "Malta": 434000, "Maldives": 434000,
...                         "Brunei": 434000, "Iceland": 337000,
...                         "Nauru": 11300, "Tuvalu": 11300,
...                         "Anguilla": 11300, "Montserrat": 5200}
>>> s = pd.Series(countries_population)
>>> s
Italy       59000000
France      65000000
Malta         434000
Maldives      434000
Brunei        434000
Iceland       337000
Nauru          11300
Tuvalu         11300
Anguilla       11300
Montserrat      5200
dtype: int64

The `n` largest elements where ``n=5`` by default.

>>> s.nlargest()
France      65000000
Italy       59000000
Malta         434000
Maldives      434000
Brunei        434000
dtype: int64

The `n` largest elements where ``n=3``. Default `keep` value is 'first'
so Malta will be kept.

>>> s.nlargest(3)
France    65000000
Italy     59000000
Malta       434000
dtype: int64

The `n` largest elements where ``n=3`` and keeping the last duplicates.
Brunei will be kept since it is the last with value 434000 based on
the index order.

>>> s.nlargest(3, keep='last')
France      65000000
Italy       59000000
Brunei        434000
dtype: int64

The `n` largest elements where ``n=3`` with all duplicates kept. Note
that the returned Series has five elements due to the three duplicates.

>>> s.nlargest(3, keep='all')
France      65000000
Italy       59000000
Malta         434000
Maldives      434000
Brunei        434000
dtype: int64

nsmallest(keep, **kwargs)[source]¶

Return the smallest n elements.

Parameters:	n (int, default 5) – Return this many ascending sorted values. keep ({'first', 'last', 'all'}, default 'first') – When there are duplicate values that cannot all fit in a DeferredSeries of n elements: `first` : return the first n occurrences in order of appearance. `last` : return the last n occurrences in reverse order of appearance. `all` : keep all occurrences. This can result in a DeferredSeries of size larger than n.
Returns:	The n smallest values in the DeferredSeries, sorted in increasing order.
Return type:	DeferredSeries

Differences from pandas

Only keep=False and keep="any" are supported. Other values of keep make this an order-sensitive operation. Note keep="any" is a Beam-specific option that guarantees only one duplicate will be kept, but unlike "first" and "last" it makes no guarantees about _which_ duplicate element is kept.

See also

DeferredSeries.nlargest(): Get the n largest elements.
DeferredSeries.sort_values(): Sort DeferredSeries by values.
DeferredSeries.head(): Return the first n rows.

Notes

Faster than .sort_values().head(n) for small n relative to the size of the DeferredSeries object.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> countries_population = {"Italy": 59000000, "France": 65000000,
...                         "Brunei": 434000, "Malta": 434000,
...                         "Maldives": 434000, "Iceland": 337000,
...                         "Nauru": 11300, "Tuvalu": 11300,
...                         "Anguilla": 11300, "Montserrat": 5200}
>>> s = pd.Series(countries_population)
>>> s
Italy       59000000
France      65000000
Brunei        434000
Malta         434000
Maldives      434000
Iceland       337000
Nauru          11300
Tuvalu         11300
Anguilla       11300
Montserrat      5200
dtype: int64

The `n` smallest elements where ``n=5`` by default.

>>> s.nsmallest()
Montserrat    5200
Nauru        11300
Tuvalu       11300
Anguilla     11300
Iceland     337000
dtype: int64

The `n` smallest elements where ``n=3``. Default `keep` value is
'first' so Nauru and Tuvalu will be kept.

>>> s.nsmallest(3)
Montserrat   5200
Nauru       11300
Tuvalu      11300
dtype: int64

The `n` smallest elements where ``n=3`` and keeping the last
duplicates. Anguilla and Tuvalu will be kept since they are the last
with value 11300 based on the index order.

>>> s.nsmallest(3, keep='last')
Montserrat   5200
Anguilla    11300
Tuvalu      11300
dtype: int64

The `n` smallest elements where ``n=3`` with all duplicates kept. Note
that the returned Series has four elements due to the three duplicates.

>>> s.nsmallest(3, keep='all')
Montserrat   5200
Nauru       11300
Tuvalu      11300
Anguilla    11300
dtype: int64

is_unique¶

Return boolean if values in the object are unique.

Returns:
Return type:	bool

Differences from pandas

This operation has no known divergences from the pandas API.

plot(**kwargs)¶

pandas.Series.plot is not supported in the Beam DataFrame API because it is a plotting tool.

For more information see {reason_data[‘url’]}.

pop(**kwargs)¶

pandas.Series.pop is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

rename_axis(**kwargs)¶

Set the name of the axis for the index or columns.

Parameters:	mapper (scalar, list-like, optional) – Value to set the axis name attribute. columns (index,) – A scalar, list-like, dict-like or functions transformations to apply to that axis’ values. Note that the `columns` parameter is not allowed if the object is a DeferredSeries. This parameter only apply for DeferredDataFrame type objects. Use either `mapper` and `axis` to specify the axis to target with `mapper`, or `index` and/or `columns`. Changed in version 0.24.0. axis ({0 or 'index', 1 or 'columns'}, default 0) – The axis to rename. copy (bool, default True) – Also copy underlying data. inplace (bool, default False) – Modifies the object directly, instead of creating a new DeferredSeries or DeferredDataFrame.
Returns:	The same type as the caller or None if `inplace=True`.
Return type:	DeferredSeries, DeferredDataFrame, or None

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.rename(): Alter DeferredSeries index labels or name.
DeferredDataFrame.rename(): Alter DeferredDataFrame index labels or name.
Index.rename(): Set new names on index.

Notes

DeferredDataFrame.rename_axis supports two calling conventions

(index=index_mapper, columns=columns_mapper, ...)
(mapper, axis={'index', 'columns'}, ...)

The first calling convention will only modify the names of the index and/or the names of the Index object that is the columns. In this case, the parameter copy is ignored.

The second calling convention will modify the names of the corresponding index if mapper is a list or a scalar. However, if mapper is dict-like or a function, it will use the deprecated behavior of modifying the axis labels.

We highly recommend using keyword arguments to clarify your intent.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

**Series**

>>> s = pd.Series(["dog", "cat", "monkey"])
>>> s
0       dog
1       cat
2    monkey
dtype: object
>>> s.rename_axis("animal")
animal
0    dog
1    cat
2    monkey
dtype: object

**DataFrame**

>>> df = pd.DataFrame({"num_legs": [4, 4, 2],
...                    "num_arms": [0, 0, 2]},
...                   ["dog", "cat", "monkey"])
>>> df
        num_legs  num_arms
dog            4         0
cat            4         0
monkey         2         2
>>> df = df.rename_axis("animal")
>>> df
        num_legs  num_arms
animal
dog            4         0
cat            4         0
monkey         2         2
>>> df = df.rename_axis("limbs", axis="columns")
>>> df
limbs   num_legs  num_arms
animal
dog            4         0
cat            4         0
monkey         2         2

**MultiIndex**

>>> df.index = pd.MultiIndex.from_product([['mammal'],
...                                        ['dog', 'cat', 'monkey']],
...                                       names=['type', 'name'])
>>> df
limbs          num_legs  num_arms
type   name
mammal dog            4         0
       cat            4         0
       monkey         2         2

>>> df.rename_axis(index={'type': 'class'})
limbs          num_legs  num_arms
class  name
mammal dog            4         0
       cat            4         0
       monkey         2         2

>>> df.rename_axis(columns=str.upper)
LIMBS          num_legs  num_arms
type   name
mammal dog            4         0
       cat            4         0
       monkey         2         2

replace(to_replace, value, limit, method, **kwargs)[source]¶

Return boolean if values in the object are unique.

Returns:
Return type:	bool

Differences from pandas

method is not supported in the Beam DataFrame API because it is order-sensitive. It cannot be specified.

If limit is specified this operation is not parallelizable.

round(**kwargs)¶

Round each value in a Series to the given number of decimals.

Parameters:	decimals (int, default 0) – Number of decimal places to round to. If decimals is negative, it specifies the number of positions to the left of the decimal point. *kwargs (args,) – Additional arguments and keywords have no effect but might be accepted for compatibility with NumPy.
Returns:	Rounded values of the DeferredSeries.
Return type:	DeferredSeries

Differences from pandas

This operation has no known divergences from the pandas API.

See also

numpy.around(): Round values of an np.array.
DeferredDataFrame.round(): Round values of a DeferredDataFrame.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series([0.1, 1.3, 2.7])
>>> s.round()
0    0.0
1    1.0
2    3.0
dtype: float64

take(**kwargs)¶: pandas.Series.take is not supported in the Beam DataFrame API because it is deprecated in pandas.

to_dict(**kwargs)¶

pandas.Series.to_dict is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

to_frame(**kwargs)¶

Convert Series to DataFrame.

Parameters:	name (object, default None) – The passed name should substitute for the series name (if it has one).
Returns:	DeferredDataFrame representation of DeferredSeries.
Return type:	DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series(["a", "b", "c"],
...               name="vals")
>>> s.to_frame()
  vals
0    a
1    b
2    c

unique(as_series=False)[source]¶

Return unique values of Series object.

Uniques are returned in order of appearance. Hash table-based unique, therefore does NOT sort.

Returns:	The unique values returned as a NumPy array. See Notes.
Return type:	ndarray or ExtensionArray

Differences from pandas

unique is not supported by default because it produces a non-deferred result: an ndarray. You can use the Beam-specific argument unique(as_series=True) to get the result as a DeferredSeries

See also

unique(): Top-level unique method for any 1-d array-like object.
Index.unique(): Return Index with unique values from an Index object.

Notes

Returns the unique values as a NumPy array. In case of an extension-array backed DeferredSeries, a new ExtensionArray of that type with just the unique values is returned. This includes

Categorical

Period

Datetime with Timezone

Interval

Sparse

IntegerNA

See Examples section.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> pd.Series([2, 1, 3, 3], name='A').unique()
array([2, 1, 3])

>>> pd.Series([pd.Timestamp('2016-01-01') for _ in range(3)]).unique()
array(['2016-01-01T00:00:00.000000000'], dtype='datetime64[ns]')

>>> pd.Series([pd.Timestamp('2016-01-01', tz='US/Eastern')
...            for _ in range(3)]).unique()
<DatetimeArray>
['2016-01-01 00:00:00-05:00']
Length: 1, dtype: datetime64[ns, US/Eastern]

An unordered Categorical will return categories in the order of
appearance.

>>> pd.Series(pd.Categorical(list('baabc'))).unique()
['b', 'a', 'c']
Categories (3, object): ['b', 'a', 'c']

An ordered Categorical preserves the category ordering.

>>> pd.Series(pd.Categorical(list('baabc'), categories=list('abc'),
...                          ordered=True)).unique()
['b', 'a', 'c']
Categories (3, object): ['a' < 'b' < 'c']

update(other)[source]¶

Modify Series in place using values from passed Series.

Uses non-NA values from passed Series to make updates. Aligns on index.

Parameters:	other (DeferredSeries, or object coercible into DeferredSeries) –

Differences from pandas

This operation has no known divergences from the pandas API.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series([1, 2, 3])
>>> s.update(pd.Series([4, 5, 6]))
>>> s
0    4
1    5
2    6
dtype: int64

>>> s = pd.Series(['a', 'b', 'c'])
>>> s.update(pd.Series(['d', 'e'], index=[0, 2]))
>>> s
0    d
1    b
2    e
dtype: object

>>> s = pd.Series([1, 2, 3])
>>> s.update(pd.Series([4, 5, 6, 7, 8]))
>>> s
0    4
1    5
2    6
dtype: int64

If ``other`` contains NaNs the corresponding values are not updated
in the original Series.

>>> s = pd.Series([1, 2, 3])
>>> s.update(pd.Series([4, np.nan, 6]))
>>> s
0    4
1    2
2    6
dtype: int64

``other`` can also be a non-Series object type
that is coercible into a Series

>>> s = pd.Series([1, 2, 3])
>>> s.update([4, np.nan, 6])
>>> s
0    4
1    2
2    6
dtype: int64

>>> s = pd.Series([1, 2, 3])
>>> s.update({1: 9})
>>> s
0    1
1    9
2    3
dtype: int64

unstack(**kwargs)¶

pandas.Series.unstack is not supported in the Beam DataFrame API because the columns in the output DataFrame depend on the data.

For more information see {reason_data[‘url’]}.

values¶

pandas.Series.values is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

view(**kwargs)¶: pandas.Series.view is not supported in the Beam DataFrame API because it relies on memory-sharing semantics that are not compatible with the Beam model.

str¶

Vectorized string functions for Series and Index.

NAs stay NA unless handled otherwise by a particular method. Patterned after Python’s string methods, with some inspiration from R’s stringr package.

Differences from pandas

This operation has no known divergences from the pandas API.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series(["A_Str_Series"])
>>> s
0    A_Str_Series
dtype: object

>>> s.str.split("_")
0    [A, Str, Series]
dtype: object

>>> s.str.replace("_", "")
0    AStrSeries
dtype: object

apply(**kwargs)¶

Invoke function on values of Series.

Can be ufunc (a NumPy function that applies to the entire Series) or a Python function that only works on single values.

Parameters:	func (function) – Python function or NumPy ufunc to apply. convert_dtype (bool, default True) – Try to find better dtype for elementwise function results. If False, leave as dtype=object. args (tuple) – Positional arguments passed to func after the series value. **kwds – Additional keyword arguments passed to func.
Returns:	If func returns a DeferredSeries object the result will be a DeferredDataFrame.
Return type:	DeferredSeries or DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.map(): For element-wise operations.
DeferredSeries.agg(): Only perform aggregating type operations.
DeferredSeries.transform(): Only perform transforming type operations.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Create a series with typical summer temperatures for each city.

>>> s = pd.Series([20, 21, 12],
...               index=['London', 'New York', 'Helsinki'])
>>> s
London      20
New York    21
Helsinki    12
dtype: int64

Square the values by defining a function and passing it as an
argument to ``apply()``.

>>> def square(x):
...     return x ** 2
>>> s.apply(square)
London      400
New York    441
Helsinki    144
dtype: int64

Square the values by passing an anonymous function as an
argument to ``apply()``.

>>> s.apply(lambda x: x ** 2)
London      400
New York    441
Helsinki    144
dtype: int64

Define a custom function that needs additional positional
arguments and pass these additional arguments using the
``args`` keyword.

>>> def subtract_custom_value(x, custom_value):
...     return x - custom_value

>>> s.apply(subtract_custom_value, args=(5,))
London      15
New York    16
Helsinki     7
dtype: int64

Define a custom function that takes keyword arguments
and pass these arguments to ``apply``.

>>> def add_custom_values(x, **kwargs):
...     for month in kwargs:
...         x += kwargs[month]
...     return x

>>> s.apply(add_custom_values, june=30, july=20, august=25)
London      95
New York    96
Helsinki    87
dtype: int64

Use a function from the Numpy library.

>>> s.apply(np.log)
London      2.995732
New York    3.044522
Helsinki    2.484907
dtype: float64

map(**kwargs)¶

Map values of Series according to input correspondence.

Used for substituting each value in a Series with another value, that may be derived from a function, a dict or a Series.

Parameters:	arg (function, collections.abc.Mapping subclass or DeferredSeries) – Mapping correspondence. na_action ({None, 'ignore'}, default None) – If ‘ignore’, propagate NaN values, without passing them to the mapping correspondence.
Returns:	Same index as caller.
Return type:	DeferredSeries

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.apply(): For applying more complex functions on a DeferredSeries.
DeferredDataFrame.apply(): Apply a function row-/column-wise.
DeferredDataFrame.applymap(): Apply a function elementwise on a whole DeferredDataFrame.

Notes

When arg is a dictionary, values in DeferredSeries that are not in the dictionary (as keys) are converted to NaN. However, if the dictionary is a dict subclass that defines __missing__ (i.e. provides a method for default values), then this default is used rather than NaN.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series(['cat', 'dog', np.nan, 'rabbit'])
>>> s
0      cat
1      dog
2      NaN
3   rabbit
dtype: object

``map`` accepts a ``dict`` or a ``Series``. Values that are not found
in the ``dict`` are converted to ``NaN``, unless the dict has a default
value (e.g. ``defaultdict``):

>>> s.map({'cat': 'kitten', 'dog': 'puppy'})
0   kitten
1    puppy
2      NaN
3      NaN
dtype: object

It also accepts a function:

>>> s.map('I am a {}'.format)
0       I am a cat
1       I am a dog
2       I am a nan
3    I am a rabbit
dtype: object

To avoid applying the function to missing values (and keep them as
``NaN``) ``na_action='ignore'`` can be used:

>>> s.map('I am a {}'.format, na_action='ignore')
0     I am a cat
1     I am a dog
2            NaN
3  I am a rabbit
dtype: object

abs(**kwargs)¶

Return a Series/DataFrame with absolute numeric value of each element.

This function only applies to elements that are all numeric.

Returns:	DeferredSeries/DeferredDataFrame containing the absolute value of each element.
Return type:	abs

Differences from pandas

This operation has no known divergences from the pandas API.

See also

numpy.absolute(): Calculate the absolute value element-wise.

Notes

For complex inputs, 1.2 + 1j, the absolute value is \(\sqrt{ a^2 + b^2 }\).

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Absolute numeric values in a Series.

>>> s = pd.Series([-1.10, 2, -3.33, 4])
>>> s.abs()
0    1.10
1    2.00
2    3.33
3    4.00
dtype: float64

Absolute numeric values in a Series with complex numbers.

>>> s = pd.Series([1.2 + 1j])
>>> s.abs()
0    1.56205
dtype: float64

Absolute numeric values in a Series with a Timedelta element.

>>> s = pd.Series([pd.Timedelta('1 days')])
>>> s.abs()
0   1 days
dtype: timedelta64[ns]

Select rows with data closest to certain value using argsort (from
`StackOverflow <https://stackoverflow.com/a/17758115>`__).

>>> df = pd.DataFrame({
...     'a': [4, 5, 6, 7],
...     'b': [10, 20, 30, 40],
...     'c': [100, 50, -30, -50]
... })
>>> df
     a    b    c
0    4   10  100
1    5   20   50
2    6   30  -30
3    7   40  -50
>>> df.loc[(df.c - 43).abs().argsort()]
     a    b    c
1    5   20   50
0    4   10  100
2    6   30  -30
3    7   40  -50

add(**kwargs)¶

Return Addition of series and other, element-wise (binary operator add).

Equivalent to series + other, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.radd(): Reverse of the Addition operator, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.add(b, fill_value=0)
a    2.0
b    1.0
c    1.0
d    1.0
e    NaN
dtype: float64

argsort(**kwargs)¶

‘argsort’ is not implemented yet.

If support for ‘argsort’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

asfreq(**kwargs)¶

‘asfreq’ is not implemented yet.

If support for ‘asfreq’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

asof(**kwargs)¶

‘asof’ is not implemented yet.

If support for ‘asof’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

astype(**kwargs)¶

Cast a pandas object to a specified dtype dtype.

Parameters:	dtype (data type, or dict of column name -> data type) – Use a numpy.dtype or Python type to cast entire pandas object to the same type. Alternatively, use {col: dtype, …}, where col is a column label and dtype is a numpy.dtype or Python type to cast one or more of the DeferredDataFrame’s columns to column-specific types. copy (bool, default True) – Return a copy when `copy=True` (be very careful setting `copy=False` as changes to values then may propagate to other pandas objects). errors ({'raise', 'ignore'}, default 'raise') – Control raising of exceptions on invalid data for provided dtype. `raise` : allow exceptions to be raised `ignore` : suppress exceptions. On error return original object.
Returns:	casted
Return type:	same type as caller

Differences from pandas

This operation has no known divergences from the pandas API.

See also

to_datetime(): Convert argument to datetime.
to_timedelta(): Convert argument to timedelta.
to_numeric(): Convert argument to a numeric type.
numpy.ndarray.astype(): Cast a numpy array to a specified type.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Create a DataFrame:

>>> d = {'col1': [1, 2], 'col2': [3, 4]}
>>> df = pd.DataFrame(data=d)
>>> df.dtypes
col1    int64
col2    int64
dtype: object

Cast all columns to int32:

>>> df.astype('int32').dtypes
col1    int32
col2    int32
dtype: object

Cast col1 to int32 using a dictionary:

>>> df.astype({'col1': 'int32'}).dtypes
col1    int32
col2    int64
dtype: object

Create a series:

>>> ser = pd.Series([1, 2], dtype='int32')
>>> ser
0    1
1    2
dtype: int32
>>> ser.astype('int64')
0    1
1    2
dtype: int64

Convert to categorical type:

>>> ser.astype('category')
0    1
1    2
dtype: category
Categories (2, int64): [1, 2]

Convert to ordered categorical type with custom ordering:

>>> cat_dtype = pd.api.types.CategoricalDtype(
...     categories=[2, 1], ordered=True)
>>> ser.astype(cat_dtype)
0    1
1    2
dtype: category
Categories (2, int64): [2 < 1]

Note that using ``copy=False`` and changing data on a new
pandas object may propagate changes:

>>> s1 = pd.Series([1, 2])
>>> s2 = s1.astype('int64', copy=False)
>>> s2[0] = 10
>>> s1  # note that s1[0] has changed too
0    10
1     2
dtype: int64

Create a series of dates:

>>> ser_date = pd.Series(pd.date_range('20200101', periods=3))
>>> ser_date
0   2020-01-01
1   2020-01-02
2   2020-01-03
dtype: datetime64[ns]

Datetimes are localized to UTC first before
converting to the specified timezone:

>>> ser_date.astype('datetime64[ns, US/Eastern]')
0   2019-12-31 19:00:00-05:00
1   2020-01-01 19:00:00-05:00
2   2020-01-02 19:00:00-05:00
dtype: datetime64[ns, US/Eastern]

at¶

‘at’ is not implemented yet.

If support for ‘at’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

at_time(**kwargs)¶

‘at_time’ is not implemented yet.

If support for ‘at_time’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

attrs¶: pandas.DataFrame.attrs is not supported in the Beam DataFrame API because it is experimental in pandas.

autocorr(**kwargs)¶

‘autocorr’ is not implemented yet.

If support for ‘autocorr’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

backfill(*args, **kwargs)¶

Synonym for DataFrame.fillna() with method='bfill'.

Returns:	Object with missing values filled or None if `inplace=True`.
Return type:	DeferredSeries/DeferredDataFrame or None

Differences from pandas

backfill is only supported for axis=”columns”. axis=”index” is order-sensitive.

between_time(**kwargs)¶

‘between_time’ is not implemented yet.

If support for ‘between_time’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

bfill(*args, **kwargs)¶

Synonym for DataFrame.fillna() with method='bfill'.

Returns:	Object with missing values filled or None if `inplace=True`.
Return type:	DeferredSeries/DeferredDataFrame or None

Differences from pandas

bfill is only supported for axis=”columns”. axis=”index” is order-sensitive.

bool()¶

Return the bool of a single element Series or DataFrame.

This must be a boolean scalar value, either True or False. It will raise a ValueError if the Series or DataFrame does not have exactly 1 element, or that element is not boolean (integer values 0 and 1 will also raise an exception).

Returns:	The value in the DeferredSeries or DeferredDataFrame.
Return type:	bool

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.astype(): Change the data type of a DeferredSeries, including to boolean.
DeferredDataFrame.astype(): Change the data type of a DeferredDataFrame, including to boolean.
numpy.bool_(): NumPy boolean data type, used by pandas for boolean values.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

The method will only work for single element objects with a boolean value:

>>> pd.Series([True]).bool()
True
>>> pd.Series([False]).bool()
False

>>> pd.DataFrame({'col': [True]}).bool()
True
>>> pd.DataFrame({'col': [False]}).bool()
False

cat¶

‘cat’ is not implemented yet.

If support for ‘cat’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

combine(**kwargs)¶

Perform column-wise combine with another DataFrame.

Combines a DataFrame with other DataFrame using func to element-wise combine columns. The row and column indexes of the resulting DataFrame will be the union of the two.

Parameters:	other (DeferredDataFrame) – The DeferredDataFrame to merge column-wise. func (function) – Function that takes two series as inputs and return a DeferredSeries or a scalar. Used to merge the two dataframes column by columns. fill_value (scalar value, default None) – The value to fill NaNs with prior to passing any column to the merge func. overwrite (bool, default True) – If True, columns in self that do not exist in other will be overwritten with NaNs.
Returns:	Combination of the provided DeferredDataFrames.
Return type:	DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.combine_first(): Combine two DeferredDataFrame objects and default to non-null values in frame calling the method.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Combine using a simple function that chooses the smaller column.

>>> df1 = pd.DataFrame({'A': [0, 0], 'B': [4, 4]})
>>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})
>>> take_smaller = lambda s1, s2: s1 if s1.sum() < s2.sum() else s2
>>> df1.combine(df2, take_smaller)
   A  B
0  0  3
1  0  3

Example using a true element-wise combine function.

>>> df1 = pd.DataFrame({'A': [5, 0], 'B': [2, 4]})
>>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})
>>> df1.combine(df2, np.minimum)
   A  B
0  1  2
1  0  3

Using `fill_value` fills Nones prior to passing the column to the
merge function.

>>> df1 = pd.DataFrame({'A': [0, 0], 'B': [None, 4]})
>>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})
>>> df1.combine(df2, take_smaller, fill_value=-5)
   A    B
0  0 -5.0
1  0  4.0

However, if the same element in both dataframes is None, that None
is preserved

>>> df1 = pd.DataFrame({'A': [0, 0], 'B': [None, 4]})
>>> df2 = pd.DataFrame({'A': [1, 1], 'B': [None, 3]})
>>> df1.combine(df2, take_smaller, fill_value=-5)
    A    B
0  0 -5.0
1  0  3.0

Example that demonstrates the use of `overwrite` and behavior when
the axis differ between the dataframes.

>>> df1 = pd.DataFrame({'A': [0, 0], 'B': [4, 4]})
>>> df2 = pd.DataFrame({'B': [3, 3], 'C': [-10, 1], }, index=[1, 2])
>>> df1.combine(df2, take_smaller)
     A    B     C
0  NaN  NaN   NaN
1  NaN  3.0 -10.0
2  NaN  3.0   1.0

>>> df1.combine(df2, take_smaller, overwrite=False)
     A    B     C
0  0.0  NaN   NaN
1  0.0  3.0 -10.0
2  NaN  3.0   1.0

Demonstrating the preference of the passed in dataframe.

>>> df2 = pd.DataFrame({'B': [3, 3], 'C': [1, 1], }, index=[1, 2])
>>> df2.combine(df1, take_smaller)
   A    B   C
0  0.0  NaN NaN
1  0.0  3.0 NaN
2  NaN  3.0 NaN

>>> df2.combine(df1, take_smaller, overwrite=False)
     A    B   C
0  0.0  NaN NaN
1  0.0  3.0 1.0
2  NaN  3.0 1.0

combine_first(**kwargs)¶

Update null elements with value in the same location in other.

Combine two DataFrame objects by filling null values in one DataFrame with non-null values from other DataFrame. The row and column indexes of the resulting DataFrame will be the union of the two.

Parameters:	other (DeferredDataFrame) – Provided DeferredDataFrame to use to fill null values.
Returns:
Return type:	DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.combine(): Perform series-wise operation on two DeferredDataFrames using a given function.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df1 = pd.DataFrame({'A': [None, 0], 'B': [None, 4]})
>>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})
>>> df1.combine_first(df2)
     A    B
0  1.0  3.0
1  0.0  4.0

Null values still persist if the location of that null value
does not exist in `other`

>>> df1 = pd.DataFrame({'A': [None, 0], 'B': [4, None]})
>>> df2 = pd.DataFrame({'B': [3, 3], 'C': [1, 1]}, index=[1, 2])
>>> df1.combine_first(df2)
     A    B    C
0  NaN  4.0  NaN
1  0.0  3.0  1.0
2  NaN  3.0  1.0

compare(**kwargs)¶

‘compare’ is not implemented yet.

If support for ‘compare’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

convert_dtypes(**kwargs)¶

‘convert_dtypes’ is not implemented yet.

If support for ‘convert_dtypes’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

copy(**kwargs)¶

Make a copy of this object’s indices and data.

When deep=True (default), a new object will be created with a copy of the calling object’s data and indices. Modifications to the data or indices of the copy will not be reflected in the original object (see notes below).

When deep=False, a new object will be created without copying the calling object’s data or index (only references to the data and index are copied). Any changes to the data of the original will be reflected in the shallow copy (and vice versa).

Parameters:	deep (bool, default True) – Make a deep copy, including a copy of the data and the indices. With `deep=False` neither the indices nor the data are copied.
Returns:	copy – Object type matches caller.
Return type:	DeferredSeries or DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

Notes

When deep=True, data is copied but actual Python objects will not be copied recursively, only the reference to the object. This is in contrast to copy.deepcopy in the Standard Library, which recursively copies object data (see examples below).

While Index objects are copied when deep=True, the underlying numpy array is not copied for performance reasons. Since Index is immutable, the underlying data can be safely shared and a copy is not needed.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series([1, 2], index=["a", "b"])
>>> s
a    1
b    2
dtype: int64

>>> s_copy = s.copy()
>>> s_copy
a    1
b    2
dtype: int64

**Shallow copy versus default (deep) copy:**

>>> s = pd.Series([1, 2], index=["a", "b"])
>>> deep = s.copy()
>>> shallow = s.copy(deep=False)

Shallow copy shares data and index with original.

>>> s is shallow
False
>>> s.values is shallow.values and s.index is shallow.index
True

Deep copy has own copy of data and index.

>>> s is deep
False
>>> s.values is deep.values or s.index is deep.index
False

Updates to the data shared by shallow copy and original is reflected
in both; deep copy remains unchanged.

>>> s[0] = 3
>>> shallow[1] = 4
>>> s
a    3
b    4
dtype: int64
>>> shallow
a    3
b    4
dtype: int64
>>> deep
a    1
b    2
dtype: int64

Note that when copying an object containing Python objects, a deep copy
will copy the data, but will not do so recursively. Updating a nested
data object will be reflected in the deep copy.

>>> s = pd.Series([[1, 2], [3, 4]])
>>> deep = s.copy()
>>> s[0][0] = 10
>>> s
0    [10, 2]
1     [3, 4]
dtype: object
>>> deep
0    [10, 2]
1     [3, 4]
dtype: object

div(**kwargs)¶

Return Floating division of series and other, element-wise (binary operator truediv).

Equivalent to series / other, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.rtruediv(): Reverse of the Floating division operator, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.divide(b, fill_value=0)
a    1.0
b    inf
c    inf
d    0.0
e    NaN
dtype: float64

divide(**kwargs)¶

‘divide’ is not implemented yet.

If support for ‘divide’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

divmod(**kwargs)¶

Return Integer division and modulo of series and other, element-wise (binary operator divmod).

Equivalent to divmod(series, other), but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	2-Tuple of DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.rdivmod(): Reverse of the Integer division and modulo operator, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.divmod(b, fill_value=0)
(a    1.0
 b    NaN
 c    NaN
 d    0.0
 e    NaN
 dtype: float64,
 a    0.0
 b    NaN
 c    NaN
 d    0.0
 e    NaN
 dtype: float64)

drop(labels, axis, index, columns, errors, **kwargs)¶

Drop specified labels from rows or columns.

Remove rows or columns by specifying label names and corresponding axis, or by specifying directly index or column names. When using a multi-index, labels on different levels can be removed by specifying the level.

Parameters:	labels (single label or list-like) – Index or column labels to drop. axis ({0 or 'index', 1 or 'columns'}, default 0) – Whether to drop labels from the index (0 or ‘index’) or columns (1 or ‘columns’). index (single label or list-like) – Alternative to specifying axis (`labels, axis=0` is equivalent to `index=labels`). columns (single label or list-like) – Alternative to specifying axis (`labels, axis=1` is equivalent to `columns=labels`). level (int or level name, optional) – For MultiIndex, level from which the labels will be removed. inplace (bool, default False) – If False, return a copy. Otherwise, do operation inplace and return None. errors ({'ignore', 'raise'}, default 'raise') – If ‘ignore’, suppress error and only existing labels are dropped.
Returns:	DeferredDataFrame without the removed index or column labels or None if `inplace=True`.
Return type:	DeferredDataFrame or None
Raises:	`KeyError` – If any of the labels is not found in the selected axis.

Differences from pandas

drop is not parallelizable when dropping from the index and errors="raise" is specified. It requires collecting all data on a single node in order to detect if one of the index values is missing.

See also

DeferredDataFrame.loc(): Label-location based indexer for selection by label.
DeferredDataFrame.dropna(): Return DeferredDataFrame with labels on given axis omitted where (all or any) data are missing.
DeferredDataFrame.drop_duplicates(): Return DeferredDataFrame with duplicate rows removed, optionally only considering certain columns.
DeferredSeries.drop(): Return DeferredSeries with specified index labels removed.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame(np.arange(12).reshape(3, 4),
...                   columns=['A', 'B', 'C', 'D'])
>>> df
   A  B   C   D
0  0  1   2   3
1  4  5   6   7
2  8  9  10  11

Drop columns

>>> df.drop(['B', 'C'], axis=1)
   A   D
0  0   3
1  4   7
2  8  11

>>> df.drop(columns=['B', 'C'])
   A   D
0  0   3
1  4   7
2  8  11

Drop a row by index

>>> df.drop([0, 1])
   A  B   C   D
2  8  9  10  11

Drop columns and/or rows of MultiIndex DataFrame

>>> midx = pd.MultiIndex(levels=[['lama', 'cow', 'falcon'],
...                              ['speed', 'weight', 'length']],
...                      codes=[[0, 0, 0, 1, 1, 1, 2, 2, 2],
...                             [0, 1, 2, 0, 1, 2, 0, 1, 2]])
>>> df = pd.DataFrame(index=midx, columns=['big', 'small'],
...                   data=[[45, 30], [200, 100], [1.5, 1], [30, 20],
...                         [250, 150], [1.5, 0.8], [320, 250],
...                         [1, 0.8], [0.3, 0.2]])
>>> df
                big     small
lama    speed   45.0    30.0
        weight  200.0   100.0
        length  1.5     1.0
cow     speed   30.0    20.0
        weight  250.0   150.0
        length  1.5     0.8
falcon  speed   320.0   250.0
        weight  1.0     0.8
        length  0.3     0.2

>>> df.drop(index='cow', columns='small')
                big
lama    speed   45.0
        weight  200.0
        length  1.5
falcon  speed   320.0
        weight  1.0
        length  0.3

>>> df.drop(index='length', level=1)
                big     small
lama    speed   45.0    30.0
        weight  200.0   100.0
cow     speed   30.0    20.0
        weight  250.0   150.0
falcon  speed   320.0   250.0
        weight  1.0     0.8

droplevel(level, axis)¶

Return DataFrame with requested index / column level(s) removed.

New in version 0.24.0.

Parameters:	level (int, str, or list-like) – If a string is given, must be the name of a level If list-like, elements must be names or positional indexes of levels. axis ({0 or 'index', 1 or 'columns'}, default 0) – Axis along which the level(s) is removed: 0 or ‘index’: remove level(s) in column. 1 or ‘columns’: remove level(s) in row.
Returns:	DeferredDataFrame with requested index / column level(s) removed.
Return type:	DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame([
...     [1, 2, 3, 4],
...     [5, 6, 7, 8],
...     [9, 10, 11, 12]
... ]).set_index([0, 1]).rename_axis(['a', 'b'])

>>> df.columns = pd.MultiIndex.from_tuples([
...     ('c', 'e'), ('d', 'f')
... ], names=['level_1', 'level_2'])

>>> df
level_1   c   d
level_2   e   f
a b
1 2      3   4
5 6      7   8
9 10    11  12

>>> df.droplevel('a')
level_1   c   d
level_2   e   f
b
2        3   4
6        7   8
10      11  12

>>> df.droplevel('level_2', axis=1)
level_1   c   d
a b
1 2      3   4
5 6      7   8
9 10    11  12

dt¶

‘dt’ is not implemented yet.

If support for ‘dt’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

empty¶

eq(**kwargs)¶

Return Equal to of series and other, element-wise (binary operator eq).

Equivalent to series == other, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

This operation has no known divergences from the pandas API.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.eq(b, fill_value=0)
a     True
b    False
c    False
d    False
e    False
dtype: bool

equals(other)¶

Test whether two objects contain the same elements.

This function allows two Series or DataFrames to be compared against each other to see if they have the same shape and elements. NaNs in the same location are considered equal.

The row/column index do not need to have the same type, as long as the values are considered equal. Corresponding columns must be of the same dtype.

Parameters:	other (DeferredSeries or DeferredDataFrame) – The other DeferredSeries or DeferredDataFrame to be compared with the first.
Returns:	True if all elements are the same in both objects, False otherwise.
Return type:	bool

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.eq(): Compare two DeferredSeries objects of the same length and return a DeferredSeries where each element is True if the element in each DeferredSeries is equal, False otherwise.
DeferredDataFrame.eq(): Compare two DeferredDataFrame objects of the same shape and return a DeferredDataFrame where each element is True if the respective element in each DeferredDataFrame is equal, False otherwise.
testing.assert_series_equal(): Raises an AssertionError if left and right are not equal. Provides an easy interface to ignore inequality in dtypes, indexes and precision among others.
testing.assert_frame_equal(): Like assert_series_equal, but targets DeferredDataFrames.
numpy.array_equal(): Return True if two arrays have the same shape and elements, False otherwise.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame({1: [10], 2: [20]})
>>> df
    1   2
0  10  20

DataFrames df and exactly_equal have the same types and values for
their elements and column labels, which will return True.

>>> exactly_equal = pd.DataFrame({1: [10], 2: [20]})
>>> exactly_equal
    1   2
0  10  20
>>> df.equals(exactly_equal)
True

DataFrames df and different_column_type have the same element
types and values, but have different types for the column labels,
which will still return True.

>>> different_column_type = pd.DataFrame({1.0: [10], 2.0: [20]})
>>> different_column_type
   1.0  2.0
0   10   20
>>> df.equals(different_column_type)
True

DataFrames df and different_data_type have different types for the
same values for their elements, and will return False even though
their column labels are the same values and types.

>>> different_data_type = pd.DataFrame({1: [10.0], 2: [20.0]})
>>> different_data_type
      1     2
0  10.0  20.0
>>> df.equals(different_data_type)
False

ewm(**kwargs)¶

‘ewm’ is not implemented yet.

If support for ‘ewm’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

expanding(**kwargs)¶

‘expanding’ is not implemented yet.

If support for ‘expanding’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

explode(**kwargs)¶

‘explode’ is not implemented yet.

If support for ‘explode’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

factorize(**kwargs)¶

‘factorize’ is not implemented yet.

If support for ‘factorize’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

ffill(*args, **kwargs)¶

Synonym for DataFrame.fillna() with method='ffill'.

Returns:	Object with missing values filled or None if `inplace=True`.
Return type:	DeferredSeries/DeferredDataFrame or None

Differences from pandas

ffill is only supported for axis=”columns”. axis=”index” is order-sensitive.

fillna(value, method, axis, limit, **kwargs)¶

Fill NA/NaN values using the specified method.

Parameters:	value (scalar, dict, DeferredSeries, or DeferredDataFrame) – Value to use to fill holes (e.g. 0), alternately a dict/DeferredSeries/DeferredDataFrame of values specifying which value to use for each index (for a DeferredSeries) or column (for a DeferredDataFrame). Values not in the dict/DeferredSeries/DeferredDataFrame will not be filled. This value cannot be a list. method ({'backfill', 'bfill', 'pad', 'ffill', None}, default None) – Method to use for filling holes in reindexed DeferredSeries pad / ffill: propagate last valid observation forward to next valid backfill / bfill: use next valid observation to fill gap. axis ({0 or 'index', 1 or 'columns'}) – Axis along which to fill missing values. inplace (bool, default False) – If True, fill in-place. Note: this will modify any other views on this object (e.g., a no-copy slice for a column in a DeferredDataFrame). limit (int, default None) – If method is specified, this is the maximum number of consecutive NaN values to forward/backward fill. In other words, if there is a gap with more than this number of consecutive NaNs, it will only be partially filled. If method is not specified, this is the maximum number of entries along the entire axis where NaNs will be filled. Must be greater than 0 if not None. downcast (dict, default is None) – A dict of item->dtype of what to downcast if possible, or the string ‘infer’ which will try to downcast to an appropriate equal type (e.g. float64 to int64 if possible).
Returns:	Object with missing values filled or None if `inplace=True`.
Return type:	DeferredDataFrame or None

Differences from pandas

When axis="index", both method and limit must be None. otherwise this operation is order-sensitive.

See also

interpolate(): Fill NaN values using interpolation.
reindex(): Conform object to new index.
asfreq(): Convert TimeDeferredSeries to specified frequency.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame([[np.nan, 2, np.nan, 0],
...                    [3, 4, np.nan, 1],
...                    [np.nan, np.nan, np.nan, 5],
...                    [np.nan, 3, np.nan, 4]],
...                   columns=list('ABCD'))
>>> df
     A    B   C  D
0  NaN  2.0 NaN  0
1  3.0  4.0 NaN  1
2  NaN  NaN NaN  5
3  NaN  3.0 NaN  4

Replace all NaN elements with 0s.

>>> df.fillna(0)
    A   B   C   D
0   0.0 2.0 0.0 0
1   3.0 4.0 0.0 1
2   0.0 0.0 0.0 5
3   0.0 3.0 0.0 4

We can also propagate non-null values forward or backward.

>>> df.fillna(method='ffill')
    A   B   C   D
0   NaN 2.0 NaN 0
1   3.0 4.0 NaN 1
2   3.0 4.0 NaN 5
3   3.0 3.0 NaN 4

Replace all NaN elements in column 'A', 'B', 'C', and 'D', with 0, 1,
2, and 3 respectively.

>>> values = {'A': 0, 'B': 1, 'C': 2, 'D': 3}
>>> df.fillna(value=values)
    A   B   C   D
0   0.0 2.0 2.0 0
1   3.0 4.0 2.0 1
2   0.0 1.0 2.0 5
3   0.0 3.0 2.0 4

Only replace the first NaN element.

>>> df.fillna(value=values, limit=1)
    A   B   C   D
0   0.0 2.0 2.0 0
1   3.0 4.0 NaN 1
2   NaN 1.0 NaN 5
3   NaN 3.0 NaN 4

first_valid_index(**kwargs)¶

‘first_valid_index’ is not implemented yet.

If support for ‘first_valid_index’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

flags¶

‘flags’ is not implemented yet.

If support for ‘flags’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

floordiv(**kwargs)¶

Return Integer division of series and other, element-wise (binary operator floordiv).

Equivalent to series // other, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.rfloordiv(): Reverse of the Integer division operator, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.floordiv(b, fill_value=0)
a    1.0
b    NaN
c    NaN
d    0.0
e    NaN
dtype: float64

ge(**kwargs)¶

Return Greater than or equal to of series and other, element-wise (binary operator ge).

Equivalent to series >= other, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

This operation has no known divergences from the pandas API.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> a = pd.Series([1, 1, 1, np.nan, 1], index=['a', 'b', 'c', 'd', 'e'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
e    1.0
dtype: float64
>>> b = pd.Series([0, 1, 2, np.nan, 1], index=['a', 'b', 'c', 'd', 'f'])
>>> b
a    0.0
b    1.0
c    2.0
d    NaN
f    1.0
dtype: float64
>>> a.ge(b, fill_value=0)
a     True
b     True
c    False
d    False
e     True
f    False
dtype: bool

get(**kwargs)¶

‘get’ is not implemented yet.

If support for ‘get’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

groupby(by, level, axis, as_index, group_keys, **kwargs)¶

Group DataFrame using a mapper or by a Series of columns.

A groupby operation involves some combination of splitting the object, applying a function, and combining the results. This can be used to group large amounts of data and compute operations on these groups.

Parameters:	by (mapping, function, label, or list of labels) – Used to determine the groups for the groupby. If `by` is a function, it’s called on each value of the object’s index. If a dict or DeferredSeries is passed, the DeferredSeries or dict VALUES will be used to determine the groups (the DeferredSeries’ values are first aligned; see `.align()` method). If an ndarray is passed, the values are used as-is to determine the groups. A label or list of labels may be passed to group by the columns in `self`. Notice that a tuple is interpreted as a (single) key. axis ({0 or 'index', 1 or 'columns'}, default 0) – Split along rows (0) or columns (1). level (int, level name, or sequence of such, default None) – If the axis is a MultiIndex (hierarchical), group by a particular level or levels. as_index (bool, default True) – For aggregated output, return object with group labels as the index. Only relevant for DeferredDataFrame input. as_index=False is effectively “SQL-style” grouped output. sort (bool, default True) – Sort group keys. Get better performance by turning this off. Note this does not influence the order of observations within each group. Groupby preserves the order of rows within each group. group_keys (bool, default True) – When calling apply, add group keys to index to identify pieces. squeeze (bool, default False) – Reduce the dimensionality of the return type if possible, otherwise return a consistent type. Deprecated since version 1.1.0. observed (bool, default False) – This only applies if any of the groupers are Categoricals. If True: only show observed values for categorical groupers. If False: show all values for categorical groupers. dropna (bool, default True) – If True, and if group keys contain NA values, NA values together with row/column will be dropped. If False, NA values will also be treated as the key in groups New in version 1.1.0.
Returns:	Returns a groupby object that contains information about the groups.
Return type:	DeferredDataFrameGroupBy

Differences from pandas

as_index and group_keys must both be True.

Aggregations grouping by a categorical column with observed=False set are not currently parallelizable (BEAM-11190<https://issues.apache.org/jira/browse/BEAM-11190>_).

See also

resample(): Convenience method for frequency conversion and resampling of time series.

Notes

See the user guide for more.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'Animal': ['Falcon', 'Falcon',
...                               'Parrot', 'Parrot'],
...                    'Max Speed': [380., 370., 24., 26.]})
>>> df
   Animal  Max Speed
0  Falcon      380.0
1  Falcon      370.0
2  Parrot       24.0
3  Parrot       26.0
>>> df.groupby(['Animal']).mean()
        Max Speed
Animal
Falcon      375.0
Parrot       25.0

**Hierarchical Indexes**

We can groupby different levels of a hierarchical index
using the `level` parameter:

>>> arrays = [['Falcon', 'Falcon', 'Parrot', 'Parrot'],
...           ['Captive', 'Wild', 'Captive', 'Wild']]
>>> index = pd.MultiIndex.from_arrays(arrays, names=('Animal', 'Type'))
>>> df = pd.DataFrame({'Max Speed': [390., 350., 30., 20.]},
...                   index=index)
>>> df
                Max Speed
Animal Type
Falcon Captive      390.0
       Wild         350.0
Parrot Captive       30.0
       Wild          20.0
>>> df.groupby(level=0).mean()
        Max Speed
Animal
Falcon      370.0
Parrot       25.0
>>> df.groupby(level="Type").mean()
         Max Speed
Type
Captive      210.0
Wild         185.0

We can also choose to include NA in group keys or not by setting
`dropna` parameter, the default setting is `True`:

>>> l = [[1, 2, 3], [1, None, 4], [2, 1, 3], [1, 2, 2]]
>>> df = pd.DataFrame(l, columns=["a", "b", "c"])

>>> df.groupby(by=["b"]).sum()
    a   c
b
1.0 2   3
2.0 2   5

>>> df.groupby(by=["b"], dropna=False).sum()
    a   c
b
1.0 2   3
2.0 2   5
NaN 1   4

>>> l = [["a", 12, 12], [None, 12.3, 33.], ["b", 12.3, 123], ["a", 1, 1]]
>>> df = pd.DataFrame(l, columns=["a", "b", "c"])

>>> df.groupby(by="a").sum()
    b     c
a
a   13.0   13.0
b   12.3  123.0

>>> df.groupby(by="a", dropna=False).sum()
    b     c
a
a   13.0   13.0
b   12.3  123.0
NaN 12.3   33.0

gt(**kwargs)¶

Return Greater than of series and other, element-wise (binary operator gt).

Equivalent to series > other, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

This operation has no known divergences from the pandas API.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> a = pd.Series([1, 1, 1, np.nan, 1], index=['a', 'b', 'c', 'd', 'e'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
e    1.0
dtype: float64
>>> b = pd.Series([0, 1, 2, np.nan, 1], index=['a', 'b', 'c', 'd', 'f'])
>>> b
a    0.0
b    1.0
c    2.0
d    NaN
f    1.0
dtype: float64
>>> a.gt(b, fill_value=0)
a     True
b    False
c    False
d    False
e     True
f    False
dtype: bool

hasnans¶

‘hasnans’ is not implemented yet.

If support for ‘hasnans’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

hist(**kwargs)¶

pandas.DataFrame.hist is not supported in the Beam DataFrame API because it is a plotting tool.

For more information see {reason_data[‘url’]}.

iat¶

‘iat’ is not implemented yet.

If support for ‘iat’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

idxmax(**kwargs)¶

‘idxmax’ is not implemented yet.

If support for ‘idxmax’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

idxmin(**kwargs)¶

‘idxmin’ is not implemented yet.

If support for ‘idxmin’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

iloc¶

‘iloc’ is not implemented yet.

If support for ‘iloc’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

index¶

The index (row labels) of the DataFrame.

Differences from pandas

This operation has no known divergences from the pandas API.

infer_objects(**kwargs)¶

‘infer_objects’ is not implemented yet.

If support for ‘infer_objects’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

is_monotonic¶

‘is_monotonic’ is not implemented yet.

If support for ‘is_monotonic’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

is_monotonic_decreasing¶

‘is_monotonic_decreasing’ is not implemented yet.

If support for ‘is_monotonic_decreasing’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

is_monotonic_increasing¶

‘is_monotonic_increasing’ is not implemented yet.

If support for ‘is_monotonic_increasing’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

isin(**kwargs)¶

Whether each element in the DataFrame is contained in values.

Parameters:	values (iterable, DeferredSeries, DeferredDataFrame or dict) – The result will only be true at a location if all the labels match. If values is a DeferredSeries, that’s the index. If values is a dict, the keys must be the column names, which must match. If values is a DeferredDataFrame, then both the index and column labels must match.
Returns:	DeferredDataFrame of booleans showing whether each element in the DeferredDataFrame is contained in values.
Return type:	DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.eq(): Equality test for DeferredDataFrame.
DeferredSeries.isin(): Equivalent method on DeferredSeries.
DeferredSeries.str.contains(): Test if pattern or regex is contained within a string of a DeferredSeries or Index.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame({'num_legs': [2, 4], 'num_wings': [2, 0]},
...                   index=['falcon', 'dog'])
>>> df
        num_legs  num_wings
falcon         2          2
dog            4          0

When ``values`` is a list check whether every value in the DataFrame
is present in the list (which animals have 0 or 2 legs or wings)

>>> df.isin([0, 2])
        num_legs  num_wings
falcon      True       True
dog        False       True

When ``values`` is a dict, we can pass values to check for each
column separately:

>>> df.isin({'num_wings': [0, 3]})
        num_legs  num_wings
falcon     False      False
dog        False       True

When ``values`` is a Series or DataFrame the index and column must
match. Note that 'falcon' does not match based on the number of legs
in df2.

>>> other = pd.DataFrame({'num_legs': [8, 2], 'num_wings': [0, 2]},
...                      index=['spider', 'falcon'])
>>> df.isin(other)
        num_legs  num_wings
falcon      True       True
dog        False      False

item(**kwargs)¶

‘item’ is not implemented yet.

If support for ‘item’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

kurt(**kwargs)¶

‘kurt’ is not implemented yet.

If support for ‘kurt’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

kurtosis(**kwargs)¶

‘kurtosis’ is not implemented yet.

If support for ‘kurtosis’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

last_valid_index(**kwargs)¶

‘last_valid_index’ is not implemented yet.

If support for ‘last_valid_index’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

le(**kwargs)¶

Return Less than or equal to of series and other, element-wise (binary operator le).

Equivalent to series <= other, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

This operation has no known divergences from the pandas API.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> a = pd.Series([1, 1, 1, np.nan, 1], index=['a', 'b', 'c', 'd', 'e'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
e    1.0
dtype: float64
>>> b = pd.Series([0, 1, 2, np.nan, 1], index=['a', 'b', 'c', 'd', 'f'])
>>> b
a    0.0
b    1.0
c    2.0
d    NaN
f    1.0
dtype: float64
>>> a.le(b, fill_value=0)
a    False
b     True
c     True
d    False
e    False
f     True
dtype: bool

loc¶

‘loc’ is not implemented yet.

If support for ‘loc’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

lt(**kwargs)¶

Return Less than of series and other, element-wise (binary operator lt).

Equivalent to series < other, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

This operation has no known divergences from the pandas API.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> a = pd.Series([1, 1, 1, np.nan, 1], index=['a', 'b', 'c', 'd', 'e'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
e    1.0
dtype: float64
>>> b = pd.Series([0, 1, 2, np.nan, 1], index=['a', 'b', 'c', 'd', 'f'])
>>> b
a    0.0
b    1.0
c    2.0
d    NaN
f    1.0
dtype: float64
>>> a.lt(b, fill_value=0)
a    False
b    False
c     True
d    False
e    False
f     True
dtype: bool

mad(**kwargs)¶

‘mad’ is not implemented yet.

If support for ‘mad’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

mask(cond, **kwargs)¶

Replace values where the condition is True.

Parameters:	cond (bool DeferredSeries/DeferredDataFrame, array-like, or callable) – Where cond is False, keep the original value. Where True, replace with corresponding value from other. If cond is callable, it is computed on the DeferredSeries/DeferredDataFrame and should return boolean DeferredSeries/DeferredDataFrame or array. The callable must not change input DeferredSeries/DeferredDataFrame (though pandas doesn’t check it). other (scalar, DeferredSeries/DeferredDataFrame, or callable) – Entries where cond is True are replaced with corresponding value from other. If other is callable, it is computed on the DeferredSeries/DeferredDataFrame and should return scalar or DeferredSeries/DeferredDataFrame. The callable must not change input DeferredSeries/DeferredDataFrame (though pandas doesn’t check it). inplace (bool, default False) – Whether to perform the operation in place on the data. axis (int, default None) – Alignment axis if needed. level (int, default None) – Alignment level if needed. errors (str, {'raise', 'ignore'}, default 'raise') – Note that currently this parameter won’t affect the results and will always coerce to a suitable dtype. ’raise’ : allow exceptions to be raised. ’ignore’ : suppress exceptions. On error return original object. try_cast (bool, default False) – Try to cast the result back to the input type (if possible).
Returns:
Return type:	Same type as caller or None if `inplace=True`.

Differences from pandas

mask is not parallelizable when errors="ignore" is specified.

See also

DeferredDataFrame.where(): Return an object of same shape as self.

Notes

The mask method is an application of the if-then idiom. For each element in the calling DeferredDataFrame, if cond is False the element is used; otherwise the corresponding element from the DeferredDataFrame other is used.

The signature for DeferredDataFrame.where() differs from numpy.where(). Roughly df1.where(m, df2) is equivalent to np.where(m, df1, df2).

For further details and examples see the mask documentation in indexing.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> s = pd.Series(range(5))
>>> s.where(s > 0)
0    NaN
1    1.0
2    2.0
3    3.0
4    4.0
dtype: float64
>>> s.mask(s > 0)
0    0.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64

>>> s.where(s > 1, 10)
0    10
1    10
2    2
3    3
4    4
dtype: int64
>>> s.mask(s > 1, 10)
0     0
1     1
2    10
3    10
4    10
dtype: int64

>>> df = pd.DataFrame(np.arange(10).reshape(-1, 2), columns=['A', 'B'])
>>> df
   A  B
0  0  1
1  2  3
2  4  5
3  6  7
4  8  9
>>> m = df % 3 == 0
>>> df.where(m, -df)
   A  B
0  0 -1
1 -2  3
2 -4 -5
3  6 -7
4 -8  9
>>> df.where(m, -df) == np.where(m, df, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True
>>> df.where(m, -df) == df.mask(~m, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True

mod(**kwargs)¶

Return Modulo of series and other, element-wise (binary operator mod).

Equivalent to series % other, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.rmod(): Reverse of the Modulo operator, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.mod(b, fill_value=0)
a    0.0
b    NaN
c    NaN
d    0.0
e    NaN
dtype: float64

mode(**kwargs)¶

‘mode’ is not implemented yet.

If support for ‘mode’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

mul(**kwargs)¶

Return Multiplication of series and other, element-wise (binary operator mul).

Equivalent to series * other, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.rmul(): Reverse of the Multiplication operator, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.multiply(b, fill_value=0)
a    1.0
b    0.0
c    0.0
d    0.0
e    NaN
dtype: float64

multiply(**kwargs)¶

Return Multiplication of series and other, element-wise (binary operator mul).

Equivalent to series * other, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.rmul(): Reverse of the Multiplication operator, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.multiply(b, fill_value=0)
a    1.0
b    0.0
c    0.0
d    0.0
e    NaN
dtype: float64

nbytes¶

‘nbytes’ is not implemented yet.

If support for ‘nbytes’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

ndim¶

Return an int representing the number of axes / array dimensions.

Return 1 if Series. Otherwise return 2 if DataFrame.

Differences from pandas

This operation has no known divergences from the pandas API.

See also

ndarray.ndim: Number of array dimensions.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series({'a': 1, 'b': 2, 'c': 3})
>>> s.ndim
1

>>> df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
>>> df.ndim
2

ne(**kwargs)¶

Return Not equal to of series and other, element-wise (binary operator ne).

Equivalent to series != other, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

This operation has no known divergences from the pandas API.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.ne(b, fill_value=0)
a    False
b     True
c     True
d     True
e     True
dtype: bool

nunique(**kwargs)¶

‘nunique’ is not implemented yet.

If support for ‘nunique’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

pad(*args, **kwargs)¶

Synonym for DataFrame.fillna() with method='ffill'.

Returns:	Object with missing values filled or None if `inplace=True`.
Return type:	DeferredSeries/DeferredDataFrame or None

Differences from pandas

pad is only supported for axis=”columns”. axis=”index” is order-sensitive.

pct_change(**kwargs)¶

‘pct_change’ is not implemented yet.

If support for ‘pct_change’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

pipe(**kwargs)¶

‘pipe’ is not implemented yet.

If support for ‘pipe’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

pow(**kwargs)¶

Return Exponential power of series and other, element-wise (binary operator pow).

Equivalent to series ** other, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.rpow(): Reverse of the Exponential power operator, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.pow(b, fill_value=0)
a    1.0
b    1.0
c    1.0
d    0.0
e    NaN
dtype: float64

quantile(**kwargs)¶

‘quantile’ is not implemented yet.

If support for ‘quantile’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

radd(**kwargs)¶

Return Addition of series and other, element-wise (binary operator radd).

Equivalent to other + series, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.add(): Element-wise Addition, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.add(b, fill_value=0)
a    2.0
b    1.0
c    1.0
d    1.0
e    NaN
dtype: float64

rank(**kwargs)¶

‘rank’ is not implemented yet.

If support for ‘rank’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

rdiv(**kwargs)¶

Return Floating division of series and other, element-wise (binary operator rtruediv).

Equivalent to other / series, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.truediv(): Element-wise Floating division, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.divide(b, fill_value=0)
a    1.0
b    inf
c    inf
d    0.0
e    NaN
dtype: float64

rdivmod(**kwargs)¶

Return Integer division and modulo of series and other, element-wise (binary operator rdivmod).

Equivalent to other divmod series, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	2-Tuple of DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.divmod(): Element-wise Integer division and modulo, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.divmod(b, fill_value=0)
(a    1.0
 b    NaN
 c    NaN
 d    0.0
 e    NaN
 dtype: float64,
 a    0.0
 b    NaN
 c    NaN
 d    0.0
 e    NaN
 dtype: float64)

reindex(**kwargs)¶

‘reindex’ is not implemented yet.

If support for ‘reindex’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

reindex_like(**kwargs)¶

‘reindex_like’ is not implemented yet.

If support for ‘reindex_like’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

reorder_levels(**kwargs)¶

Rearrange index levels using input order. May not drop or duplicate levels.

Parameters:	order (list of int or list of str) – List representing new level order. Reference level by number (position) or by key (label). axis ({0 or 'index', 1 or 'columns'}, default 0) – Where to reorder levels.
Returns:
Return type:	DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

repeat(**kwargs)¶

‘repeat’ is not implemented yet.

If support for ‘repeat’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

resample(**kwargs)¶

‘resample’ is not implemented yet.

If support for ‘resample’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

reset_index(**kwargs)¶

‘reset_index’ is not implemented yet.

If support for ‘reset_index’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

rfloordiv(**kwargs)¶

Return Integer division of series and other, element-wise (binary operator rfloordiv).

Equivalent to other // series, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.floordiv(): Element-wise Integer division, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.floordiv(b, fill_value=0)
a    1.0
b    NaN
c    NaN
d    0.0
e    NaN
dtype: float64

rmod(**kwargs)¶

Return Modulo of series and other, element-wise (binary operator rmod).

Equivalent to other % series, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.mod(): Element-wise Modulo, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.mod(b, fill_value=0)
a    0.0
b    NaN
c    NaN
d    0.0
e    NaN
dtype: float64

rmul(**kwargs)¶

Return Multiplication of series and other, element-wise (binary operator rmul).

Equivalent to other * series, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.mul(): Element-wise Multiplication, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.multiply(b, fill_value=0)
a    1.0
b    0.0
c    0.0
d    0.0
e    NaN
dtype: float64

rolling(**kwargs)¶

‘rolling’ is not implemented yet.

If support for ‘rolling’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

rpow(**kwargs)¶

Return Exponential power of series and other, element-wise (binary operator rpow).

Equivalent to other ** series, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.pow(): Element-wise Exponential power, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.pow(b, fill_value=0)
a    1.0
b    1.0
c    1.0
d    0.0
e    NaN
dtype: float64

rsub(**kwargs)¶

Return Subtraction of series and other, element-wise (binary operator rsub).

Equivalent to other - series, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.sub(): Element-wise Subtraction, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.subtract(b, fill_value=0)
a    0.0
b    1.0
c    1.0
d   -1.0
e    NaN
dtype: float64

rtruediv(**kwargs)¶

Return Floating division of series and other, element-wise (binary operator rtruediv).

Equivalent to other / series, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.truediv(): Element-wise Floating division, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.divide(b, fill_value=0)
a    1.0
b    inf
c    inf
d    0.0
e    NaN
dtype: float64

sample(**kwargs)¶

‘sample’ is not implemented yet.

If support for ‘sample’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

sem(**kwargs)¶

‘sem’ is not implemented yet.

If support for ‘sem’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

set_axis(**kwargs)¶

‘set_axis’ is not implemented yet.

If support for ‘set_axis’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

set_flags(**kwargs)¶

‘set_flags’ is not implemented yet.

If support for ‘set_flags’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

shape¶

‘shape’ is not implemented yet.

If support for ‘shape’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

size¶

Return an int representing the number of elements in this object.

Return the number of rows if Series. Otherwise return the number of rows times number of columns if DataFrame.

Differences from pandas

This operation has no known divergences from the pandas API.

See also

ndarray.size: Number of elements in the array.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series({'a': 1, 'b': 2, 'c': 3})
>>> s.size
3

>>> df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
>>> df.size
4

skew(**kwargs)¶

‘skew’ is not implemented yet.

If support for ‘skew’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

slice_shift(**kwargs)¶

‘slice_shift’ is not implemented yet.

If support for ‘slice_shift’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

sort_index(axis, **kwargs)¶

Sort object by labels (along an axis).

Returns a new DataFrame sorted by label if inplace argument is False, otherwise updates the original DataFrame and returns None.

Parameters:	axis ({0 or 'index', 1 or 'columns'}, default 0) – The axis along which to sort. The value 0 identifies the rows, and 1 identifies the columns. level (int or level name or list of ints or list of level names) – If not None, sort on values in specified index level(s). ascending (bool or list-like of bools, default True) – Sort ascending vs. descending. When the index is a MultiIndex the sort direction can be controlled for each level individually. inplace (bool, default False) – If True, perform operation in-place. kind ({'quicksort', 'mergesort', 'heapsort'}, default 'quicksort') – Choice of sorting algorithm. See also ndarray.np.sort for more information. mergesort is the only stable algorithm. For DeferredDataFrames, this option is only applied when sorting on a single column or label. na_position ({'first', 'last'}, default 'last') – Puts NaNs at the beginning if first; last puts NaNs at the end. Not implemented for MultiIndex. sort_remaining (bool, default True) – If True and sorting by level and index is multilevel, sort by other levels too (in order) after sorting by specified level. ignore_index (bool, default False) – If True, the resulting axis will be labeled 0, 1, …, n - 1. New in version 1.0.0. key (callable, optional) – If not None, apply the key function to the index values before sorting. This is similar to the key argument in the builtin `sorted()` function, with the notable difference that this key function should be vectorized. It should expect an `Index` and return an `Index` of the same shape. For MultiIndex inputs, the key is applied per level. New in version 1.1.0.
Returns:	The original DeferredDataFrame sorted by the labels or None if `inplace=True`.
Return type:	DeferredDataFrame or None

Differences from pandas

axis=index is not allowed because it imposes an ordering on the dataset, and we cannot guarantee it will be maintained (see https://s.apache.org/dataframe-order-sensitive-operations). Only axis=columns is allowed.

See also

DeferredSeries.sort_index(): Sort DeferredSeries by the index.
DeferredDataFrame.sort_values(): Sort DeferredDataFrame by the value.
DeferredSeries.sort_values(): Sort DeferredSeries by the value.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame([1, 2, 3, 4, 5], index=[100, 29, 234, 1, 150],
...                   columns=['A'])
>>> df.sort_index()
     A
1    4
29   2
100  1
150  5
234  3

By default, it sorts in ascending order, to sort in descending order,
use ``ascending=False``

>>> df.sort_index(ascending=False)
     A
234  3
150  5
100  1
29   2
1    4

A key function can be specified which is applied to the index before
sorting. For a ``MultiIndex`` this is applied to each level separately.

>>> df = pd.DataFrame({"a": [1, 2, 3, 4]}, index=['A', 'b', 'C', 'd'])
>>> df.sort_index(key=lambda x: x.str.lower())
   a
A  1
b  2
C  3
d  4

sort_values(axis, **kwargs)¶

sort_values is not implemented.

It is not implemented for axis=index because it imposes an ordering on the dataset, and we cannot guarantee it will be maintained (see https://s.apache.org/dataframe-order-sensitive-operations).

It is not implemented for axis=columns because it makes the order of the columns depend on the data (see https://s.apache.org/dataframe-non-deferred-column-names).

sparse¶

‘sparse’ is not implemented yet.

If support for ‘sparse’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

squeeze(**kwargs)¶

‘squeeze’ is not implemented yet.

If support for ‘squeeze’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

sub(**kwargs)¶

Return Subtraction of series and other, element-wise (binary operator sub).

Equivalent to series - other, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.rsub(): Reverse of the Subtraction operator, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.subtract(b, fill_value=0)
a    0.0
b    1.0
c    1.0
d   -1.0
e    NaN
dtype: float64

subtract(**kwargs)¶

‘subtract’ is not implemented yet.

If support for ‘subtract’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

swapaxes(**kwargs)¶

‘swapaxes’ is not implemented yet.

If support for ‘swapaxes’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

swaplevel(**kwargs)¶

‘swaplevel’ is not implemented yet.

If support for ‘swaplevel’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_clipboard(**kwargs)¶

‘clipboard’ is not implemented yet.

If support for ‘clipboard’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_csv(path, *args, **kwargs)¶

Write object to a comma-separated values (csv) file.

Changed in version 0.24.0: The order of arguments for Series was changed.

Parameters:	path_or_buf (str or file handle, default None) – File path or object, if None is provided the result is returned as a string. If a non-binary file object is passed, it should be opened with newline=’’, disabling universal newlines. If a binary file object is passed, mode might need to contain a ‘b’. Changed in version 0.24.0: Was previously named “path” for DeferredSeries. Changed in version 1.2.0: Support for binary file objects was introduced. sep (str, default ',') – String of length 1. Field delimiter for the output file. na_rep (str, default '') – Missing data representation. float_format (str, default None) – Format string for floating point numbers. columns (sequence, optional) – Columns to write. header (bool or list of str, default True) – Write out the column names. If a list of strings is given it is assumed to be aliases for the column names. Changed in version 0.24.0: Previously defaulted to False for DeferredSeries. index (bool, default True) – Write row names (index). index_label (str or sequence, or False, default None) – Column label for index column(s) if desired. If None is given, and header and index are True, then the index names are used. A sequence should be given if the object uses MultiIndex. If False do not print fields for index names. Use index_label=False for easier importing in R. mode (str) – Python write mode, default ‘w’. encoding (str, optional) – A string representing the encoding to use in the output file, defaults to ‘utf-8’. encoding is not supported if path_or_buf is a non-binary file object. compression (str or dict, default 'infer') – If str, represents compression mode. If dict, value at ‘method’ is the compression mode. Compression mode may be any of the following possible values: {‘infer’, ‘gzip’, ‘bz2’, ‘zip’, ‘xz’, None}. If compression mode is ‘infer’ and path_or_buf is path-like, then detect compression mode from the following extensions: ‘.gz’, ‘.bz2’, ‘.zip’ or ‘.xz’. (otherwise no compression). If dict given and mode is one of {‘zip’, ‘gzip’, ‘bz2’}, or inferred as one of the above, other entries passed as additional compression options. Changed in version 1.0.0: May now be a dict with key ‘method’ as compression mode and other entries as additional compression options if compression mode is ‘zip’. Changed in version 1.1.0: Passing compression options as keys in dict is supported for compression modes ‘gzip’ and ‘bz2’ as well as ‘zip’. Changed in version 1.2.0: Compression is supported for binary file objects. Changed in version 1.2.0: Previous versions forwarded dict entries for ‘gzip’ to gzip.open instead of gzip.GzipFile which prevented setting mtime. quoting (optional constant from csv module) – Defaults to csv.QUOTE_MINIMAL. If you have set a float_format then floats are converted to strings and thus csv.QUOTE_NONNUMERIC will treat them as non-numeric. quotechar (str, default '"') – String of length 1. Character used to quote fields. line_terminator (str, optional) – The newline character or character sequence to use in the output file. Defaults to os.linesep, which depends on the OS in which this method is called (‘n’ for linux, ‘rn’ for Windows, i.e.). Changed in version 0.24.0. chunksize (int or None) – Rows to write at a time. date_format (str, default None) – Format string for datetime objects. doublequote (bool, default True) – Control quoting of quotechar inside a field. escapechar (str, default None) – String of length 1. Character used to escape sep and quotechar when appropriate. decimal (str, default '.') – Character recognized as decimal separator. E.g. use ‘,’ for European data. errors (str, default 'strict') – Specifies how encoding and decoding errors are to be handled. See the errors argument for `open()` for a full list of options. New in version 1.1.0. storage_options (dict, optional) – Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc., if using a URL that will be parsed by `fsspec`, e.g., starting “s3://”, “gcs://”. An error will be raised if providing this argument with a non-fsspec URL. See the fsspec and backend storage implementation docs for the set of allowed keys and values. New in version 1.2.0.
Returns:	If path_or_buf is None, returns the resulting csv format as a string. Otherwise returns None.
Return type:	None or str

Differences from pandas

This operation has no known divergences from the pandas API.

See also

read_csv(): Load a CSV file into a DeferredDataFrame.
to_excel(): Write DeferredDataFrame to an Excel file.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame({'name': ['Raphael', 'Donatello'],
...                    'mask': ['red', 'purple'],
...                    'weapon': ['sai', 'bo staff']})
>>> df.to_csv(index=False)
'name,mask,weapon\nRaphael,red,sai\nDonatello,purple,bo staff\n'

Create 'out.zip' containing 'out.csv'

>>> compression_opts = dict(method='zip',
...                         archive_name='out.csv')  
>>> df.to_csv('out.zip', index=False,
...           compression=compression_opts)  

to_excel(path, *args, **kwargs)¶

Write object to an Excel sheet.

To write a single object to an Excel .xlsx file it is only necessary to specify a target file name. To write to multiple sheets it is necessary to create an ExcelWriter object with a target file name, and specify a sheet in the file to write to.

Multiple sheets may be written to by specifying unique sheet_name. With all data written to the file it is necessary to save the changes. Note that creating an ExcelWriter object with a file name that already exists will result in the contents of the existing file being erased.

Parameters:

excel_writer (path-like, file-like, or ExcelWriter object) – File path or existing ExcelWriter.
sheet_name (str, default 'Sheet1') – Name of sheet which will contain DeferredDataFrame.
na_rep (str, default '') – Missing data representation.
float_format (str, optional) – Format string for floating point numbers. For example float_format="%.2f" will format 0.1234 to 0.12.
columns (sequence or list of str, optional) – Columns to write.
header (bool or list of str, default True) – Write out the column names. If a list of string is given it is assumed to be aliases for the column names.
index (bool, default True) – Write row names (index).
index_label (str or sequence, optional) – Column label for index column(s) if desired. If not specified, and header and index are True, then the index names are used. A sequence should be given if the DeferredDataFrame uses MultiIndex.
startrow (int, default 0) – Upper left cell row to dump data frame.
startcol (int, default 0) – Upper left cell column to dump data frame.
engine (str, optional) –
Write engine to use, ‘openpyxl’ or ‘xlsxwriter’. You can also set this via the options io.excel.xlsx.writer, io.excel.xls.writer, and io.excel.xlsm.writer.

Deprecated since version 1.2.0: As the xlwt package is no longer maintained, the xlwt engine will be removed in a future version of pandas.
merge_cells (bool, default True) – Write MultiIndex and Hierarchical Rows as merged cells.
encoding (str, optional) – Encoding of the resulting excel file. Only necessary for xlwt, other writers support unicode natively.
inf_rep (str, default 'inf') – Representation for infinity (there is no native representation for infinity in Excel).
verbose (bool, default True) – Display more information in the error logs.
freeze_panes (tuple of int (length 2), optional) – Specifies the one-based bottommost row and rightmost column that is to be frozen.
storage_options (dict, optional) –
Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc., if using a URL that will be parsed by fsspec, e.g., starting “s3://”, “gcs://”. An error will be raised if providing this argument with a non-fsspec URL. See the fsspec and backend storage implementation docs for the set of allowed keys and values.

New in version 1.2.0.

Differences from pandas

This operation has no known divergences from the pandas API.

See also

to_csv(): Write DeferredDataFrame to a comma-separated values (csv) file.
ExcelWriter(): Class for writing DeferredDataFrame objects into excel sheets.
read_excel(): Read an Excel file into a pandas DeferredDataFrame.
read_csv(): Read a comma-separated values (csv) file into DeferredDataFrame.

Notes

For compatibility with to_csv(), to_excel serializes lists and dicts to strings before writing.

Once a workbook has been saved it is not possible write further data without rewriting the whole workbook.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Create, write to and save a workbook:

>>> df1 = pd.DataFrame([['a', 'b'], ['c', 'd']],
...                    index=['row 1', 'row 2'],
...                    columns=['col 1', 'col 2'])
>>> df1.to_excel("output.xlsx")  

To specify the sheet name:

>>> df1.to_excel("output.xlsx",
...              sheet_name='Sheet_name_1')  

If you wish to write to more than one sheet in the workbook, it is
necessary to specify an ExcelWriter object:

>>> df2 = df1.copy()
>>> with pd.ExcelWriter('output.xlsx') as writer:  
...     df1.to_excel(writer, sheet_name='Sheet_name_1')
...     df2.to_excel(writer, sheet_name='Sheet_name_2')

ExcelWriter can also be used to append to an existing Excel file:

>>> with pd.ExcelWriter('output.xlsx',
...                     mode='a') as writer:  
...     df.to_excel(writer, sheet_name='Sheet_name_3')

To set the library that is used to write the Excel file,
you can pass the `engine` keyword (the default engine is
automatically chosen depending on the file extension):

>>> df1.to_excel('output1.xlsx', engine='xlsxwriter')  

to_feather(path, *args, **kwargs)¶

Write a DataFrame to the binary Feather format.

Parameters:	path (str or file-like object) – If a string, it will be used as Root Directory path. **kwargs – Additional keywords passed to `pyarrow.feather.write_feather()`. Starting with pyarrow 0.17, this includes the compression, compression_level, chunksize and version keywords. New in version 1.1.0.

Differences from pandas

This operation has no known divergences from the pandas API.

to_hdf(**kwargs)¶: pandas.DataFrame.to_hdf is not supported in the Beam DataFrame API because HDF5 is a random access file format.

to_html(path, *args, **kwargs)¶

Render a DataFrame as an HTML table.

Parameters:	buf (str, Path or StringIO-like, optional, default None) – Buffer to write to. If None, the output is returned as a string. columns (sequence, optional, default None) – The subset of columns to write. Writes all columns by default. col_space (str or int, list or dict of int or str, optional) – The minimum width of each column in CSS length units. An int is assumed to be px units. New in version 0.25.0: Ability to use str. header (bool, optional) – Whether to print column labels, default True. index (bool, optional, default True) – Whether to print index (row) labels. na_rep (str, optional, default 'NaN') – String representation of `NaN` to use. formatters (list, tuple or dict of one-param. functions, optional) – Formatter functions to apply to columns’ elements by position or name. The result of each function must be a unicode string. List/tuple must be of length equal to the number of columns. float_format (one-parameter function, optional, default None) – Formatter function to apply to columns’ elements if they are floats. This function must return a unicode string and will be applied only to the non-`NaN` elements, with `NaN` being handled by `na_rep`. Changed in version 1.2.0. sparsify (bool, optional, default True) – Set to False for a DeferredDataFrame with a hierarchical index to print every multiindex key at each row. index_names (bool, optional, default True) – Prints the names of the indexes. justify (str, default None) – How to justify the column labels. If None uses the option from the print configuration (controlled by set_option), ‘right’ out of the box. Valid values are left right center justify justify-all start end inherit match-parent initial unset. max_rows (int, optional) – Maximum number of rows to display in the console. min_rows (int, optional) – The number of rows to display in the console in a truncated repr (when number of rows is above max_rows). max_cols (int, optional) – Maximum number of columns to display in the console. show_dimensions (bool, default False) – Display DeferredDataFrame dimensions (number of rows by number of columns). decimal (str, default '.') – Character recognized as decimal separator, e.g. ‘,’ in Europe. bold_rows (bool, default True) – Make the row labels bold in the output. classes (str or list or tuple, default None) – CSS class(es) to apply to the resulting html table. escape (bool, default True) – Convert the characters <, >, and & to HTML-safe sequences. notebook ({True, False}, default False) – Whether the generated HTML is for IPython Notebook. border (int) – A `border=border` attribute is included in the opening <table> tag. Default `pd.options.display.html.border`. encoding (str, default "utf-8") – Set character encoding. New in version 1.0. table_id (str, optional) – A css id is included in the opening <table> tag if specified. render_links (bool, default False) – Convert URLs to HTML links. New in version 0.24.0.
Returns:	If buf is None, returns the result as a string. Otherwise returns None.
Return type:	str or None

Differences from pandas

This operation has no known divergences from the pandas API.

See also

to_string(): Convert DeferredDataFrame to a string.

to_json(path, orient=None, *args, **kwargs)¶

Convert the object to a JSON string.

Note NaN’s and None will be converted to null and datetime objects will be converted to UNIX timestamps.

Parameters:	path_or_buf (str or file handle, optional) – File path or object. If not specified, the result is returned as a string. orient (str) – Indication of expected JSON string format. DeferredSeries: default is ‘index’ allowed values are: {‘split’, ‘records’, ‘index’, ‘table’}. DeferredDataFrame: default is ‘columns’ allowed values are: {‘split’, ‘records’, ‘index’, ‘columns’, ‘values’, ‘table’}. The format of the JSON string: ’split’ : dict like {‘index’ -> [index], ‘columns’ -> [columns], ‘data’ -> [values]} ’records’ : list like [{column -> value}, … , {column -> value}] ’index’ : dict like {index -> {column -> value}} ’columns’ : dict like {column -> {index -> value}} ’values’ : just the values array ’table’ : dict like {‘schema’: {schema}, ‘data’: {data}} Describing the data, where data component is like `orient='records'`. date_format ({None, 'epoch', 'iso'}) – Type of date conversion. ‘epoch’ = epoch milliseconds, ‘iso’ = ISO8601. The default depends on the orient. For `orient='table'`, the default is ‘iso’. For all other orients, the default is ‘epoch’. double_precision (int, default 10) – The number of decimal places to use when encoding floating point values. force_ascii (bool, default True) – Force encoded string to be ASCII. date_unit (str, default 'ms' (milliseconds)) – The time unit to encode to, governs timestamp and ISO8601 precision. One of ‘s’, ‘ms’, ‘us’, ‘ns’ for second, millisecond, microsecond, and nanosecond respectively. default_handler (callable, default None) – Handler to call if object cannot otherwise be converted to a suitable format for JSON. Should receive a single argument which is the object to convert and return a serialisable object. lines (bool, default False) – If ‘orient’ is ‘records’ write out line delimited json format. Will throw ValueError if incorrect ‘orient’ since others are not list like. compression ({'infer', 'gzip', 'bz2', 'zip', 'xz', None}) – A string representing the compression to use in the output file, only used when the first argument is a filename. By default, the compression is inferred from the filename. Changed in version 0.24.0: ‘infer’ option added and set to default index (bool, default True) – Whether to include the index values in the JSON string. Not including the index (`index=False`) is only supported when orient is ‘split’ or ‘table’. indent (int, optional) – Length of whitespace used to indent each record. New in version 1.0.0. storage_options (dict, optional) – Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc., if using a URL that will be parsed by `fsspec`, e.g., starting “s3://”, “gcs://”. An error will be raised if providing this argument with a non-fsspec URL. See the fsspec and backend storage implementation docs for the set of allowed keys and values. New in version 1.2.0.
Returns:	If path_or_buf is None, returns the resulting json format as a string. Otherwise returns None.
Return type:	None or str

Differences from pandas

This operation has no known divergences from the pandas API.

See also

read_json(): Convert a JSON string to pandas object.

Notes

The behavior of indent=0 varies from the stdlib, which does not indent the output but does insert newlines. Currently, indent=0 and the default indent=None are equivalent in pandas, though this may change in a future release.

orient='table' contains a ‘pandas_version’ field under ‘schema’. This stores the version of pandas used in the latest revision of the schema.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> import json
>>> df = pd.DataFrame(
...     [["a", "b"], ["c", "d"]],
...     index=["row 1", "row 2"],
...     columns=["col 1", "col 2"],
... )

>>> result = df.to_json(orient="split")
>>> parsed = json.loads(result)
>>> json.dumps(parsed, indent=4)  
{
    "columns": [
        "col 1",
        "col 2"
    ],
    "index": [
        "row 1",
        "row 2"
    ],
    "data": [
        [
            "a",
            "b"
        ],
        [
            "c",
            "d"
        ]
    ]
}

Encoding/decoding a Dataframe using ``'records'`` formatted JSON.
Note that index labels are not preserved with this encoding.

>>> result = df.to_json(orient="records")
>>> parsed = json.loads(result)
>>> json.dumps(parsed, indent=4)  
[
    {
        "col 1": "a",
        "col 2": "b"
    },
    {
        "col 1": "c",
        "col 2": "d"
    }
]

Encoding/decoding a Dataframe using ``'index'`` formatted JSON:

>>> result = df.to_json(orient="index")
>>> parsed = json.loads(result)
>>> json.dumps(parsed, indent=4)  
{
    "row 1": {
        "col 1": "a",
        "col 2": "b"
    },
    "row 2": {
        "col 1": "c",
        "col 2": "d"
    }
}

Encoding/decoding a Dataframe using ``'columns'`` formatted JSON:

>>> result = df.to_json(orient="columns")
>>> parsed = json.loads(result)
>>> json.dumps(parsed, indent=4)  
{
    "col 1": {
        "row 1": "a",
        "row 2": "c"
    },
    "col 2": {
        "row 1": "b",
        "row 2": "d"
    }
}

Encoding/decoding a Dataframe using ``'values'`` formatted JSON:

>>> result = df.to_json(orient="values")
>>> parsed = json.loads(result)
>>> json.dumps(parsed, indent=4)  
[
    [
        "a",
        "b"
    ],
    [
        "c",
        "d"
    ]
]

Encoding with Table Schema:

>>> result = df.to_json(orient="table")
>>> parsed = json.loads(result)
>>> json.dumps(parsed, indent=4)  
{
    "schema": {
        "fields": [
            {
                "name": "index",
                "type": "string"
            },
            {
                "name": "col 1",
                "type": "string"
            },
            {
                "name": "col 2",
                "type": "string"
            }
        ],
        "primaryKey": [
            "index"
        ],
        "pandas_version": "0.20.0"
    },
    "data": [
        {
            "index": "row 1",
            "col 1": "a",
            "col 2": "b"
        },
        {
            "index": "row 2",
            "col 1": "c",
            "col 2": "d"
        }
    ]
}

to_latex(**kwargs)¶

‘to_latex’ is not implemented yet.

If support for ‘to_latex’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_list(**kwargs)¶

‘to_list’ is not implemented yet.

If support for ‘to_list’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_markdown(**kwargs)¶

‘to_markdown’ is not implemented yet.

If support for ‘to_markdown’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_msgpack(**kwargs)¶: pandas.DataFrame.to_msgpack is not supported in the Beam DataFrame API because it is deprecated in pandas.

to_parquet(path, *args, **kwargs)¶

Write a DataFrame to the binary parquet format.

This function writes the dataframe as a parquet file. You can choose different parquet backends, and have the option of compression. See the user guide for more details.

Parameters:	path (str or file-like object, default None) – If a string, it will be used as Root Directory path when writing a partitioned dataset. By file-like object, we refer to objects with a write() method, such as a file handle (e.g. via builtin open function) or io.BytesIO. The engine fastparquet does not accept file-like objects. If path is None, a bytes object is returned. Changed in version 1.2.0. Previously this was “fname” engine ({'auto', 'pyarrow', 'fastparquet'}, default 'auto') – Parquet library to use. If ‘auto’, then the option `io.parquet.engine` is used. The default `io.parquet.engine` behavior is to try ‘pyarrow’, falling back to ‘fastparquet’ if ‘pyarrow’ is unavailable. compression ({'snappy', 'gzip', 'brotli', None}, default 'snappy') – Name of the compression to use. Use `None` for no compression. index (bool, default None) – If `True`, include the dataframe’s index(es) in the file output. If `False`, they will not be written to the file. If `None`, similar to `True` the dataframe’s index(es) will be saved. However, instead of being saved as values, the RangeIndex will be stored as a range in the metadata so it doesn’t require much space and is faster. Other indexes will be included as columns in the file output. New in version 0.24.0. partition_cols (list, optional, default None) – Column names by which to partition the dataset. Columns are partitioned in the order they are given. Must be None if path is not a string. New in version 0.24.0. storage_options (dict, optional) – Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc., if using a URL that will be parsed by `fsspec`, e.g., starting “s3://”, “gcs://”. An error will be raised if providing this argument with a non-fsspec URL. See the fsspec and backend storage implementation docs for the set of allowed keys and values. New in version 1.2.0. **kwargs – Additional arguments passed to the parquet library. See pandas io for more details.
Returns:
Return type:	bytes if no path argument is provided else None

Differences from pandas

This operation has no known divergences from the pandas API.

See also

read_parquet(): Read a parquet file.
DeferredDataFrame.to_csv(): Write a csv file.
DeferredDataFrame.to_sql(): Write to a sql table.
DeferredDataFrame.to_hdf(): Write to hdf.

Notes

This function requires either the fastparquet or pyarrow library.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame(data={'col1': [1, 2], 'col2': [3, 4]})
>>> df.to_parquet('df.parquet.gzip',
...               compression='gzip')  
>>> pd.read_parquet('df.parquet.gzip')  
   col1  col2
0     1     3
1     2     4

If you want to get a buffer to the parquet content you can use a io.BytesIO
object, as long as you don't use partition_cols, which creates multiple files.

>>> import io
>>> f = io.BytesIO()
>>> df.to_parquet(f)
>>> f.seek(0)
0
>>> content = f.read()

to_period(**kwargs)¶

‘to_period’ is not implemented yet.

If support for ‘to_period’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_pickle(**kwargs)¶

‘to_pickle’ is not implemented yet.

If support for ‘to_pickle’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_sql(**kwargs)¶

‘to_sql’ is not implemented yet.

If support for ‘to_sql’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_stata(path, *args, **kwargs)¶

Export DataFrame object to Stata dta format.

Writes the DataFrame to a Stata dataset file. “dta” files contain a Stata dataset.

Parameters:

path (str, buffer or path object) –
String, path object (pathlib.Path or py._path.local.LocalPath) or object implementing a binary write() function. If using a buffer then the buffer will not be automatically closed after the file data has been written.

Changed in version 1.0.0.

Previously this was “fname”
convert_dates (dict) – Dictionary mapping columns containing datetime types to stata internal format to use when writing the dates. Options are ‘tc’, ‘td’, ‘tm’, ‘tw’, ‘th’, ‘tq’, ‘ty’. Column can be either an integer or a name. Datetime columns that do not have a conversion type specified will be converted to ‘tc’. Raises NotImplementedError if a datetime column has timezone information.
write_index (bool) – Write the index to Stata dataset.
byteorder (str) – Can be “>”, “<”, “little”, or “big”. default is sys.byteorder.
time_stamp (datetime) – A datetime to use as file creation date. Default is the current time.
data_label (str, optional) – A label for the data set. Must be 80 characters or smaller.
variable_labels (dict) – Dictionary containing columns as keys and variable labels as values. Each label must be 80 characters or smaller.
version ({114, 117, 118, 119, None}, default 114) –
Version to use in the output dta file. Set to None to let pandas decide between 118 or 119 formats depending on the number of columns in the frame. pandas Version 114 can be read by Stata 10 and later. pandas Version 117 can be read by Stata 13 or later. pandas Version 118 is supported in Stata 14 and later. pandas Version 119 is supported in Stata 15 and later. pandas Version 114 limits string variables to 244 characters or fewer while versions 117 and later allow strings with lengths up to 2,000,000 characters. Versions 118 and 119 support Unicode characters, and pandas version 119 supports more than 32,767 variables.

pandas Version 119 should usually only be used when the number of variables exceeds the capacity of dta format 118. Exporting smaller datasets in format 119 may have unintended consequences, and, as of November 2020, Stata SE cannot read pandas version 119 files.

Changed in version 1.0.0: Added support for formats 118 and 119.
convert_strl (list, optional) – List of column names to convert to string columns to Stata StrL format. Only available if version is 117. Storing strings in the StrL format can produce smaller dta files if strings have more than 8 characters and values are repeated.
compression (str or dict, default 'infer') –
For on-the-fly compression of the output dta. If string, specifies compression mode. If dict, value at key ‘method’ specifies compression mode. Compression mode must be one of {‘infer’, ‘gzip’, ‘bz2’, ‘zip’, ‘xz’, None}. If compression mode is ‘infer’ and fname is path-like, then detect compression from the following extensions: ‘.gz’, ‘.bz2’, ‘.zip’, or ‘.xz’ (otherwise no compression). If dict and compression mode is one of {‘zip’, ‘gzip’, ‘bz2’}, or inferred as one of the above, other entries passed as additional compression options.

New in version 1.1.0.
storage_options (dict, optional) –
Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc., if using a URL that will be parsed by fsspec, e.g., starting “s3://”, “gcs://”. An error will be raised if providing this argument with a non-fsspec URL. See the fsspec and backend storage implementation docs for the set of allowed keys and values.

New in version 1.2.0.

Raises:

NotImplementedError – * If datetimes contain timezone information * Column dtype is not representable in Stata
ValueError – * Columns listed in convert_dates are neither datetime64[ns]

or datetime.datetime
- Column listed in convert_dates is not in DeferredDataFrame
- Categorical label contains more than 32,000 characters

Differences from pandas

This operation has no known divergences from the pandas API.

See also

read_stata(): Import Stata data files.
io.stata.StataWriter(): Low-level writer for Stata data files.
io.stata.StataWriter117(): Low-level writer for pandas version 117 files.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame({'animal': ['falcon', 'parrot', 'falcon',
...                               'parrot'],
...                    'speed': [350, 18, 361, 15]})
>>> df.to_stata('animals.dta')  

to_timestamp(**kwargs)¶

‘to_timestamp’ is not implemented yet.

If support for ‘to_timestamp’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_xarray(**kwargs)¶

‘to_xarray’ is not implemented yet.

If support for ‘to_xarray’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

transform(**kwargs)¶

‘transform’ is not implemented yet.

If support for ‘transform’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

transpose(**kwargs)¶

‘transpose’ is not implemented yet.

If support for ‘transpose’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

truediv(**kwargs)¶

Return Floating division of series and other, element-wise (binary operator truediv).

Equivalent to series / other, but with support to substitute a fill_value for missing data in either one of the inputs.

Parameters:	other (DeferredSeries or scalar value) – fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful DeferredSeries alignment, with this value before computation. If data in both corresponding DeferredSeries locations is missing the result of filling (at that location) will be missing. level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	The result of the operation.
Return type:	DeferredSeries

Differences from pandas

Only level=None is supported

See also

DeferredSeries.rtruediv(): Reverse of the Floating division operator, see Python documentation for more details.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
>>> a
a    1.0
b    1.0
c    1.0
d    NaN
dtype: float64
>>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
>>> b
a    1.0
b    NaN
d    1.0
e    NaN
dtype: float64
>>> a.divide(b, fill_value=0)
a    1.0
b    inf
c    inf
d    0.0
e    NaN
dtype: float64

truncate(**kwargs)¶

‘truncate’ is not implemented yet.

If support for ‘truncate’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

tshift(**kwargs)¶

‘tshift’ is not implemented yet.

If support for ‘tshift’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

tz_convert(**kwargs)¶

‘tz_convert’ is not implemented yet.

If support for ‘tz_convert’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

tz_localize(ambiguous, **kwargs)¶

Localize tz-naive index of a Series or DataFrame to target time zone.

This operation localizes the Index. To localize the values in a timezone-naive Series, use Series.dt.tz_localize().

Parameters:	tz (str or tzinfo) – axis (the axis to localize) – level (int, str, default None) – If axis ia a MultiIndex, localize a specific level. Otherwise must be None. copy (bool, default True) – Also make a copy of the underlying data. ambiguous ('infer', bool-ndarray, 'NaT', default 'raise') – When clocks moved backward due to DST, ambiguous times may arise. For example in Central European Time (UTC+01), when going from 03:00 DST to 02:00 non-DST, 02:30:00 local time occurs both at 00:30:00 UTC and at 01:30:00 UTC. In such a situation, the ambiguous parameter dictates how ambiguous times should be handled. ’infer’ will attempt to infer fall dst-transition hours based on order bool-ndarray where True signifies a DST time, False designates a non-DST time (note that this flag is only applicable for ambiguous times) ’NaT’ will return NaT where there are ambiguous times ’raise’ will raise an AmbiguousTimeError if there are ambiguous times. nonexistent (str, default 'raise') – A nonexistent time does not exist in a particular timezone where clocks moved forward due to DST. Valid values are: ’shift_forward’ will shift the nonexistent time forward to the closest existing time ’shift_backward’ will shift the nonexistent time backward to the closest existing time ’NaT’ will return NaT where there are nonexistent times timedelta objects will shift nonexistent times by the timedelta ’raise’ will raise an NonExistentTimeError if there are nonexistent times. New in version 0.24.0.
Returns:	Same type as the input.
Return type:	DeferredSeries or DeferredDataFrame
Raises:	`TypeError` – If the TimeDeferredSeries is tz-aware and tz is not None.

Differences from pandas

ambiguous cannot be set to "infer" as its semantics are order-sensitive. Similarly, specifying ambiguous as an ndarray is order-sensitive, but you can achieve similar functionality by specifying ambiguous as a Series.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

Localize local times:

>>> s = pd.Series([1],
...               index=pd.DatetimeIndex(['2018-09-15 01:30:00']))
>>> s.tz_localize('CET')
2018-09-15 01:30:00+02:00    1
dtype: int64

Be careful with DST changes. When there is sequential data, pandas
can infer the DST time:

>>> s = pd.Series(range(7),
...               index=pd.DatetimeIndex(['2018-10-28 01:30:00',
...                                       '2018-10-28 02:00:00',
...                                       '2018-10-28 02:30:00',
...                                       '2018-10-28 02:00:00',
...                                       '2018-10-28 02:30:00',
...                                       '2018-10-28 03:00:00',
...                                       '2018-10-28 03:30:00']))
>>> s.tz_localize('CET', ambiguous='infer')
2018-10-28 01:30:00+02:00    0
2018-10-28 02:00:00+02:00    1
2018-10-28 02:30:00+02:00    2
2018-10-28 02:00:00+01:00    3
2018-10-28 02:30:00+01:00    4
2018-10-28 03:00:00+01:00    5
2018-10-28 03:30:00+01:00    6
dtype: int64

In some cases, inferring the DST is impossible. In such cases, you can
pass an ndarray to the ambiguous parameter to set the DST explicitly

>>> s = pd.Series(range(3),
...               index=pd.DatetimeIndex(['2018-10-28 01:20:00',
...                                       '2018-10-28 02:36:00',
...                                       '2018-10-28 03:46:00']))
>>> s.tz_localize('CET', ambiguous=np.array([True, True, False]))
2018-10-28 01:20:00+02:00    0
2018-10-28 02:36:00+02:00    1
2018-10-28 03:46:00+01:00    2
dtype: int64

If the DST transition causes nonexistent times, you can shift these
dates forward or backward with a timedelta object or `'shift_forward'`
or `'shift_backward'`.

>>> s = pd.Series(range(2),
...               index=pd.DatetimeIndex(['2015-03-29 02:30:00',
...                                       '2015-03-29 03:30:00']))
>>> s.tz_localize('Europe/Warsaw', nonexistent='shift_forward')
2015-03-29 03:00:00+02:00    0
2015-03-29 03:30:00+02:00    1
dtype: int64
>>> s.tz_localize('Europe/Warsaw', nonexistent='shift_backward')
2015-03-29 01:59:59.999999999+01:00    0
2015-03-29 03:30:00+02:00              1
dtype: int64
>>> s.tz_localize('Europe/Warsaw', nonexistent=pd.Timedelta('1H'))
2015-03-29 03:30:00+02:00    0
2015-03-29 03:30:00+02:00    1
dtype: int64

value_counts(**kwargs)¶

‘value_counts’ is not implemented yet.

If support for ‘value_counts’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

where(cond, other, errors, **kwargs)¶

Replace values where the condition is False.

Parameters:	cond (bool DeferredSeries/DeferredDataFrame, array-like, or callable) – Where cond is True, keep the original value. Where False, replace with corresponding value from other. If cond is callable, it is computed on the DeferredSeries/DeferredDataFrame and should return boolean DeferredSeries/DeferredDataFrame or array. The callable must not change input DeferredSeries/DeferredDataFrame (though pandas doesn’t check it). other (scalar, DeferredSeries/DeferredDataFrame, or callable) – Entries where cond is False are replaced with corresponding value from other. If other is callable, it is computed on the DeferredSeries/DeferredDataFrame and should return scalar or DeferredSeries/DeferredDataFrame. The callable must not change input DeferredSeries/DeferredDataFrame (though pandas doesn’t check it). inplace (bool, default False) – Whether to perform the operation in place on the data. axis (int, default None) – Alignment axis if needed. level (int, default None) – Alignment level if needed. errors (str, {'raise', 'ignore'}, default 'raise') – Note that currently this parameter won’t affect the results and will always coerce to a suitable dtype. ’raise’ : allow exceptions to be raised. ’ignore’ : suppress exceptions. On error return original object. try_cast (bool, default False) – Try to cast the result back to the input type (if possible).
Returns:
Return type:	Same type as caller or None if `inplace=True`.

Differences from pandas

where is not parallelizable when errors="ignore" is specified.

See also

DeferredDataFrame.mask(): Return an object of same shape as self.

Notes

The where method is an application of the if-then idiom. For each element in the calling DeferredDataFrame, if cond is True the element is used; otherwise the corresponding element from the DeferredDataFrame other is used.

The signature for DeferredDataFrame.where() differs from numpy.where(). Roughly df1.where(m, df2) is equivalent to np.where(m, df1, df2).

For further details and examples see the where documentation in indexing.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> s = pd.Series(range(5))
>>> s.where(s > 0)
0    NaN
1    1.0
2    2.0
3    3.0
4    4.0
dtype: float64
>>> s.mask(s > 0)
0    0.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64

>>> s.where(s > 1, 10)
0    10
1    10
2    2
3    3
4    4
dtype: int64
>>> s.mask(s > 1, 10)
0     0
1     1
2    10
3    10
4    10
dtype: int64

>>> df = pd.DataFrame(np.arange(10).reshape(-1, 2), columns=['A', 'B'])
>>> df
   A  B
0  0  1
1  2  3
2  4  5
3  6  7
4  8  9
>>> m = df % 3 == 0
>>> df.where(m, -df)
   A  B
0  0 -1
1 -2  3
2 -4 -5
3  6 -7
4 -8  9
>>> df.where(m, -df) == np.where(m, df, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True
>>> df.where(m, -df) == df.mask(~m, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True

classmethod wrap(expr, split_tuples=True)¶

xs(**kwargs)¶

‘xs’ is not implemented yet.

If support for ‘xs’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

class apache_beam.dataframe.frames.DeferredDataFrame(expr)[source]¶

Bases: apache_beam.dataframe.frames.DeferredDataFrameOrSeries

T¶

columns¶

keys()[source]¶

align(other, join, axis, copy, level, method, **kwargs)[source]¶

append(other, ignore_index, verify_integrity, sort, **kwargs)[source]¶

set_index(keys, **kwargs)[source]¶

loc¶

iloc¶

axes¶

dtypes¶

assign(**kwargs)[source]¶

explode(column, ignore_index)[source]¶

insert(value, **kwargs)[source]¶

duplicated(keep, subset)[source]¶

Return boolean Series denoting duplicate rows.

Considering certain columns is optional.

Parameters:	subset (column label or sequence of labels, optional) – Only consider certain columns for identifying duplicates, by default use all of the columns. keep ({'first', 'last', False}, default 'first') – Determines which duplicates (if any) to mark. `first` : Mark duplicates as `True` except for the first occurrence. `last` : Mark duplicates as `True` except for the last occurrence. False : Mark all duplicates as `True`.
Returns:	Boolean series for each duplicated rows.
Return type:	DeferredSeries

Differences from pandas

Only keep=False and keep="any" are supported. Other values of keep make this an order-sensitive operation. Note keep="any" is a Beam-specific option that guarantees only one duplicate will be kept, but unlike "first" and "last" it makes no guarantees about _which_ duplicate element is kept.

See also

Index.duplicated(): Equivalent method on index.
DeferredSeries.duplicated(): Equivalent method on DeferredSeries.
DeferredSeries.drop_duplicates(): Remove duplicate values from DeferredSeries.
DeferredDataFrame.drop_duplicates(): Remove duplicate values from DeferredDataFrame.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

Consider dataset containing ramen rating.

>>> df = pd.DataFrame({
...     'brand': ['Yum Yum', 'Yum Yum', 'Indomie', 'Indomie', 'Indomie'],
...     'style': ['cup', 'cup', 'cup', 'pack', 'pack'],
...     'rating': [4, 4, 3.5, 15, 5]
... })
>>> df
    brand style  rating
0  Yum Yum   cup     4.0
1  Yum Yum   cup     4.0
2  Indomie   cup     3.5
3  Indomie  pack    15.0
4  Indomie  pack     5.0

By default, for each set of duplicated values, the first occurrence
is set on False and all others on True.

>>> df.duplicated()
0    False
1     True
2    False
3    False
4    False
dtype: bool

By using 'last', the last occurrence of each set of duplicated values
is set on False and all others on True.

>>> df.duplicated(keep='last')
0     True
1    False
2    False
3    False
4    False
dtype: bool

By setting ``keep`` on False, all duplicates are True.

>>> df.duplicated(keep=False)
0     True
1     True
2    False
3    False
4    False
dtype: bool

To find duplicates on specific column(s), use ``subset``.

>>> df.duplicated(subset=['brand'])
0    False
1     True
2    False
3     True
4     True
dtype: bool

drop_duplicates(keep, subset, ignore_index)[source]¶

Return DataFrame with duplicate rows removed.

Considering certain columns is optional. Indexes, including time indexes are ignored.

Parameters:	subset (column label or sequence of labels, optional) – Only consider certain columns for identifying duplicates, by default use all of the columns. keep ({'first', 'last', False}, default 'first') – Determines which duplicates (if any) to keep. - `first` : Drop duplicates except for the first occurrence. - `last` : Drop duplicates except for the last occurrence. - False : Drop all duplicates. inplace (bool, default False) – Whether to drop duplicates in place or to return a copy. ignore_index (bool, default False) – If True, the resulting axis will be labeled 0, 1, …, n - 1. New in version 1.0.0.
Returns:	DeferredDataFrame with duplicates removed or None if `inplace=True`.
Return type:	DeferredDataFrame or None

Differences from pandas

Only keep=False and keep="any" are supported. Other values of keep make this an order-sensitive operation. Note keep="any" is a Beam-specific option that guarantees only one duplicate will be kept, but unlike "first" and "last" it makes no guarantees about _which_ duplicate element is kept.

See also

DeferredDataFrame.value_counts(): Count unique combinations of columns.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

Consider dataset containing ramen rating.

>>> df = pd.DataFrame({
...     'brand': ['Yum Yum', 'Yum Yum', 'Indomie', 'Indomie', 'Indomie'],
...     'style': ['cup', 'cup', 'cup', 'pack', 'pack'],
...     'rating': [4, 4, 3.5, 15, 5]
... })
>>> df
    brand style  rating
0  Yum Yum   cup     4.0
1  Yum Yum   cup     4.0
2  Indomie   cup     3.5
3  Indomie  pack    15.0
4  Indomie  pack     5.0

By default, it removes duplicate rows based on all columns.

>>> df.drop_duplicates()
    brand style  rating
0  Yum Yum   cup     4.0
2  Indomie   cup     3.5
3  Indomie  pack    15.0
4  Indomie  pack     5.0

To remove duplicates on specific column(s), use ``subset``.

>>> df.drop_duplicates(subset=['brand'])
    brand style  rating
0  Yum Yum   cup     4.0
2  Indomie   cup     3.5

To remove duplicates and keep last occurrences, use ``keep``.

>>> df.drop_duplicates(subset=['brand', 'style'], keep='last')
    brand style  rating
1  Yum Yum   cup     4.0
2  Indomie   cup     3.5
4  Indomie  pack     5.0

aggregate(func, axis, *args, **kwargs)[source]¶

Aggregate using one or more operations over the specified axis.

Parameters:

func (function, str, list or dict) –
Function to use for aggregating the data. If a function, must either work when passed a DeferredDataFrame or when passed to DeferredDataFrame.apply.

Accepted combinations are:
- function
- string function name
- list of functions and/or function names, e.g. [np.sum, 'mean']
- dict of axis labels -> functions, function names or list of such.
axis ({0 or 'index', 1 or 'columns'}, default 0) – If 0 or ‘index’: apply function to each column. If 1 or ‘columns’: apply function to each row.
*args – Positional arguments to pass to func.
**kwargs – Keyword arguments to pass to func.

Returns:

scalar, DeferredSeries or DeferredDataFrame – The return can be:
- scalar : when DeferredSeries.agg is called with single function
- DeferredSeries : when DeferredDataFrame.agg is called with a single function
- DeferredDataFrame : when DeferredDataFrame.agg is called with several functions
Return scalar, DeferredSeries or DeferredDataFrame.
The aggregation operations are always performed over an axis, either the
index (default) or the column axis. This behavior is different from
numpy aggregation functions (mean, median, prod, sum, std,
var), where the default is to compute the aggregation of the flattened
array, e.g., numpy.mean(arr_2d) as opposed to
numpy.mean(arr_2d, axis=0).
agg is an alias for aggregate. Use the alias.

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.apply(): Perform any type of operations.
DeferredDataFrame.transform(): Perform transformation type operations.
core.groupby.GroupBy(): Perform operations over groups.
core.resample.Resampler(): Perform operations over resampled bins.
core.window.Rolling(): Perform operations over rolling window.
core.window.Expanding(): Perform operations over expanding window.
core.window.ExponentialMovingWindow(): Perform operation over exponential weighted window.

Notes

agg is an alias for aggregate. Use the alias.

A passed user-defined-function will be passed a DeferredSeries for evaluation.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame([[1, 2, 3],
...                    [4, 5, 6],
...                    [7, 8, 9],
...                    [np.nan, np.nan, np.nan]],
...                   columns=['A', 'B', 'C'])

Aggregate these functions over the rows.

>>> df.agg(['sum', 'min'])
        A     B     C
sum  12.0  15.0  18.0
min   1.0   2.0   3.0

Different aggregations per column.

>>> df.agg({'A' : ['sum', 'min'], 'B' : ['min', 'max']})
        A    B
sum  12.0  NaN
min   1.0  2.0
max   NaN  8.0

Aggregate different functions over the columns and rename the index of the resulting
DataFrame.

>>> df.agg(x=('A', max), y=('B', 'min'), z=('C', np.mean))
     A    B    C
x  7.0  NaN  NaN
y  NaN  2.0  NaN
z  NaN  NaN  6.0

Aggregate over the columns.

>>> df.agg("mean", axis="columns")
0    2.0
1    5.0
2    8.0
3    NaN
dtype: float64

agg(func, axis, *args, **kwargs)¶

Aggregate using one or more operations over the specified axis.

Parameters:

func (function, str, list or dict) –
Function to use for aggregating the data. If a function, must either work when passed a DeferredDataFrame or when passed to DeferredDataFrame.apply.

Accepted combinations are:
- function
- string function name
- list of functions and/or function names, e.g. [np.sum, 'mean']
- dict of axis labels -> functions, function names or list of such.
axis ({0 or 'index', 1 or 'columns'}, default 0) – If 0 or ‘index’: apply function to each column. If 1 or ‘columns’: apply function to each row.
*args – Positional arguments to pass to func.
**kwargs – Keyword arguments to pass to func.

Returns:

scalar, DeferredSeries or DeferredDataFrame – The return can be:
- scalar : when DeferredSeries.agg is called with single function
- DeferredSeries : when DeferredDataFrame.agg is called with a single function
- DeferredDataFrame : when DeferredDataFrame.agg is called with several functions
Return scalar, DeferredSeries or DeferredDataFrame.
The aggregation operations are always performed over an axis, either the
index (default) or the column axis. This behavior is different from
numpy aggregation functions (mean, median, prod, sum, std,
var), where the default is to compute the aggregation of the flattened
array, e.g., numpy.mean(arr_2d) as opposed to
numpy.mean(arr_2d, axis=0).
agg is an alias for aggregate. Use the alias.

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.apply(): Perform any type of operations.
DeferredDataFrame.transform(): Perform transformation type operations.
core.groupby.GroupBy(): Perform operations over groups.
core.resample.Resampler(): Perform operations over resampled bins.
core.window.Rolling(): Perform operations over rolling window.
core.window.Expanding(): Perform operations over expanding window.
core.window.ExponentialMovingWindow(): Perform operation over exponential weighted window.

Notes

agg is an alias for aggregate. Use the alias.

A passed user-defined-function will be passed a DeferredSeries for evaluation.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame([[1, 2, 3],
...                    [4, 5, 6],
...                    [7, 8, 9],
...                    [np.nan, np.nan, np.nan]],
...                   columns=['A', 'B', 'C'])

Aggregate these functions over the rows.

>>> df.agg(['sum', 'min'])
        A     B     C
sum  12.0  15.0  18.0
min   1.0   2.0   3.0

Different aggregations per column.

>>> df.agg({'A' : ['sum', 'min'], 'B' : ['min', 'max']})
        A    B
sum  12.0  NaN
min   1.0  2.0
max   NaN  8.0

Aggregate different functions over the columns and rename the index of the resulting
DataFrame.

>>> df.agg(x=('A', max), y=('B', 'min'), z=('C', np.mean))
     A    B    C
x  7.0  NaN  NaN
y  NaN  2.0  NaN
z  NaN  NaN  6.0

Aggregate over the columns.

>>> df.agg("mean", axis="columns")
0    2.0
1    5.0
2    8.0
3    NaN
dtype: float64

applymap(**kwargs)¶

Apply a function to a Dataframe elementwise.

This method applies a function that accepts and returns a scalar to every element of a DataFrame.

Parameters:	func (callable) – Python function, returns a single value from a single value. na_action ({None, 'ignore'}, default None) – If ‘ignore’, propagate NaN values, without passing them to func. New in version 1.2.
Returns:	Transformed DeferredDataFrame.
Return type:	DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.apply(): Apply a function along input axis of DeferredDataFrame.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame([[1, 2.12], [3.356, 4.567]])
>>> df
       0      1
0  1.000  2.120
1  3.356  4.567

>>> df.applymap(lambda x: len(str(x)))
   0  1
0  3  4
1  5  5

Like Series.map, NA values can be ignored:

>>> df_copy = df.copy()
>>> df_copy.iloc[0, 0] = pd.NA
>>> df_copy.applymap(lambda x: len(str(x)), na_action='ignore')
      0  1
0  <NA>  4
1     5  5

Note that a vectorized version of `func` often exists, which will
be much faster. You could square each number elementwise.

>>> df.applymap(lambda x: x**2)
           0          1
0   1.000000   4.494400
1  11.262736  20.857489

But it's better to avoid applymap in that case.

>>> df ** 2
           0          1
0   1.000000   4.494400
1  11.262736  20.857489

add_prefix(**kwargs)¶

Prefix labels with string prefix.

For Series, the row labels are prefixed. For DataFrame, the column labels are prefixed.

Parameters:	prefix (str) – The string to add before each label.
Returns:	New DeferredSeries or DeferredDataFrame with updated labels.
Return type:	DeferredSeries or DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.add_suffix(): Suffix row labels with string suffix.
DeferredDataFrame.add_suffix(): Suffix column labels with string suffix.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series([1, 2, 3, 4])
>>> s
0    1
1    2
2    3
3    4
dtype: int64

>>> s.add_prefix('item_')
item_0    1
item_1    2
item_2    3
item_3    4
dtype: int64

>>> df = pd.DataFrame({'A': [1, 2, 3, 4], 'B': [3, 4, 5, 6]})
>>> df
   A  B
0  1  3
1  2  4
2  3  5
3  4  6

>>> df.add_prefix('col_')
     col_A  col_B
0       1       3
1       2       4
2       3       5
3       4       6

add_suffix(**kwargs)¶

Suffix labels with string suffix.

For Series, the row labels are suffixed. For DataFrame, the column labels are suffixed.

Parameters:	suffix (str) – The string to add after each label.
Returns:	New DeferredSeries or DeferredDataFrame with updated labels.
Return type:	DeferredSeries or DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.add_prefix(): Prefix row labels with string prefix.
DeferredDataFrame.add_prefix(): Prefix column labels with string prefix.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series([1, 2, 3, 4])
>>> s
0    1
1    2
2    3
3    4
dtype: int64

>>> s.add_suffix('_item')
0_item    1
1_item    2
2_item    3
3_item    4
dtype: int64

>>> df = pd.DataFrame({'A': [1, 2, 3, 4], 'B': [3, 4, 5, 6]})
>>> df
   A  B
0  1  3
1  2  4
2  3  5
3  4  6

>>> df.add_suffix('_col')
     A_col  B_col
0       1       3
1       2       4
2       3       5
3       4       6

memory_usage(**kwargs)¶

pandas.DataFrame.memory_usage is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

info(**kwargs)¶

pandas.DataFrame.info is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

clip(axis, **kwargs)[source]¶: lower and upper must be DeferredSeries instances, or constants. Array-like arguments are not supported because they are order-sensitive.

corr(method, min_periods)[source]¶

Compute pairwise correlation of columns, excluding NA/null values.

Parameters:	method ({'pearson', 'kendall', 'spearman'} or callable) – Method of correlation: pearson : standard correlation coefficient kendall : Kendall Tau correlation coefficient spearman : Spearman rank correlation callable: callable with input two 1d ndarrays and returning a float. Note that the returned matrix from corr will have 1 along the diagonals and will be symmetric regardless of the callable’s behavior. New in version 0.24.0. min_periods (int, optional) – Minimum number of observations required per pair of columns to have a valid result. Currently only available for Pearson and Spearman correlation.
Returns:	Correlation matrix.
Return type:	DeferredDataFrame

Differences from pandas

Only method="pearson" can be parallelized. Other methods require collecting all data on a single worker (see https://s.apache.org/dataframe-non-parallelizable-operations for details).

See also

DeferredDataFrame.corrwith(): Compute pairwise correlation with another DeferredDataFrame or DeferredSeries.
DeferredSeries.corr(): Compute the correlation between two DeferredSeries.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> def histogram_intersection(a, b):
...     v = np.minimum(a, b).sum().round(decimals=1)
...     return v
>>> df = pd.DataFrame([(.2, .3), (.0, .6), (.6, .0), (.2, .1)],
...                   columns=['dogs', 'cats'])
>>> df.corr(method=histogram_intersection)
      dogs  cats
dogs   1.0   0.3
cats   0.3   1.0

cov(min_periods, ddof)[source]¶

corrwith(other, axis, drop, method)[source]¶

cummax(**kwargs)¶

pandas.DataFrame.cummax is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

cummin(**kwargs)¶

pandas.DataFrame.cummin is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

cumprod(**kwargs)¶

pandas.DataFrame.cumprod is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

cumsum(**kwargs)¶

pandas.DataFrame.cumsum is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

diff(**kwargs)¶

pandas.DataFrame.diff is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

first(**kwargs)¶

pandas.DataFrame.first is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

head(**kwargs)¶

pandas.DataFrame.head is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

interpolate(**kwargs)¶

pandas.DataFrame.interpolate is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

last(**kwargs)¶

pandas.DataFrame.last is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

tail(**kwargs)¶

pandas.DataFrame.tail is not supported in the Beam DataFrame API because it is sensitive to the order of the data.

For more information see {reason_data[‘url’]}.

dot(other)[source]¶

mode(axis=0, *args, **kwargs)[source]¶

dropna(axis, **kwargs)[source]¶

eval(expr, inplace, **kwargs)[source]¶

query(expr, inplace, **kwargs)[source]¶

isnull(**kwargs)¶

Detect missing values.

Return a boolean same-sized object indicating if the values are NA. NA values, such as None or numpy.NaN, gets mapped to True values. Everything else gets mapped to False values. Characters such as empty strings '' or numpy.inf are not considered NA values (unless you set pandas.options.mode.use_inf_as_na = True).

Returns:	Mask of bool values for each element in DeferredDataFrame that indicates whether an element is an NA value.
Return type:	DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.isnull(): Alias of isna.
DeferredDataFrame.notna(): Boolean inverse of isna.
DeferredDataFrame.dropna(): Omit axes labels with missing values.
isna(): Top-level isna.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Show which entries in a DataFrame are NA.

>>> df = pd.DataFrame(dict(age=[5, 6, np.NaN],
...                    born=[pd.NaT, pd.Timestamp('1939-05-27'),
...                          pd.Timestamp('1940-04-25')],
...                    name=['Alfred', 'Batman', ''],
...                    toy=[None, 'Batmobile', 'Joker']))
>>> df
   age       born    name        toy
0  5.0        NaT  Alfred       None
1  6.0 1939-05-27  Batman  Batmobile
2  NaN 1940-04-25              Joker

>>> df.isna()
     age   born   name    toy
0  False   True  False   True
1  False  False  False  False
2   True  False  False  False

Show which entries in a Series are NA.

>>> ser = pd.Series([5, 6, np.NaN])
>>> ser
0    5.0
1    6.0
2    NaN
dtype: float64

>>> ser.isna()
0    False
1    False
2     True
dtype: bool

isna(**kwargs)¶

Detect missing values.

Return a boolean same-sized object indicating if the values are NA. NA values, such as None or numpy.NaN, gets mapped to True values. Everything else gets mapped to False values. Characters such as empty strings '' or numpy.inf are not considered NA values (unless you set pandas.options.mode.use_inf_as_na = True).

Returns:	Mask of bool values for each element in DeferredDataFrame that indicates whether an element is an NA value.
Return type:	DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.isnull(): Alias of isna.
DeferredDataFrame.notna(): Boolean inverse of isna.
DeferredDataFrame.dropna(): Omit axes labels with missing values.
isna(): Top-level isna.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Show which entries in a DataFrame are NA.

>>> df = pd.DataFrame(dict(age=[5, 6, np.NaN],
...                    born=[pd.NaT, pd.Timestamp('1939-05-27'),
...                          pd.Timestamp('1940-04-25')],
...                    name=['Alfred', 'Batman', ''],
...                    toy=[None, 'Batmobile', 'Joker']))
>>> df
   age       born    name        toy
0  5.0        NaT  Alfred       None
1  6.0 1939-05-27  Batman  Batmobile
2  NaN 1940-04-25              Joker

>>> df.isna()
     age   born   name    toy
0  False   True  False   True
1  False  False  False  False
2   True  False  False  False

Show which entries in a Series are NA.

>>> ser = pd.Series([5, 6, np.NaN])
>>> ser
0    5.0
1    6.0
2    NaN
dtype: float64

>>> ser.isna()
0    False
1    False
2     True
dtype: bool

notnull(**kwargs)¶

Detect existing (non-missing) values.

Return a boolean same-sized object indicating if the values are not NA. Non-missing values get mapped to True. Characters such as empty strings '' or numpy.inf are not considered NA values (unless you set pandas.options.mode.use_inf_as_na = True). NA values, such as None or numpy.NaN, get mapped to False values.

Returns:	Mask of bool values for each element in DeferredDataFrame that indicates whether an element is not an NA value.
Return type:	DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.notnull(): Alias of notna.
DeferredDataFrame.isna(): Boolean inverse of notna.
DeferredDataFrame.dropna(): Omit axes labels with missing values.
notna(): Top-level notna.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Show which entries in a DataFrame are not NA.

>>> df = pd.DataFrame(dict(age=[5, 6, np.NaN],
...                    born=[pd.NaT, pd.Timestamp('1939-05-27'),
...                          pd.Timestamp('1940-04-25')],
...                    name=['Alfred', 'Batman', ''],
...                    toy=[None, 'Batmobile', 'Joker']))
>>> df
   age       born    name        toy
0  5.0        NaT  Alfred       None
1  6.0 1939-05-27  Batman  Batmobile
2  NaN 1940-04-25              Joker

>>> df.notna()
     age   born  name    toy
0   True  False  True  False
1   True   True  True   True
2  False   True  True   True

Show which entries in a Series are not NA.

>>> ser = pd.Series([5, 6, np.NaN])
>>> ser
0    5.0
1    6.0
2    NaN
dtype: float64

>>> ser.notna()
0     True
1     True
2    False
dtype: bool

notna(**kwargs)¶

Detect existing (non-missing) values.

Return a boolean same-sized object indicating if the values are not NA. Non-missing values get mapped to True. Characters such as empty strings '' or numpy.inf are not considered NA values (unless you set pandas.options.mode.use_inf_as_na = True). NA values, such as None or numpy.NaN, get mapped to False values.

Returns:	Mask of bool values for each element in DeferredDataFrame that indicates whether an element is not an NA value.
Return type:	DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.notnull(): Alias of notna.
DeferredDataFrame.isna(): Boolean inverse of notna.
DeferredDataFrame.dropna(): Omit axes labels with missing values.
notna(): Top-level notna.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Show which entries in a DataFrame are not NA.

>>> df = pd.DataFrame(dict(age=[5, 6, np.NaN],
...                    born=[pd.NaT, pd.Timestamp('1939-05-27'),
...                          pd.Timestamp('1940-04-25')],
...                    name=['Alfred', 'Batman', ''],
...                    toy=[None, 'Batmobile', 'Joker']))
>>> df
   age       born    name        toy
0  5.0        NaT  Alfred       None
1  6.0 1939-05-27  Batman  Batmobile
2  NaN 1940-04-25              Joker

>>> df.notna()
     age   born  name    toy
0   True  False  True  False
1   True   True  True   True
2  False   True  True   True

Show which entries in a Series are not NA.

>>> ser = pd.Series([5, 6, np.NaN])
>>> ser
0    5.0
1    6.0
2    NaN
dtype: float64

>>> ser.notna()
0     True
1     True
2    False
dtype: bool

items(**kwargs)¶

pandas.DataFrame.items is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

itertuples(**kwargs)¶

pandas.DataFrame.itertuples is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

iterrows(**kwargs)¶

pandas.DataFrame.iterrows is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

iteritems(**kwargs)¶

pandas.DataFrame.iteritems is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

join(other, on, **kwargs)[source]¶

merge(right, on, left_on, right_on, left_index, right_index, suffixes, **kwargs)[source]¶

nlargest(keep, **kwargs)[source]¶

Return the first n rows ordered by columns in descending order.

Return the first n rows with the largest values in columns, in descending order. The columns that are not specified are returned as well, but not used for ordering.

This method is equivalent to df.sort_values(columns, ascending=False).head(n), but more performant.

Parameters:	n (int) – Number of rows to return. columns (label or list of labels) – Column label(s) to order by. keep ({'first', 'last', 'all'}, default 'first') – Where there are duplicate values: first : prioritize the first occurrence(s) last : prioritize the last occurrence(s) `all` : do not drop any duplicates, even it means selecting more than n items. New in version 0.24.0.
Returns:	The first n rows ordered by the given columns in descending order.
Return type:	DeferredDataFrame

Differences from pandas

Only keep=False and keep="any" are supported. Other values of keep make this an order-sensitive operation. Note keep="any" is a Beam-specific option that guarantees only one duplicate will be kept, but unlike "first" and "last" it makes no guarantees about _which_ duplicate element is kept.

See also

DeferredDataFrame.nsmallest(): Return the first n rows ordered by columns in ascending order.
DeferredDataFrame.sort_values(): Sort DeferredDataFrame by the values.
DeferredDataFrame.head(): Return the first n rows without re-ordering.

Notes

This function cannot be used with all column types. For example, when specifying columns with object or category dtypes, TypeError is raised.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'population': [59000000, 65000000, 434000,
...                                   434000, 434000, 337000, 11300,
...                                   11300, 11300],
...                    'GDP': [1937894, 2583560 , 12011, 4520, 12128,
...                            17036, 182, 38, 311],
...                    'alpha-2': ["IT", "FR", "MT", "MV", "BN",
...                                "IS", "NR", "TV", "AI"]},
...                   index=["Italy", "France", "Malta",
...                          "Maldives", "Brunei", "Iceland",
...                          "Nauru", "Tuvalu", "Anguilla"])
>>> df
          population      GDP alpha-2
Italy       59000000  1937894      IT
France      65000000  2583560      FR
Malta         434000    12011      MT
Maldives      434000     4520      MV
Brunei        434000    12128      BN
Iceland       337000    17036      IS
Nauru          11300      182      NR
Tuvalu         11300       38      TV
Anguilla       11300      311      AI

In the following example, we will use ``nlargest`` to select the three
rows having the largest values in column "population".

>>> df.nlargest(3, 'population')
        population      GDP alpha-2
France    65000000  2583560      FR
Italy     59000000  1937894      IT
Malta       434000    12011      MT

When using ``keep='last'``, ties are resolved in reverse order:

>>> df.nlargest(3, 'population', keep='last')
        population      GDP alpha-2
France    65000000  2583560      FR
Italy     59000000  1937894      IT
Brunei      434000    12128      BN

When using ``keep='all'``, all duplicate items are maintained:

>>> df.nlargest(3, 'population', keep='all')
          population      GDP alpha-2
France      65000000  2583560      FR
Italy       59000000  1937894      IT
Malta         434000    12011      MT
Maldives      434000     4520      MV
Brunei        434000    12128      BN

To order by the largest values in column "population" and then "GDP",
we can specify multiple columns like in the next example.

>>> df.nlargest(3, ['population', 'GDP'])
        population      GDP alpha-2
France    65000000  2583560      FR
Italy     59000000  1937894      IT
Brunei      434000    12128      BN

nsmallest(keep, **kwargs)[source]¶

Return the first n rows ordered by columns in ascending order.

Return the first n rows with the smallest values in columns, in ascending order. The columns that are not specified are returned as well, but not used for ordering.

This method is equivalent to df.sort_values(columns, ascending=True).head(n), but more performant.

Parameters:	n (int) – Number of items to retrieve. columns (list or str) – Column name or names to order by. keep ({'first', 'last', 'all'}, default 'first') – Where there are duplicate values: `first` : take the first occurrence. `last` : take the last occurrence. `all` : do not drop any duplicates, even it means selecting more than n items. New in version 0.24.0.
Returns:
Return type:	DeferredDataFrame

Differences from pandas

Only keep=False and keep="any" are supported. Other values of keep make this an order-sensitive operation. Note keep="any" is a Beam-specific option that guarantees only one duplicate will be kept, but unlike "first" and "last" it makes no guarantees about _which_ duplicate element is kept.

See also

DeferredDataFrame.nlargest(): Return the first n rows ordered by columns in descending order.
DeferredDataFrame.sort_values(): Sort DeferredDataFrame by the values.
DeferredDataFrame.head(): Return the first n rows without re-ordering.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'population': [59000000, 65000000, 434000,
...                                   434000, 434000, 337000, 337000,
...                                   11300, 11300],
...                    'GDP': [1937894, 2583560 , 12011, 4520, 12128,
...                            17036, 182, 38, 311],
...                    'alpha-2': ["IT", "FR", "MT", "MV", "BN",
...                                "IS", "NR", "TV", "AI"]},
...                   index=["Italy", "France", "Malta",
...                          "Maldives", "Brunei", "Iceland",
...                          "Nauru", "Tuvalu", "Anguilla"])
>>> df
          population      GDP alpha-2
Italy       59000000  1937894      IT
France      65000000  2583560      FR
Malta         434000    12011      MT
Maldives      434000     4520      MV
Brunei        434000    12128      BN
Iceland       337000    17036      IS
Nauru         337000      182      NR
Tuvalu         11300       38      TV
Anguilla       11300      311      AI

In the following example, we will use ``nsmallest`` to select the
three rows having the smallest values in column "population".

>>> df.nsmallest(3, 'population')
          population    GDP alpha-2
Tuvalu         11300     38      TV
Anguilla       11300    311      AI
Iceland       337000  17036      IS

When using ``keep='last'``, ties are resolved in reverse order:

>>> df.nsmallest(3, 'population', keep='last')
          population  GDP alpha-2
Anguilla       11300  311      AI
Tuvalu         11300   38      TV
Nauru         337000  182      NR

When using ``keep='all'``, all duplicate items are maintained:

>>> df.nsmallest(3, 'population', keep='all')
          population    GDP alpha-2
Tuvalu         11300     38      TV
Anguilla       11300    311      AI
Iceland       337000  17036      IS
Nauru         337000    182      NR

To order by the smallest values in column "population" and then "GDP", we can
specify multiple columns like in the next example.

>>> df.nsmallest(3, ['population', 'GDP'])
          population  GDP alpha-2
Tuvalu         11300   38      TV
Anguilla       11300  311      AI
Nauru         337000  182      NR

nunique(**kwargs)[source]¶

plot(**kwargs)¶

pandas.DataFrame.plot is not supported in the Beam DataFrame API because it is a plotting tool.

For more information see {reason_data[‘url’]}.

pop(item)[source]¶

quantile(q, axis, **kwargs)[source]¶

rename(**kwargs)[source]¶

rename_axis(**kwargs)¶

Set the name of the axis for the index or columns.

Parameters:	mapper (scalar, list-like, optional) – Value to set the axis name attribute. columns (index,) – A scalar, list-like, dict-like or functions transformations to apply to that axis’ values. Note that the `columns` parameter is not allowed if the object is a DeferredSeries. This parameter only apply for DeferredDataFrame type objects. Use either `mapper` and `axis` to specify the axis to target with `mapper`, or `index` and/or `columns`. Changed in version 0.24.0. axis ({0 or 'index', 1 or 'columns'}, default 0) – The axis to rename. copy (bool, default True) – Also copy underlying data. inplace (bool, default False) – Modifies the object directly, instead of creating a new DeferredSeries or DeferredDataFrame.
Returns:	The same type as the caller or None if `inplace=True`.
Return type:	DeferredSeries, DeferredDataFrame, or None

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.rename(): Alter DeferredSeries index labels or name.
DeferredDataFrame.rename(): Alter DeferredDataFrame index labels or name.
Index.rename(): Set new names on index.

Notes

DeferredDataFrame.rename_axis supports two calling conventions

(index=index_mapper, columns=columns_mapper, ...)
(mapper, axis={'index', 'columns'}, ...)

The first calling convention will only modify the names of the index and/or the names of the Index object that is the columns. In this case, the parameter copy is ignored.

The second calling convention will modify the names of the corresponding index if mapper is a list or a scalar. However, if mapper is dict-like or a function, it will use the deprecated behavior of modifying the axis labels.

We highly recommend using keyword arguments to clarify your intent.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

**Series**

>>> s = pd.Series(["dog", "cat", "monkey"])
>>> s
0       dog
1       cat
2    monkey
dtype: object
>>> s.rename_axis("animal")
animal
0    dog
1    cat
2    monkey
dtype: object

**DataFrame**

>>> df = pd.DataFrame({"num_legs": [4, 4, 2],
...                    "num_arms": [0, 0, 2]},
...                   ["dog", "cat", "monkey"])
>>> df
        num_legs  num_arms
dog            4         0
cat            4         0
monkey         2         2
>>> df = df.rename_axis("animal")
>>> df
        num_legs  num_arms
animal
dog            4         0
cat            4         0
monkey         2         2
>>> df = df.rename_axis("limbs", axis="columns")
>>> df
limbs   num_legs  num_arms
animal
dog            4         0
cat            4         0
monkey         2         2

**MultiIndex**

>>> df.index = pd.MultiIndex.from_product([['mammal'],
...                                        ['dog', 'cat', 'monkey']],
...                                       names=['type', 'name'])
>>> df
limbs          num_legs  num_arms
type   name
mammal dog            4         0
       cat            4         0
       monkey         2         2

>>> df.rename_axis(index={'type': 'class'})
limbs          num_legs  num_arms
class  name
mammal dog            4         0
       cat            4         0
       monkey         2         2

>>> df.rename_axis(columns=str.upper)
LIMBS          num_legs  num_arms
type   name
mammal dog            4         0
       cat            4         0
       monkey         2         2

replace(limit, **kwargs)[source]¶

reset_index(level=None, **kwargs)[source]¶

round(decimals, *args, **kwargs)[source]¶

select_dtypes(**kwargs)¶

Return a subset of the DataFrame’s columns based on the column dtypes.

Parameters:	exclude (include,) – A selection of dtypes or strings to be included/excluded. At least one of these parameters must be supplied.
Returns:	The subset of the frame including the dtypes in `include` and excluding the dtypes in `exclude`.
Return type:	DeferredDataFrame
Raises:	`ValueError` – * If both of `include` and `exclude` are empty * If `include` and `exclude` have overlapping elements * If any kind of string dtype is passed in.

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.dtypes(): Return DeferredSeries with the data type of each column.

Notes

To select all numeric types, use np.number or 'number'
To select strings you must use the object dtype, but note that this will return all object dtype columns
See the numpy dtype hierarchy
To select datetimes, use np.datetime64, 'datetime' or 'datetime64'
To select timedeltas, use np.timedelta64, 'timedelta' or 'timedelta64'
To select Pandas categorical dtypes, use 'category'
To select Pandas datetimetz dtypes, use 'datetimetz' (new in 0.20.0) or 'datetime64[ns, tz]'

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame({'a': [1, 2] * 3,
...                    'b': [True, False] * 3,
...                    'c': [1.0, 2.0] * 3})
>>> df
        a      b  c
0       1   True  1.0
1       2  False  2.0
2       1   True  1.0
3       2  False  2.0
4       1   True  1.0
5       2  False  2.0

>>> df.select_dtypes(include='bool')
   b
0  True
1  False
2  True
3  False
4  True
5  False

>>> df.select_dtypes(include=['float64'])
   c
0  1.0
1  2.0
2  1.0
3  2.0
4  1.0
5  2.0

>>> df.select_dtypes(exclude=['int64'])
       b    c
0   True  1.0
1  False  2.0
2   True  1.0
3  False  2.0
4   True  1.0
5  False  2.0

shift(axis, freq, **kwargs)[source]¶

shape¶

pandas.DataFrame.shape is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

stack(**kwargs)¶

Stack the prescribed level(s) from columns to index.

Return a reshaped DataFrame or Series having a multi-level index with one or more new inner-most levels compared to the current DataFrame. The new inner-most levels are created by pivoting the columns of the current dataframe:

if the columns have a single level, the output is a Series;

if the columns have multiple levels, the new index level(s) is (are) taken from the prescribed level(s) and the output is a DataFrame.

Parameters:	level (int, str, list, default -1) – Level(s) to stack from the column axis onto the index axis, defined as one index or label, or a list of indices or labels. dropna (bool, default True) – Whether to drop rows in the resulting Frame/DeferredSeries with missing values. Stacking a column level onto the index axis can create combinations of index and column values that are missing from the original dataframe. See Examples section.
Returns:	Stacked dataframe or series.
Return type:	DeferredDataFrame or DeferredSeries

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.unstack(): Unstack prescribed level(s) from index axis onto column axis.
DeferredDataFrame.pivot(): Reshape dataframe from long format to wide format.
DeferredDataFrame.pivot_table(): Create a spreadsheet-style pivot table as a DeferredDataFrame.

Notes

The function is named by analogy with a collection of books being reorganized from being side by side on a horizontal position (the columns of the dataframe) to being stacked vertically on top of each other (in the index of the dataframe).

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

**Single level columns**

>>> df_single_level_cols = pd.DataFrame([[0, 1], [2, 3]],
...                                     index=['cat', 'dog'],
...                                     columns=['weight', 'height'])

Stacking a dataframe with a single level column axis returns a Series:

>>> df_single_level_cols
     weight height
cat       0      1
dog       2      3
>>> df_single_level_cols.stack()
cat  weight    0
     height    1
dog  weight    2
     height    3
dtype: int64

**Multi level columns: simple case**

>>> multicol1 = pd.MultiIndex.from_tuples([('weight', 'kg'),
...                                        ('weight', 'pounds')])
>>> df_multi_level_cols1 = pd.DataFrame([[1, 2], [2, 4]],
...                                     index=['cat', 'dog'],
...                                     columns=multicol1)

Stacking a dataframe with a multi-level column axis:

>>> df_multi_level_cols1
     weight
         kg    pounds
cat       1        2
dog       2        4
>>> df_multi_level_cols1.stack()
            weight
cat kg           1
    pounds       2
dog kg           2
    pounds       4

**Missing values**

>>> multicol2 = pd.MultiIndex.from_tuples([('weight', 'kg'),
...                                        ('height', 'm')])
>>> df_multi_level_cols2 = pd.DataFrame([[1.0, 2.0], [3.0, 4.0]],
...                                     index=['cat', 'dog'],
...                                     columns=multicol2)

It is common to have missing values when stacking a dataframe
with multi-level columns, as the stacked dataframe typically
has more values than the original dataframe. Missing values
are filled with NaNs:

>>> df_multi_level_cols2
    weight height
        kg      m
cat    1.0    2.0
dog    3.0    4.0
>>> df_multi_level_cols2.stack()
        height  weight
cat kg     NaN     1.0
    m      2.0     NaN
dog kg     NaN     3.0
    m      4.0     NaN

**Prescribing the level(s) to be stacked**

The first parameter controls which level or levels are stacked:

>>> df_multi_level_cols2.stack(0)
             kg    m
cat height  NaN  2.0
    weight  1.0  NaN
dog height  NaN  4.0
    weight  3.0  NaN
>>> df_multi_level_cols2.stack([0, 1])
cat  height  m     2.0
     weight  kg    1.0
dog  height  m     4.0
     weight  kg    3.0
dtype: float64

**Dropping missing values**

>>> df_multi_level_cols3 = pd.DataFrame([[None, 1.0], [2.0, 3.0]],
...                                     index=['cat', 'dog'],
...                                     columns=multicol2)

Note that rows where all values are missing are dropped by
default but this behaviour can be controlled via the dropna
keyword parameter:

>>> df_multi_level_cols3
    weight height
        kg      m
cat    NaN    1.0
dog    2.0    3.0
>>> df_multi_level_cols3.stack(dropna=False)
        height  weight
cat kg     NaN     NaN
    m      1.0     NaN
dog kg     NaN     2.0
    m      3.0     NaN
>>> df_multi_level_cols3.stack(dropna=True)
        height  weight
cat m      1.0     NaN
dog kg     NaN     2.0
    m      3.0     NaN

all(*args, **kwargs)¶

Return whether all elements are True, potentially over an axis.

Returns True unless there at least one element within a series or along a Dataframe axis that is False or equivalent (e.g. zero or empty).

Parameters:	axis ({0 or 'index', 1 or 'columns', None}, default 0) – Indicate which axis or axes should be reduced. 0 / ‘index’ : reduce the index, return a DeferredSeries whose index is the original column labels. 1 / ‘columns’ : reduce the columns, return a DeferredSeries whose index is the original index. None : reduce all axes, return a scalar. bool_only (bool, default None) – Include only boolean columns. If None, will attempt to use everything, then use only boolean data. Not implemented for DeferredSeries. skipna (bool, default True) – Exclude NA/null values. If the entire row/column is NA and skipna is True, then the result will be True, as for an empty row/column. If skipna is False, then NA are treated as True, because these are not equal to zero. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DeferredSeries. *kwargs (any, default None*) – Additional keywords have no effect but might be accepted for compatibility with NumPy.
Returns:	If level is specified, then, DeferredDataFrame is returned; otherwise, DeferredSeries is returned.
Return type:	DeferredSeries or DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.all(): Return True if all elements are True.
DeferredDataFrame.any(): Return True if one (or more) elements are True.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

**Series**

>>> pd.Series([True, True]).all()
True
>>> pd.Series([True, False]).all()
False
>>> pd.Series([]).all()
True
>>> pd.Series([np.nan]).all()
True
>>> pd.Series([np.nan]).all(skipna=False)
True

**DataFrames**

Create a dataframe from a dictionary.

>>> df = pd.DataFrame({'col1': [True, True], 'col2': [True, False]})
>>> df
   col1   col2
0  True   True
1  True  False

Default behaviour checks if column-wise values all return True.

>>> df.all()
col1     True
col2    False
dtype: bool

Specify ``axis='columns'`` to check if row-wise values all return True.

>>> df.all(axis='columns')
0     True
1    False
dtype: bool

Or ``axis=None`` for whether every value is True.

>>> df.all(axis=None)
False

any(*args, **kwargs)¶

Return whether any element is True, potentially over an axis.

Returns False unless there is at least one element within a series or along a Dataframe axis that is True or equivalent (e.g. non-zero or non-empty).

Parameters:	axis ({0 or 'index', 1 or 'columns', None}, default 0) – Indicate which axis or axes should be reduced. 0 / ‘index’ : reduce the index, return a DeferredSeries whose index is the original column labels. 1 / ‘columns’ : reduce the columns, return a DeferredSeries whose index is the original index. None : reduce all axes, return a scalar. bool_only (bool, default None) – Include only boolean columns. If None, will attempt to use everything, then use only boolean data. Not implemented for DeferredSeries. skipna (bool, default True) – Exclude NA/null values. If the entire row/column is NA and skipna is True, then the result will be False, as for an empty row/column. If skipna is False, then NA are treated as True, because these are not equal to zero. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DeferredSeries. *kwargs (any, default None*) – Additional keywords have no effect but might be accepted for compatibility with NumPy.
Returns:	If level is specified, then, DeferredDataFrame is returned; otherwise, DeferredSeries is returned.
Return type:	DeferredSeries or DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

numpy.any(): Numpy version of this method.
DeferredSeries.any(): Return whether any element is True.
DeferredSeries.all(): Return whether all elements are True.
DeferredDataFrame.any(): Return whether any element is True over requested axis.
DeferredDataFrame.all(): Return whether all elements are True over requested axis.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

**Series**

For Series input, the output is a scalar indicating whether any element
is True.

>>> pd.Series([False, False]).any()
False
>>> pd.Series([True, False]).any()
True
>>> pd.Series([]).any()
False
>>> pd.Series([np.nan]).any()
False
>>> pd.Series([np.nan]).any(skipna=False)
True

**DataFrame**

Whether each column contains at least one True element (the default).

>>> df = pd.DataFrame({"A": [1, 2], "B": [0, 2], "C": [0, 0]})
>>> df
   A  B  C
0  1  0  0
1  2  2  0

>>> df.any()
A     True
B     True
C    False
dtype: bool

Aggregating over the columns.

>>> df = pd.DataFrame({"A": [True, False], "B": [1, 2]})
>>> df
       A  B
0   True  1
1  False  2

>>> df.any(axis='columns')
0    True
1    True
dtype: bool

>>> df = pd.DataFrame({"A": [True, False], "B": [1, 0]})
>>> df
       A  B
0   True  1
1  False  0

>>> df.any(axis='columns')
0    True
1    False
dtype: bool

Aggregating over the entire DataFrame with ``axis=None``.

>>> df.any(axis=None)
True

`any` for an empty DataFrame is an empty Series.

>>> pd.DataFrame([]).any()
Series([], dtype: bool)

count(*args, **kwargs)¶

Count non-NA cells for each column or row.

The values None, NaN, NaT, and optionally numpy.inf (depending on pandas.options.mode.use_inf_as_na) are considered NA.

Parameters:	axis ({0 or 'index', 1 or 'columns'}, default 0) – If 0 or ‘index’ counts are generated for each column. If 1 or ‘columns’ counts are generated for each row. level (int or str, optional) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DeferredDataFrame. A str specifies the level name. numeric_only (bool, default False) – Include only float, int or boolean data.
Returns:	For each column/row the number of non-NA/null entries. If level is specified returns a DeferredDataFrame.
Return type:	DeferredSeries or DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.count(): Number of non-NA elements in a DeferredSeries.
DeferredDataFrame.value_counts(): Count unique combinations of columns.
DeferredDataFrame.shape(): Number of DeferredDataFrame rows and columns (including NA elements).
DeferredDataFrame.isna(): Boolean same-sized DeferredDataFrame showing places of NA elements.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Constructing DataFrame from a dictionary:

>>> df = pd.DataFrame({"Person":
...                    ["John", "Myla", "Lewis", "John", "Myla"],
...                    "Age": [24., np.nan, 21., 33, 26],
...                    "Single": [False, True, True, True, False]})
>>> df
   Person   Age  Single
0    John  24.0   False
1    Myla   NaN    True
2   Lewis  21.0    True
3    John  33.0    True
4    Myla  26.0   False

Notice the uncounted NA values:

>>> df.count()
Person    5
Age       4
Single    5
dtype: int64

Counts for each **row**:

>>> df.count(axis='columns')
0    3
1    2
2    3
3    3
4    3
dtype: int64

Counts for one level of a `MultiIndex`:

>>> df.set_index(["Person", "Single"]).count(level="Person")
        Age
Person
John      2
Lewis     1
Myla      1

describe(*args, **kwargs)¶

Generate descriptive statistics.

Descriptive statistics include those that summarize the central tendency, dispersion and shape of a dataset’s distribution, excluding NaN values.

Analyzes both numeric and object series, as well as DataFrame column sets of mixed data types. The output will vary depending on what is provided. Refer to the notes below for more detail.

Parameters:	percentiles (list-like of numbers, optional) – The percentiles to include in the output. All should fall between 0 and 1. The default is `[.25, .5, .75]`, which returns the 25th, 50th, and 75th percentiles. include ('all', list-like of dtypes or None (default), optional) – A white list of data types to include in the result. Ignored for `DeferredSeries`. Here are the options: ’all’ : All columns of the input will be included in the output. A list-like of dtypes : Limits the results to the provided data types. To limit the result to numeric types submit `numpy.number`. To limit it instead to object columns submit the `numpy.object` data type. Strings can also be used in the style of `select_dtypes` (e.g. `df.describe(include=['O'])`). To select pandas categorical columns, use `'category'` None (default) : The result will include all numeric columns. exclude (list-like of dtypes or None (default), optional,) – A black list of data types to omit from the result. Ignored for `DeferredSeries`. Here are the options: A list-like of dtypes : Excludes the provided data types from the result. To exclude numeric types submit `numpy.number`. To exclude object columns submit the data type `numpy.object`. Strings can also be used in the style of `select_dtypes` (e.g. `df.describe(include=['O'])`). To exclude pandas categorical columns, use `'category'` None (default) : The result will exclude nothing. datetime_is_numeric (bool, default False) – Whether to treat datetime dtypes as numeric. This affects statistics calculated for the column. For DeferredDataFrame input, this also controls whether datetime columns are included by default. New in version 1.1.0.
Returns:	Summary statistics of the DeferredSeries or Dataframe provided.
Return type:	DeferredSeries or DeferredDataFrame

Differences from pandas

describe cannot currently be parallelized. It will require collecting all data on a single node.

See also

DeferredDataFrame.count(): Count number of non-NA/null observations.
DeferredDataFrame.max(): Maximum of the values in the object.
DeferredDataFrame.min(): Minimum of the values in the object.
DeferredDataFrame.mean(): Mean of the values.
DeferredDataFrame.std(): Standard deviation of the observations.
DeferredDataFrame.select_dtypes(): Subset of a DeferredDataFrame including/excluding columns based on their dtype.

Notes

For numeric data, the result’s index will include count, mean, std, min, max as well as lower, 50 and upper percentiles. By default the lower percentile is 25 and the upper percentile is 75. The 50 percentile is the same as the median.

For object data (e.g. strings or timestamps), the result’s index will include count, unique, top, and freq. The top is the most common value. The freq is the most common value’s frequency. Timestamps also include the first and last items.

If multiple object values have the highest count, then the count and top results will be arbitrarily chosen from among those with the highest count.

For mixed data types provided via a DeferredDataFrame, the default is to return only an analysis of numeric columns. If the dataframe consists only of object and categorical data without any numeric columns, the default is to return an analysis of both the object and categorical columns. If include='all' is provided as an option, the result will include a union of attributes of each type.

The include and exclude parameters can be used to limit which columns in a DeferredDataFrame are analyzed for the output. The parameters are ignored when analyzing a DeferredSeries.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

Describing a numeric ``Series``.

>>> s = pd.Series([1, 2, 3])
>>> s.describe()
count    3.0
mean     2.0
std      1.0
min      1.0
25%      1.5
50%      2.0
75%      2.5
max      3.0
dtype: float64

Describing a categorical ``Series``.

>>> s = pd.Series(['a', 'a', 'b', 'c'])
>>> s.describe()
count     4
unique    3
top       a
freq      2
dtype: object

Describing a timestamp ``Series``.

>>> s = pd.Series([
...   np.datetime64("2000-01-01"),
...   np.datetime64("2010-01-01"),
...   np.datetime64("2010-01-01")
... ])
>>> s.describe(datetime_is_numeric=True)
count                      3
mean     2006-09-01 08:00:00
min      2000-01-01 00:00:00
25%      2004-12-31 12:00:00
50%      2010-01-01 00:00:00
75%      2010-01-01 00:00:00
max      2010-01-01 00:00:00
dtype: object

Describing a ``DataFrame``. By default only numeric fields
are returned.

>>> df = pd.DataFrame({'categorical': pd.Categorical(['d','e','f']),
...                    'numeric': [1, 2, 3],
...                    'object': ['a', 'b', 'c']
...                   })
>>> df.describe()
       numeric
count      3.0
mean       2.0
std        1.0
min        1.0
25%        1.5
50%        2.0
75%        2.5
max        3.0

Describing all columns of a ``DataFrame`` regardless of data type.

>>> df.describe(include='all')  
       categorical  numeric object
count            3      3.0      3
unique           3      NaN      3
top              f      NaN      a
freq             1      NaN      1
mean           NaN      2.0    NaN
std            NaN      1.0    NaN
min            NaN      1.0    NaN
25%            NaN      1.5    NaN
50%            NaN      2.0    NaN
75%            NaN      2.5    NaN
max            NaN      3.0    NaN

Describing a column from a ``DataFrame`` by accessing it as
an attribute.

>>> df.numeric.describe()
count    3.0
mean     2.0
std      1.0
min      1.0
25%      1.5
50%      2.0
75%      2.5
max      3.0
Name: numeric, dtype: float64

Including only numeric columns in a ``DataFrame`` description.

>>> df.describe(include=[np.number])
       numeric
count      3.0
mean       2.0
std        1.0
min        1.0
25%        1.5
50%        2.0
75%        2.5
max        3.0

Including only string columns in a ``DataFrame`` description.

>>> df.describe(include=[object])  
       object
count       3
unique      3
top         a
freq        1

Including only categorical columns from a ``DataFrame`` description.

>>> df.describe(include=['category'])
       categorical
count            3
unique           3
top              d
freq             1

Excluding numeric columns from a ``DataFrame`` description.

>>> df.describe(exclude=[np.number])  
       categorical object
count            3      3
unique           3      3
top              f      a
freq             1      1

Excluding object columns from a ``DataFrame`` description.

>>> df.describe(exclude=[object])  
       categorical  numeric
count            3      3.0
unique           3      NaN
top              f      NaN
freq             1      NaN
mean           NaN      2.0
std            NaN      1.0
min            NaN      1.0
25%            NaN      1.5
50%            NaN      2.0
75%            NaN      2.5
max            NaN      3.0

max(*args, **kwargs)¶

Return the maximum of the values over the requested axis.

If you want the index of the maximum, use idxmax. This isthe equivalent of the numpy.ndarray method argmax.

Parameters:	axis ({index (0), columns (1)}) – Axis for the function to be applied on. skipna (bool, default True) – Exclude NA/null values when computing the result. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DeferredSeries. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries. **kwargs – Additional keyword arguments to be passed to the function.
Returns:
Return type:	DeferredSeries or DeferredDataFrame (if level specified)

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.sum(): Return the sum.
DeferredSeries.min(): Return the minimum.
DeferredSeries.max(): Return the maximum.
DeferredSeries.idxmin(): Return the index of the minimum.
DeferredSeries.idxmax(): Return the index of the maximum.
DeferredDataFrame.sum(): Return the sum over the requested axis.
DeferredDataFrame.min(): Return the minimum over the requested axis.
DeferredDataFrame.max(): Return the maximum over the requested axis.
DeferredDataFrame.idxmin(): Return the index of the minimum over the requested axis.
DeferredDataFrame.idxmax(): Return the index of the maximum over the requested axis.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> idx = pd.MultiIndex.from_arrays([
...     ['warm', 'warm', 'cold', 'cold'],
...     ['dog', 'falcon', 'fish', 'spider']],
...     names=['blooded', 'animal'])
>>> s = pd.Series([4, 2, 0, 8], name='legs', index=idx)
>>> s
blooded  animal
warm     dog       4
         falcon    2
cold     fish      0
         spider    8
Name: legs, dtype: int64

>>> s.max()
8

Max using level names, as well as indices.

>>> s.max(level='blooded')
blooded
warm    4
cold    8
Name: legs, dtype: int64

>>> s.max(level=0)
blooded
warm    4
cold    8
Name: legs, dtype: int64

min(*args, **kwargs)¶

Return the minimum of the values over the requested axis.

If you want the index of the minimum, use idxmin. This isthe equivalent of the numpy.ndarray method argmin.

Parameters:	axis ({index (0), columns (1)}) – Axis for the function to be applied on. skipna (bool, default True) – Exclude NA/null values when computing the result. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DeferredSeries. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries. **kwargs – Additional keyword arguments to be passed to the function.
Returns:
Return type:	DeferredSeries or DeferredDataFrame (if level specified)

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.sum(): Return the sum.
DeferredSeries.min(): Return the minimum.
DeferredSeries.max(): Return the maximum.
DeferredSeries.idxmin(): Return the index of the minimum.
DeferredSeries.idxmax(): Return the index of the maximum.
DeferredDataFrame.sum(): Return the sum over the requested axis.
DeferredDataFrame.min(): Return the minimum over the requested axis.
DeferredDataFrame.max(): Return the maximum over the requested axis.
DeferredDataFrame.idxmin(): Return the index of the minimum over the requested axis.
DeferredDataFrame.idxmax(): Return the index of the maximum over the requested axis.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> idx = pd.MultiIndex.from_arrays([
...     ['warm', 'warm', 'cold', 'cold'],
...     ['dog', 'falcon', 'fish', 'spider']],
...     names=['blooded', 'animal'])
>>> s = pd.Series([4, 2, 0, 8], name='legs', index=idx)
>>> s
blooded  animal
warm     dog       4
         falcon    2
cold     fish      0
         spider    8
Name: legs, dtype: int64

>>> s.min()
0

Min using level names, as well as indices.

>>> s.min(level='blooded')
blooded
warm    2
cold    0
Name: legs, dtype: int64

>>> s.min(level=0)
blooded
warm    2
cold    0
Name: legs, dtype: int64

prod(*args, **kwargs)¶

Return the product of the values over the requested axis.

Parameters:	axis ({index (0), columns (1)}) – Axis for the function to be applied on. skipna (bool, default True) – Exclude NA/null values when computing the result. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DeferredSeries. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries. min_count (int, default 0) – The required number of valid values to perform the operation. If fewer than `min_count` non-NA values are present the result will be NA. **kwargs – Additional keyword arguments to be passed to the function.
Returns:
Return type:	DeferredSeries or DeferredDataFrame (if level specified)

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.sum(): Return the sum.
DeferredSeries.min(): Return the minimum.
DeferredSeries.max(): Return the maximum.
DeferredSeries.idxmin(): Return the index of the minimum.
DeferredSeries.idxmax(): Return the index of the maximum.
DeferredDataFrame.sum(): Return the sum over the requested axis.
DeferredDataFrame.min(): Return the minimum over the requested axis.
DeferredDataFrame.max(): Return the maximum over the requested axis.
DeferredDataFrame.idxmin(): Return the index of the minimum over the requested axis.
DeferredDataFrame.idxmax(): Return the index of the maximum over the requested axis.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

By default, the product of an empty or all-NA Series is ``1``

>>> pd.Series([]).prod()
1.0

This can be controlled with the ``min_count`` parameter

>>> pd.Series([]).prod(min_count=1)
nan

Thanks to the ``skipna`` parameter, ``min_count`` handles all-NA and
empty series identically.

>>> pd.Series([np.nan]).prod()
1.0

>>> pd.Series([np.nan]).prod(min_count=1)
nan

product(*args, **kwargs)¶

Return the product of the values over the requested axis.

Parameters:	axis ({index (0), columns (1)}) – Axis for the function to be applied on. skipna (bool, default True) – Exclude NA/null values when computing the result. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DeferredSeries. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries. min_count (int, default 0) – The required number of valid values to perform the operation. If fewer than `min_count` non-NA values are present the result will be NA. **kwargs – Additional keyword arguments to be passed to the function.
Returns:
Return type:	DeferredSeries or DeferredDataFrame (if level specified)

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.sum(): Return the sum.
DeferredSeries.min(): Return the minimum.
DeferredSeries.max(): Return the maximum.
DeferredSeries.idxmin(): Return the index of the minimum.
DeferredSeries.idxmax(): Return the index of the maximum.
DeferredDataFrame.sum(): Return the sum over the requested axis.
DeferredDataFrame.min(): Return the minimum over the requested axis.
DeferredDataFrame.max(): Return the maximum over the requested axis.
DeferredDataFrame.idxmin(): Return the index of the minimum over the requested axis.
DeferredDataFrame.idxmax(): Return the index of the maximum over the requested axis.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

By default, the product of an empty or all-NA Series is ``1``

>>> pd.Series([]).prod()
1.0

This can be controlled with the ``min_count`` parameter

>>> pd.Series([]).prod(min_count=1)
nan

Thanks to the ``skipna`` parameter, ``min_count`` handles all-NA and
empty series identically.

>>> pd.Series([np.nan]).prod()
1.0

>>> pd.Series([np.nan]).prod(min_count=1)
nan

sum(*args, **kwargs)¶

Return the sum of the values over the requested axis.

This is equivalent to the method numpy.sum.

Parameters:	axis ({index (0), columns (1)}) – Axis for the function to be applied on. skipna (bool, default True) – Exclude NA/null values when computing the result. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DeferredSeries. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries. min_count (int, default 0) – The required number of valid values to perform the operation. If fewer than `min_count` non-NA values are present the result will be NA. **kwargs – Additional keyword arguments to be passed to the function.
Returns:
Return type:	DeferredSeries or DeferredDataFrame (if level specified)

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.sum(): Return the sum.
DeferredSeries.min(): Return the minimum.
DeferredSeries.max(): Return the maximum.
DeferredSeries.idxmin(): Return the index of the minimum.
DeferredSeries.idxmax(): Return the index of the maximum.
DeferredDataFrame.sum(): Return the sum over the requested axis.
DeferredDataFrame.min(): Return the minimum over the requested axis.
DeferredDataFrame.max(): Return the maximum over the requested axis.
DeferredDataFrame.idxmin(): Return the index of the minimum over the requested axis.
DeferredDataFrame.idxmax(): Return the index of the maximum over the requested axis.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> idx = pd.MultiIndex.from_arrays([
...     ['warm', 'warm', 'cold', 'cold'],
...     ['dog', 'falcon', 'fish', 'spider']],
...     names=['blooded', 'animal'])
>>> s = pd.Series([4, 2, 0, 8], name='legs', index=idx)
>>> s
blooded  animal
warm     dog       4
         falcon    2
cold     fish      0
         spider    8
Name: legs, dtype: int64

>>> s.sum()
14

Sum using level names, as well as indices.

>>> s.sum(level='blooded')
blooded
warm    6
cold    8
Name: legs, dtype: int64

>>> s.sum(level=0)
blooded
warm    6
cold    8
Name: legs, dtype: int64

By default, the sum of an empty or all-NA Series is ``0``.

>>> pd.Series([]).sum()  # min_count=0 is the default
0.0

This can be controlled with the ``min_count`` parameter. For example, if
you'd like the sum of an empty series to be NaN, pass ``min_count=1``.

>>> pd.Series([]).sum(min_count=1)
nan

Thanks to the ``skipna`` parameter, ``min_count`` handles all-NA and
empty series identically.

>>> pd.Series([np.nan]).sum()
0.0

>>> pd.Series([np.nan]).sum(min_count=1)
nan

mean(*args, **kwargs)¶

Return the mean of the values over the requested axis.

Parameters:	axis ({index (0), columns (1)}) – Axis for the function to be applied on. skipna (bool, default True) – Exclude NA/null values when computing the result. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DeferredSeries. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries. **kwargs – Additional keyword arguments to be passed to the function.
Returns:
Return type:	DeferredSeries or DeferredDataFrame (if level specified)

Differences from pandas

mean cannot currently be parallelized. It will require collecting all data on a single node.

median(*args, **kwargs)¶

Return the median of the values over the requested axis.

Parameters:	axis ({index (0), columns (1)}) – Axis for the function to be applied on. skipna (bool, default True) – Exclude NA/null values when computing the result. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DeferredSeries. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries. **kwargs – Additional keyword arguments to be passed to the function.
Returns:
Return type:	DeferredSeries or DeferredDataFrame (if level specified)

Differences from pandas

median cannot currently be parallelized. It will require collecting all data on a single node.

std(*args, **kwargs)¶

Return sample standard deviation over requested axis.

Normalized by N-1 by default. This can be changed using the ddof argument

Parameters:	axis ({index (0), columns (1)}) – skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DeferredSeries. ddof (int, default 1) – Delta Degrees of Freedom. The divisor used in calculations is N - ddof, where N represents the number of elements. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries.
Returns:
Return type:	DeferredSeries or DeferredDataFrame (if level specified)

Differences from pandas

std cannot currently be parallelized. It will require collecting all data on a single node.

Notes

To have the same behaviour as numpy.std, use ddof=0 (instead of the default ddof=1)

var(*args, **kwargs)¶

Return unbiased variance over requested axis.

Normalized by N-1 by default. This can be changed using the ddof argument

Parameters:	axis ({index (0), columns (1)}) – skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA. level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DeferredSeries. ddof (int, default 1) – Delta Degrees of Freedom. The divisor used in calculations is N - ddof, where N represents the number of elements. numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for DeferredSeries.
Returns:
Return type:	DeferredSeries or DeferredDataFrame (if level specified)

Differences from pandas

var cannot currently be parallelized. It will require collecting all data on a single node.

Notes

To have the same behaviour as numpy.std, use ddof=0 (instead of the default ddof=1)

take(**kwargs)¶: pandas.DataFrame.take is not supported in the Beam DataFrame API because it is deprecated in pandas.

to_records(**kwargs)¶

pandas.DataFrame.to_records is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

to_dict(**kwargs)¶

pandas.DataFrame.to_dict is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

to_numpy(**kwargs)¶

pandas.DataFrame.to_numpy is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

to_string(**kwargs)¶

pandas.DataFrame.to_string is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

to_sparse(**kwargs)¶

pandas.DataFrame.to_sparse is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

transpose(**kwargs)¶

pandas.DataFrame.transpose is not supported in the Beam DataFrame API because the columns in the output DataFrame depend on the data.

For more information see {reason_data[‘url’]}.

unstack(*args, **kwargs)[source]¶

update(**kwargs)¶

Modify in place using non-NA values from another DataFrame.

Aligns on indices. There is no return value.

Parameters:	other (DeferredDataFrame, or object coercible into a DeferredDataFrame) – Should have at least one matching index/column label with the original DeferredDataFrame. If a DeferredSeries is passed, its name attribute must be set, and that will be used as the column name to align with the original DeferredDataFrame. join ({'left'}, default 'left') – Only left join is implemented, keeping the index and columns of the original object. overwrite (bool, default True) – How to handle non-NA values for overlapping keys: True: overwrite original DeferredDataFrame’s values with values from other. False: only update values that are NA in the original DeferredDataFrame. filter_func (callable(1d-array) -> bool 1d-array, optional) – Can choose to replace values other than NA. Return True for values that should be updated. errors ({'raise', 'ignore'}, default 'ignore') – If ‘raise’, will raise a ValueError if the DeferredDataFrame and other both contain non-NA data in the same place. Changed in version 0.24.0: Changed from raise_conflict=False\|True to errors=’ignore’\|’raise’.
Returns:	None
Return type:	method directly changes calling object
Raises:	`ValueError` – * When errors=’raise’ and there’s overlapping non-NA data. * When errors is not either ‘ignore’ or ‘raise’ `NotImplementedError` – * If join != ‘left’

Differences from pandas

This operation has no known divergences from the pandas API.

See also

dict.update(): Similar method for dictionaries.
DeferredDataFrame.merge(): For column(s)-on-column(s) operations.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame({'A': [1, 2, 3],
...                    'B': [400, 500, 600]})
>>> new_df = pd.DataFrame({'B': [4, 5, 6],
...                        'C': [7, 8, 9]})
>>> df.update(new_df)
>>> df
   A  B
0  1  4
1  2  5
2  3  6

The DataFrame's length does not increase as a result of the update,
only values at matching index/column labels are updated.

>>> df = pd.DataFrame({'A': ['a', 'b', 'c'],
...                    'B': ['x', 'y', 'z']})
>>> new_df = pd.DataFrame({'B': ['d', 'e', 'f', 'g', 'h', 'i']})
>>> df.update(new_df)
>>> df
   A  B
0  a  d
1  b  e
2  c  f

For Series, its name attribute must be set.

>>> df = pd.DataFrame({'A': ['a', 'b', 'c'],
...                    'B': ['x', 'y', 'z']})
>>> new_column = pd.Series(['d', 'e'], name='B', index=[0, 2])
>>> df.update(new_column)
>>> df
   A  B
0  a  d
1  b  y
2  c  e
>>> df = pd.DataFrame({'A': ['a', 'b', 'c'],
...                    'B': ['x', 'y', 'z']})
>>> new_df = pd.DataFrame({'B': ['d', 'e']}, index=[1, 2])
>>> df.update(new_df)
>>> df
   A  B
0  a  x
1  b  d
2  c  e

If `other` contains NaNs the corresponding values are not updated
in the original dataframe.

>>> df = pd.DataFrame({'A': [1, 2, 3],
...                    'B': [400, 500, 600]})
>>> new_df = pd.DataFrame({'B': [4, np.nan, 6]})
>>> df.update(new_df)
>>> df
   A      B
0  1    4.0
1  2  500.0
2  3    6.0

values¶

pandas.DataFrame.values is not supported in the Beam DataFrame API because it produces an output type that is not deferred.

For more information see {reason_data[‘url’]}.

melt(ignore_index, **kwargs)[source]¶

abs(**kwargs)¶

Return a Series/DataFrame with absolute numeric value of each element.

This function only applies to elements that are all numeric.

Returns:	DeferredSeries/DeferredDataFrame containing the absolute value of each element.
Return type:	abs

Differences from pandas

This operation has no known divergences from the pandas API.

See also

numpy.absolute(): Calculate the absolute value element-wise.

Notes

For complex inputs, 1.2 + 1j, the absolute value is \(\sqrt{ a^2 + b^2 }\).

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Absolute numeric values in a Series.

>>> s = pd.Series([-1.10, 2, -3.33, 4])
>>> s.abs()
0    1.10
1    2.00
2    3.33
3    4.00
dtype: float64

Absolute numeric values in a Series with complex numbers.

>>> s = pd.Series([1.2 + 1j])
>>> s.abs()
0    1.56205
dtype: float64

Absolute numeric values in a Series with a Timedelta element.

>>> s = pd.Series([pd.Timedelta('1 days')])
>>> s.abs()
0   1 days
dtype: timedelta64[ns]

Select rows with data closest to certain value using argsort (from
`StackOverflow <https://stackoverflow.com/a/17758115>`__).

>>> df = pd.DataFrame({
...     'a': [4, 5, 6, 7],
...     'b': [10, 20, 30, 40],
...     'c': [100, 50, -30, -50]
... })
>>> df
     a    b    c
0    4   10  100
1    5   20   50
2    6   30  -30
3    7   40  -50
>>> df.loc[(df.c - 43).abs().argsort()]
     a    b    c
1    5   20   50
0    4   10  100
2    6   30  -30
3    7   40  -50

add(**kwargs)¶

Get Addition of dataframe and other, element-wise (binary operator add).

Equivalent to dataframe + other, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, radd.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For DeferredSeries input, axis to match DeferredSeries index on. level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level. fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DeferredDataFrame alignment, with this value before computation. If data in both corresponding DeferredDataFrame locations is missing the result will be missing.
Returns:	Result of the arithmetic operation.
Return type:	DeferredDataFrame

Differences from pandas

Only level=None is supported

See also

DeferredDataFrame.add(): Add DeferredDataFrames.
DeferredDataFrame.sub(): Subtract DeferredDataFrames.
DeferredDataFrame.mul(): Multiply DeferredDataFrames.
DeferredDataFrame.div(): Divide DeferredDataFrames (float division).
DeferredDataFrame.truediv(): Divide DeferredDataFrames (float division).
DeferredDataFrame.floordiv(): Divide DeferredDataFrames (integer division).
DeferredDataFrame.mod(): Calculate modulo (remainder after division).
DeferredDataFrame.pow(): Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same
results.

>>> df + 1
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1)
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10)
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10)
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> df - [1, 2]
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns')
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
           angles
circle          0
triangle        3
rectangle       4

>>> df * other
           angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN

>>> df.mul(other, fill_value=0)
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

Divide by a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
             angles  degrees
A circle          0      360
  triangle        3      180
  rectangle       4      360
B square          4      360
  pentagon        5      540
  hexagon         6      720

>>> df.div(df_multindex, level=1, fill_value=0)
             angles  degrees
A circle        NaN      1.0
  triangle      1.0      1.0
  rectangle     1.0      1.0
B square        0.0      0.0
  pentagon      0.0      0.0
  hexagon       0.0      0.0

apply(**kwargs)¶

‘apply’ is not implemented yet.

If support for ‘apply’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

asfreq(**kwargs)¶

‘asfreq’ is not implemented yet.

If support for ‘asfreq’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

asof(**kwargs)¶

‘asof’ is not implemented yet.

If support for ‘asof’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

astype(**kwargs)¶

Cast a pandas object to a specified dtype dtype.

Parameters:	dtype (data type, or dict of column name -> data type) – Use a numpy.dtype or Python type to cast entire pandas object to the same type. Alternatively, use {col: dtype, …}, where col is a column label and dtype is a numpy.dtype or Python type to cast one or more of the DeferredDataFrame’s columns to column-specific types. copy (bool, default True) – Return a copy when `copy=True` (be very careful setting `copy=False` as changes to values then may propagate to other pandas objects). errors ({'raise', 'ignore'}, default 'raise') – Control raising of exceptions on invalid data for provided dtype. `raise` : allow exceptions to be raised `ignore` : suppress exceptions. On error return original object.
Returns:	casted
Return type:	same type as caller

Differences from pandas

This operation has no known divergences from the pandas API.

See also

to_datetime(): Convert argument to datetime.
to_timedelta(): Convert argument to timedelta.
to_numeric(): Convert argument to a numeric type.
numpy.ndarray.astype(): Cast a numpy array to a specified type.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Create a DataFrame:

>>> d = {'col1': [1, 2], 'col2': [3, 4]}
>>> df = pd.DataFrame(data=d)
>>> df.dtypes
col1    int64
col2    int64
dtype: object

Cast all columns to int32:

>>> df.astype('int32').dtypes
col1    int32
col2    int32
dtype: object

Cast col1 to int32 using a dictionary:

>>> df.astype({'col1': 'int32'}).dtypes
col1    int32
col2    int64
dtype: object

Create a series:

>>> ser = pd.Series([1, 2], dtype='int32')
>>> ser
0    1
1    2
dtype: int32
>>> ser.astype('int64')
0    1
1    2
dtype: int64

Convert to categorical type:

>>> ser.astype('category')
0    1
1    2
dtype: category
Categories (2, int64): [1, 2]

Convert to ordered categorical type with custom ordering:

>>> cat_dtype = pd.api.types.CategoricalDtype(
...     categories=[2, 1], ordered=True)
>>> ser.astype(cat_dtype)
0    1
1    2
dtype: category
Categories (2, int64): [2 < 1]

Note that using ``copy=False`` and changing data on a new
pandas object may propagate changes:

>>> s1 = pd.Series([1, 2])
>>> s2 = s1.astype('int64', copy=False)
>>> s2[0] = 10
>>> s1  # note that s1[0] has changed too
0    10
1     2
dtype: int64

Create a series of dates:

>>> ser_date = pd.Series(pd.date_range('20200101', periods=3))
>>> ser_date
0   2020-01-01
1   2020-01-02
2   2020-01-03
dtype: datetime64[ns]

Datetimes are localized to UTC first before
converting to the specified timezone:

>>> ser_date.astype('datetime64[ns, US/Eastern]')
0   2019-12-31 19:00:00-05:00
1   2020-01-01 19:00:00-05:00
2   2020-01-02 19:00:00-05:00
dtype: datetime64[ns, US/Eastern]

at¶

‘at’ is not implemented yet.

If support for ‘at’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

at_time(**kwargs)¶

‘at_time’ is not implemented yet.

If support for ‘at_time’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

attrs¶: pandas.DataFrame.attrs is not supported in the Beam DataFrame API because it is experimental in pandas.

backfill(*args, **kwargs)¶

Synonym for DataFrame.fillna() with method='bfill'.

Returns:	Object with missing values filled or None if `inplace=True`.
Return type:	DeferredSeries/DeferredDataFrame or None

Differences from pandas

backfill is only supported for axis=”columns”. axis=”index” is order-sensitive.

between_time(**kwargs)¶

‘between_time’ is not implemented yet.

If support for ‘between_time’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

bfill(*args, **kwargs)¶

Synonym for DataFrame.fillna() with method='bfill'.

Returns:	Object with missing values filled or None if `inplace=True`.
Return type:	DeferredSeries/DeferredDataFrame or None

Differences from pandas

bfill is only supported for axis=”columns”. axis=”index” is order-sensitive.

bool()¶

Return the bool of a single element Series or DataFrame.

This must be a boolean scalar value, either True or False. It will raise a ValueError if the Series or DataFrame does not have exactly 1 element, or that element is not boolean (integer values 0 and 1 will also raise an exception).

Returns:	The value in the DeferredSeries or DeferredDataFrame.
Return type:	bool

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.astype(): Change the data type of a DeferredSeries, including to boolean.
DeferredDataFrame.astype(): Change the data type of a DeferredDataFrame, including to boolean.
numpy.bool_(): NumPy boolean data type, used by pandas for boolean values.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

The method will only work for single element objects with a boolean value:

>>> pd.Series([True]).bool()
True
>>> pd.Series([False]).bool()
False

>>> pd.DataFrame({'col': [True]}).bool()
True
>>> pd.DataFrame({'col': [False]}).bool()
False

boxplot(**kwargs)¶

‘boxplot’ is not implemented yet.

If support for ‘boxplot’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

combine(**kwargs)¶

Perform column-wise combine with another DataFrame.

Combines a DataFrame with other DataFrame using func to element-wise combine columns. The row and column indexes of the resulting DataFrame will be the union of the two.

Parameters:	other (DeferredDataFrame) – The DeferredDataFrame to merge column-wise. func (function) – Function that takes two series as inputs and return a DeferredSeries or a scalar. Used to merge the two dataframes column by columns. fill_value (scalar value, default None) – The value to fill NaNs with prior to passing any column to the merge func. overwrite (bool, default True) – If True, columns in self that do not exist in other will be overwritten with NaNs.
Returns:	Combination of the provided DeferredDataFrames.
Return type:	DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.combine_first(): Combine two DeferredDataFrame objects and default to non-null values in frame calling the method.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Combine using a simple function that chooses the smaller column.

>>> df1 = pd.DataFrame({'A': [0, 0], 'B': [4, 4]})
>>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})
>>> take_smaller = lambda s1, s2: s1 if s1.sum() < s2.sum() else s2
>>> df1.combine(df2, take_smaller)
   A  B
0  0  3
1  0  3

Example using a true element-wise combine function.

>>> df1 = pd.DataFrame({'A': [5, 0], 'B': [2, 4]})
>>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})
>>> df1.combine(df2, np.minimum)
   A  B
0  1  2
1  0  3

Using `fill_value` fills Nones prior to passing the column to the
merge function.

>>> df1 = pd.DataFrame({'A': [0, 0], 'B': [None, 4]})
>>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})
>>> df1.combine(df2, take_smaller, fill_value=-5)
   A    B
0  0 -5.0
1  0  4.0

However, if the same element in both dataframes is None, that None
is preserved

>>> df1 = pd.DataFrame({'A': [0, 0], 'B': [None, 4]})
>>> df2 = pd.DataFrame({'A': [1, 1], 'B': [None, 3]})
>>> df1.combine(df2, take_smaller, fill_value=-5)
    A    B
0  0 -5.0
1  0  3.0

Example that demonstrates the use of `overwrite` and behavior when
the axis differ between the dataframes.

>>> df1 = pd.DataFrame({'A': [0, 0], 'B': [4, 4]})
>>> df2 = pd.DataFrame({'B': [3, 3], 'C': [-10, 1], }, index=[1, 2])
>>> df1.combine(df2, take_smaller)
     A    B     C
0  NaN  NaN   NaN
1  NaN  3.0 -10.0
2  NaN  3.0   1.0

>>> df1.combine(df2, take_smaller, overwrite=False)
     A    B     C
0  0.0  NaN   NaN
1  0.0  3.0 -10.0
2  NaN  3.0   1.0

Demonstrating the preference of the passed in dataframe.

>>> df2 = pd.DataFrame({'B': [3, 3], 'C': [1, 1], }, index=[1, 2])
>>> df2.combine(df1, take_smaller)
   A    B   C
0  0.0  NaN NaN
1  0.0  3.0 NaN
2  NaN  3.0 NaN

>>> df2.combine(df1, take_smaller, overwrite=False)
     A    B   C
0  0.0  NaN NaN
1  0.0  3.0 1.0
2  NaN  3.0 1.0

combine_first(**kwargs)¶

Update null elements with value in the same location in other.

Combine two DataFrame objects by filling null values in one DataFrame with non-null values from other DataFrame. The row and column indexes of the resulting DataFrame will be the union of the two.

Parameters:	other (DeferredDataFrame) – Provided DeferredDataFrame to use to fill null values.
Returns:
Return type:	DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.combine(): Perform series-wise operation on two DeferredDataFrames using a given function.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df1 = pd.DataFrame({'A': [None, 0], 'B': [None, 4]})
>>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})
>>> df1.combine_first(df2)
     A    B
0  1.0  3.0
1  0.0  4.0

Null values still persist if the location of that null value
does not exist in `other`

>>> df1 = pd.DataFrame({'A': [None, 0], 'B': [4, None]})
>>> df2 = pd.DataFrame({'B': [3, 3], 'C': [1, 1]}, index=[1, 2])
>>> df1.combine_first(df2)
     A    B    C
0  NaN  4.0  NaN
1  0.0  3.0  1.0
2  NaN  3.0  1.0

compare(**kwargs)¶

‘compare’ is not implemented yet.

If support for ‘compare’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

convert_dtypes(**kwargs)¶

‘convert_dtypes’ is not implemented yet.

If support for ‘convert_dtypes’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

copy(**kwargs)¶

Make a copy of this object’s indices and data.

When deep=True (default), a new object will be created with a copy of the calling object’s data and indices. Modifications to the data or indices of the copy will not be reflected in the original object (see notes below).

When deep=False, a new object will be created without copying the calling object’s data or index (only references to the data and index are copied). Any changes to the data of the original will be reflected in the shallow copy (and vice versa).

Parameters:	deep (bool, default True) – Make a deep copy, including a copy of the data and the indices. With `deep=False` neither the indices nor the data are copied.
Returns:	copy – Object type matches caller.
Return type:	DeferredSeries or DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

Notes

When deep=True, data is copied but actual Python objects will not be copied recursively, only the reference to the object. This is in contrast to copy.deepcopy in the Standard Library, which recursively copies object data (see examples below).

While Index objects are copied when deep=True, the underlying numpy array is not copied for performance reasons. Since Index is immutable, the underlying data can be safely shared and a copy is not needed.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series([1, 2], index=["a", "b"])
>>> s
a    1
b    2
dtype: int64

>>> s_copy = s.copy()
>>> s_copy
a    1
b    2
dtype: int64

**Shallow copy versus default (deep) copy:**

>>> s = pd.Series([1, 2], index=["a", "b"])
>>> deep = s.copy()
>>> shallow = s.copy(deep=False)

Shallow copy shares data and index with original.

>>> s is shallow
False
>>> s.values is shallow.values and s.index is shallow.index
True

Deep copy has own copy of data and index.

>>> s is deep
False
>>> s.values is deep.values or s.index is deep.index
False

Updates to the data shared by shallow copy and original is reflected
in both; deep copy remains unchanged.

>>> s[0] = 3
>>> shallow[1] = 4
>>> s
a    3
b    4
dtype: int64
>>> shallow
a    3
b    4
dtype: int64
>>> deep
a    1
b    2
dtype: int64

Note that when copying an object containing Python objects, a deep copy
will copy the data, but will not do so recursively. Updating a nested
data object will be reflected in the deep copy.

>>> s = pd.Series([[1, 2], [3, 4]])
>>> deep = s.copy()
>>> s[0][0] = 10
>>> s
0    [10, 2]
1     [3, 4]
dtype: object
>>> deep
0    [10, 2]
1     [3, 4]
dtype: object

div(**kwargs)¶

Get Floating division of dataframe and other, element-wise (binary operator truediv).

Equivalent to dataframe / other, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rtruediv.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For DeferredSeries input, axis to match DeferredSeries index on. level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level. fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DeferredDataFrame alignment, with this value before computation. If data in both corresponding DeferredDataFrame locations is missing the result will be missing.
Returns:	Result of the arithmetic operation.
Return type:	DeferredDataFrame

Differences from pandas

Only level=None is supported

See also

DeferredDataFrame.add(): Add DeferredDataFrames.
DeferredDataFrame.sub(): Subtract DeferredDataFrames.
DeferredDataFrame.mul(): Multiply DeferredDataFrames.
DeferredDataFrame.div(): Divide DeferredDataFrames (float division).
DeferredDataFrame.truediv(): Divide DeferredDataFrames (float division).
DeferredDataFrame.floordiv(): Divide DeferredDataFrames (integer division).
DeferredDataFrame.mod(): Calculate modulo (remainder after division).
DeferredDataFrame.pow(): Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same
results.

>>> df + 1
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1)
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10)
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10)
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> df - [1, 2]
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns')
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
           angles
circle          0
triangle        3
rectangle       4

>>> df * other
           angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN

>>> df.mul(other, fill_value=0)
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

Divide by a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
             angles  degrees
A circle          0      360
  triangle        3      180
  rectangle       4      360
B square          4      360
  pentagon        5      540
  hexagon         6      720

>>> df.div(df_multindex, level=1, fill_value=0)
             angles  degrees
A circle        NaN      1.0
  triangle      1.0      1.0
  rectangle     1.0      1.0
B square        0.0      0.0
  pentagon      0.0      0.0
  hexagon       0.0      0.0

divide(**kwargs)¶

‘divide’ is not implemented yet.

If support for ‘divide’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

drop(labels, axis, index, columns, errors, **kwargs)¶

Drop specified labels from rows or columns.

Remove rows or columns by specifying label names and corresponding axis, or by specifying directly index or column names. When using a multi-index, labels on different levels can be removed by specifying the level.

Parameters:	labels (single label or list-like) – Index or column labels to drop. axis ({0 or 'index', 1 or 'columns'}, default 0) – Whether to drop labels from the index (0 or ‘index’) or columns (1 or ‘columns’). index (single label or list-like) – Alternative to specifying axis (`labels, axis=0` is equivalent to `index=labels`). columns (single label or list-like) – Alternative to specifying axis (`labels, axis=1` is equivalent to `columns=labels`). level (int or level name, optional) – For MultiIndex, level from which the labels will be removed. inplace (bool, default False) – If False, return a copy. Otherwise, do operation inplace and return None. errors ({'ignore', 'raise'}, default 'raise') – If ‘ignore’, suppress error and only existing labels are dropped.
Returns:	DeferredDataFrame without the removed index or column labels or None if `inplace=True`.
Return type:	DeferredDataFrame or None
Raises:	`KeyError` – If any of the labels is not found in the selected axis.

Differences from pandas

drop is not parallelizable when dropping from the index and errors="raise" is specified. It requires collecting all data on a single node in order to detect if one of the index values is missing.

See also

DeferredDataFrame.loc(): Label-location based indexer for selection by label.
DeferredDataFrame.dropna(): Return DeferredDataFrame with labels on given axis omitted where (all or any) data are missing.
DeferredDataFrame.drop_duplicates(): Return DeferredDataFrame with duplicate rows removed, optionally only considering certain columns.
DeferredSeries.drop(): Return DeferredSeries with specified index labels removed.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame(np.arange(12).reshape(3, 4),
...                   columns=['A', 'B', 'C', 'D'])
>>> df
   A  B   C   D
0  0  1   2   3
1  4  5   6   7
2  8  9  10  11

Drop columns

>>> df.drop(['B', 'C'], axis=1)
   A   D
0  0   3
1  4   7
2  8  11

>>> df.drop(columns=['B', 'C'])
   A   D
0  0   3
1  4   7
2  8  11

Drop a row by index

>>> df.drop([0, 1])
   A  B   C   D
2  8  9  10  11

Drop columns and/or rows of MultiIndex DataFrame

>>> midx = pd.MultiIndex(levels=[['lama', 'cow', 'falcon'],
...                              ['speed', 'weight', 'length']],
...                      codes=[[0, 0, 0, 1, 1, 1, 2, 2, 2],
...                             [0, 1, 2, 0, 1, 2, 0, 1, 2]])
>>> df = pd.DataFrame(index=midx, columns=['big', 'small'],
...                   data=[[45, 30], [200, 100], [1.5, 1], [30, 20],
...                         [250, 150], [1.5, 0.8], [320, 250],
...                         [1, 0.8], [0.3, 0.2]])
>>> df
                big     small
lama    speed   45.0    30.0
        weight  200.0   100.0
        length  1.5     1.0
cow     speed   30.0    20.0
        weight  250.0   150.0
        length  1.5     0.8
falcon  speed   320.0   250.0
        weight  1.0     0.8
        length  0.3     0.2

>>> df.drop(index='cow', columns='small')
                big
lama    speed   45.0
        weight  200.0
        length  1.5
falcon  speed   320.0
        weight  1.0
        length  0.3

>>> df.drop(index='length', level=1)
                big     small
lama    speed   45.0    30.0
        weight  200.0   100.0
cow     speed   30.0    20.0
        weight  250.0   150.0
falcon  speed   320.0   250.0
        weight  1.0     0.8

droplevel(level, axis)¶

Return DataFrame with requested index / column level(s) removed.

New in version 0.24.0.

Parameters:	level (int, str, or list-like) – If a string is given, must be the name of a level If list-like, elements must be names or positional indexes of levels. axis ({0 or 'index', 1 or 'columns'}, default 0) – Axis along which the level(s) is removed: 0 or ‘index’: remove level(s) in column. 1 or ‘columns’: remove level(s) in row.
Returns:	DeferredDataFrame with requested index / column level(s) removed.
Return type:	DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame([
...     [1, 2, 3, 4],
...     [5, 6, 7, 8],
...     [9, 10, 11, 12]
... ]).set_index([0, 1]).rename_axis(['a', 'b'])

>>> df.columns = pd.MultiIndex.from_tuples([
...     ('c', 'e'), ('d', 'f')
... ], names=['level_1', 'level_2'])

>>> df
level_1   c   d
level_2   e   f
a b
1 2      3   4
5 6      7   8
9 10    11  12

>>> df.droplevel('a')
level_1   c   d
level_2   e   f
b
2        3   4
6        7   8
10      11  12

>>> df.droplevel('level_2', axis=1)
level_1   c   d
a b
1 2      3   4
5 6      7   8
9 10    11  12

dtype¶

empty¶

eq(**kwargs)¶

Get Equal to of dataframe and other, element-wise (binary operator eq).

Among flexible wrappers (eq, ne, le, lt, ge, gt) to comparison operators.

Equivalent to ==, !=, <=, <, >=, > with support to choose axis (rows or columns) and level for comparison.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}, default 'columns') – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	Result of the comparison.
Return type:	DeferredDataFrame of bool

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.eq(): Compare DeferredDataFrames for equality elementwise.
DeferredDataFrame.ne(): Compare DeferredDataFrames for inequality elementwise.
DeferredDataFrame.le(): Compare DeferredDataFrames for less than inequality or equality elementwise.
DeferredDataFrame.lt(): Compare DeferredDataFrames for strictly less than inequality elementwise.
DeferredDataFrame.ge(): Compare DeferredDataFrames for greater than inequality or equality elementwise.
DeferredDataFrame.gt(): Compare DeferredDataFrames for strictly greater than inequality elementwise.

Notes

Mismatched indices will be unioned together. NaN values are considered different (i.e. NaN != NaN).

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame({'cost': [250, 150, 100],
...                    'revenue': [100, 250, 300]},
...                   index=['A', 'B', 'C'])
>>> df
   cost  revenue
A   250      100
B   150      250
C   100      300

Comparison with a scalar, using either the operator or method:

>>> df == 100
    cost  revenue
A  False     True
B  False    False
C   True    False

>>> df.eq(100)
    cost  revenue
A  False     True
B  False    False
C   True    False

When `other` is a :class:`Series`, the columns of a DataFrame are aligned
with the index of `other` and broadcast:

>>> df != pd.Series([100, 250], index=["cost", "revenue"])
    cost  revenue
A   True     True
B   True    False
C  False     True

Use the method to control the broadcast axis:

>>> df.ne(pd.Series([100, 300], index=["A", "D"]), axis='index')
   cost  revenue
A  True    False
B  True     True
C  True     True
D  True     True

When comparing to an arbitrary sequence, the number of columns must
match the number elements in `other`:

>>> df == [250, 100]
    cost  revenue
A   True     True
B  False    False
C  False    False

Use the method to control the axis:

>>> df.eq([250, 250, 100], axis='index')
    cost  revenue
A   True    False
B  False     True
C   True    False

Compare to a DataFrame of different shape.

>>> other = pd.DataFrame({'revenue': [300, 250, 100, 150]},
...                      index=['A', 'B', 'C', 'D'])
>>> other
   revenue
A      300
B      250
C      100
D      150

>>> df.gt(other)
    cost  revenue
A  False    False
B  False    False
C  False     True
D  False    False

Compare to a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'cost': [250, 150, 100, 150, 300, 220],
...                              'revenue': [100, 250, 300, 200, 175, 225]},
...                             index=[['Q1', 'Q1', 'Q1', 'Q2', 'Q2', 'Q2'],
...                                    ['A', 'B', 'C', 'A', 'B', 'C']])
>>> df_multindex
      cost  revenue
Q1 A   250      100
   B   150      250
   C   100      300
Q2 A   150      200
   B   300      175
   C   220      225

>>> df.le(df_multindex, level=1)
       cost  revenue
Q1 A   True     True
   B   True     True
   C   True     True
Q2 A  False     True
   B   True    False
   C   True    False

equals(other)¶

Test whether two objects contain the same elements.

This function allows two Series or DataFrames to be compared against each other to see if they have the same shape and elements. NaNs in the same location are considered equal.

The row/column index do not need to have the same type, as long as the values are considered equal. Corresponding columns must be of the same dtype.

Parameters:	other (DeferredSeries or DeferredDataFrame) – The other DeferredSeries or DeferredDataFrame to be compared with the first.
Returns:	True if all elements are the same in both objects, False otherwise.
Return type:	bool

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredSeries.eq(): Compare two DeferredSeries objects of the same length and return a DeferredSeries where each element is True if the element in each DeferredSeries is equal, False otherwise.
DeferredDataFrame.eq(): Compare two DeferredDataFrame objects of the same shape and return a DeferredDataFrame where each element is True if the respective element in each DeferredDataFrame is equal, False otherwise.
testing.assert_series_equal(): Raises an AssertionError if left and right are not equal. Provides an easy interface to ignore inequality in dtypes, indexes and precision among others.
testing.assert_frame_equal(): Like assert_series_equal, but targets DeferredDataFrames.
numpy.array_equal(): Return True if two arrays have the same shape and elements, False otherwise.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame({1: [10], 2: [20]})
>>> df
    1   2
0  10  20

DataFrames df and exactly_equal have the same types and values for
their elements and column labels, which will return True.

>>> exactly_equal = pd.DataFrame({1: [10], 2: [20]})
>>> exactly_equal
    1   2
0  10  20
>>> df.equals(exactly_equal)
True

DataFrames df and different_column_type have the same element
types and values, but have different types for the column labels,
which will still return True.

>>> different_column_type = pd.DataFrame({1.0: [10], 2.0: [20]})
>>> different_column_type
   1.0  2.0
0   10   20
>>> df.equals(different_column_type)
True

DataFrames df and different_data_type have different types for the
same values for their elements, and will return False even though
their column labels are the same values and types.

>>> different_data_type = pd.DataFrame({1: [10.0], 2: [20.0]})
>>> different_data_type
      1     2
0  10.0  20.0
>>> df.equals(different_data_type)
False

ewm(**kwargs)¶

‘ewm’ is not implemented yet.

If support for ‘ewm’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

expanding(**kwargs)¶

‘expanding’ is not implemented yet.

If support for ‘expanding’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

ffill(*args, **kwargs)¶

Synonym for DataFrame.fillna() with method='ffill'.

Returns:	Object with missing values filled or None if `inplace=True`.
Return type:	DeferredSeries/DeferredDataFrame or None

Differences from pandas

ffill is only supported for axis=”columns”. axis=”index” is order-sensitive.

fillna(value, method, axis, limit, **kwargs)¶

Fill NA/NaN values using the specified method.

Parameters:	value (scalar, dict, DeferredSeries, or DeferredDataFrame) – Value to use to fill holes (e.g. 0), alternately a dict/DeferredSeries/DeferredDataFrame of values specifying which value to use for each index (for a DeferredSeries) or column (for a DeferredDataFrame). Values not in the dict/DeferredSeries/DeferredDataFrame will not be filled. This value cannot be a list. method ({'backfill', 'bfill', 'pad', 'ffill', None}, default None) – Method to use for filling holes in reindexed DeferredSeries pad / ffill: propagate last valid observation forward to next valid backfill / bfill: use next valid observation to fill gap. axis ({0 or 'index', 1 or 'columns'}) – Axis along which to fill missing values. inplace (bool, default False) – If True, fill in-place. Note: this will modify any other views on this object (e.g., a no-copy slice for a column in a DeferredDataFrame). limit (int, default None) – If method is specified, this is the maximum number of consecutive NaN values to forward/backward fill. In other words, if there is a gap with more than this number of consecutive NaNs, it will only be partially filled. If method is not specified, this is the maximum number of entries along the entire axis where NaNs will be filled. Must be greater than 0 if not None. downcast (dict, default is None) – A dict of item->dtype of what to downcast if possible, or the string ‘infer’ which will try to downcast to an appropriate equal type (e.g. float64 to int64 if possible).
Returns:	Object with missing values filled or None if `inplace=True`.
Return type:	DeferredDataFrame or None

Differences from pandas

When axis="index", both method and limit must be None. otherwise this operation is order-sensitive.

See also

interpolate(): Fill NaN values using interpolation.
reindex(): Conform object to new index.
asfreq(): Convert TimeDeferredSeries to specified frequency.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame([[np.nan, 2, np.nan, 0],
...                    [3, 4, np.nan, 1],
...                    [np.nan, np.nan, np.nan, 5],
...                    [np.nan, 3, np.nan, 4]],
...                   columns=list('ABCD'))
>>> df
     A    B   C  D
0  NaN  2.0 NaN  0
1  3.0  4.0 NaN  1
2  NaN  NaN NaN  5
3  NaN  3.0 NaN  4

Replace all NaN elements with 0s.

>>> df.fillna(0)
    A   B   C   D
0   0.0 2.0 0.0 0
1   3.0 4.0 0.0 1
2   0.0 0.0 0.0 5
3   0.0 3.0 0.0 4

We can also propagate non-null values forward or backward.

>>> df.fillna(method='ffill')
    A   B   C   D
0   NaN 2.0 NaN 0
1   3.0 4.0 NaN 1
2   3.0 4.0 NaN 5
3   3.0 3.0 NaN 4

Replace all NaN elements in column 'A', 'B', 'C', and 'D', with 0, 1,
2, and 3 respectively.

>>> values = {'A': 0, 'B': 1, 'C': 2, 'D': 3}
>>> df.fillna(value=values)
    A   B   C   D
0   0.0 2.0 2.0 0
1   3.0 4.0 2.0 1
2   0.0 1.0 2.0 5
3   0.0 3.0 2.0 4

Only replace the first NaN element.

>>> df.fillna(value=values, limit=1)
    A   B   C   D
0   0.0 2.0 2.0 0
1   3.0 4.0 NaN 1
2   NaN 1.0 NaN 5
3   NaN 3.0 NaN 4

filter(**kwargs)¶

Subset the dataframe rows or columns according to the specified index labels.

Note that this routine does not filter a dataframe on its contents. The filter is applied to the labels of the index.

Parameters:	items (list-like) – Keep labels from axis which are in items. like (str) – Keep labels from axis for which “like in label == True”. regex (str (regular expression)) – Keep labels from axis for which re.search(regex, label) == True. axis ({0 or ‘index’, 1 or ‘columns’, None}, default None) – The axis to filter on, expressed either as an index (int) or axis name (str). By default this is the info axis, ‘index’ for DeferredSeries, ‘columns’ for DeferredDataFrame.
Returns:
Return type:	same type as input object

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.loc(): Access a group of rows and columns by label(s) or a boolean array.

Notes

The items, like, and regex parameters are enforced to be mutually exclusive.

axis defaults to the info axis that is used when indexing with [].

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame(np.array(([1, 2, 3], [4, 5, 6])),
...                   index=['mouse', 'rabbit'],
...                   columns=['one', 'two', 'three'])
>>> df
        one  two  three
mouse     1    2      3
rabbit    4    5      6

>>> # select columns by name
>>> df.filter(items=['one', 'three'])
         one  three
mouse     1      3
rabbit    4      6

>>> # select columns by regular expression
>>> df.filter(regex='e$', axis=1)
         one  three
mouse     1      3
rabbit    4      6

>>> # select rows containing 'bbi'
>>> df.filter(like='bbi', axis=0)
         one  two  three
rabbit    4    5      6

first_valid_index(**kwargs)¶

‘first_valid_index’ is not implemented yet.

If support for ‘first_valid_index’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

flags¶

‘flags’ is not implemented yet.

If support for ‘flags’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

floordiv(**kwargs)¶

Get Integer division of dataframe and other, element-wise (binary operator floordiv).

Equivalent to dataframe // other, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rfloordiv.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For DeferredSeries input, axis to match DeferredSeries index on. level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level. fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DeferredDataFrame alignment, with this value before computation. If data in both corresponding DeferredDataFrame locations is missing the result will be missing.
Returns:	Result of the arithmetic operation.
Return type:	DeferredDataFrame

Differences from pandas

Only level=None is supported

See also

DeferredDataFrame.add(): Add DeferredDataFrames.
DeferredDataFrame.sub(): Subtract DeferredDataFrames.
DeferredDataFrame.mul(): Multiply DeferredDataFrames.
DeferredDataFrame.div(): Divide DeferredDataFrames (float division).
DeferredDataFrame.truediv(): Divide DeferredDataFrames (float division).
DeferredDataFrame.floordiv(): Divide DeferredDataFrames (integer division).
DeferredDataFrame.mod(): Calculate modulo (remainder after division).
DeferredDataFrame.pow(): Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same
results.

>>> df + 1
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1)
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10)
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10)
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> df - [1, 2]
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns')
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
           angles
circle          0
triangle        3
rectangle       4

>>> df * other
           angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN

>>> df.mul(other, fill_value=0)
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

Divide by a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
             angles  degrees
A circle          0      360
  triangle        3      180
  rectangle       4      360
B square          4      360
  pentagon        5      540
  hexagon         6      720

>>> df.div(df_multindex, level=1, fill_value=0)
             angles  degrees
A circle        NaN      1.0
  triangle      1.0      1.0
  rectangle     1.0      1.0
B square        0.0      0.0
  pentagon      0.0      0.0
  hexagon       0.0      0.0

from_dict(**kwargs)¶

‘from_dict’ is not implemented yet.

If support for ‘from_dict’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

from_records(**kwargs)¶

‘from_records’ is not implemented yet.

If support for ‘from_records’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

ge(**kwargs)¶

Get Greater than or equal to of dataframe and other, element-wise (binary operator ge).

Among flexible wrappers (eq, ne, le, lt, ge, gt) to comparison operators.

Equivalent to ==, !=, <=, <, >=, > with support to choose axis (rows or columns) and level for comparison.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}, default 'columns') – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	Result of the comparison.
Return type:	DeferredDataFrame of bool

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.eq(): Compare DeferredDataFrames for equality elementwise.
DeferredDataFrame.ne(): Compare DeferredDataFrames for inequality elementwise.
DeferredDataFrame.le(): Compare DeferredDataFrames for less than inequality or equality elementwise.
DeferredDataFrame.lt(): Compare DeferredDataFrames for strictly less than inequality elementwise.
DeferredDataFrame.ge(): Compare DeferredDataFrames for greater than inequality or equality elementwise.
DeferredDataFrame.gt(): Compare DeferredDataFrames for strictly greater than inequality elementwise.

Notes

Mismatched indices will be unioned together. NaN values are considered different (i.e. NaN != NaN).

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame({'cost': [250, 150, 100],
...                    'revenue': [100, 250, 300]},
...                   index=['A', 'B', 'C'])
>>> df
   cost  revenue
A   250      100
B   150      250
C   100      300

Comparison with a scalar, using either the operator or method:

>>> df == 100
    cost  revenue
A  False     True
B  False    False
C   True    False

>>> df.eq(100)
    cost  revenue
A  False     True
B  False    False
C   True    False

When `other` is a :class:`Series`, the columns of a DataFrame are aligned
with the index of `other` and broadcast:

>>> df != pd.Series([100, 250], index=["cost", "revenue"])
    cost  revenue
A   True     True
B   True    False
C  False     True

Use the method to control the broadcast axis:

>>> df.ne(pd.Series([100, 300], index=["A", "D"]), axis='index')
   cost  revenue
A  True    False
B  True     True
C  True     True
D  True     True

When comparing to an arbitrary sequence, the number of columns must
match the number elements in `other`:

>>> df == [250, 100]
    cost  revenue
A   True     True
B  False    False
C  False    False

Use the method to control the axis:

>>> df.eq([250, 250, 100], axis='index')
    cost  revenue
A   True    False
B  False     True
C   True    False

Compare to a DataFrame of different shape.

>>> other = pd.DataFrame({'revenue': [300, 250, 100, 150]},
...                      index=['A', 'B', 'C', 'D'])
>>> other
   revenue
A      300
B      250
C      100
D      150

>>> df.gt(other)
    cost  revenue
A  False    False
B  False    False
C  False     True
D  False    False

Compare to a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'cost': [250, 150, 100, 150, 300, 220],
...                              'revenue': [100, 250, 300, 200, 175, 225]},
...                             index=[['Q1', 'Q1', 'Q1', 'Q2', 'Q2', 'Q2'],
...                                    ['A', 'B', 'C', 'A', 'B', 'C']])
>>> df_multindex
      cost  revenue
Q1 A   250      100
   B   150      250
   C   100      300
Q2 A   150      200
   B   300      175
   C   220      225

>>> df.le(df_multindex, level=1)
       cost  revenue
Q1 A   True     True
   B   True     True
   C   True     True
Q2 A  False     True
   B   True    False
   C   True    False

get(**kwargs)¶

‘get’ is not implemented yet.

If support for ‘get’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

groupby(by, level, axis, as_index, group_keys, **kwargs)¶

Group DataFrame using a mapper or by a Series of columns.

A groupby operation involves some combination of splitting the object, applying a function, and combining the results. This can be used to group large amounts of data and compute operations on these groups.

Parameters:	by (mapping, function, label, or list of labels) – Used to determine the groups for the groupby. If `by` is a function, it’s called on each value of the object’s index. If a dict or DeferredSeries is passed, the DeferredSeries or dict VALUES will be used to determine the groups (the DeferredSeries’ values are first aligned; see `.align()` method). If an ndarray is passed, the values are used as-is to determine the groups. A label or list of labels may be passed to group by the columns in `self`. Notice that a tuple is interpreted as a (single) key. axis ({0 or 'index', 1 or 'columns'}, default 0) – Split along rows (0) or columns (1). level (int, level name, or sequence of such, default None) – If the axis is a MultiIndex (hierarchical), group by a particular level or levels. as_index (bool, default True) – For aggregated output, return object with group labels as the index. Only relevant for DeferredDataFrame input. as_index=False is effectively “SQL-style” grouped output. sort (bool, default True) – Sort group keys. Get better performance by turning this off. Note this does not influence the order of observations within each group. Groupby preserves the order of rows within each group. group_keys (bool, default True) – When calling apply, add group keys to index to identify pieces. squeeze (bool, default False) – Reduce the dimensionality of the return type if possible, otherwise return a consistent type. Deprecated since version 1.1.0. observed (bool, default False) – This only applies if any of the groupers are Categoricals. If True: only show observed values for categorical groupers. If False: show all values for categorical groupers. dropna (bool, default True) – If True, and if group keys contain NA values, NA values together with row/column will be dropped. If False, NA values will also be treated as the key in groups New in version 1.1.0.
Returns:	Returns a groupby object that contains information about the groups.
Return type:	DeferredDataFrameGroupBy

Differences from pandas

as_index and group_keys must both be True.

Aggregations grouping by a categorical column with observed=False set are not currently parallelizable (BEAM-11190<https://issues.apache.org/jira/browse/BEAM-11190>_).

See also

resample(): Convenience method for frequency conversion and resampling of time series.

Notes

See the user guide for more.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'Animal': ['Falcon', 'Falcon',
...                               'Parrot', 'Parrot'],
...                    'Max Speed': [380., 370., 24., 26.]})
>>> df
   Animal  Max Speed
0  Falcon      380.0
1  Falcon      370.0
2  Parrot       24.0
3  Parrot       26.0
>>> df.groupby(['Animal']).mean()
        Max Speed
Animal
Falcon      375.0
Parrot       25.0

**Hierarchical Indexes**

We can groupby different levels of a hierarchical index
using the `level` parameter:

>>> arrays = [['Falcon', 'Falcon', 'Parrot', 'Parrot'],
...           ['Captive', 'Wild', 'Captive', 'Wild']]
>>> index = pd.MultiIndex.from_arrays(arrays, names=('Animal', 'Type'))
>>> df = pd.DataFrame({'Max Speed': [390., 350., 30., 20.]},
...                   index=index)
>>> df
                Max Speed
Animal Type
Falcon Captive      390.0
       Wild         350.0
Parrot Captive       30.0
       Wild          20.0
>>> df.groupby(level=0).mean()
        Max Speed
Animal
Falcon      370.0
Parrot       25.0
>>> df.groupby(level="Type").mean()
         Max Speed
Type
Captive      210.0
Wild         185.0

We can also choose to include NA in group keys or not by setting
`dropna` parameter, the default setting is `True`:

>>> l = [[1, 2, 3], [1, None, 4], [2, 1, 3], [1, 2, 2]]
>>> df = pd.DataFrame(l, columns=["a", "b", "c"])

>>> df.groupby(by=["b"]).sum()
    a   c
b
1.0 2   3
2.0 2   5

>>> df.groupby(by=["b"], dropna=False).sum()
    a   c
b
1.0 2   3
2.0 2   5
NaN 1   4

>>> l = [["a", 12, 12], [None, 12.3, 33.], ["b", 12.3, 123], ["a", 1, 1]]
>>> df = pd.DataFrame(l, columns=["a", "b", "c"])

>>> df.groupby(by="a").sum()
    b     c
a
a   13.0   13.0
b   12.3  123.0

>>> df.groupby(by="a", dropna=False).sum()
    b     c
a
a   13.0   13.0
b   12.3  123.0
NaN 12.3   33.0

gt(**kwargs)¶

Get Greater than of dataframe and other, element-wise (binary operator gt).

Among flexible wrappers (eq, ne, le, lt, ge, gt) to comparison operators.

Equivalent to ==, !=, <=, <, >=, > with support to choose axis (rows or columns) and level for comparison.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}, default 'columns') – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	Result of the comparison.
Return type:	DeferredDataFrame of bool

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.eq(): Compare DeferredDataFrames for equality elementwise.
DeferredDataFrame.ne(): Compare DeferredDataFrames for inequality elementwise.
DeferredDataFrame.le(): Compare DeferredDataFrames for less than inequality or equality elementwise.
DeferredDataFrame.lt(): Compare DeferredDataFrames for strictly less than inequality elementwise.
DeferredDataFrame.ge(): Compare DeferredDataFrames for greater than inequality or equality elementwise.
DeferredDataFrame.gt(): Compare DeferredDataFrames for strictly greater than inequality elementwise.

Notes

Mismatched indices will be unioned together. NaN values are considered different (i.e. NaN != NaN).

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame({'cost': [250, 150, 100],
...                    'revenue': [100, 250, 300]},
...                   index=['A', 'B', 'C'])
>>> df
   cost  revenue
A   250      100
B   150      250
C   100      300

Comparison with a scalar, using either the operator or method:

>>> df == 100
    cost  revenue
A  False     True
B  False    False
C   True    False

>>> df.eq(100)
    cost  revenue
A  False     True
B  False    False
C   True    False

When `other` is a :class:`Series`, the columns of a DataFrame are aligned
with the index of `other` and broadcast:

>>> df != pd.Series([100, 250], index=["cost", "revenue"])
    cost  revenue
A   True     True
B   True    False
C  False     True

Use the method to control the broadcast axis:

>>> df.ne(pd.Series([100, 300], index=["A", "D"]), axis='index')
   cost  revenue
A  True    False
B  True     True
C  True     True
D  True     True

When comparing to an arbitrary sequence, the number of columns must
match the number elements in `other`:

>>> df == [250, 100]
    cost  revenue
A   True     True
B  False    False
C  False    False

Use the method to control the axis:

>>> df.eq([250, 250, 100], axis='index')
    cost  revenue
A   True    False
B  False     True
C   True    False

Compare to a DataFrame of different shape.

>>> other = pd.DataFrame({'revenue': [300, 250, 100, 150]},
...                      index=['A', 'B', 'C', 'D'])
>>> other
   revenue
A      300
B      250
C      100
D      150

>>> df.gt(other)
    cost  revenue
A  False    False
B  False    False
C  False     True
D  False    False

Compare to a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'cost': [250, 150, 100, 150, 300, 220],
...                              'revenue': [100, 250, 300, 200, 175, 225]},
...                             index=[['Q1', 'Q1', 'Q1', 'Q2', 'Q2', 'Q2'],
...                                    ['A', 'B', 'C', 'A', 'B', 'C']])
>>> df_multindex
      cost  revenue
Q1 A   250      100
   B   150      250
   C   100      300
Q2 A   150      200
   B   300      175
   C   220      225

>>> df.le(df_multindex, level=1)
       cost  revenue
Q1 A   True     True
   B   True     True
   C   True     True
Q2 A  False     True
   B   True    False
   C   True    False

hist(**kwargs)¶

pandas.DataFrame.hist is not supported in the Beam DataFrame API because it is a plotting tool.

For more information see {reason_data[‘url’]}.

iat¶

‘iat’ is not implemented yet.

If support for ‘iat’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

idxmax(**kwargs)¶

‘idxmax’ is not implemented yet.

If support for ‘idxmax’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

idxmin(**kwargs)¶

‘idxmin’ is not implemented yet.

If support for ‘idxmin’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

index¶

The index (row labels) of the DataFrame.

Differences from pandas

This operation has no known divergences from the pandas API.

infer_objects(**kwargs)¶

‘infer_objects’ is not implemented yet.

If support for ‘infer_objects’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

isin(**kwargs)¶

Whether each element in the DataFrame is contained in values.

Parameters:	values (iterable, DeferredSeries, DeferredDataFrame or dict) – The result will only be true at a location if all the labels match. If values is a DeferredSeries, that’s the index. If values is a dict, the keys must be the column names, which must match. If values is a DeferredDataFrame, then both the index and column labels must match.
Returns:	DeferredDataFrame of booleans showing whether each element in the DeferredDataFrame is contained in values.
Return type:	DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.eq(): Equality test for DeferredDataFrame.
DeferredSeries.isin(): Equivalent method on DeferredSeries.
DeferredSeries.str.contains(): Test if pattern or regex is contained within a string of a DeferredSeries or Index.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame({'num_legs': [2, 4], 'num_wings': [2, 0]},
...                   index=['falcon', 'dog'])
>>> df
        num_legs  num_wings
falcon         2          2
dog            4          0

When ``values`` is a list check whether every value in the DataFrame
is present in the list (which animals have 0 or 2 legs or wings)

>>> df.isin([0, 2])
        num_legs  num_wings
falcon      True       True
dog        False       True

When ``values`` is a dict, we can pass values to check for each
column separately:

>>> df.isin({'num_wings': [0, 3]})
        num_legs  num_wings
falcon     False      False
dog        False       True

When ``values`` is a Series or DataFrame the index and column must
match. Note that 'falcon' does not match based on the number of legs
in df2.

>>> other = pd.DataFrame({'num_legs': [8, 2], 'num_wings': [0, 2]},
...                      index=['spider', 'falcon'])
>>> df.isin(other)
        num_legs  num_wings
falcon      True       True
dog        False      False

kurt(**kwargs)¶

‘kurt’ is not implemented yet.

If support for ‘kurt’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

kurtosis(**kwargs)¶

‘kurtosis’ is not implemented yet.

If support for ‘kurtosis’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

last_valid_index(**kwargs)¶

‘last_valid_index’ is not implemented yet.

If support for ‘last_valid_index’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

le(**kwargs)¶

Get Less than or equal to of dataframe and other, element-wise (binary operator le).

Among flexible wrappers (eq, ne, le, lt, ge, gt) to comparison operators.

Equivalent to ==, !=, <=, <, >=, > with support to choose axis (rows or columns) and level for comparison.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}, default 'columns') – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	Result of the comparison.
Return type:	DeferredDataFrame of bool

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.eq(): Compare DeferredDataFrames for equality elementwise.
DeferredDataFrame.ne(): Compare DeferredDataFrames for inequality elementwise.
DeferredDataFrame.le(): Compare DeferredDataFrames for less than inequality or equality elementwise.
DeferredDataFrame.lt(): Compare DeferredDataFrames for strictly less than inequality elementwise.
DeferredDataFrame.ge(): Compare DeferredDataFrames for greater than inequality or equality elementwise.
DeferredDataFrame.gt(): Compare DeferredDataFrames for strictly greater than inequality elementwise.

Notes

Mismatched indices will be unioned together. NaN values are considered different (i.e. NaN != NaN).

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame({'cost': [250, 150, 100],
...                    'revenue': [100, 250, 300]},
...                   index=['A', 'B', 'C'])
>>> df
   cost  revenue
A   250      100
B   150      250
C   100      300

Comparison with a scalar, using either the operator or method:

>>> df == 100
    cost  revenue
A  False     True
B  False    False
C   True    False

>>> df.eq(100)
    cost  revenue
A  False     True
B  False    False
C   True    False

When `other` is a :class:`Series`, the columns of a DataFrame are aligned
with the index of `other` and broadcast:

>>> df != pd.Series([100, 250], index=["cost", "revenue"])
    cost  revenue
A   True     True
B   True    False
C  False     True

Use the method to control the broadcast axis:

>>> df.ne(pd.Series([100, 300], index=["A", "D"]), axis='index')
   cost  revenue
A  True    False
B  True     True
C  True     True
D  True     True

When comparing to an arbitrary sequence, the number of columns must
match the number elements in `other`:

>>> df == [250, 100]
    cost  revenue
A   True     True
B  False    False
C  False    False

Use the method to control the axis:

>>> df.eq([250, 250, 100], axis='index')
    cost  revenue
A   True    False
B  False     True
C   True    False

Compare to a DataFrame of different shape.

>>> other = pd.DataFrame({'revenue': [300, 250, 100, 150]},
...                      index=['A', 'B', 'C', 'D'])
>>> other
   revenue
A      300
B      250
C      100
D      150

>>> df.gt(other)
    cost  revenue
A  False    False
B  False    False
C  False     True
D  False    False

Compare to a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'cost': [250, 150, 100, 150, 300, 220],
...                              'revenue': [100, 250, 300, 200, 175, 225]},
...                             index=[['Q1', 'Q1', 'Q1', 'Q2', 'Q2', 'Q2'],
...                                    ['A', 'B', 'C', 'A', 'B', 'C']])
>>> df_multindex
      cost  revenue
Q1 A   250      100
   B   150      250
   C   100      300
Q2 A   150      200
   B   300      175
   C   220      225

>>> df.le(df_multindex, level=1)
       cost  revenue
Q1 A   True     True
   B   True     True
   C   True     True
Q2 A  False     True
   B   True    False
   C   True    False

lookup(**kwargs)¶

‘lookup’ is not implemented yet.

If support for ‘lookup’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

lt(**kwargs)¶

Get Less than of dataframe and other, element-wise (binary operator lt).

Among flexible wrappers (eq, ne, le, lt, ge, gt) to comparison operators.

Equivalent to ==, !=, <=, <, >=, > with support to choose axis (rows or columns) and level for comparison.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}, default 'columns') – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	Result of the comparison.
Return type:	DeferredDataFrame of bool

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.eq(): Compare DeferredDataFrames for equality elementwise.
DeferredDataFrame.ne(): Compare DeferredDataFrames for inequality elementwise.
DeferredDataFrame.le(): Compare DeferredDataFrames for less than inequality or equality elementwise.
DeferredDataFrame.lt(): Compare DeferredDataFrames for strictly less than inequality elementwise.
DeferredDataFrame.ge(): Compare DeferredDataFrames for greater than inequality or equality elementwise.
DeferredDataFrame.gt(): Compare DeferredDataFrames for strictly greater than inequality elementwise.

Notes

Mismatched indices will be unioned together. NaN values are considered different (i.e. NaN != NaN).

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame({'cost': [250, 150, 100],
...                    'revenue': [100, 250, 300]},
...                   index=['A', 'B', 'C'])
>>> df
   cost  revenue
A   250      100
B   150      250
C   100      300

Comparison with a scalar, using either the operator or method:

>>> df == 100
    cost  revenue
A  False     True
B  False    False
C   True    False

>>> df.eq(100)
    cost  revenue
A  False     True
B  False    False
C   True    False

When `other` is a :class:`Series`, the columns of a DataFrame are aligned
with the index of `other` and broadcast:

>>> df != pd.Series([100, 250], index=["cost", "revenue"])
    cost  revenue
A   True     True
B   True    False
C  False     True

Use the method to control the broadcast axis:

>>> df.ne(pd.Series([100, 300], index=["A", "D"]), axis='index')
   cost  revenue
A  True    False
B  True     True
C  True     True
D  True     True

When comparing to an arbitrary sequence, the number of columns must
match the number elements in `other`:

>>> df == [250, 100]
    cost  revenue
A   True     True
B  False    False
C  False    False

Use the method to control the axis:

>>> df.eq([250, 250, 100], axis='index')
    cost  revenue
A   True    False
B  False     True
C   True    False

Compare to a DataFrame of different shape.

>>> other = pd.DataFrame({'revenue': [300, 250, 100, 150]},
...                      index=['A', 'B', 'C', 'D'])
>>> other
   revenue
A      300
B      250
C      100
D      150

>>> df.gt(other)
    cost  revenue
A  False    False
B  False    False
C  False     True
D  False    False

Compare to a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'cost': [250, 150, 100, 150, 300, 220],
...                              'revenue': [100, 250, 300, 200, 175, 225]},
...                             index=[['Q1', 'Q1', 'Q1', 'Q2', 'Q2', 'Q2'],
...                                    ['A', 'B', 'C', 'A', 'B', 'C']])
>>> df_multindex
      cost  revenue
Q1 A   250      100
   B   150      250
   C   100      300
Q2 A   150      200
   B   300      175
   C   220      225

>>> df.le(df_multindex, level=1)
       cost  revenue
Q1 A   True     True
   B   True     True
   C   True     True
Q2 A  False     True
   B   True    False
   C   True    False

mad(**kwargs)¶

‘mad’ is not implemented yet.

If support for ‘mad’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

mask(cond, **kwargs)¶

Replace values where the condition is True.

Parameters:	cond (bool DeferredSeries/DeferredDataFrame, array-like, or callable) – Where cond is False, keep the original value. Where True, replace with corresponding value from other. If cond is callable, it is computed on the DeferredSeries/DeferredDataFrame and should return boolean DeferredSeries/DeferredDataFrame or array. The callable must not change input DeferredSeries/DeferredDataFrame (though pandas doesn’t check it). other (scalar, DeferredSeries/DeferredDataFrame, or callable) – Entries where cond is True are replaced with corresponding value from other. If other is callable, it is computed on the DeferredSeries/DeferredDataFrame and should return scalar or DeferredSeries/DeferredDataFrame. The callable must not change input DeferredSeries/DeferredDataFrame (though pandas doesn’t check it). inplace (bool, default False) – Whether to perform the operation in place on the data. axis (int, default None) – Alignment axis if needed. level (int, default None) – Alignment level if needed. errors (str, {'raise', 'ignore'}, default 'raise') – Note that currently this parameter won’t affect the results and will always coerce to a suitable dtype. ’raise’ : allow exceptions to be raised. ’ignore’ : suppress exceptions. On error return original object. try_cast (bool, default False) – Try to cast the result back to the input type (if possible).
Returns:
Return type:	Same type as caller or None if `inplace=True`.

Differences from pandas

mask is not parallelizable when errors="ignore" is specified.

See also

DeferredDataFrame.where(): Return an object of same shape as self.

Notes

The mask method is an application of the if-then idiom. For each element in the calling DeferredDataFrame, if cond is False the element is used; otherwise the corresponding element from the DeferredDataFrame other is used.

The signature for DeferredDataFrame.where() differs from numpy.where(). Roughly df1.where(m, df2) is equivalent to np.where(m, df1, df2).

For further details and examples see the mask documentation in indexing.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> s = pd.Series(range(5))
>>> s.where(s > 0)
0    NaN
1    1.0
2    2.0
3    3.0
4    4.0
dtype: float64
>>> s.mask(s > 0)
0    0.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64

>>> s.where(s > 1, 10)
0    10
1    10
2    2
3    3
4    4
dtype: int64
>>> s.mask(s > 1, 10)
0     0
1     1
2    10
3    10
4    10
dtype: int64

>>> df = pd.DataFrame(np.arange(10).reshape(-1, 2), columns=['A', 'B'])
>>> df
   A  B
0  0  1
1  2  3
2  4  5
3  6  7
4  8  9
>>> m = df % 3 == 0
>>> df.where(m, -df)
   A  B
0  0 -1
1 -2  3
2 -4 -5
3  6 -7
4 -8  9
>>> df.where(m, -df) == np.where(m, df, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True
>>> df.where(m, -df) == df.mask(~m, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True

mod(**kwargs)¶

Get Modulo of dataframe and other, element-wise (binary operator mod).

Equivalent to dataframe % other, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rmod.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For DeferredSeries input, axis to match DeferredSeries index on. level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level. fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DeferredDataFrame alignment, with this value before computation. If data in both corresponding DeferredDataFrame locations is missing the result will be missing.
Returns:	Result of the arithmetic operation.
Return type:	DeferredDataFrame

Differences from pandas

Only level=None is supported

See also

DeferredDataFrame.add(): Add DeferredDataFrames.
DeferredDataFrame.sub(): Subtract DeferredDataFrames.
DeferredDataFrame.mul(): Multiply DeferredDataFrames.
DeferredDataFrame.div(): Divide DeferredDataFrames (float division).
DeferredDataFrame.truediv(): Divide DeferredDataFrames (float division).
DeferredDataFrame.floordiv(): Divide DeferredDataFrames (integer division).
DeferredDataFrame.mod(): Calculate modulo (remainder after division).
DeferredDataFrame.pow(): Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same
results.

>>> df + 1
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1)
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10)
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10)
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> df - [1, 2]
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns')
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
           angles
circle          0
triangle        3
rectangle       4

>>> df * other
           angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN

>>> df.mul(other, fill_value=0)
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

Divide by a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
             angles  degrees
A circle          0      360
  triangle        3      180
  rectangle       4      360
B square          4      360
  pentagon        5      540
  hexagon         6      720

>>> df.div(df_multindex, level=1, fill_value=0)
             angles  degrees
A circle        NaN      1.0
  triangle      1.0      1.0
  rectangle     1.0      1.0
B square        0.0      0.0
  pentagon      0.0      0.0
  hexagon       0.0      0.0

mul(**kwargs)¶

Get Multiplication of dataframe and other, element-wise (binary operator mul).

Equivalent to dataframe * other, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rmul.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For DeferredSeries input, axis to match DeferredSeries index on. level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level. fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DeferredDataFrame alignment, with this value before computation. If data in both corresponding DeferredDataFrame locations is missing the result will be missing.
Returns:	Result of the arithmetic operation.
Return type:	DeferredDataFrame

Differences from pandas

Only level=None is supported

See also

DeferredDataFrame.add(): Add DeferredDataFrames.
DeferredDataFrame.sub(): Subtract DeferredDataFrames.
DeferredDataFrame.mul(): Multiply DeferredDataFrames.
DeferredDataFrame.div(): Divide DeferredDataFrames (float division).
DeferredDataFrame.truediv(): Divide DeferredDataFrames (float division).
DeferredDataFrame.floordiv(): Divide DeferredDataFrames (integer division).
DeferredDataFrame.mod(): Calculate modulo (remainder after division).
DeferredDataFrame.pow(): Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same
results.

>>> df + 1
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1)
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10)
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10)
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> df - [1, 2]
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns')
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
           angles
circle          0
triangle        3
rectangle       4

>>> df * other
           angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN

>>> df.mul(other, fill_value=0)
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

Divide by a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
             angles  degrees
A circle          0      360
  triangle        3      180
  rectangle       4      360
B square          4      360
  pentagon        5      540
  hexagon         6      720

>>> df.div(df_multindex, level=1, fill_value=0)
             angles  degrees
A circle        NaN      1.0
  triangle      1.0      1.0
  rectangle     1.0      1.0
B square        0.0      0.0
  pentagon      0.0      0.0
  hexagon       0.0      0.0

multiply(**kwargs)¶

Get Multiplication of dataframe and other, element-wise (binary operator mul).

Equivalent to dataframe * other, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rmul.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For DeferredSeries input, axis to match DeferredSeries index on. level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level. fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DeferredDataFrame alignment, with this value before computation. If data in both corresponding DeferredDataFrame locations is missing the result will be missing.
Returns:	Result of the arithmetic operation.
Return type:	DeferredDataFrame

Differences from pandas

Only level=None is supported

See also

DeferredDataFrame.add(): Add DeferredDataFrames.
DeferredDataFrame.sub(): Subtract DeferredDataFrames.
DeferredDataFrame.mul(): Multiply DeferredDataFrames.
DeferredDataFrame.div(): Divide DeferredDataFrames (float division).
DeferredDataFrame.truediv(): Divide DeferredDataFrames (float division).
DeferredDataFrame.floordiv(): Divide DeferredDataFrames (integer division).
DeferredDataFrame.mod(): Calculate modulo (remainder after division).
DeferredDataFrame.pow(): Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same
results.

>>> df + 1
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1)
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10)
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10)
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> df - [1, 2]
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns')
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
           angles
circle          0
triangle        3
rectangle       4

>>> df * other
           angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN

>>> df.mul(other, fill_value=0)
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

Divide by a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
             angles  degrees
A circle          0      360
  triangle        3      180
  rectangle       4      360
B square          4      360
  pentagon        5      540
  hexagon         6      720

>>> df.div(df_multindex, level=1, fill_value=0)
             angles  degrees
A circle        NaN      1.0
  triangle      1.0      1.0
  rectangle     1.0      1.0
B square        0.0      0.0
  pentagon      0.0      0.0
  hexagon       0.0      0.0

ndim¶

Return an int representing the number of axes / array dimensions.

Return 1 if Series. Otherwise return 2 if DataFrame.

Differences from pandas

This operation has no known divergences from the pandas API.

See also

ndarray.ndim: Number of array dimensions.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series({'a': 1, 'b': 2, 'c': 3})
>>> s.ndim
1

>>> df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
>>> df.ndim
2

ne(**kwargs)¶

Get Not equal to of dataframe and other, element-wise (binary operator ne).

Among flexible wrappers (eq, ne, le, lt, ge, gt) to comparison operators.

Equivalent to ==, !=, <=, <, >=, > with support to choose axis (rows or columns) and level for comparison.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}, default 'columns') – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
Returns:	Result of the comparison.
Return type:	DeferredDataFrame of bool

Differences from pandas

This operation has no known divergences from the pandas API.

See also

DeferredDataFrame.eq(): Compare DeferredDataFrames for equality elementwise.
DeferredDataFrame.ne(): Compare DeferredDataFrames for inequality elementwise.
DeferredDataFrame.le(): Compare DeferredDataFrames for less than inequality or equality elementwise.
DeferredDataFrame.lt(): Compare DeferredDataFrames for strictly less than inequality elementwise.
DeferredDataFrame.ge(): Compare DeferredDataFrames for greater than inequality or equality elementwise.
DeferredDataFrame.gt(): Compare DeferredDataFrames for strictly greater than inequality elementwise.

Notes

Mismatched indices will be unioned together. NaN values are considered different (i.e. NaN != NaN).

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame({'cost': [250, 150, 100],
...                    'revenue': [100, 250, 300]},
...                   index=['A', 'B', 'C'])
>>> df
   cost  revenue
A   250      100
B   150      250
C   100      300

Comparison with a scalar, using either the operator or method:

>>> df == 100
    cost  revenue
A  False     True
B  False    False
C   True    False

>>> df.eq(100)
    cost  revenue
A  False     True
B  False    False
C   True    False

When `other` is a :class:`Series`, the columns of a DataFrame are aligned
with the index of `other` and broadcast:

>>> df != pd.Series([100, 250], index=["cost", "revenue"])
    cost  revenue
A   True     True
B   True    False
C  False     True

Use the method to control the broadcast axis:

>>> df.ne(pd.Series([100, 300], index=["A", "D"]), axis='index')
   cost  revenue
A  True    False
B  True     True
C  True     True
D  True     True

When comparing to an arbitrary sequence, the number of columns must
match the number elements in `other`:

>>> df == [250, 100]
    cost  revenue
A   True     True
B  False    False
C  False    False

Use the method to control the axis:

>>> df.eq([250, 250, 100], axis='index')
    cost  revenue
A   True    False
B  False     True
C   True    False

Compare to a DataFrame of different shape.

>>> other = pd.DataFrame({'revenue': [300, 250, 100, 150]},
...                      index=['A', 'B', 'C', 'D'])
>>> other
   revenue
A      300
B      250
C      100
D      150

>>> df.gt(other)
    cost  revenue
A  False    False
B  False    False
C  False     True
D  False    False

Compare to a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'cost': [250, 150, 100, 150, 300, 220],
...                              'revenue': [100, 250, 300, 200, 175, 225]},
...                             index=[['Q1', 'Q1', 'Q1', 'Q2', 'Q2', 'Q2'],
...                                    ['A', 'B', 'C', 'A', 'B', 'C']])
>>> df_multindex
      cost  revenue
Q1 A   250      100
   B   150      250
   C   100      300
Q2 A   150      200
   B   300      175
   C   220      225

>>> df.le(df_multindex, level=1)
       cost  revenue
Q1 A   True     True
   B   True     True
   C   True     True
Q2 A  False     True
   B   True    False
   C   True    False

pad(*args, **kwargs)¶

Synonym for DataFrame.fillna() with method='ffill'.

Returns:	Object with missing values filled or None if `inplace=True`.
Return type:	DeferredSeries/DeferredDataFrame or None

Differences from pandas

pad is only supported for axis=”columns”. axis=”index” is order-sensitive.

pct_change(**kwargs)¶

‘pct_change’ is not implemented yet.

If support for ‘pct_change’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

pipe(**kwargs)¶

‘pipe’ is not implemented yet.

If support for ‘pipe’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

pivot(**kwargs)¶

‘pivot’ is not implemented yet.

If support for ‘pivot’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

pivot_table(**kwargs)¶

‘pivot_table’ is not implemented yet.

If support for ‘pivot_table’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

pow(**kwargs)¶

Get Exponential power of dataframe and other, element-wise (binary operator pow).

Equivalent to dataframe ** other, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rpow.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For DeferredSeries input, axis to match DeferredSeries index on. level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level. fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DeferredDataFrame alignment, with this value before computation. If data in both corresponding DeferredDataFrame locations is missing the result will be missing.
Returns:	Result of the arithmetic operation.
Return type:	DeferredDataFrame

Differences from pandas

Only level=None is supported

See also

DeferredDataFrame.add(): Add DeferredDataFrames.
DeferredDataFrame.sub(): Subtract DeferredDataFrames.
DeferredDataFrame.mul(): Multiply DeferredDataFrames.
DeferredDataFrame.div(): Divide DeferredDataFrames (float division).
DeferredDataFrame.truediv(): Divide DeferredDataFrames (float division).
DeferredDataFrame.floordiv(): Divide DeferredDataFrames (integer division).
DeferredDataFrame.mod(): Calculate modulo (remainder after division).
DeferredDataFrame.pow(): Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same
results.

>>> df + 1
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1)
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10)
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10)
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> df - [1, 2]
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns')
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
           angles
circle          0
triangle        3
rectangle       4

>>> df * other
           angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN

>>> df.mul(other, fill_value=0)
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

Divide by a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
             angles  degrees
A circle          0      360
  triangle        3      180
  rectangle       4      360
B square          4      360
  pentagon        5      540
  hexagon         6      720

>>> df.div(df_multindex, level=1, fill_value=0)
             angles  degrees
A circle        NaN      1.0
  triangle      1.0      1.0
  rectangle     1.0      1.0
B square        0.0      0.0
  pentagon      0.0      0.0
  hexagon       0.0      0.0

radd(**kwargs)¶

Get Addition of dataframe and other, element-wise (binary operator radd).

Equivalent to other + dataframe, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, add.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For DeferredSeries input, axis to match DeferredSeries index on. level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level. fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DeferredDataFrame alignment, with this value before computation. If data in both corresponding DeferredDataFrame locations is missing the result will be missing.
Returns:	Result of the arithmetic operation.
Return type:	DeferredDataFrame

Differences from pandas

Only level=None is supported

See also

DeferredDataFrame.add(): Add DeferredDataFrames.
DeferredDataFrame.sub(): Subtract DeferredDataFrames.
DeferredDataFrame.mul(): Multiply DeferredDataFrames.
DeferredDataFrame.div(): Divide DeferredDataFrames (float division).
DeferredDataFrame.truediv(): Divide DeferredDataFrames (float division).
DeferredDataFrame.floordiv(): Divide DeferredDataFrames (integer division).
DeferredDataFrame.mod(): Calculate modulo (remainder after division).
DeferredDataFrame.pow(): Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same
results.

>>> df + 1
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1)
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10)
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10)
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> df - [1, 2]
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns')
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
           angles
circle          0
triangle        3
rectangle       4

>>> df * other
           angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN

>>> df.mul(other, fill_value=0)
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

Divide by a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
             angles  degrees
A circle          0      360
  triangle        3      180
  rectangle       4      360
B square          4      360
  pentagon        5      540
  hexagon         6      720

>>> df.div(df_multindex, level=1, fill_value=0)
             angles  degrees
A circle        NaN      1.0
  triangle      1.0      1.0
  rectangle     1.0      1.0
B square        0.0      0.0
  pentagon      0.0      0.0
  hexagon       0.0      0.0

rank(**kwargs)¶

‘rank’ is not implemented yet.

If support for ‘rank’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

rdiv(**kwargs)¶

Get Floating division of dataframe and other, element-wise (binary operator rtruediv).

Equivalent to other / dataframe, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, truediv.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For DeferredSeries input, axis to match DeferredSeries index on. level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level. fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DeferredDataFrame alignment, with this value before computation. If data in both corresponding DeferredDataFrame locations is missing the result will be missing.
Returns:	Result of the arithmetic operation.
Return type:	DeferredDataFrame

Differences from pandas

Only level=None is supported

See also

DeferredDataFrame.add(): Add DeferredDataFrames.
DeferredDataFrame.sub(): Subtract DeferredDataFrames.
DeferredDataFrame.mul(): Multiply DeferredDataFrames.
DeferredDataFrame.div(): Divide DeferredDataFrames (float division).
DeferredDataFrame.truediv(): Divide DeferredDataFrames (float division).
DeferredDataFrame.floordiv(): Divide DeferredDataFrames (integer division).
DeferredDataFrame.mod(): Calculate modulo (remainder after division).
DeferredDataFrame.pow(): Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same
results.

>>> df + 1
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1)
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10)
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10)
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> df - [1, 2]
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns')
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
           angles
circle          0
triangle        3
rectangle       4

>>> df * other
           angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN

>>> df.mul(other, fill_value=0)
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

Divide by a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
             angles  degrees
A circle          0      360
  triangle        3      180
  rectangle       4      360
B square          4      360
  pentagon        5      540
  hexagon         6      720

>>> df.div(df_multindex, level=1, fill_value=0)
             angles  degrees
A circle        NaN      1.0
  triangle      1.0      1.0
  rectangle     1.0      1.0
B square        0.0      0.0
  pentagon      0.0      0.0
  hexagon       0.0      0.0

reindex(**kwargs)¶

‘reindex’ is not implemented yet.

If support for ‘reindex’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

reindex_like(**kwargs)¶

‘reindex_like’ is not implemented yet.

If support for ‘reindex_like’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

reorder_levels(**kwargs)¶

Rearrange index levels using input order. May not drop or duplicate levels.

Parameters:	order (list of int or list of str) – List representing new level order. Reference level by number (position) or by key (label). axis ({0 or 'index', 1 or 'columns'}, default 0) – Where to reorder levels.
Returns:
Return type:	DeferredDataFrame

Differences from pandas

This operation has no known divergences from the pandas API.

resample(**kwargs)¶

‘resample’ is not implemented yet.

If support for ‘resample’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

rfloordiv(**kwargs)¶

Get Integer division of dataframe and other, element-wise (binary operator rfloordiv).

Equivalent to other // dataframe, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, floordiv.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For DeferredSeries input, axis to match DeferredSeries index on. level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level. fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DeferredDataFrame alignment, with this value before computation. If data in both corresponding DeferredDataFrame locations is missing the result will be missing.
Returns:	Result of the arithmetic operation.
Return type:	DeferredDataFrame

Differences from pandas

Only level=None is supported

See also

DeferredDataFrame.add(): Add DeferredDataFrames.
DeferredDataFrame.sub(): Subtract DeferredDataFrames.
DeferredDataFrame.mul(): Multiply DeferredDataFrames.
DeferredDataFrame.div(): Divide DeferredDataFrames (float division).
DeferredDataFrame.truediv(): Divide DeferredDataFrames (float division).
DeferredDataFrame.floordiv(): Divide DeferredDataFrames (integer division).
DeferredDataFrame.mod(): Calculate modulo (remainder after division).
DeferredDataFrame.pow(): Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same
results.

>>> df + 1
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1)
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10)
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10)
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> df - [1, 2]
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns')
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
           angles
circle          0
triangle        3
rectangle       4

>>> df * other
           angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN

>>> df.mul(other, fill_value=0)
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

Divide by a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
             angles  degrees
A circle          0      360
  triangle        3      180
  rectangle       4      360
B square          4      360
  pentagon        5      540
  hexagon         6      720

>>> df.div(df_multindex, level=1, fill_value=0)
             angles  degrees
A circle        NaN      1.0
  triangle      1.0      1.0
  rectangle     1.0      1.0
B square        0.0      0.0
  pentagon      0.0      0.0
  hexagon       0.0      0.0

rmod(**kwargs)¶

Get Modulo of dataframe and other, element-wise (binary operator rmod).

Equivalent to other % dataframe, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, mod.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For DeferredSeries input, axis to match DeferredSeries index on. level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level. fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DeferredDataFrame alignment, with this value before computation. If data in both corresponding DeferredDataFrame locations is missing the result will be missing.
Returns:	Result of the arithmetic operation.
Return type:	DeferredDataFrame

Differences from pandas

Only level=None is supported

See also

DeferredDataFrame.add(): Add DeferredDataFrames.
DeferredDataFrame.sub(): Subtract DeferredDataFrames.
DeferredDataFrame.mul(): Multiply DeferredDataFrames.
DeferredDataFrame.div(): Divide DeferredDataFrames (float division).
DeferredDataFrame.truediv(): Divide DeferredDataFrames (float division).
DeferredDataFrame.floordiv(): Divide DeferredDataFrames (integer division).
DeferredDataFrame.mod(): Calculate modulo (remainder after division).
DeferredDataFrame.pow(): Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same
results.

>>> df + 1
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1)
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10)
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10)
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> df - [1, 2]
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns')
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
           angles
circle          0
triangle        3
rectangle       4

>>> df * other
           angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN

>>> df.mul(other, fill_value=0)
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

Divide by a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
             angles  degrees
A circle          0      360
  triangle        3      180
  rectangle       4      360
B square          4      360
  pentagon        5      540
  hexagon         6      720

>>> df.div(df_multindex, level=1, fill_value=0)
             angles  degrees
A circle        NaN      1.0
  triangle      1.0      1.0
  rectangle     1.0      1.0
B square        0.0      0.0
  pentagon      0.0      0.0
  hexagon       0.0      0.0

rmul(**kwargs)¶

Get Multiplication of dataframe and other, element-wise (binary operator rmul).

Equivalent to other * dataframe, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, mul.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For DeferredSeries input, axis to match DeferredSeries index on. level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level. fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DeferredDataFrame alignment, with this value before computation. If data in both corresponding DeferredDataFrame locations is missing the result will be missing.
Returns:	Result of the arithmetic operation.
Return type:	DeferredDataFrame

Differences from pandas

Only level=None is supported

See also

DeferredDataFrame.add(): Add DeferredDataFrames.
DeferredDataFrame.sub(): Subtract DeferredDataFrames.
DeferredDataFrame.mul(): Multiply DeferredDataFrames.
DeferredDataFrame.div(): Divide DeferredDataFrames (float division).
DeferredDataFrame.truediv(): Divide DeferredDataFrames (float division).
DeferredDataFrame.floordiv(): Divide DeferredDataFrames (integer division).
DeferredDataFrame.mod(): Calculate modulo (remainder after division).
DeferredDataFrame.pow(): Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same
results.

>>> df + 1
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1)
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10)
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10)
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> df - [1, 2]
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns')
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
           angles
circle          0
triangle        3
rectangle       4

>>> df * other
           angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN

>>> df.mul(other, fill_value=0)
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

Divide by a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
             angles  degrees
A circle          0      360
  triangle        3      180
  rectangle       4      360
B square          4      360
  pentagon        5      540
  hexagon         6      720

>>> df.div(df_multindex, level=1, fill_value=0)
             angles  degrees
A circle        NaN      1.0
  triangle      1.0      1.0
  rectangle     1.0      1.0
B square        0.0      0.0
  pentagon      0.0      0.0
  hexagon       0.0      0.0

rolling(**kwargs)¶

‘rolling’ is not implemented yet.

If support for ‘rolling’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

rpow(**kwargs)¶

Get Exponential power of dataframe and other, element-wise (binary operator rpow).

Equivalent to other ** dataframe, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, pow.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For DeferredSeries input, axis to match DeferredSeries index on. level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level. fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DeferredDataFrame alignment, with this value before computation. If data in both corresponding DeferredDataFrame locations is missing the result will be missing.
Returns:	Result of the arithmetic operation.
Return type:	DeferredDataFrame

Differences from pandas

Only level=None is supported

See also

DeferredDataFrame.add(): Add DeferredDataFrames.
DeferredDataFrame.sub(): Subtract DeferredDataFrames.
DeferredDataFrame.mul(): Multiply DeferredDataFrames.
DeferredDataFrame.div(): Divide DeferredDataFrames (float division).
DeferredDataFrame.truediv(): Divide DeferredDataFrames (float division).
DeferredDataFrame.floordiv(): Divide DeferredDataFrames (integer division).
DeferredDataFrame.mod(): Calculate modulo (remainder after division).
DeferredDataFrame.pow(): Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same
results.

>>> df + 1
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1)
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10)
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10)
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> df - [1, 2]
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns')
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
           angles
circle          0
triangle        3
rectangle       4

>>> df * other
           angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN

>>> df.mul(other, fill_value=0)
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

Divide by a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
             angles  degrees
A circle          0      360
  triangle        3      180
  rectangle       4      360
B square          4      360
  pentagon        5      540
  hexagon         6      720

>>> df.div(df_multindex, level=1, fill_value=0)
             angles  degrees
A circle        NaN      1.0
  triangle      1.0      1.0
  rectangle     1.0      1.0
B square        0.0      0.0
  pentagon      0.0      0.0
  hexagon       0.0      0.0

rsub(**kwargs)¶

Get Subtraction of dataframe and other, element-wise (binary operator rsub).

Equivalent to other - dataframe, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, sub.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For DeferredSeries input, axis to match DeferredSeries index on. level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level. fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DeferredDataFrame alignment, with this value before computation. If data in both corresponding DeferredDataFrame locations is missing the result will be missing.
Returns:	Result of the arithmetic operation.
Return type:	DeferredDataFrame

Differences from pandas

Only level=None is supported

See also

DeferredDataFrame.add(): Add DeferredDataFrames.
DeferredDataFrame.sub(): Subtract DeferredDataFrames.
DeferredDataFrame.mul(): Multiply DeferredDataFrames.
DeferredDataFrame.div(): Divide DeferredDataFrames (float division).
DeferredDataFrame.truediv(): Divide DeferredDataFrames (float division).
DeferredDataFrame.floordiv(): Divide DeferredDataFrames (integer division).
DeferredDataFrame.mod(): Calculate modulo (remainder after division).
DeferredDataFrame.pow(): Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same
results.

>>> df + 1
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1)
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10)
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10)
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> df - [1, 2]
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns')
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
           angles
circle          0
triangle        3
rectangle       4

>>> df * other
           angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN

>>> df.mul(other, fill_value=0)
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

Divide by a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
             angles  degrees
A circle          0      360
  triangle        3      180
  rectangle       4      360
B square          4      360
  pentagon        5      540
  hexagon         6      720

>>> df.div(df_multindex, level=1, fill_value=0)
             angles  degrees
A circle        NaN      1.0
  triangle      1.0      1.0
  rectangle     1.0      1.0
B square        0.0      0.0
  pentagon      0.0      0.0
  hexagon       0.0      0.0

rtruediv(**kwargs)¶

Get Floating division of dataframe and other, element-wise (binary operator rtruediv).

Equivalent to other / dataframe, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, truediv.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For DeferredSeries input, axis to match DeferredSeries index on. level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level. fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DeferredDataFrame alignment, with this value before computation. If data in both corresponding DeferredDataFrame locations is missing the result will be missing.
Returns:	Result of the arithmetic operation.
Return type:	DeferredDataFrame

Differences from pandas

Only level=None is supported

See also

DeferredDataFrame.add(): Add DeferredDataFrames.
DeferredDataFrame.sub(): Subtract DeferredDataFrames.
DeferredDataFrame.mul(): Multiply DeferredDataFrames.
DeferredDataFrame.div(): Divide DeferredDataFrames (float division).
DeferredDataFrame.truediv(): Divide DeferredDataFrames (float division).
DeferredDataFrame.floordiv(): Divide DeferredDataFrames (integer division).
DeferredDataFrame.mod(): Calculate modulo (remainder after division).
DeferredDataFrame.pow(): Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same
results.

>>> df + 1
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1)
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10)
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10)
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> df - [1, 2]
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns')
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
           angles
circle          0
triangle        3
rectangle       4

>>> df * other
           angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN

>>> df.mul(other, fill_value=0)
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

Divide by a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
             angles  degrees
A circle          0      360
  triangle        3      180
  rectangle       4      360
B square          4      360
  pentagon        5      540
  hexagon         6      720

>>> df.div(df_multindex, level=1, fill_value=0)
             angles  degrees
A circle        NaN      1.0
  triangle      1.0      1.0
  rectangle     1.0      1.0
B square        0.0      0.0
  pentagon      0.0      0.0
  hexagon       0.0      0.0

sample(**kwargs)¶

‘sample’ is not implemented yet.

If support for ‘sample’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

sem(**kwargs)¶

‘sem’ is not implemented yet.

If support for ‘sem’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

set_axis(**kwargs)¶

‘set_axis’ is not implemented yet.

If support for ‘set_axis’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

set_flags(**kwargs)¶

‘set_flags’ is not implemented yet.

If support for ‘set_flags’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

size¶

Return an int representing the number of elements in this object.

Return the number of rows if Series. Otherwise return the number of rows times number of columns if DataFrame.

Differences from pandas

This operation has no known divergences from the pandas API.

See also

ndarray.size: Number of elements in the array.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> s = pd.Series({'a': 1, 'b': 2, 'c': 3})
>>> s.size
3

>>> df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
>>> df.size
4

skew(**kwargs)¶

‘skew’ is not implemented yet.

If support for ‘skew’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

slice_shift(**kwargs)¶

‘slice_shift’ is not implemented yet.

If support for ‘slice_shift’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

sort_index(axis, **kwargs)¶

Sort object by labels (along an axis).

Returns a new DataFrame sorted by label if inplace argument is False, otherwise updates the original DataFrame and returns None.

Parameters:	axis ({0 or 'index', 1 or 'columns'}, default 0) – The axis along which to sort. The value 0 identifies the rows, and 1 identifies the columns. level (int or level name or list of ints or list of level names) – If not None, sort on values in specified index level(s). ascending (bool or list-like of bools, default True) – Sort ascending vs. descending. When the index is a MultiIndex the sort direction can be controlled for each level individually. inplace (bool, default False) – If True, perform operation in-place. kind ({'quicksort', 'mergesort', 'heapsort'}, default 'quicksort') – Choice of sorting algorithm. See also ndarray.np.sort for more information. mergesort is the only stable algorithm. For DeferredDataFrames, this option is only applied when sorting on a single column or label. na_position ({'first', 'last'}, default 'last') – Puts NaNs at the beginning if first; last puts NaNs at the end. Not implemented for MultiIndex. sort_remaining (bool, default True) – If True and sorting by level and index is multilevel, sort by other levels too (in order) after sorting by specified level. ignore_index (bool, default False) – If True, the resulting axis will be labeled 0, 1, …, n - 1. New in version 1.0.0. key (callable, optional) – If not None, apply the key function to the index values before sorting. This is similar to the key argument in the builtin `sorted()` function, with the notable difference that this key function should be vectorized. It should expect an `Index` and return an `Index` of the same shape. For MultiIndex inputs, the key is applied per level. New in version 1.1.0.
Returns:	The original DeferredDataFrame sorted by the labels or None if `inplace=True`.
Return type:	DeferredDataFrame or None

Differences from pandas

axis=index is not allowed because it imposes an ordering on the dataset, and we cannot guarantee it will be maintained (see https://s.apache.org/dataframe-order-sensitive-operations). Only axis=columns is allowed.

See also

DeferredSeries.sort_index(): Sort DeferredSeries by the index.
DeferredDataFrame.sort_values(): Sort DeferredDataFrame by the value.
DeferredSeries.sort_values(): Sort DeferredSeries by the value.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame([1, 2, 3, 4, 5], index=[100, 29, 234, 1, 150],
...                   columns=['A'])
>>> df.sort_index()
     A
1    4
29   2
100  1
150  5
234  3

By default, it sorts in ascending order, to sort in descending order,
use ``ascending=False``

>>> df.sort_index(ascending=False)
     A
234  3
150  5
100  1
29   2
1    4

A key function can be specified which is applied to the index before
sorting. For a ``MultiIndex`` this is applied to each level separately.

>>> df = pd.DataFrame({"a": [1, 2, 3, 4]}, index=['A', 'b', 'C', 'd'])
>>> df.sort_index(key=lambda x: x.str.lower())
   a
A  1
b  2
C  3
d  4

sort_values(axis, **kwargs)¶

sort_values is not implemented.

It is not implemented for axis=index because it imposes an ordering on the dataset, and we cannot guarantee it will be maintained (see https://s.apache.org/dataframe-order-sensitive-operations).

It is not implemented for axis=columns because it makes the order of the columns depend on the data (see https://s.apache.org/dataframe-non-deferred-column-names).

sparse¶

‘sparse’ is not implemented yet.

If support for ‘sparse’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

squeeze(**kwargs)¶

‘squeeze’ is not implemented yet.

If support for ‘squeeze’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

style¶

‘style’ is not implemented yet.

If support for ‘style’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

sub(**kwargs)¶

Get Subtraction of dataframe and other, element-wise (binary operator sub).

Equivalent to dataframe - other, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rsub.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For DeferredSeries input, axis to match DeferredSeries index on. level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level. fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DeferredDataFrame alignment, with this value before computation. If data in both corresponding DeferredDataFrame locations is missing the result will be missing.
Returns:	Result of the arithmetic operation.
Return type:	DeferredDataFrame

Differences from pandas

Only level=None is supported

See also

DeferredDataFrame.add(): Add DeferredDataFrames.
DeferredDataFrame.sub(): Subtract DeferredDataFrames.
DeferredDataFrame.mul(): Multiply DeferredDataFrames.
DeferredDataFrame.div(): Divide DeferredDataFrames (float division).
DeferredDataFrame.truediv(): Divide DeferredDataFrames (float division).
DeferredDataFrame.floordiv(): Divide DeferredDataFrames (integer division).
DeferredDataFrame.mod(): Calculate modulo (remainder after division).
DeferredDataFrame.pow(): Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same
results.

>>> df + 1
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1)
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10)
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10)
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> df - [1, 2]
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns')
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
           angles
circle          0
triangle        3
rectangle       4

>>> df * other
           angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN

>>> df.mul(other, fill_value=0)
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

Divide by a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
             angles  degrees
A circle          0      360
  triangle        3      180
  rectangle       4      360
B square          4      360
  pentagon        5      540
  hexagon         6      720

>>> df.div(df_multindex, level=1, fill_value=0)
             angles  degrees
A circle        NaN      1.0
  triangle      1.0      1.0
  rectangle     1.0      1.0
B square        0.0      0.0
  pentagon      0.0      0.0
  hexagon       0.0      0.0

subtract(**kwargs)¶

‘subtract’ is not implemented yet.

If support for ‘subtract’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

swapaxes(**kwargs)¶

‘swapaxes’ is not implemented yet.

If support for ‘swapaxes’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

swaplevel(**kwargs)¶

‘swaplevel’ is not implemented yet.

If support for ‘swaplevel’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_clipboard(**kwargs)¶

‘clipboard’ is not implemented yet.

If support for ‘clipboard’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_csv(path, *args, **kwargs)¶

Write object to a comma-separated values (csv) file.

Changed in version 0.24.0: The order of arguments for Series was changed.

Parameters:	path_or_buf (str or file handle, default None) – File path or object, if None is provided the result is returned as a string. If a non-binary file object is passed, it should be opened with newline=’’, disabling universal newlines. If a binary file object is passed, mode might need to contain a ‘b’. Changed in version 0.24.0: Was previously named “path” for DeferredSeries. Changed in version 1.2.0: Support for binary file objects was introduced. sep (str, default ',') – String of length 1. Field delimiter for the output file. na_rep (str, default '') – Missing data representation. float_format (str, default None) – Format string for floating point numbers. columns (sequence, optional) – Columns to write. header (bool or list of str, default True) – Write out the column names. If a list of strings is given it is assumed to be aliases for the column names. Changed in version 0.24.0: Previously defaulted to False for DeferredSeries. index (bool, default True) – Write row names (index). index_label (str or sequence, or False, default None) – Column label for index column(s) if desired. If None is given, and header and index are True, then the index names are used. A sequence should be given if the object uses MultiIndex. If False do not print fields for index names. Use index_label=False for easier importing in R. mode (str) – Python write mode, default ‘w’. encoding (str, optional) – A string representing the encoding to use in the output file, defaults to ‘utf-8’. encoding is not supported if path_or_buf is a non-binary file object. compression (str or dict, default 'infer') – If str, represents compression mode. If dict, value at ‘method’ is the compression mode. Compression mode may be any of the following possible values: {‘infer’, ‘gzip’, ‘bz2’, ‘zip’, ‘xz’, None}. If compression mode is ‘infer’ and path_or_buf is path-like, then detect compression mode from the following extensions: ‘.gz’, ‘.bz2’, ‘.zip’ or ‘.xz’. (otherwise no compression). If dict given and mode is one of {‘zip’, ‘gzip’, ‘bz2’}, or inferred as one of the above, other entries passed as additional compression options. Changed in version 1.0.0: May now be a dict with key ‘method’ as compression mode and other entries as additional compression options if compression mode is ‘zip’. Changed in version 1.1.0: Passing compression options as keys in dict is supported for compression modes ‘gzip’ and ‘bz2’ as well as ‘zip’. Changed in version 1.2.0: Compression is supported for binary file objects. Changed in version 1.2.0: Previous versions forwarded dict entries for ‘gzip’ to gzip.open instead of gzip.GzipFile which prevented setting mtime. quoting (optional constant from csv module) – Defaults to csv.QUOTE_MINIMAL. If you have set a float_format then floats are converted to strings and thus csv.QUOTE_NONNUMERIC will treat them as non-numeric. quotechar (str, default '"') – String of length 1. Character used to quote fields. line_terminator (str, optional) – The newline character or character sequence to use in the output file. Defaults to os.linesep, which depends on the OS in which this method is called (‘n’ for linux, ‘rn’ for Windows, i.e.). Changed in version 0.24.0. chunksize (int or None) – Rows to write at a time. date_format (str, default None) – Format string for datetime objects. doublequote (bool, default True) – Control quoting of quotechar inside a field. escapechar (str, default None) – String of length 1. Character used to escape sep and quotechar when appropriate. decimal (str, default '.') – Character recognized as decimal separator. E.g. use ‘,’ for European data. errors (str, default 'strict') – Specifies how encoding and decoding errors are to be handled. See the errors argument for `open()` for a full list of options. New in version 1.1.0. storage_options (dict, optional) – Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc., if using a URL that will be parsed by `fsspec`, e.g., starting “s3://”, “gcs://”. An error will be raised if providing this argument with a non-fsspec URL. See the fsspec and backend storage implementation docs for the set of allowed keys and values. New in version 1.2.0.
Returns:	If path_or_buf is None, returns the resulting csv format as a string. Otherwise returns None.
Return type:	None or str

Differences from pandas

This operation has no known divergences from the pandas API.

See also

read_csv(): Load a CSV file into a DeferredDataFrame.
to_excel(): Write DeferredDataFrame to an Excel file.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame({'name': ['Raphael', 'Donatello'],
...                    'mask': ['red', 'purple'],
...                    'weapon': ['sai', 'bo staff']})
>>> df.to_csv(index=False)
'name,mask,weapon\nRaphael,red,sai\nDonatello,purple,bo staff\n'

Create 'out.zip' containing 'out.csv'

>>> compression_opts = dict(method='zip',
...                         archive_name='out.csv')  
>>> df.to_csv('out.zip', index=False,
...           compression=compression_opts)  

to_excel(path, *args, **kwargs)¶

Write object to an Excel sheet.

To write a single object to an Excel .xlsx file it is only necessary to specify a target file name. To write to multiple sheets it is necessary to create an ExcelWriter object with a target file name, and specify a sheet in the file to write to.

Multiple sheets may be written to by specifying unique sheet_name. With all data written to the file it is necessary to save the changes. Note that creating an ExcelWriter object with a file name that already exists will result in the contents of the existing file being erased.

Parameters:

excel_writer (path-like, file-like, or ExcelWriter object) – File path or existing ExcelWriter.
sheet_name (str, default 'Sheet1') – Name of sheet which will contain DeferredDataFrame.
na_rep (str, default '') – Missing data representation.
float_format (str, optional) – Format string for floating point numbers. For example float_format="%.2f" will format 0.1234 to 0.12.
columns (sequence or list of str, optional) – Columns to write.
header (bool or list of str, default True) – Write out the column names. If a list of string is given it is assumed to be aliases for the column names.
index (bool, default True) – Write row names (index).
index_label (str or sequence, optional) – Column label for index column(s) if desired. If not specified, and header and index are True, then the index names are used. A sequence should be given if the DeferredDataFrame uses MultiIndex.
startrow (int, default 0) – Upper left cell row to dump data frame.
startcol (int, default 0) – Upper left cell column to dump data frame.
engine (str, optional) –
Write engine to use, ‘openpyxl’ or ‘xlsxwriter’. You can also set this via the options io.excel.xlsx.writer, io.excel.xls.writer, and io.excel.xlsm.writer.

Deprecated since version 1.2.0: As the xlwt package is no longer maintained, the xlwt engine will be removed in a future version of pandas.
merge_cells (bool, default True) – Write MultiIndex and Hierarchical Rows as merged cells.
encoding (str, optional) – Encoding of the resulting excel file. Only necessary for xlwt, other writers support unicode natively.
inf_rep (str, default 'inf') – Representation for infinity (there is no native representation for infinity in Excel).
verbose (bool, default True) – Display more information in the error logs.
freeze_panes (tuple of int (length 2), optional) – Specifies the one-based bottommost row and rightmost column that is to be frozen.
storage_options (dict, optional) –
Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc., if using a URL that will be parsed by fsspec, e.g., starting “s3://”, “gcs://”. An error will be raised if providing this argument with a non-fsspec URL. See the fsspec and backend storage implementation docs for the set of allowed keys and values.

New in version 1.2.0.

Differences from pandas

This operation has no known divergences from the pandas API.

See also

to_csv(): Write DeferredDataFrame to a comma-separated values (csv) file.
ExcelWriter(): Class for writing DeferredDataFrame objects into excel sheets.
read_excel(): Read an Excel file into a pandas DeferredDataFrame.
read_csv(): Read a comma-separated values (csv) file into DeferredDataFrame.

Notes

For compatibility with to_csv(), to_excel serializes lists and dicts to strings before writing.

Once a workbook has been saved it is not possible write further data without rewriting the whole workbook.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

Create, write to and save a workbook:

>>> df1 = pd.DataFrame([['a', 'b'], ['c', 'd']],
...                    index=['row 1', 'row 2'],
...                    columns=['col 1', 'col 2'])
>>> df1.to_excel("output.xlsx")  

To specify the sheet name:

>>> df1.to_excel("output.xlsx",
...              sheet_name='Sheet_name_1')  

If you wish to write to more than one sheet in the workbook, it is
necessary to specify an ExcelWriter object:

>>> df2 = df1.copy()
>>> with pd.ExcelWriter('output.xlsx') as writer:  
...     df1.to_excel(writer, sheet_name='Sheet_name_1')
...     df2.to_excel(writer, sheet_name='Sheet_name_2')

ExcelWriter can also be used to append to an existing Excel file:

>>> with pd.ExcelWriter('output.xlsx',
...                     mode='a') as writer:  
...     df.to_excel(writer, sheet_name='Sheet_name_3')

To set the library that is used to write the Excel file,
you can pass the `engine` keyword (the default engine is
automatically chosen depending on the file extension):

>>> df1.to_excel('output1.xlsx', engine='xlsxwriter')  

to_feather(path, *args, **kwargs)¶

Write a DataFrame to the binary Feather format.

Parameters:	path (str or file-like object) – If a string, it will be used as Root Directory path. **kwargs – Additional keywords passed to `pyarrow.feather.write_feather()`. Starting with pyarrow 0.17, this includes the compression, compression_level, chunksize and version keywords. New in version 1.1.0.

Differences from pandas

This operation has no known divergences from the pandas API.

to_gbq(**kwargs)¶

‘to_gbq’ is not implemented yet.

If support for ‘to_gbq’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_hdf(**kwargs)¶: pandas.DataFrame.to_hdf is not supported in the Beam DataFrame API because HDF5 is a random access file format.

to_html(path, *args, **kwargs)¶

Render a DataFrame as an HTML table.

Parameters:	buf (str, Path or StringIO-like, optional, default None) – Buffer to write to. If None, the output is returned as a string. columns (sequence, optional, default None) – The subset of columns to write. Writes all columns by default. col_space (str or int, list or dict of int or str, optional) – The minimum width of each column in CSS length units. An int is assumed to be px units. New in version 0.25.0: Ability to use str. header (bool, optional) – Whether to print column labels, default True. index (bool, optional, default True) – Whether to print index (row) labels. na_rep (str, optional, default 'NaN') – String representation of `NaN` to use. formatters (list, tuple or dict of one-param. functions, optional) – Formatter functions to apply to columns’ elements by position or name. The result of each function must be a unicode string. List/tuple must be of length equal to the number of columns. float_format (one-parameter function, optional, default None) – Formatter function to apply to columns’ elements if they are floats. This function must return a unicode string and will be applied only to the non-`NaN` elements, with `NaN` being handled by `na_rep`. Changed in version 1.2.0. sparsify (bool, optional, default True) – Set to False for a DeferredDataFrame with a hierarchical index to print every multiindex key at each row. index_names (bool, optional, default True) – Prints the names of the indexes. justify (str, default None) – How to justify the column labels. If None uses the option from the print configuration (controlled by set_option), ‘right’ out of the box. Valid values are left right center justify justify-all start end inherit match-parent initial unset. max_rows (int, optional) – Maximum number of rows to display in the console. min_rows (int, optional) – The number of rows to display in the console in a truncated repr (when number of rows is above max_rows). max_cols (int, optional) – Maximum number of columns to display in the console. show_dimensions (bool, default False) – Display DeferredDataFrame dimensions (number of rows by number of columns). decimal (str, default '.') – Character recognized as decimal separator, e.g. ‘,’ in Europe. bold_rows (bool, default True) – Make the row labels bold in the output. classes (str or list or tuple, default None) – CSS class(es) to apply to the resulting html table. escape (bool, default True) – Convert the characters <, >, and & to HTML-safe sequences. notebook ({True, False}, default False) – Whether the generated HTML is for IPython Notebook. border (int) – A `border=border` attribute is included in the opening <table> tag. Default `pd.options.display.html.border`. encoding (str, default "utf-8") – Set character encoding. New in version 1.0. table_id (str, optional) – A css id is included in the opening <table> tag if specified. render_links (bool, default False) – Convert URLs to HTML links. New in version 0.24.0.
Returns:	If buf is None, returns the result as a string. Otherwise returns None.
Return type:	str or None

Differences from pandas

This operation has no known divergences from the pandas API.

See also

to_string(): Convert DeferredDataFrame to a string.

to_json(path, orient=None, *args, **kwargs)¶

Convert the object to a JSON string.

Note NaN’s and None will be converted to null and datetime objects will be converted to UNIX timestamps.

Parameters:	path_or_buf (str or file handle, optional) – File path or object. If not specified, the result is returned as a string. orient (str) – Indication of expected JSON string format. DeferredSeries: default is ‘index’ allowed values are: {‘split’, ‘records’, ‘index’, ‘table’}. DeferredDataFrame: default is ‘columns’ allowed values are: {‘split’, ‘records’, ‘index’, ‘columns’, ‘values’, ‘table’}. The format of the JSON string: ’split’ : dict like {‘index’ -> [index], ‘columns’ -> [columns], ‘data’ -> [values]} ’records’ : list like [{column -> value}, … , {column -> value}] ’index’ : dict like {index -> {column -> value}} ’columns’ : dict like {column -> {index -> value}} ’values’ : just the values array ’table’ : dict like {‘schema’: {schema}, ‘data’: {data}} Describing the data, where data component is like `orient='records'`. date_format ({None, 'epoch', 'iso'}) – Type of date conversion. ‘epoch’ = epoch milliseconds, ‘iso’ = ISO8601. The default depends on the orient. For `orient='table'`, the default is ‘iso’. For all other orients, the default is ‘epoch’. double_precision (int, default 10) – The number of decimal places to use when encoding floating point values. force_ascii (bool, default True) – Force encoded string to be ASCII. date_unit (str, default 'ms' (milliseconds)) – The time unit to encode to, governs timestamp and ISO8601 precision. One of ‘s’, ‘ms’, ‘us’, ‘ns’ for second, millisecond, microsecond, and nanosecond respectively. default_handler (callable, default None) – Handler to call if object cannot otherwise be converted to a suitable format for JSON. Should receive a single argument which is the object to convert and return a serialisable object. lines (bool, default False) – If ‘orient’ is ‘records’ write out line delimited json format. Will throw ValueError if incorrect ‘orient’ since others are not list like. compression ({'infer', 'gzip', 'bz2', 'zip', 'xz', None}) – A string representing the compression to use in the output file, only used when the first argument is a filename. By default, the compression is inferred from the filename. Changed in version 0.24.0: ‘infer’ option added and set to default index (bool, default True) – Whether to include the index values in the JSON string. Not including the index (`index=False`) is only supported when orient is ‘split’ or ‘table’. indent (int, optional) – Length of whitespace used to indent each record. New in version 1.0.0. storage_options (dict, optional) – Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc., if using a URL that will be parsed by `fsspec`, e.g., starting “s3://”, “gcs://”. An error will be raised if providing this argument with a non-fsspec URL. See the fsspec and backend storage implementation docs for the set of allowed keys and values. New in version 1.2.0.
Returns:	If path_or_buf is None, returns the resulting json format as a string. Otherwise returns None.
Return type:	None or str

Differences from pandas

This operation has no known divergences from the pandas API.

See also

read_json(): Convert a JSON string to pandas object.

Notes

The behavior of indent=0 varies from the stdlib, which does not indent the output but does insert newlines. Currently, indent=0 and the default indent=None are equivalent in pandas, though this may change in a future release.

orient='table' contains a ‘pandas_version’ field under ‘schema’. This stores the version of pandas used in the latest revision of the schema.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> import json
>>> df = pd.DataFrame(
...     [["a", "b"], ["c", "d"]],
...     index=["row 1", "row 2"],
...     columns=["col 1", "col 2"],
... )

>>> result = df.to_json(orient="split")
>>> parsed = json.loads(result)
>>> json.dumps(parsed, indent=4)  
{
    "columns": [
        "col 1",
        "col 2"
    ],
    "index": [
        "row 1",
        "row 2"
    ],
    "data": [
        [
            "a",
            "b"
        ],
        [
            "c",
            "d"
        ]
    ]
}

Encoding/decoding a Dataframe using ``'records'`` formatted JSON.
Note that index labels are not preserved with this encoding.

>>> result = df.to_json(orient="records")
>>> parsed = json.loads(result)
>>> json.dumps(parsed, indent=4)  
[
    {
        "col 1": "a",
        "col 2": "b"
    },
    {
        "col 1": "c",
        "col 2": "d"
    }
]

Encoding/decoding a Dataframe using ``'index'`` formatted JSON:

>>> result = df.to_json(orient="index")
>>> parsed = json.loads(result)
>>> json.dumps(parsed, indent=4)  
{
    "row 1": {
        "col 1": "a",
        "col 2": "b"
    },
    "row 2": {
        "col 1": "c",
        "col 2": "d"
    }
}

Encoding/decoding a Dataframe using ``'columns'`` formatted JSON:

>>> result = df.to_json(orient="columns")
>>> parsed = json.loads(result)
>>> json.dumps(parsed, indent=4)  
{
    "col 1": {
        "row 1": "a",
        "row 2": "c"
    },
    "col 2": {
        "row 1": "b",
        "row 2": "d"
    }
}

Encoding/decoding a Dataframe using ``'values'`` formatted JSON:

>>> result = df.to_json(orient="values")
>>> parsed = json.loads(result)
>>> json.dumps(parsed, indent=4)  
[
    [
        "a",
        "b"
    ],
    [
        "c",
        "d"
    ]
]

Encoding with Table Schema:

>>> result = df.to_json(orient="table")
>>> parsed = json.loads(result)
>>> json.dumps(parsed, indent=4)  
{
    "schema": {
        "fields": [
            {
                "name": "index",
                "type": "string"
            },
            {
                "name": "col 1",
                "type": "string"
            },
            {
                "name": "col 2",
                "type": "string"
            }
        ],
        "primaryKey": [
            "index"
        ],
        "pandas_version": "0.20.0"
    },
    "data": [
        {
            "index": "row 1",
            "col 1": "a",
            "col 2": "b"
        },
        {
            "index": "row 2",
            "col 1": "c",
            "col 2": "d"
        }
    ]
}

to_latex(**kwargs)¶

‘to_latex’ is not implemented yet.

If support for ‘to_latex’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_markdown(**kwargs)¶

‘to_markdown’ is not implemented yet.

If support for ‘to_markdown’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_msgpack(**kwargs)¶: pandas.DataFrame.to_msgpack is not supported in the Beam DataFrame API because it is deprecated in pandas.

to_parquet(path, *args, **kwargs)¶

Write a DataFrame to the binary parquet format.

This function writes the dataframe as a parquet file. You can choose different parquet backends, and have the option of compression. See the user guide for more details.

Parameters:	path (str or file-like object, default None) – If a string, it will be used as Root Directory path when writing a partitioned dataset. By file-like object, we refer to objects with a write() method, such as a file handle (e.g. via builtin open function) or io.BytesIO. The engine fastparquet does not accept file-like objects. If path is None, a bytes object is returned. Changed in version 1.2.0. Previously this was “fname” engine ({'auto', 'pyarrow', 'fastparquet'}, default 'auto') – Parquet library to use. If ‘auto’, then the option `io.parquet.engine` is used. The default `io.parquet.engine` behavior is to try ‘pyarrow’, falling back to ‘fastparquet’ if ‘pyarrow’ is unavailable. compression ({'snappy', 'gzip', 'brotli', None}, default 'snappy') – Name of the compression to use. Use `None` for no compression. index (bool, default None) – If `True`, include the dataframe’s index(es) in the file output. If `False`, they will not be written to the file. If `None`, similar to `True` the dataframe’s index(es) will be saved. However, instead of being saved as values, the RangeIndex will be stored as a range in the metadata so it doesn’t require much space and is faster. Other indexes will be included as columns in the file output. New in version 0.24.0. partition_cols (list, optional, default None) – Column names by which to partition the dataset. Columns are partitioned in the order they are given. Must be None if path is not a string. New in version 0.24.0. storage_options (dict, optional) – Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc., if using a URL that will be parsed by `fsspec`, e.g., starting “s3://”, “gcs://”. An error will be raised if providing this argument with a non-fsspec URL. See the fsspec and backend storage implementation docs for the set of allowed keys and values. New in version 1.2.0. **kwargs – Additional arguments passed to the parquet library. See pandas io for more details.
Returns:
Return type:	bytes if no path argument is provided else None

Differences from pandas

This operation has no known divergences from the pandas API.

See also

read_parquet(): Read a parquet file.
DeferredDataFrame.to_csv(): Write a csv file.
DeferredDataFrame.to_sql(): Write to a sql table.
DeferredDataFrame.to_hdf(): Write to hdf.

Notes

This function requires either the fastparquet or pyarrow library.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame(data={'col1': [1, 2], 'col2': [3, 4]})
>>> df.to_parquet('df.parquet.gzip',
...               compression='gzip')  
>>> pd.read_parquet('df.parquet.gzip')  
   col1  col2
0     1     3
1     2     4

If you want to get a buffer to the parquet content you can use a io.BytesIO
object, as long as you don't use partition_cols, which creates multiple files.

>>> import io
>>> f = io.BytesIO()
>>> df.to_parquet(f)
>>> f.seek(0)
0
>>> content = f.read()

to_period(**kwargs)¶

‘to_period’ is not implemented yet.

If support for ‘to_period’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_pickle(**kwargs)¶

‘to_pickle’ is not implemented yet.

If support for ‘to_pickle’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_sql(**kwargs)¶

‘to_sql’ is not implemented yet.

If support for ‘to_sql’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_stata(path, *args, **kwargs)¶

Export DataFrame object to Stata dta format.

Writes the DataFrame to a Stata dataset file. “dta” files contain a Stata dataset.

Parameters:

path (str, buffer or path object) –
String, path object (pathlib.Path or py._path.local.LocalPath) or object implementing a binary write() function. If using a buffer then the buffer will not be automatically closed after the file data has been written.

Changed in version 1.0.0.

Previously this was “fname”
convert_dates (dict) – Dictionary mapping columns containing datetime types to stata internal format to use when writing the dates. Options are ‘tc’, ‘td’, ‘tm’, ‘tw’, ‘th’, ‘tq’, ‘ty’. Column can be either an integer or a name. Datetime columns that do not have a conversion type specified will be converted to ‘tc’. Raises NotImplementedError if a datetime column has timezone information.
write_index (bool) – Write the index to Stata dataset.
byteorder (str) – Can be “>”, “<”, “little”, or “big”. default is sys.byteorder.
time_stamp (datetime) – A datetime to use as file creation date. Default is the current time.
data_label (str, optional) – A label for the data set. Must be 80 characters or smaller.
variable_labels (dict) – Dictionary containing columns as keys and variable labels as values. Each label must be 80 characters or smaller.
version ({114, 117, 118, 119, None}, default 114) –
Version to use in the output dta file. Set to None to let pandas decide between 118 or 119 formats depending on the number of columns in the frame. pandas Version 114 can be read by Stata 10 and later. pandas Version 117 can be read by Stata 13 or later. pandas Version 118 is supported in Stata 14 and later. pandas Version 119 is supported in Stata 15 and later. pandas Version 114 limits string variables to 244 characters or fewer while versions 117 and later allow strings with lengths up to 2,000,000 characters. Versions 118 and 119 support Unicode characters, and pandas version 119 supports more than 32,767 variables.

pandas Version 119 should usually only be used when the number of variables exceeds the capacity of dta format 118. Exporting smaller datasets in format 119 may have unintended consequences, and, as of November 2020, Stata SE cannot read pandas version 119 files.

Changed in version 1.0.0: Added support for formats 118 and 119.
convert_strl (list, optional) – List of column names to convert to string columns to Stata StrL format. Only available if version is 117. Storing strings in the StrL format can produce smaller dta files if strings have more than 8 characters and values are repeated.
compression (str or dict, default 'infer') –
For on-the-fly compression of the output dta. If string, specifies compression mode. If dict, value at key ‘method’ specifies compression mode. Compression mode must be one of {‘infer’, ‘gzip’, ‘bz2’, ‘zip’, ‘xz’, None}. If compression mode is ‘infer’ and fname is path-like, then detect compression from the following extensions: ‘.gz’, ‘.bz2’, ‘.zip’, or ‘.xz’ (otherwise no compression). If dict and compression mode is one of {‘zip’, ‘gzip’, ‘bz2’}, or inferred as one of the above, other entries passed as additional compression options.

New in version 1.1.0.
storage_options (dict, optional) –
Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc., if using a URL that will be parsed by fsspec, e.g., starting “s3://”, “gcs://”. An error will be raised if providing this argument with a non-fsspec URL. See the fsspec and backend storage implementation docs for the set of allowed keys and values.

New in version 1.2.0.

Raises:

NotImplementedError – * If datetimes contain timezone information * Column dtype is not representable in Stata
ValueError – * Columns listed in convert_dates are neither datetime64[ns]

or datetime.datetime
- Column listed in convert_dates is not in DeferredDataFrame
- Categorical label contains more than 32,000 characters

Differences from pandas

This operation has no known divergences from the pandas API.

See also

read_stata(): Import Stata data files.
io.stata.StataWriter(): Low-level writer for Stata data files.
io.stata.StataWriter117(): Low-level writer for pandas version 117 files.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API.

>>> df = pd.DataFrame({'animal': ['falcon', 'parrot', 'falcon',
...                               'parrot'],
...                    'speed': [350, 18, 361, 15]})
>>> df.to_stata('animals.dta')  

to_timestamp(**kwargs)¶

‘to_timestamp’ is not implemented yet.

If support for ‘to_timestamp’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

to_xarray(**kwargs)¶

‘to_xarray’ is not implemented yet.

If support for ‘to_xarray’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

transform(**kwargs)¶

‘transform’ is not implemented yet.

If support for ‘transform’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

truediv(**kwargs)¶

Get Floating division of dataframe and other, element-wise (binary operator truediv).

Equivalent to dataframe / other, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rtruediv.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters:	other (scalar, sequence, DeferredSeries, or DeferredDataFrame) – Any single or multiple element data structure, or list-like object. axis ({0 or 'index', 1 or 'columns'}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For DeferredSeries input, axis to match DeferredSeries index on. level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level. fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DeferredDataFrame alignment, with this value before computation. If data in both corresponding DeferredDataFrame locations is missing the result will be missing.
Returns:	Result of the arithmetic operation.
Return type:	DeferredDataFrame

Differences from pandas

Only level=None is supported

See also

DeferredDataFrame.add(): Add DeferredDataFrames.
DeferredDataFrame.sub(): Subtract DeferredDataFrames.
DeferredDataFrame.mul(): Multiply DeferredDataFrames.
DeferredDataFrame.div(): Divide DeferredDataFrames (float division).
DeferredDataFrame.truediv(): Divide DeferredDataFrames (float division).
DeferredDataFrame.floordiv(): Divide DeferredDataFrames (integer division).
DeferredDataFrame.mod(): Calculate modulo (remainder after division).
DeferredDataFrame.pow(): Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
           angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360

Add a scalar with operator version which return the same
results.

>>> df + 1
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

>>> df.add(1)
           angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361

Divide by constant with reverse version.

>>> df.div(10)
           angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0

>>> df.rdiv(10)
             angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778

Subtract a list and Series by axis with operator version.

>>> df - [1, 2]
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub([1, 2], axis='columns')
           angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358

>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
           angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359

Multiply a DataFrame of different shape with operator version.

>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
           angles
circle          0
triangle        3
rectangle       4

>>> df * other
           angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN

>>> df.mul(other, fill_value=0)
           angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0

Divide by a MultiIndex by level.

>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
             angles  degrees
A circle          0      360
  triangle        3      180
  rectangle       4      360
B square          4      360
  pentagon        5      540
  hexagon         6      720

>>> df.div(df_multindex, level=1, fill_value=0)
             angles  degrees
A circle        NaN      1.0
  triangle      1.0      1.0
  rectangle     1.0      1.0
B square        0.0      0.0
  pentagon      0.0      0.0
  hexagon       0.0      0.0

truncate(**kwargs)¶

‘truncate’ is not implemented yet.

If support for ‘truncate’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

tshift(**kwargs)¶

‘tshift’ is not implemented yet.

If support for ‘tshift’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

tz_convert(**kwargs)¶

‘tz_convert’ is not implemented yet.

If support for ‘tz_convert’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

tz_localize(ambiguous, **kwargs)¶

Localize tz-naive index of a Series or DataFrame to target time zone.

This operation localizes the Index. To localize the values in a timezone-naive Series, use Series.dt.tz_localize().

Parameters:	tz (str or tzinfo) – axis (the axis to localize) – level (int, str, default None) – If axis ia a MultiIndex, localize a specific level. Otherwise must be None. copy (bool, default True) – Also make a copy of the underlying data. ambiguous ('infer', bool-ndarray, 'NaT', default 'raise') – When clocks moved backward due to DST, ambiguous times may arise. For example in Central European Time (UTC+01), when going from 03:00 DST to 02:00 non-DST, 02:30:00 local time occurs both at 00:30:00 UTC and at 01:30:00 UTC. In such a situation, the ambiguous parameter dictates how ambiguous times should be handled. ’infer’ will attempt to infer fall dst-transition hours based on order bool-ndarray where True signifies a DST time, False designates a non-DST time (note that this flag is only applicable for ambiguous times) ’NaT’ will return NaT where there are ambiguous times ’raise’ will raise an AmbiguousTimeError if there are ambiguous times. nonexistent (str, default 'raise') – A nonexistent time does not exist in a particular timezone where clocks moved forward due to DST. Valid values are: ’shift_forward’ will shift the nonexistent time forward to the closest existing time ’shift_backward’ will shift the nonexistent time backward to the closest existing time ’NaT’ will return NaT where there are nonexistent times timedelta objects will shift nonexistent times by the timedelta ’raise’ will raise an NonExistentTimeError if there are nonexistent times. New in version 0.24.0.
Returns:	Same type as the input.
Return type:	DeferredSeries or DeferredDataFrame
Raises:	`TypeError` – If the TimeDeferredSeries is tz-aware and tz is not None.

Differences from pandas

ambiguous cannot be set to "infer" as its semantics are order-sensitive. Similarly, specifying ambiguous as an ndarray is order-sensitive, but you can achieve similar functionality by specifying ambiguous as a Series.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

Localize local times:

>>> s = pd.Series([1],
...               index=pd.DatetimeIndex(['2018-09-15 01:30:00']))
>>> s.tz_localize('CET')
2018-09-15 01:30:00+02:00    1
dtype: int64

Be careful with DST changes. When there is sequential data, pandas
can infer the DST time:

>>> s = pd.Series(range(7),
...               index=pd.DatetimeIndex(['2018-10-28 01:30:00',
...                                       '2018-10-28 02:00:00',
...                                       '2018-10-28 02:30:00',
...                                       '2018-10-28 02:00:00',
...                                       '2018-10-28 02:30:00',
...                                       '2018-10-28 03:00:00',
...                                       '2018-10-28 03:30:00']))
>>> s.tz_localize('CET', ambiguous='infer')
2018-10-28 01:30:00+02:00    0
2018-10-28 02:00:00+02:00    1
2018-10-28 02:30:00+02:00    2
2018-10-28 02:00:00+01:00    3
2018-10-28 02:30:00+01:00    4
2018-10-28 03:00:00+01:00    5
2018-10-28 03:30:00+01:00    6
dtype: int64

In some cases, inferring the DST is impossible. In such cases, you can
pass an ndarray to the ambiguous parameter to set the DST explicitly

>>> s = pd.Series(range(3),
...               index=pd.DatetimeIndex(['2018-10-28 01:20:00',
...                                       '2018-10-28 02:36:00',
...                                       '2018-10-28 03:46:00']))
>>> s.tz_localize('CET', ambiguous=np.array([True, True, False]))
2018-10-28 01:20:00+02:00    0
2018-10-28 02:36:00+02:00    1
2018-10-28 03:46:00+01:00    2
dtype: int64

If the DST transition causes nonexistent times, you can shift these
dates forward or backward with a timedelta object or `'shift_forward'`
or `'shift_backward'`.

>>> s = pd.Series(range(2),
...               index=pd.DatetimeIndex(['2015-03-29 02:30:00',
...                                       '2015-03-29 03:30:00']))
>>> s.tz_localize('Europe/Warsaw', nonexistent='shift_forward')
2015-03-29 03:00:00+02:00    0
2015-03-29 03:30:00+02:00    1
dtype: int64
>>> s.tz_localize('Europe/Warsaw', nonexistent='shift_backward')
2015-03-29 01:59:59.999999999+01:00    0
2015-03-29 03:30:00+02:00              1
dtype: int64
>>> s.tz_localize('Europe/Warsaw', nonexistent=pd.Timedelta('1H'))
2015-03-29 03:30:00+02:00    0
2015-03-29 03:30:00+02:00    1
dtype: int64

value_counts(**kwargs)¶

‘value_counts’ is not implemented yet.

If support for ‘value_counts’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.

where(cond, other, errors, **kwargs)¶

Replace values where the condition is False.

Parameters:	cond (bool DeferredSeries/DeferredDataFrame, array-like, or callable) – Where cond is True, keep the original value. Where False, replace with corresponding value from other. If cond is callable, it is computed on the DeferredSeries/DeferredDataFrame and should return boolean DeferredSeries/DeferredDataFrame or array. The callable must not change input DeferredSeries/DeferredDataFrame (though pandas doesn’t check it). other (scalar, DeferredSeries/DeferredDataFrame, or callable) – Entries where cond is False are replaced with corresponding value from other. If other is callable, it is computed on the DeferredSeries/DeferredDataFrame and should return scalar or DeferredSeries/DeferredDataFrame. The callable must not change input DeferredSeries/DeferredDataFrame (though pandas doesn’t check it). inplace (bool, default False) – Whether to perform the operation in place on the data. axis (int, default None) – Alignment axis if needed. level (int, default None) – Alignment level if needed. errors (str, {'raise', 'ignore'}, default 'raise') – Note that currently this parameter won’t affect the results and will always coerce to a suitable dtype. ’raise’ : allow exceptions to be raised. ’ignore’ : suppress exceptions. On error return original object. try_cast (bool, default False) – Try to cast the result back to the input type (if possible).
Returns:
Return type:	Same type as caller or None if `inplace=True`.

Differences from pandas

where is not parallelizable when errors="ignore" is specified.

See also

DeferredDataFrame.mask(): Return an object of same shape as self.

Notes

The where method is an application of the if-then idiom. For each element in the calling DeferredDataFrame, if cond is True the element is used; otherwise the corresponding element from the DeferredDataFrame other is used.

The signature for DeferredDataFrame.where() differs from numpy.where(). Roughly df1.where(m, df2) is equivalent to np.where(m, df1, df2).

For further details and examples see the where documentation in indexing.

Examples

NOTE: These examples are pulled directly from the pandas documentation for convenience. Usage of the Beam DataFrame API will look different because it is a deferred API. In addition, some arguments shown here may not be supported, see ‘Differences from pandas’ for details.

>>> s = pd.Series(range(5))
>>> s.where(s > 0)
0    NaN
1    1.0
2    2.0
3    3.0
4    4.0
dtype: float64
>>> s.mask(s > 0)
0    0.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64

>>> s.where(s > 1, 10)
0    10
1    10
2    2
3    3
4    4
dtype: int64
>>> s.mask(s > 1, 10)
0     0
1     1
2    10
3    10
4    10
dtype: int64

>>> df = pd.DataFrame(np.arange(10).reshape(-1, 2), columns=['A', 'B'])
>>> df
   A  B
0  0  1
1  2  3
2  4  5
3  6  7
4  8  9
>>> m = df % 3 == 0
>>> df.where(m, -df)
   A  B
0  0 -1
1 -2  3
2 -4 -5
3  6 -7
4 -8  9
>>> df.where(m, -df) == np.where(m, df, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True
>>> df.where(m, -df) == df.mask(~m, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True

classmethod wrap(expr, split_tuples=True)¶

xs(**kwargs)¶

‘xs’ is not implemented yet.

If support for ‘xs’ is important to you, please let the Beam community know by writing to user@beam.apache.org or commenting on BEAM-9547.