apache_beam.dataframe package

Beam DataFrame API

• For high-level documentation see https://beam.apache.org/documentation/dsls/dataframes/overview/

• apache_beam.dataframe.io: DataFrame I/Os

• apache_beam.dataframe.frames: DataFrame operations

• apache_beam.dataframe.convert: Conversion between PCollection and DeferredDataFrame.

• apache_beam.dataframe.schemas: Mapping from Beam Schemas to pandas dtypes.

• apache_beam.dataframe.transforms: Use DataFrame operations within a Beam pipeline with DataframeTransform.

Submodules

• apache_beam.dataframe.convert module
  • to_dataframe()
  • RowsToDataFrameFn
    • RowsToDataFrameFn.process_batch()
  • ElementsToSeriesFn
    • ElementsToSeriesFn.process_batch()
  • DataFrameToRowsFn
    • DataFrameToRowsFn.process()
    • DataFrameToRowsFn.infer_output_type()
  • SeriesToElementsFn
    • SeriesToElementsFn.process()
    • SeriesToElementsFn.infer_output_type()
  • to_pcollection()
• apache_beam.dataframe.doctests module
  • FakePandasObject
  • TestEnvironment
    • TestEnvironment.fake_pandas_module()
    • TestEnvironment.context()
  • BeamDataframeDoctestRunner
    • BeamDataframeDoctestRunner.run()
    • BeamDataframeDoctestRunner.report_success()
    • BeamDataframeDoctestRunner.fake_pandas_module()
    • BeamDataframeDoctestRunner.summarize()
    • BeamDataframeDoctestRunner.summary()
  • AugmentedTestResults
  • Summary
    • Summary.result()
    • Summary.summarize()
  • parse_rst_ipython_tests()
  • test_rst_ipython()
  • teststring()
  • teststrings()
  • set_pandas_options()
  • with_run_patched_docstring()
  • testfile()
  • testmod()
• apache_beam.dataframe.expressions module
  • Session
    • Session.evaluate()
    • Session.lookup()
  • PartitioningSession
    • PartitioningSession.evaluate()
  • output_partitioning()
  • Expression
    • Expression.proxy()
    • Expression.placeholders()
    • Expression.evaluate_at()
    • Expression.requires_partition_by()
    • Expression.preserves_partition_by()
  • PlaceholderExpression
    • PlaceholderExpression.placeholders()
    • PlaceholderExpression.args()
    • PlaceholderExpression.evaluate_at()
    • PlaceholderExpression.requires_partition_by()
    • PlaceholderExpression.preserves_partition_by()
  • ConstantExpression
    • ConstantExpression.placeholders()
    • ConstantExpression.args()
    • ConstantExpression.evaluate_at()
    • ConstantExpression.requires_partition_by()
    • ConstantExpression.preserves_partition_by()
  • ComputedExpression
    • ComputedExpression.placeholders()
    • ComputedExpression.args()
    • ComputedExpression.evaluate_at()
    • ComputedExpression.requires_partition_by()
    • ComputedExpression.preserves_partition_by()
  • elementwise_expression()
  • allow_non_parallel_operations()
  • NonParallelOperation
• apache_beam.dataframe.frame_base module
  • DeferredBase
    • DeferredBase.wrap()
  • UnusableUnpickledDeferredBase
  • DeferredFrame
  • name_and_func()
  • wont_implement_method()
  • not_implemented_method()
  • maybe_inplace()
  • args_to_kwargs()
  • with_docs_from()
  • populate_defaults()
  • WontImplementError
• apache_beam.dataframe.frames module
  • DeferredSeries
    • DeferredSeries.name
    • DeferredSeries.hasnans
    • DeferredSeries.dtype
    • DeferredSeries.dtypes
    • DeferredSeries.keys()
    • DeferredSeries.T()
    • DeferredSeries.transpose()
    • DeferredSeries.shape
    • DeferredSeries.append()
    • DeferredSeries.align()
    • DeferredSeries.argsort()
    • DeferredSeries.array
    • DeferredSeries.get()
    • DeferredSeries.ravel()
    • DeferredSeries.slice_shift()
    • DeferredSeries.tshift()
    • DeferredSeries.rename()
    • DeferredSeries.between()
    • DeferredSeries.add_suffix()
    • DeferredSeries.add_prefix()
    • DeferredSeries.info()
    • DeferredSeries.idxmin()
    • DeferredSeries.idxmax()
    • DeferredSeries.explode()
    • DeferredSeries.dot()
    • DeferredSeries.nunique()
    • DeferredSeries.quantile()
    • DeferredSeries.std()
    • DeferredSeries.mean()
    • DeferredSeries.var()
    • DeferredSeries.corr()
    • DeferredSeries.skew()
    • DeferredSeries.kurtosis()
    • DeferredSeries.kurt()
    • DeferredSeries.cov()
    • DeferredSeries.dropna()
    • DeferredSeries.set_axis()
    • DeferredSeries.isnull()
    • DeferredSeries.isna()
    • DeferredSeries.notnull()
    • DeferredSeries.notna()
    • DeferredSeries.items()
    • DeferredSeries.iteritems()
    • DeferredSeries.tolist()
    • DeferredSeries.to_numpy()
    • DeferredSeries.to_string()
    • DeferredSeries.duplicated()
    • DeferredSeries.drop_duplicates()
    • DeferredSeries.sample()
    • DeferredSeries.aggregate()
    • DeferredSeries.agg()
    • DeferredSeries.axes
    • DeferredSeries.clip()
    • DeferredSeries.all()
    • DeferredSeries.any()
    • DeferredSeries.count()
    • DeferredSeries.describe()
    • DeferredSeries.min()
    • DeferredSeries.max()
    • DeferredSeries.prod()
    • DeferredSeries.product()
    • DeferredSeries.sum()
    • DeferredSeries.median()
    • DeferredSeries.sem()
    • DeferredSeries.argmax()
    • DeferredSeries.argmin()
    • DeferredSeries.cummax()
    • DeferredSeries.cummin()
    • DeferredSeries.cumprod()
    • DeferredSeries.cumsum()
    • DeferredSeries.diff()
    • DeferredSeries.interpolate()
    • DeferredSeries.searchsorted()
    • DeferredSeries.shift()
    • DeferredSeries.pct_change()
    • DeferredSeries.is_monotonic()
    • DeferredSeries.is_monotonic_increasing()
    • DeferredSeries.is_monotonic_decreasing()
    • DeferredSeries.asof()
    • DeferredSeries.first_valid_index()
    • DeferredSeries.last_valid_index()
    • DeferredSeries.autocorr()
    • DeferredSeries.iat
    • DeferredSeries.head()
    • DeferredSeries.tail()
    • DeferredSeries.filter()
    • DeferredSeries.memory_usage()
    • DeferredSeries.nbytes()
    • DeferredSeries.to_list()
    • DeferredSeries.factorize()
    • DeferredSeries.nlargest()
    • DeferredSeries.nsmallest()
    • DeferredSeries.is_unique
    • DeferredSeries.plot()
    • DeferredSeries.pop()
    • DeferredSeries.rename_axis()
    • DeferredSeries.round()
    • DeferredSeries.take()
    • DeferredSeries.to_dict()
    • DeferredSeries.to_frame()
    • DeferredSeries.unique()
    • DeferredSeries.update()
    • DeferredSeries.value_counts()
    • DeferredSeries.values
    • DeferredSeries.view()
    • DeferredSeries.str
    • DeferredSeries.cat
    • DeferredSeries.dt
    • DeferredSeries.mode()
    • DeferredSeries.apply()
    • DeferredSeries.map()
    • DeferredSeries.repeat()
    • DeferredSeries.compare()
    • DeferredSeries.add()
    • DeferredSeries.asfreq()
    • DeferredSeries.at
    • DeferredSeries.convert_dtypes()
    • DeferredSeries.div()
    • DeferredSeries.divide()
    • DeferredSeries.divmod()
    • DeferredSeries.eq()
    • DeferredSeries.flags
    • DeferredSeries.floordiv()
    • DeferredSeries.ge()
    • DeferredSeries.gt()
    • DeferredSeries.infer_objects()
    • DeferredSeries.item()
    • DeferredSeries.le()
    • DeferredSeries.lt()
    • DeferredSeries.mod()
    • DeferredSeries.mul()
    • DeferredSeries.multiply()
    • DeferredSeries.ne()
    • DeferredSeries.pow()
    • DeferredSeries.radd()
    • DeferredSeries.rank()
    • DeferredSeries.rdiv()
    • DeferredSeries.rdivmod()
    • DeferredSeries.reindex_like()
    • DeferredSeries.rfloordiv()
    • DeferredSeries.rmod()
    • DeferredSeries.rmul()
    • DeferredSeries.rpow()
    • DeferredSeries.rsub()
    • DeferredSeries.rtruediv()
    • DeferredSeries.set_flags()
    • DeferredSeries.squeeze()
    • DeferredSeries.sub()
    • DeferredSeries.subtract()
    • DeferredSeries.to_clipboard()
    • DeferredSeries.to_csv()
    • DeferredSeries.to_excel()
    • DeferredSeries.to_feather()
    • DeferredSeries.to_hdf()
    • DeferredSeries.to_html()
    • DeferredSeries.to_json()
    • DeferredSeries.to_latex()
    • DeferredSeries.to_markdown()
    • DeferredSeries.to_msgpack()
    • DeferredSeries.to_parquet()
    • DeferredSeries.to_period()
    • DeferredSeries.to_pickle()
    • DeferredSeries.to_sql()
    • DeferredSeries.to_stata()
    • DeferredSeries.to_timestamp()
    • DeferredSeries.truediv()
  • DeferredDataFrame
    • DeferredDataFrame.columns
    • DeferredDataFrame.keys()
    • DeferredDataFrame.align()
    • DeferredDataFrame.append()
    • DeferredDataFrame.get()
    • DeferredDataFrame.set_index()
    • DeferredDataFrame.set_axis()
    • DeferredDataFrame.axes
    • DeferredDataFrame.dtypes
    • DeferredDataFrame.assign()
    • DeferredDataFrame.explode()
    • DeferredDataFrame.insert()
    • DeferredDataFrame.from_dict()
    • DeferredDataFrame.from_records()
    • DeferredDataFrame.duplicated()
    • DeferredDataFrame.drop_duplicates()
    • DeferredDataFrame.aggregate()
    • DeferredDataFrame.agg()
    • DeferredDataFrame.applymap()
    • DeferredDataFrame.map()
    • DeferredDataFrame.add_prefix()
    • DeferredDataFrame.add_suffix()
    • DeferredDataFrame.memory_usage()
    • DeferredDataFrame.info()
    • DeferredDataFrame.clip()
    • DeferredDataFrame.corr()
    • DeferredDataFrame.cov()
    • DeferredDataFrame.corrwith()
    • DeferredDataFrame.cummax()
    • DeferredDataFrame.cummin()
    • DeferredDataFrame.cumprod()
    • DeferredDataFrame.cumsum()
    • DeferredDataFrame.diff()
    • DeferredDataFrame.interpolate()
    • DeferredDataFrame.pct_change()
    • DeferredDataFrame.asof()
    • DeferredDataFrame.first_valid_index()
    • DeferredDataFrame.last_valid_index()
    • DeferredDataFrame.iat
    • DeferredDataFrame.lookup()
    • DeferredDataFrame.head()
    • DeferredDataFrame.tail()
    • DeferredDataFrame.sample()
    • DeferredDataFrame.dot()
    • DeferredDataFrame.mode()
    • DeferredDataFrame.dropna()
    • DeferredDataFrame.eval()
    • DeferredDataFrame.query()
    • DeferredDataFrame.isnull()
    • DeferredDataFrame.isna()
    • DeferredDataFrame.notnull()
    • DeferredDataFrame.notna()
    • DeferredDataFrame.items()
    • DeferredDataFrame.itertuples()
    • DeferredDataFrame.iterrows()
    • DeferredDataFrame.iteritems()
    • DeferredDataFrame.join()
    • DeferredDataFrame.merge()
    • DeferredDataFrame.nlargest()
    • DeferredDataFrame.nsmallest()
    • DeferredDataFrame.plot()
    • DeferredDataFrame.pop()
    • DeferredDataFrame.quantile()
    • DeferredDataFrame.rename()
    • DeferredDataFrame.rename_axis()
    • DeferredDataFrame.round()
    • DeferredDataFrame.select_dtypes()
    • DeferredDataFrame.shift()
    • DeferredDataFrame.shape
    • DeferredDataFrame.stack()
    • DeferredDataFrame.all()
    • DeferredDataFrame.any()
    • DeferredDataFrame.count()
    • DeferredDataFrame.describe()
    • DeferredDataFrame.max()
    • DeferredDataFrame.min()
    • DeferredDataFrame.pivot()
    • DeferredDataFrame.prod()
    • DeferredDataFrame.product()
    • DeferredDataFrame.sum()
    • DeferredDataFrame.mean()
    • DeferredDataFrame.median()
    • DeferredDataFrame.nunique()
    • DeferredDataFrame.std()
    • DeferredDataFrame.var()
    • DeferredDataFrame.sem()
    • DeferredDataFrame.skew()
    • DeferredDataFrame.kurt()
    • DeferredDataFrame.kurtosis()
    • DeferredDataFrame.take()
    • DeferredDataFrame.to_records()
    • DeferredDataFrame.to_dict()
    • DeferredDataFrame.to_numpy()
    • DeferredDataFrame.to_string()
    • DeferredDataFrame.to_sparse()
    • DeferredDataFrame.transpose()
    • DeferredDataFrame.T
    • DeferredDataFrame.update()
    • DeferredDataFrame.values
    • DeferredDataFrame.style
    • DeferredDataFrame.melt()
    • DeferredDataFrame.value_counts()
    • DeferredDataFrame.compare()
    • DeferredDataFrame.idxmin()
    • DeferredDataFrame.idxmax()
    • DeferredDataFrame.add()
    • DeferredDataFrame.apply()
    • DeferredDataFrame.asfreq()
    • DeferredDataFrame.at
    • DeferredDataFrame.boxplot()
    • DeferredDataFrame.convert_dtypes()
    • DeferredDataFrame.div()
    • DeferredDataFrame.divide()
    • DeferredDataFrame.eq()
    • DeferredDataFrame.filter()
    • DeferredDataFrame.flags
    • DeferredDataFrame.floordiv()
    • DeferredDataFrame.ge()
    • DeferredDataFrame.gt()
    • DeferredDataFrame.infer_objects()
    • DeferredDataFrame.isetitem()
    • DeferredDataFrame.le()
    • DeferredDataFrame.lt()
    • DeferredDataFrame.mod()
    • DeferredDataFrame.mul()
    • DeferredDataFrame.multiply()
    • DeferredDataFrame.ne()
    • DeferredDataFrame.pivot_table()
    • DeferredDataFrame.pow()
    • DeferredDataFrame.radd()
    • DeferredDataFrame.rank()
    • DeferredDataFrame.rdiv()
    • DeferredDataFrame.reindex_like()
    • DeferredDataFrame.rfloordiv()
    • DeferredDataFrame.rmod()
    • DeferredDataFrame.rmul()
    • DeferredDataFrame.rpow()
    • DeferredDataFrame.rsub()
    • DeferredDataFrame.rtruediv()
    • DeferredDataFrame.set_flags()
    • DeferredDataFrame.squeeze()
    • DeferredDataFrame.sub()
    • DeferredDataFrame.subtract()
    • DeferredDataFrame.to_clipboard()
    • DeferredDataFrame.to_csv()
    • DeferredDataFrame.to_excel()
    • DeferredDataFrame.to_feather()
    • DeferredDataFrame.to_gbq()
    • DeferredDataFrame.to_hdf()
    • DeferredDataFrame.to_html()
    • DeferredDataFrame.to_json()
    • DeferredDataFrame.to_latex()
    • DeferredDataFrame.to_markdown()
    • DeferredDataFrame.to_msgpack()
    • DeferredDataFrame.to_orc()
    • DeferredDataFrame.to_parquet()
    • DeferredDataFrame.to_period()
    • DeferredDataFrame.to_pickle()
    • DeferredDataFrame.to_sql()
    • DeferredDataFrame.to_stata()
    • DeferredDataFrame.to_timestamp()
    • DeferredDataFrame.to_xml()
    • DeferredDataFrame.truediv()
• apache_beam.dataframe.io module
  • Sources
  • Sinks
  • read_gbq()
  • read_csv()
  • to_csv()
  • read_fwf()
  • read_json()
  • to_json()
  • read_html()
  • to_html()
  • ReadViaPandas
    • ReadViaPandas.expand()
  • WriteViaPandas
    • WriteViaPandas.expand()
  • read_excel()
  • read_feather()
  • read_parquet()
  • read_sas()
  • read_spss()
  • read_stata()
  • to_excel()
  • to_feather()
  • to_parquet()
  • to_stata()
• apache_beam.dataframe.pandas_top_level_functions module
  • DeferredPandasModule
    • DeferredPandasModule.array()
    • DeferredPandasModule.bdate_range()
    • DeferredPandasModule.concat()
    • DeferredPandasModule.date_range()
    • DeferredPandasModule.describe_option()
    • DeferredPandasModule.factorize()
    • DeferredPandasModule.get_option()
    • DeferredPandasModule.interval_range()
    • DeferredPandasModule.isna()
    • DeferredPandasModule.isnull()
    • DeferredPandasModule.json_normalize()
    • DeferredPandasModule.merge()
    • DeferredPandasModule.melt()
    • DeferredPandasModule.merge_ordered()
    • DeferredPandasModule.notnull()
    • DeferredPandasModule.option_context()
    • DeferredPandasModule.period_range()
    • DeferredPandasModule.pivot()
    • DeferredPandasModule.pivot_table()
    • DeferredPandasModule.set_eng_float_format()
    • DeferredPandasModule.show_versions()
    • DeferredPandasModule.test()
    • DeferredPandasModule.timedelta_range()
    • DeferredPandasModule.to_pickle()
    • DeferredPandasModule.to_datetime()
    • DeferredPandasModule.notna()
• apache_beam.dataframe.partitionings module
  • Partitioning
    • Partitioning.is_subpartitioning_of()
    • Partitioning.partition_fn()
    • Partitioning.test_partition_fn()
  • Index
    • Index.is_subpartitioning_of()
    • Index.partition_fn()
    • Index.check()
  • Singleton
    • Singleton.reason
    • Singleton.is_subpartitioning_of()
    • Singleton.partition_fn()
    • Singleton.check()
  • JoinIndex
    • JoinIndex.is_subpartitioning_of()
    • JoinIndex.test_partition_fn()
    • JoinIndex.check()
  • Arbitrary
    • Arbitrary.is_subpartitioning_of()
    • Arbitrary.test_partition_fn()
    • Arbitrary.check()
• apache_beam.dataframe.schemas module
  • BatchRowsAsDataFrame
    • BatchRowsAsDataFrame.expand()
  • generate_proxy()
  • UnbatchPandas
    • UnbatchPandas.expand()
  • element_type_from_dataframe()
• apache_beam.dataframe.transforms module
  • DataframeTransform
    • DataframeTransform.expand()