apache_beam.ml.rag.ingestion package

Vector storage ingestion components for RAG pipelines. This module provides components for storing vectors in RAG pipelines.

Submodules

• apache_beam.ml.rag.ingestion.alloydb module
  • AlloyDBLanguageConnectorConfig
    • AlloyDBLanguageConnectorConfig.username
    • AlloyDBLanguageConnectorConfig.password
    • AlloyDBLanguageConnectorConfig.database_name
    • AlloyDBLanguageConnectorConfig.instance_name
    • AlloyDBLanguageConnectorConfig.ip_type
    • AlloyDBLanguageConnectorConfig.enable_iam_auth
    • AlloyDBLanguageConnectorConfig.target_principal
    • AlloyDBLanguageConnectorConfig.delegates
    • AlloyDBLanguageConnectorConfig.admin_service_endpoint
    • AlloyDBLanguageConnectorConfig.quota_project
    • AlloyDBLanguageConnectorConfig.connection_properties
    • AlloyDBLanguageConnectorConfig.additional_properties
    • AlloyDBLanguageConnectorConfig.username
    • AlloyDBLanguageConnectorConfig.password
    • AlloyDBLanguageConnectorConfig.database_name
    • AlloyDBLanguageConnectorConfig.instance_name
    • AlloyDBLanguageConnectorConfig.ip_type
    • AlloyDBLanguageConnectorConfig.enable_iam_auth
    • AlloyDBLanguageConnectorConfig.target_principal
    • AlloyDBLanguageConnectorConfig.delegates
    • AlloyDBLanguageConnectorConfig.admin_service_endpoint
    • AlloyDBLanguageConnectorConfig.quota_project
    • AlloyDBLanguageConnectorConfig.connection_properties
    • AlloyDBLanguageConnectorConfig.additional_properties
    • AlloyDBLanguageConnectorConfig.to_jdbc_url()
    • AlloyDBLanguageConnectorConfig.to_connection_config()
    • AlloyDBLanguageConnectorConfig.additional_jdbc_args()
  • AlloyDBVectorWriterConfig
• apache_beam.ml.rag.ingestion.base module
  • VectorDatabaseWriteConfig
    • VectorDatabaseWriteConfig.create_write_transform()
  • VectorDatabaseWriteTransform
    • VectorDatabaseWriteTransform.expand()
• apache_beam.ml.rag.ingestion.bigquery module
  • SchemaConfig
    • SchemaConfig.schema
    • SchemaConfig.chunk_to_dict_fn
    • SchemaConfig.schema
    • SchemaConfig.chunk_to_dict_fn
  • BigQueryVectorWriterConfig
    • BigQueryVectorWriterConfig.create_write_transform()
• apache_beam.ml.rag.ingestion.cloudsql module
  • LanguageConnectorConfig
    • LanguageConnectorConfig.username
    • LanguageConnectorConfig.password
    • LanguageConnectorConfig.database_name
    • LanguageConnectorConfig.instance_name
    • LanguageConnectorConfig.ip_type
    • LanguageConnectorConfig.enable_iam_auth
    • LanguageConnectorConfig.target_principal
    • LanguageConnectorConfig.delegates
    • LanguageConnectorConfig.admin_service_endpoint
    • LanguageConnectorConfig.quota_project
    • LanguageConnectorConfig.connection_properties
    • LanguageConnectorConfig.additional_properties
    • LanguageConnectorConfig.username
    • LanguageConnectorConfig.password
    • LanguageConnectorConfig.database_name
    • LanguageConnectorConfig.instance_name
    • LanguageConnectorConfig.ip_types
    • LanguageConnectorConfig.enable_iam_auth
    • LanguageConnectorConfig.target_principal
    • LanguageConnectorConfig.delegates
    • LanguageConnectorConfig.quota_project
    • LanguageConnectorConfig.connection_properties
    • LanguageConnectorConfig.additional_properties
    • LanguageConnectorConfig.to_connection_config()
    • LanguageConnectorConfig.additional_jdbc_args()
  • CloudSQLPostgresVectorWriterConfig
• apache_beam.ml.rag.ingestion.jdbc_common module
  • ConnectionConfig
    • ConnectionConfig.jdbc_url
    • ConnectionConfig.username
    • ConnectionConfig.password
    • ConnectionConfig.connection_properties
    • ConnectionConfig.connection_init_sqls
    • ConnectionConfig.additional_jdbc_args
    • ConnectionConfig.jdbc_url
    • ConnectionConfig.username
    • ConnectionConfig.password
    • ConnectionConfig.connection_properties
    • ConnectionConfig.connection_init_sqls
    • ConnectionConfig.additional_jdbc_args
  • WriteConfig
    • WriteConfig.autosharding
    • WriteConfig.max_connections
    • WriteConfig.write_batch_size
    • WriteConfig.autosharding
    • WriteConfig.max_connections
    • WriteConfig.write_batch_size
• apache_beam.ml.rag.ingestion.postgres module
  • PostgresVectorWriterConfig
    • PostgresVectorWriterConfig.create_write_transform()
• apache_beam.ml.rag.ingestion.postgres_common module
  • chunk_embedding_fn()
  • ColumnSpec
    • ColumnSpec.column_name
    • ColumnSpec.python_type
    • ColumnSpec.value_fn
    • ColumnSpec.sql_typecast
    • ColumnSpec.column_name
    • ColumnSpec.python_type
    • ColumnSpec.value_fn
    • ColumnSpec.sql_typecast
    • ColumnSpec.placeholder
    • ColumnSpec.text()
    • ColumnSpec.integer()
    • ColumnSpec.float()
    • ColumnSpec.vector()
    • ColumnSpec.jsonb()
  • ColumnSpecsBuilder
    • ColumnSpecsBuilder.with_defaults()
    • ColumnSpecsBuilder.with_id_spec()
    • ColumnSpecsBuilder.with_content_spec()
    • ColumnSpecsBuilder.with_metadata_spec()
    • ColumnSpecsBuilder.with_embedding_spec()
    • ColumnSpecsBuilder.add_metadata_field()
    • ColumnSpecsBuilder.add_custom_column_spec()
    • ColumnSpecsBuilder.build()
  • ConflictResolution
    • ConflictResolution.on_conflict_fields
    • ConflictResolution.action
    • ConflictResolution.update_fields
    • ConflictResolution.on_conflict_fields
    • ConflictResolution.action
    • ConflictResolution.update_fields
    • ConflictResolution.maybe_set_default_update_fields()
    • ConflictResolution.get_conflict_clause()
• apache_beam.ml.rag.ingestion.test_utils module
  • TestRow
    • TestRow.content
    • TestRow.embedding
    • TestRow.id
    • TestRow.metadata
  • row_to_chunk()
  • ChunkTestUtils
    • ChunkTestUtils.from_seed()
    • ChunkTestUtils.get_expected_values()
  • HashingFn
    • HashingFn.create_accumulator()
    • HashingFn.add_input()
    • HashingFn.merge_accumulators()
    • HashingFn.extract_output()
  • generate_expected_hash()
  • key_on_id()