apache_beam.ml.rag package

Apache Beam RAG (Retrieval Augmented Generation) components. This package provides components for building RAG pipelines in Apache Beam, including: - Chunking - Embedding generation - Vector storage - Vector search enrichment

Subpackages

• apache_beam.ml.rag.chunking package
  • Submodules
    • apache_beam.ml.rag.chunking.base module
      • ChunkingTransformProvider
    • apache_beam.ml.rag.chunking.langchain module
      • LangChainChunker
• apache_beam.ml.rag.embeddings package
  • Submodules
    • apache_beam.ml.rag.embeddings.base module
      • create_rag_adapter()
    • apache_beam.ml.rag.embeddings.huggingface module
      • HuggingfaceTextEmbeddings
    • apache_beam.ml.rag.embeddings.vertex_ai module
      • VertexAITextEmbeddings
• apache_beam.ml.rag.enrichment package
  • Submodules
    • apache_beam.ml.rag.enrichment.bigquery_vector_search module
      • BigQueryVectorSearchParameters
      • BigQueryVectorSearchEnrichmentHandler
      • join_fn()
    • apache_beam.ml.rag.enrichment.milvus_search module
      • SearchStrategy
      • KeywordSearchMetrics
      • VectorSearchMetrics
      • MilvusBaseRanker
      • MilvusConnectionParameters
      • BaseSearchParameters
      • VectorSearchParameters
      • KeywordSearchParameters
      • HybridSearchParameters
      • MilvusSearchParameters
      • MilvusCollectionLoadParameters
      • MilvusSearchResult
      • MilvusSearchEnrichmentHandler
      • join_fn()
      • unpack_dataclass_with_kwargs()
• apache_beam.ml.rag.ingestion package
  • Submodules
    • apache_beam.ml.rag.ingestion.alloydb module
      • AlloyDBLanguageConnectorConfig
      • AlloyDBVectorWriterConfig
    • apache_beam.ml.rag.ingestion.base module
      • VectorDatabaseWriteConfig
      • VectorDatabaseWriteTransform
    • apache_beam.ml.rag.ingestion.bigquery module
      • SchemaConfig
      • BigQueryVectorWriterConfig
    • apache_beam.ml.rag.ingestion.cloudsql module
      • LanguageConnectorConfig
      • CloudSQLPostgresVectorWriterConfig
      • CloudSQLMySQLVectorWriterConfig
    • apache_beam.ml.rag.ingestion.jdbc_common module
      • ConnectionConfig
      • WriteConfig
    • apache_beam.ml.rag.ingestion.mysql module
      • MySQLVectorWriterConfig
    • apache_beam.ml.rag.ingestion.mysql_common module
      • chunk_embedding_fn()
      • ColumnSpec
      • embedding_to_string()
      • ColumnSpecsBuilder
      • ConflictResolution
    • apache_beam.ml.rag.ingestion.postgres module
      • PostgresVectorWriterConfig
    • apache_beam.ml.rag.ingestion.postgres_common module
      • chunk_embedding_fn()
      • ColumnSpec
      • ColumnSpecsBuilder
      • ConflictResolution
    • apache_beam.ml.rag.ingestion.test_utils module
      • row_to_chunk()
      • ChunkTestUtils
      • HashingFn
      • generate_expected_hash()
      • key_on_id()

Submodules

• apache_beam.ml.rag.types module
  • Content
    • Content.text
  • Embedding
    • Embedding.dense_embedding
    • Embedding.sparse_embedding
  • Chunk
    • Chunk.content
    • Chunk.id
    • Chunk.index
    • Chunk.metadata
    • Chunk.embedding
    • Chunk.dense_embedding
    • Chunk.sparse_embedding