Description
Learn how to build end-to-end AI agent retrieval with Lakebase Search, bringing vector search, full-text search, and hybrid search directly into Lakebase Postgres.
In this video, we explore how Lakebase Search enables scalable, low-latency retrieval with pgvector-compatible vector search, BM25 ranking, and hybrid search. Through hands-on demos, you'll see how AI agents can search operational data and memories, and how to build an automated retrieval pipeline using Lakebase CDF, Spark Declarative Pipelines, AI Functions, and Synced Tables to continuously chunk and embed as they change.
All powered by the Databricks platform, without managing separate vector databases or complex data synchronization pipelines.
Chapters:
0:00 Intro: Lakebase Search for AI agents
0:15 Why AI agents turn search into an OLTP workload
0:40 The limits of pgvector and tsvector in Postgres (BM25, vector databases)
1:33 Introducing Lakebase Search: lakebase_vector and lakebase_text
2:36 Hybrid search: semantic + full-text search in SQL
3:17 Reciprocal Rank Fusion (RRF) explained
3:43 Demo: search-as-you-type with full-text search
4:07 Demo: hybrid search results
4:25 Agentic search: let the agent choose the search tool
4:50 Demo: AI support agent with agentic search
5:14 Fast vector search at scale with RaBitQ
5:52 Decoupled storage and compute: indexes that don't need to fit in RAM
6:19 Bursty workloads and scale to zero
6:46 Making agent-written data searchable instantly
7:43 LTAP: offload embedding generation to Spark
8:26 Architecture: an automated embedding pipeline (CDF, Lakeflow Jobs, Synced Tables)
10:18 Demo: an AI agent updates a document
10:44 Demo: the Spark Declarative Pipeline is triggered
11:11 Demo: pipeline code with ai_prep_search() and ai_query()
11:24 Demo: Unity Catalog and the Synced Table to Lakebase
11:47 Demo: hybrid search on the synced documents in Lakebase
12:15 Recap: one backend for OLTP, search and AI agents
Learn more about developing with Lakebase on: https://developers.databricks.com/