Databricks Data + AI Summit 2025 – Full Executive Summary & Key Takeaways - EXC Summary

Databricks continues to champion the "Lakehouse" architecture, which combines the benefits of data lakes (cost-effective storage, open formats) and data warehouses (structured data, governance). This approach was initially met with skepticism but has since gained widespread industry acceptance.

Industry Acceptance of Open Formats (Delta Lake & Iceberg): Five years after its introduction, the lakehouse concept is widely embraced. "Now everybody loves open everybody's talking about it... there's even books by the creators of data warehousing talking about the lakehouse and how that's the future."

Emphasis on Data Ownership: Databricks strongly advocates for customers to "own the data. Don't give it to Databricks. Don't give it to other vendors. Own it, store it on open lakes like S3 on Amazon or like Azure data lake in storage ADLS on Microsoft Azure or like GCS Google cloud storage." This prevents vendor lock-in.

Unified Support for Open Formats (Delta Lake & Iceberg): Databricks now offers "100% support for both" Delta Lake and Apache Iceberg formats. Through the acquisition of Tabular, founded by the original creators of Apache Iceberg, Databricks has worked to integrate these formats, making their differences "largely now negligible."

Unity Catalog as the Open Standard for Governance: Unity Catalog, open-sourced last year, is presented as the "most open approach you can take to the lakehouse." It implements standard open-source interfaces like Hive Metastore and Iceberg Rest Catalog, enabling any system to interact with it.

A significant focus of the summit is "Data Intelligence," defined by two core objectives: democratizing access to data and democratizing AI.

English as the New Programming Language: The goal is to allow users to "just speak English to your data or any other mother tongue or natural language that you have and we should be able to get answers for you."

• Genie: Used by "81% of our customers," Genie allows users to ask questions in natural language within "data rooms" in Unity Catalog. It uses an "ensemble of agents" to "write code for you and then executes that," eliminating the need for coding knowledge.
• Assistant: Used by "98% of our customers," the Databricks Assistant "understand[s] every aspect of the platform" and the user's "data in the organization," aiding with error diagnosis, code explanation, and code generation.

Building Proprietary AI: Databricks aims to help companies "build your own AI for your companies so you can innovate AI that can reason and answer questions on your proprietary enterprise data or organizational data." This focuses on "domain specific intelligence or we call it data intelligence."

AI Adoption Statistics: Classic ML is used by "95% of our customers," while generative AI has quickly picked up, now at "81% of our customer base."

A major new architectural announcement is "Lakebase," a paradigm shift for traditional transactional databases.

Addressing Legacy Database Limitations: Traditional transactional databases (SQL Server, Oracle, MySQL, Postgres) are criticized for:

• Lock-in: "Data is so sticky... it's just nearly impossible for you to move off of it."
• High Cost: Due to lock-in and "clunky big instances that you buy."
• Outdated Architecture: "Built for that on-prem era... they were built pre-AI."

Lakebase Architecture: Splits the database into a "base and a lake layer," storing data in "cheap data lakes in an open format." Transaction processing occurs in the "base layer on top."

• Key Lakebase Attributes:Open Source (Postgres): Built on "open source Postgress" due to its massive traction and ecosystem of extensions.
• Proper Separation of Compute and Storage: Achieves low-latency (single-digit millisecond) and high QPS (millions) by introducing a "middle layer storage that's actually only have soft state and it acts as a write through cache for all the data to the object stores."
• Built for the AI Era: Designed for rapid launch, scalability to zero, and pay-per-use, supporting the launch of "hundreds or thousands of these databases" by AI agents.

Neon Acquisition: Databricks acquired Neon, a company that developed this "new novel separation of storage from compute architecture," to power Lakebase.

Modern Developer Workflow: Lakebase enables "serverless" and "branching" capabilities for databases.

• Serverless: Databases can be launched in less than a second, auto-scaling up or down to zero, meaning "you only pay for when the duration you actually need the compute."
• Branching: Instantly creates a "whole clone of the database... including both the data and the schema" in less than a second, leveraging copy-on-write. This "completely change[s] the way you think about database development."

AI Agent Database Creation: Notably, "80% of the databases created on neon.com were created by AI agents, not humans." This is expected to reach "99% of all the databases on the platform will be created by AI agents" in a couple of years.

Databricks Apps: Initially a skepticism, Databricks Apps, launched in November 2023, now have "over 2,500 customers actually building their own applications." They provide a secure and governed way to connect front-end applications to data and AI, simplifying "productionalizing of these applications next to your data and AI."

Agent Bricks: A higher-level framework than the agent framework announced a year ago, Agent Bricks "maps to the business problems you have and then it uses agent framework under the hood."

• Automated LLM Judges: Agent Bricks automatically creates "LLM judges for your specific problem" to evaluate agent performance. This is based on the insight that "large language models are much better judges than creators."
• Auto-optimization: It "automatically search[es] through and compose[s] different optimization methods and settings to deliver high quality," including fine-tuning, prompt optimization, and reinforcement learning.
• Cost vs. Quality Graph: Users are presented with a "cost versus quality graph" to choose the optimal AI system for their needs.
• Agent Learning: A technique that "helps you get feedback into the system and generalize it and keep optimizing it on new data" so the system "gets smarter and smarter over time."
• Use Cases: Agent Bricks supports tasks like information extraction, knowledge assistant (Q&A), multi-agent supervision (orchestrating multiple expert agents), and custom LLMs. Virgin Atlantic leverages AI for "triage... summarization and categorization of health and safety messages," saving teams "one to two hours per day."

The core data analytics capabilities of Databricks are also seeing significant innovation.

Apache Spark 4.0 & Open Sourcing DLT: Databricks is contributing "real time mode" and "declarative pipelines" (formerly Delta Live Tables, DLT) to Apache Spark.

• Real-time Mode: Dramatically reduces latency from "many seconds down to a couple of milliseconds" for operational workloads.
• Declarative Pipelines: Simplifies ETL by allowing users to "build an end-to-end production pipeline" with "just a few lines of SQL," abstracting away underlying complexities.

LakeFlow: A "generally available" platform that simplifies ETL and data engineering. It includes an "IDE for data engineering" with AI assistance, data exploration, debugging, and production readiness. It leverages declarative pipelines and serverless compute.

Databricks SQL (DBSQL): Databricks has improved DBSQL performance by "25% and we didn't change the price at all."

• Lowest TCO: Continues to claim the "best TCO in the market" compared to other cloud data warehouses.
• Open by Design: Built on "100% open data formats" (Delta Lake, Iceberg) and the "open catalog" (Unity Catalog), ensuring full interoperability and avoiding lock-in.
• Unified with Data Intelligence Platform: Natively integrated with Unity Catalog and AI capabilities, allowing users to "leverage pretty much any LLM that you want directly through SQL."

Lakebridge: A free system that uses LLMs and AB testing to automate migrations from proprietary data warehouses to the Databricks lakehouse, achieving "much higher accuracy on the code" and faster, lower-cost migrations.

AIBI (AI + BI): Databricks' BI platform is "exploding in usage," with "500% user growth" in the last year.

• Key Differentiators: Free of charge, full-fledged BI platform, blazing fast, and "secure by design" through Unity Catalog integration.
• AI Integration: Built with AI from the ground up, notably with Genie, which allows users to ask questions in natural language.
• AI Forecasting & Top Drivers: New features enable business users to "add an accurate forecast right into your dashboard" and "explain the difference" for anomalous data points using AI.
• Deep Research Mode (Genie): Entering preview, this mode leverages LLM reasoning and Genie's knowledge store to tackle "deep open research questions" by creating and executing research plans.
• Shifting Semantics Left: Databricks advocates for semantic layers to reside within the data platform (Unity Catalog Metrics) rather than proprietary BI tools, ensuring consistent metric definitions across all tools and users.
• Databricks One: A "brand new experience for Databricks designed specifically for business users," providing "the one place they go to to get data and AI," with a simplified UI and curated content.
• General Availability: AIBI Genie is now "generally available to everyone today."

Databricks continues to expand its ecosystem and partnerships to drive its mission.

Hyperscaler Sponsors: AWS, Google, and Microsoft are key "legend sponsors."

• Microsoft Azure Partnership: An "eight years" long partnership, with Databricks being a "first-party service" on Azure, integrating with Foundry, Power Platform, and SAP. Satya Nadella emphasizes that the collaboration leads to "GDP growth in the real world."
• Google Cloud (GCP) Partnership: Bringing "Gemini models natively to Databricks," enabling enterprises to build AI use cases on their data on GCP. Gemini's strengths in "reasoning" and "tool selection" are highlighted.
• Agent-to-Agent (A2A) Protocol: Google has made available an "A2A protocol" in open source, allowing "one agent talking to another agent and being able to understand what's the API to the other agent." Databricks is partnering on this to foster a diverse ecosystem of interoperable agents.

Mastercard: Leveraging Databricks for "data intelligence," with a focus on scaling use cases, consistent evaluations, and trust in data handling. Mastercard processes "159 billion transactions on our network" and aims to bring more data assets together into a "data mesh."

JP Morgan Chase (JPMC): Jamie Dimon, CEO of JPMC, discusses their $18 billion annual IT budget, with AI as a major priority.

• Internal AI Adoption: 200,000 JPMC employees are using LLMs on their "internal data," which "dwarfs some of the stuff you have in the web." Examples include reviewing legal documents and optimizing money movement using Databricks systems ("brie").
• AI and Jobs: Dimon sees AI reducing jobs but emphasizes retraining and redeployment, using attrition as a "friend."
• Cybersecurity Concerns: AI is seen as a significant threat in cybersecurity, with "bad guys already using it" to penetrate major companies. JPMC spends "almost a billion dollars a dollars a year in cyber protection."
• Risk Management: JPMC's approach is to "look at risk management not guessing the future but laying out the wide range of possibilities and then studying can you handle that." They perform "a hundred stress tests a week" compared to the Fed's one a year.
• US Leadership in AI: Dimon views American military and technological leadership as "critical to the health of the future free and democratic world."

Databricks Free Edition: A new offering that allows anyone to "get a free slice of Databricks forever," without requiring credit card or business email. This aims to "democratize data and AI."

$100 Million Investment in Training & Education: Databricks is investing in training and education, open-sourcing all its self-paced learning content and making it free, particularly for universities.