Databricks AI Summit 2023 Databricks Session

Data science and AI are moving fast, with giants like Google out in front. Tech giants and startups have very different budgets, and Databricks pitches itself as the thing that closes that gap. These are my notes from the session.

Data and AI win

The big claim was that the winners in every industry will be data and AI companies. They also admitted that many companies still struggle to handle their data, usually for lack of people and infrastructure.

How data handling changed

Structured data used to be easy with Excel, BI, and data warehouses. Once unstructured, unsemantic data piled up, you needed one platform for data lakes, orchestration, governance, data science, warehousing, streaming, and BI.

Governance and silos

• Bad governance leads to bad engineering.
• Data silos drive up operating costs.
• Inconsistent policies and scattered tools make people trust the data less and get in the way of teams working together.

The Lakehouse

Databricks calls the Lakehouse the "iPhone of data." It puts every use of data on a single layer, with one copy of the data and one place for governance. The whole idea rests on unification.

Open source and portability

Open source isn't an advantage by itself, they said. What matters is that "open" means portable, so you don't get locked in.

More data, always

They expect the amount of data to keep growing, and to explode.

Cost

At scale, operations like ETL get pricier on platforms like Snowflake than on Databricks, so Databricks comes out cheaper.

Real-time streaming

Over 50% of Databricks customers use its real-time streaming for critical risk profiling.

AI/ML on the Lakehouse

Lakehouse AI puts data and AI under one security and governance model.

Dolly

Databricks introduced Dolly, which they called the first truly open instruction-tuned LLM licensed for commercial use.

Build vs. buy

They closed on build vs. buy, with a checklist: do you have plenty of engineers, enough time, enough money, and do you need to stay on a single cloud?

Takeaway

Data keeps getting bigger and more important, so companies need something that scales, is unified, and doesn't cost a fortune. The Lakehouse is Databricks' answer to that.