Databricks AI Summit 2023 Definitive Healthcare Session

Definitive Healthcare, a healthcare AI company, talked about running and optimizing data at scale, and what they ended up doing about it.

Problems

1. The goal was clear, but the road to optimized data was messy.
2. They had hundreds of assets, each needing its own ETL and moving through zones from landing to curated.
3. HIPAA and GDPR compliance forced them into data silos.
4. End users didn't know enough, especially about query optimization.
5. Humans and machines were querying a huge amount of data all the time.

Data and stakeholders

Their tables ran from 111 records (Patient Transactions) to 15M Expert Profiles (Scientific Affairs Data). Stakeholders included product teams, data science, custom predictive analytics, and data analysts, so the needs ranged from ETL ingestion to custom predictive models.

Why the Databricks Lakehouse

It covered every priority they had.

1. Solid pipelines for large daily ETL ingestion.
2. Full database restatements.
3. An enterprise data warehouse for hundreds of inbound sources and 24/7 aggregation.
4. Tools for everyone, from data engineering and data management to analytics and data science.
5. SQL, Python, R, and Scala.
6. Scaling up as needed to get the most work done.

Query performance mattered a lot, but never at the cost of data integrity. The trick was balancing the two.

What they used

1. Unity Catalog, for data changes and restatements.
2. ACID transaction logs and time travel, to track changes and roll back.
3. Dynamic partitions and Z-order, to tune performance per application.
4. Reserved and on-demand capacity, for flexibility.
5. Multi-level security and data sharing with audit logs and tooling, which compliance and governance depend on.