Grammarly Work Note 2023-05-30
Warning
This post is more than a year old. Information may be outdated.
Optimal Resource Utilization and Cost Reduction Strategies for Databricks Clusters
Today Grammarly Experimentations Team discussed strategies to reduce cost in a large-scale Databricks environment, where we experienced a recent spike in usage.
Background
Grammarly Experimentations Team detected a surge in resource usage in our Databricks environment around May. Since we added five default metrics to each experiment and are conducting around 60 experiments simultaneously, the daily volume of metrics being processed has significantly increased. As a result, we have observed a corresponding rise in costs.
Solutions to Explore
- Auto-Scaling Reduction. Given the considerable setup time, we can reduce the minimum number for auto-scaling from 10 to 1 as we use
4xlargenodes. - Creation of Instance Pools. Grammarly Experimentations Team aim to create instance pools for worker nodes with a minimum item instance set to 0. This configuration will ensure that nodes do not persist after job completion, improving cost efficiency. Instance pools also provide the flexibility to add new jobs on the fly.
- Utilization of Cheaper AWS Instances. Grammarly Experimentations Team plans to explore using more cost-effective instances. Our analysis suggests that the currently utilized nodes may not be necessary for these jobs. We could save costs by transitioning to more compute-optimized machines instead of the current storage-optimized ones.
Backlinks1
230530Comments