Grammarly Work Note 2023-06-01
Warning
A large portion of an ML Engineer's work is handling data, sourcing it, examining it, cleansing it, and computing features. Training takes up only a small percent of an ML Engineer's time, and it is generally performed offline. Currently, Grammarly uses two different feature stores in our operations, one for DynamoDB (suitable for costs) and one for Redis (ideal for speed). Merging these into a single, unified feature store is gaining appeal for the potential efficiencies it could bring. An online feature store SDK is also available, which further adds to this consideration, although our focus will likely be solely on the online part of the process.
In 2015, Grammarly Experiments moved from Mixpanel to a homegrown solution to better address our needs. This homegrown solution soon showed its limitations. It struggled once the scale got excessive and required constant maintenance. Its high degree of coupling also posed issues, with more than 3,000 event streams writing to it and no schema enforcement. Grammarly used a demultiplexer to manage the heterogeneous events, filter each event type, enforce a schema, and resolve any schema incompatibility between the batch and delta tables. More than 40 clients were using this complex system, adding to its management challenges.