Grammarly Work Note 2023-05-31
Warning
Gate vs. Publish
'Publish' is the traditional way to release. You roll a new application or feature out to the entire user base, often through an app store release. That typically takes a considerable amount of time, often due to app store processes or user uptake. 'Gate' works differently. All of the code necessary for the new feature ships all at once, but the back-end controls the interfaces and enables them gradually throughout the entire demographic. Getting from the experimental phase to the final roll-out is more efficient this way.
Data Infrastructure Team: Focus on Data Ingestion
Data ingestion, importing data from existing sources, is a critical part of the Data Infrastructure team's role. The team's CDC project turns existing SQL data into a format data scientists can use. GNAR, the previous method for ingesting analytical data, is 60% deprecated and is being replaced by the end of Summer 2023. The new entity in charge of data ingestion is called 'Sungari.' The data ingestion process follows this flow:
- Data Engineering (Extraction)
- Data Processing
- Featurization
- Analytics (Feature Store → Serve data from Databricks)
Linguistic Data Check and Scaling
Next to the central data ingestion pipeline, there is a separate system for Linguistic Data Checks. This system may perform quality control or contextual interpretation for linguistic data. The infrastructure deals with significant data logistics, scaling, and cost operations, and it tries to bridge real-time systems and batch systems. Currently, processes are in a 'Code Red' situation, a challenging period requiring compromises and rapid changes.