Sungari

Sungari is a Schema Managed Data Ingestion at Grammarly. The old system at GNAR had several significant issues. Low scalability, poor stability, a mix of events in every Kinesis shard, and JSON serialization/deserialization eating high amounts of CPU time and causing inconsistencies. Sungari addresses these like this.

1. Scalability and Stability. Switching from Kinesis to Apache Kafka, and decoupling ingestion from indexing, will improve scalability and stability.
2. Mixed Events and CPU Consumption. Avro instead of JSON, plus event-level schema registration. The new pipeline will consume less CPU time and avoid data inconsistencies.
3. Data Accessibility. Data lands in rawEventTables at DeltaLake, so non-GNAR query engines can access it, and Upsert operations work. Analytic data will also be stored in parquet format in partitioned storage, eliminating the need for GNAR indexing jobs and further increasing data accessibility.
4. Consolidation and Simplification. The new pipeline also plans to migrate all types of ingestion pipelines to the schema-based one, reducing the number of schemas to under 2000 during migration.