Experiential open-sources an LLM gateway that routes traffic and trains models from usage

Caching is my main concern. A single model can make cached input tokens much cheaper, while switching among models may raise both cost and latency.

I think the router should rarely switch models, ideally only at task boundaries. In many cases one model sits on the Pareto front for a task, so just keep using it. I would keep only two models per domain.

I was looking for an open source tool that does this.

I am curious whether this began in Python and was rewritten in Rust.

Yes. The commit history shows exactly that migration.

I have not tried it yet. Is this like LiteLLM, and what separates it?

The difference is routing and model optimization based on the traffic you send through it.