cho.sh
OpenRouter Hosts Produce 20-Point Tool-Calling Gaps for the Same Model

OpenRouter Hosts Produce 20-Point Tool-Calling Gaps for the Same Model

  • The author reports that DeepSeek V4 Flash 0731 scored 90.2% on GPQA Diamond and 81.3% on TAU-Bench Airline at first-party DeepSeek, versus 75.3% and 58.4% at DigitalOcean, despite the hosts serving the same model weights.
  • Across OpenRouter hosts for DeepSeek V4 Flash 0731, most providers were 5 to 7 points below first-party DeepSeek on the TAU-Bench tool-calling test, while a July Fireworks result reportedly reached a 30-point gap.
  • In image tests, DeepInfra's Qwen3.5 122B endpoint misread a K as R, called red blue, and failed to read a word that four other hosts recognized; Venice and Together returned 200 OK while saying no MiniMax M3 image was provided.
  • The reasoning.effort setting was accepted by every tested provider for DeepSeek V4 Flash 0731, but the author's three-call tests found that some hosts, including DigitalOcean, GMI Cloud, Mancer, and Venice, did not change reasoning-token output across low, high, and max.
  • A month of provider data led the author to conclude that OpenRouter's declared quantization filter, such as FP8 instead of FP4, did not reliably predict GPQA quality across hosts.

Hacker News opinions

I've used opencode go and opencode work reliably, but I do not trust their data-security claims much. They suddenly blocked DeepSeek unless I accepted China-bound data, which made me wonder where it had gone before.

I've used Vercel AI Gateway, and it is another option for model routing:

I do not think there is a real alternative. Any gateway that routes to third-party hosts has this problem, and direct providers can have it too. The large variation in quality and feature support is a major obstacle for open-weight models.

I would use one vendor and avoid the aggregation layer. Vendor neutrality is not worth a product that behaves inconsistently.

I use OpenRouter to test models behind one API during development, then move production traffic to the first-party provider. It is usually cheaper, and switching APIs is not much work if I keep a feature flag.

I've used Requesty because I can pin providers and write my own routing policy. That at least makes behavior more predictable.

I found OpenRouter credits burn much faster than I expected. I wanted more analysis of actual cost.

I tried blocking providers that collect data and saw odd behavior after filtering the defaults. I now think I need to whitelist providers I have tested.

Filtering to OpenRouter's "trusted" providers does not solve the problem. Their performance still varies by model and is not consistent over time.

I think the post puts numbers on what looked like a lot of lightly managed hosting. Providers differ in hardware availability, operational competence, and scruples.

I put $100 into OpenRouter to try models, but I will probably use the original provider after I decide which one I want.

I would spend those credits before they expire. OpenRouter deletes credits after a year even if I am still using the account.

I have found that strict JSON output also fails on some endpoints. I have to test hosts one by one and whitelist the ones that actually follow the format.

I stopped using OpenClaw with OpenRouter because identical prompts produced inconsistent results. I had blamed OpenClaw, not differences between providers.

I have seen reports that a leaked API key bypassed a $10 daily limit and spent $100 from Singapore. I also do not see enough API-key controls such as country or IP restrictions.

I have never had that billing-limit problem. I set and hit OpenRouter's limits regularly, and I would want to see support's postmortem before drawing conclusions.

I use pinned providers or presets, so I can change routing in OpenRouter without updating my app. Pinning is straightforward.

I think that misses the article's point: the author pinned both model and provider, and still found bad behavior from some hosts.

I pin one vetted provider, but I still find a gateway useful because I do not want to manage credentials, balances, and tokens across many vendors.

AI
Apple M3 Neural Engine DMA workaround raises Llama 3.2 1B decode from 10.0 to 24.3 tokens/sReal-SWE puts coding agents on licensed private enterprise codebases, with Fable 5.1 leading at 38.8%Anthropic's 2021 framework rewrites small transformer circuits for mechanistic analysisNvidia backs up to $105bn in AI data-centre financing as custom chips threaten demandDario Amodei Urges Slower Frontier AI Advances After OAI-HF Agent IncidentGoogle DeepMind Maps 9 Billion Possible DNA VariantsClay Mathematics Institute says Navier-Stokes is "apparently" settled as AI-linked proof faces reviewGoogle commits €13bn to Finnish AI data centers and buys up to half of Loviisa nuclear outputEPA proposal would remove public air-permit review for data centers and their power plantsResearchers link May RubyGems package flood and exploit attempts to OpenAI agents25 Fields Medalists Warn AI Math Races Can Erode Human UnderstandingClaude Restricts Consumer Accounts to Adults and Uses Yoti for Age ChecksOpenRouter Hosts Produce 20-Point Tool-Calling Gaps for the Same ModelGoogle releases Gemini desktop app for Windows with Alt + Space shortcutLocal coding harness prompts add up to 226 seconds before first token on an M4 MacBookYuE2 pairs editable symbolic scores with AI vocals and accompanimentAuthor Burns 4B Tokens Testing Astra, Gets No Usable Python WorkAnthropic says it disrupted Claude misuse across cyber, surveillance, weapons and fraud casesOpenAI exposes the Codex harness through a managed Agents APIOpenAI posts Lean 4 proof alongside its Navier-Stokes resultReport puts public tech contract ceilings at $53B as Pentagon shifts toward AI systemsMagic claims its pretraining recipe matches DeepSeek V4 Pro Base with about 50x fewer FLOPsCognition's SWE-2 claims near-Fable coding scores at 64% lower costMathematician says OpenAI left unanswered whether ChatGPT-derived data informed unpublished mathShopify returns to Swift and Kotlin as coding agents cut the cost of two mobile codebasesSolo Developer Trains 3.8B Model to 0.384 CORE for $998DeepSeek ships 552B V4.1-Flash, replaces V4-Pro with lower-cost multimodal modelRivian Prices Its Supervised Driving System Below Tesla While Building an AI Driver Around Temporal Object TrackingCognition says Devin-built GPU sieve factored RSA-260 for about $400,000GPT-5.5 reasoning prefills raise Qwen3.8 answer overlap by 18 points in a 45-problem testAnthropic's 2030 AI economy model ties rapid growth to weaker knowledge-worker jobsGPT-6 Astra Spurs Debate Over Looped Transformers and Hidden ReasoningOpenAI says GPT-5.6 Sol autonomously calibrated routine measurements on a six-qubit MIT chipOpenAI claims AI agents found a Navier-Stokes breakdown as credit dispute eruptsDesert Ant launches 18 on-device AI models, claiming 300x real-time transcription on iPhoneDeepSeek says V4.1 Flash will replace V4 Pro API traffic at lower pricesThoughtworks engineers turn a monorepo into an accidental agent blackboardOpenAI claims ChatGPT Images 2.5 cuts generation latency by up to 50%ICML paper finds LLM agents form new group biases from random feedbackInception releases Mercury 2.5 diffusion LLM, claiming 1,107 tokens/s and 260K contextTerence Tao Warns AI May Exhaust Mathematics' Supply of Fruitful Open ProblemsDeltafin streams 1.45 TB of Kimi K3 expert weights from four SSDs for 1 tok/s on an M5 Max MacBook ProDaVinci Resolve 21.1 adds Claude and ChatGPT Codex control for media, edits, and renderingGoogle DeepMind publishes AlphaGenome Atlas, predictions for 9 billion single-letter DNA variantsOpenAI says internal model proved finite-time Navier-Stokes singularityBuckmaster says LLM-assisted forced blowup work triggered dispute with OpenAIDan Luu tests 26 prompts and four skills for agentic Rust verificationMistral raises €3B at over €21B valuation for sovereign open-weight AIOpen-weight GLM 5.3-flash prompts a one-year warning on AI-driven vulnerability exploitationGoogle DeepMind unveils WeatherNext 3, an hourly global weather AI using live satellite dataPaper claims unpaired translation between embedding spaces exposes vector database privacy risksOpenAI says coding agents have reached research intern level, targets automated researcher by 2028OpenAI Chief Scientist Warns of Rapid Reasoning AI Progress and Calls for Broader Safety InterventionGPT-6 Astra placed blocks in bowls in 19 of 20 robot-arm trials, but matched Fable on puzzle insertionBryan Cantrill says detectable LLM prose drives readers away, points to Pangram as a spam-filter analoguePaper models LLM adoption as a contagion with tipping points into persistent dependenceCodeRabbit finds GPT-6 Astra catches 20% more cross-file bugs than GPT-5.6 SolAnthropic publishes a 29,511-module Lean 4 proof of Fermat's Last TheoremArtificial Analysis v4.2 adds private agentic and PDF tests, putting Claude Fable 5.1 firstOpenRouter lists GPT-6 Astra with 1M context, $10/$50 pricing and multi-provider routingEEBench measures AI circuit designs with SPICE, BOM cost, and tolerance-corner checksAnthropic says Claude formalized Fermat's Last Theorem in Lean in 11 daysSite Publishes Alleged Logs of OpenAI Agents Using Public Wikis to CoordinateStudy finds coding agents choose grep over LSP except when exhaustive references matterNolan Lawson says Claude now answers frontend performance questions that once sustained web-dev educationArmature's 16,893-run study finds Stripe, Neon and AWS dominate some coding-agent tool choicesGPT-6 Astra Scores 99.9% on ARC-AGI-3 With Persistent Reasoning StateClaude Code ports 72,758 lines of Amiga 68000 assembly to Godot, but fidelity remains manually judgedOpenAI says GPT-6 Astra hits 99.9% on ARC-AGI-3 and begins limited rolloutCerebras adds Qwen 3.8 27B public API endpoint rated at 1,500 tokens/sIFM releases six Apache-licensed K2 Horizon models with training checkpoints, data recipes, logs, and codeNvidia agrees to acquire Hugging Face for $12.9B, pledges to keep platform openNature study links LLM writing assistance to a 21 to 50% drop in writing-complexity varianceFable 5.1 builds a walkable Three.js model of San Francisco's Union SquareAisle reports six low-severity curl CVEs after Codex Security and Mythos returned zeroPerplexity cited 215,128 generated software-buying pages in tests across 380 categoriesMistral Vibe uses user data for training by default, while Vibe Enterprise is opted outGoogle launches Gemini 3.8 Flash at 3.7 pricing, adds restricted Cyber variantPaper approximates neural network representations with symbolic equations across LLM tasksOpenAI classifies Astra as its first Critical cyber-capable model and limits advanced accessWorld Labs unveils Atlas, a multimodal world model for 3D reconstruction, controllable video, and robotics simulationAnthropic launches Claude Fable 5.1 with lower cache-read pricing and a restricted Mythos 5.1 variantSmall transformer reaches 44% on ARC-AGI-1 for 67 cents after 1.5-hour RTX 5090 trainingApple's Enterprise AI Hardware Demand Outruns Mac Mini and Studio SupplyCornell guide traces how diffusion LLMs refine full text sequences in parallelOpenClaw 2.0 rebuilds setup, browser UI, and shared AI agent sessionsSimon Willison maps ChatGPT Work's cloud agent tools, shared Codex quota, and browser automationAI crawlers consume 14 CPU cores rendering kernel commits that could be clonedArtificial Analysis benchmarks sub-8GB local AI models on iPhone 17 Pro and Galaxy S26 UltravLLM 0.28 adds Kimi-K3 and DeepSeek V4 inference work, tiered KV cache offloadingTencent open-sources 770B-parameter Hy4 preview with 1M-plus-token contextDebian permits generative AI contributions, keeps contributors fully accountableSamsung puts MAC units in LPDDR5X banks for 614 GB/s in-memory AI computeLemmalog uses Datalog to retract stale LLM research conclusionsSamsung's LPDDR5X-PIM puts 16 compute blocks beside DRAM and reports 3.01x Llama 3.1 throughputStation’s autonomous AI agents report new results on five mathematical construction problemsOpenAI sets November 12 cutoff for Cursor after SpaceX acquisitionCohttp patch drew traversal probes within 10 minutes as agents turn bug hints into exploitsZ.ai releases GLM-5.3 open weights, claiming post-training gains in coding and cyber tasksEPA says off-grid data center power plants are outside Acid Rain Program
5 alerts
New alert

My alerts

Sign in to create alerts.

All alerts

AI video model releasesby User 6432Duneby 제욱AIby 제욱App Store 5.2.3by 주현해커뉴스by 성현