literally every single competitor will do it because they don't control for them at all. Most of them don't even control for traffic lights or stop signs. Also https://www.youtube.com/watch?v=9lc-HUZUiQg
Interesting because I already felt like current models spit out too much garbage verbose code that a human would write in a far more terse, beautiful and grokable way
I had a case yesterday where Claude wrote me a series of if/elses in python. I asked it if it could use some newer constructs instead, and it told me that I was on a new enough python version that I could use match/case. Great!
And then it proceeded to rewrite the block with a dict lookup plus if-elses, instead of using match/case. I had to nag it to actually rewrite the code the way it said it would!
My early takeaway is that Gemma 26B-A4B is the best tuned out of the bunch, but being small and with few active params, it's severely constrained by context (large inputs and tasks with large required outputs tank Gemma 26B's performance). We're working on a clean visualization for this; the data is there.
It's not uncommon for a sub-release of a model to show improvements across the board on its model card, but actually have mixed real performance compared to its predecessor (sometimes even being worse on average).
In early tests the performance of gemma-4-31B was affected by tokenizer bugs in many of the existing backends, like llama.cpp, which were later corrected by their maintainers.
Moreover, tool invocation had problems that were later corrected by Google in an updated chat template.
So any early benchmarks that have shown the dense model as inferior to the MoE model are likely to be flawed and they must be repeated after updating both the inference backend and the model.
All benchmarks that I have seen after the bugs were fixed have shown the dense model as clearly superior in quality, even if much slower.
We add samples every week, so I'm curious if the numbers will move.
They did a similar re-release during the Gemini 3.1 Pro Preview rollout, and released a custom-tools version with its own slug, which performs MUCH better on custom harnesses (mostly because the original release could not figure out tool call formatting at all).
Not really. A portion of users will randomly tap that just to get rid of the question. They don’t read.
The easiest way to experience that yourself is to set your device to a language you barely understand. You’ll find yourself dismissing dialogs just like all those illiterate normies.
reply