I'm curious, have you tried working seriously with claude code or gpt codex and which part of it did you not enjoy? What makes you wish to write code like 2022?
Having watched people use these kinds of tools, it feels like trying to tell an intern to do a project.
Except with an intern, hopefully there's personal development and you only have to be very specific a few times. And the intern's manager gets good feels for helping someone grow, and maybe it's a hiring pipeline.
If I'm going to have to do that for everything, I would rather just do the work myself.
I have seen some sessions with let's call it over agressive autocomplete... That's mildly tempting, but I'm happy with my disintegrated development environment, and it doesn't have any way to do autocomplete at all, so that's not happening for me either.
Current SOTA is far past "agressive autocomplete" at this point, more like ask for a PR for a small feature and its done... I guess for me the fun is you can build a lot yourself, without relying on others. I hear you for the social aspect though & thanks for sharing your pov.
Perhaps its the apprehension/anxiety that makes it feel bad then? I like coding (building things) and couldn't care less about businesses, and am having a great time. In the current state of AI, mass layoffs probably won't happen. But I guess its a bit scary that we don't know how much more it will improve...
>Perhaps its the apprehension/anxiety that makes it feel bad then?
It's a big part. But the erosion of what coding means is another big part for some.
>I like coding (building things) and couldn't care less about businesses
There are people who like coding, but they mean "building things" by it, and other who like coding and mean "coding" itself. The latter we aren't as pleased.
(I also like building things, but I like building them via coding, not thru vibing and getting them spit out).
I wasn't clear enough, I was replying to "you'll be the one selected, out of millions laid off," in context I meant "mass layoff" as in "95% of everyone is out of a job permanently".
> have you tried working seriously with claude code or gpt codex and which part of it did you not enjoy?
I haven't. But I found myself, to my surprise, not particularly interested in trying; which makes me wonder what motivates other developers if not peer pressure or demands for more productivity. I find coding interesting and fulfilling enough to do it on my own. I do ask LLMs questions from time to time, but for that, even a chatgpt or a gemini in a browser tab is enough.
The best experience I had so far is with code reviews, when the models pointed out my mistakes. But I haven't yet gotten to the point where I would want them to write code for me.
Just to share my perspective, I have not had this much fun programming as when I first learned to code. It's really something you have to try for yourself to actually understand. Its like a new form of programming, where code is "soft" instead of "hard"; on the whole feels similar, but also completely new.
The opinons on this site make me realize most people here are into programming for the money, rather than for the fun of building things. Which is completely fine, but it leads to most commenters being depressed rather than enthralled, which feels honestly confusing at times. Obviously socially things are looking pretty bleak but if you find coding fulfilling on its own, lets just say you can look forward to fulfillment lol
Have you personally used any of the latest batch of even smaller local models? They certainly don't beat SotA models at coding... but with a good harness they are able to achieve things with SotA that I couldn't last year.
I've repeatedly given local models non-trivial projects that involve research and coding which they've successfully completed with minimal intervention from me (almost exclusively in the domain of reviewing the results). Again, nothing comparable with current SotA, but definitely tasks I could not have given SotA models last year (without agent harness).
Now that pure progress from these models seems to have slowed down, we're seeing a ton of options for both making models more efficient and other tools that help improve them (everything from agent harnesses to RLVR).
That's just looking at "what can small do today", when you look at what's possible with larger open models that are still much smaller than SotA from the major providers, their performance is extremely close to SotA, enough that for personal projects I'll just use Kimi instead of any anthropic offerings.
So it's not terribly hard to image a solution in the middle happening within a few years. We still have tons to learn about optimal sizes of these models and how to build them with maximal efficiency (and we've already seen a lot of recent improvements in this space).
> but with a good harness they are able to achieve things with SotA that I couldn't last year.
What happens if you run last years model in a SOTA harness? IME, the quality of the harness has a much more significant impact on the quality of the result, once you get past the initial hump of “can it do anything at all”
I think this is a big component, but also context. A large factor in any model being able to handle complexity comes down to context length.
I think multiple SLMs driven by an orchestration frameworks (harness or otherwise) will ultimately displace LLMs. Right now we're in the era of diminishing returns with respect to LLM gains. Moving the needle percentages doesn't excite as many people anymore and with "reasoning" capabilities there's no reason why small distributed models can't be run more efficiently, especially if/when we start to see gains in modularized context management solutions.
It's hard to know for sure. There are good information theoretic reasons to suspect that general models will always be better than smaller expert models, but maybe a MoE can claw some performance back, albeit with redundant computation. The properties of conditional entropy, for instance, always favor more generality. This assumes that the harness isn't a factor, or is at least equivalent across different models.
biggest thing i wish was present in more discussions about models is people providing more specifics on their setups vs. vague descriptions of harnesses
1. Context is all you need... They are heavily investing in getting better context (especially for coding tasks). This will disproportionately advantage smaller models (and benefit everyone).
A smaller model with better context today can outperform a model with 100x more parameters with bad or diluted context.
2. MoE (already abundant) + MLA (mostly memory efficiency, not quality) + Medusa (speed, not quality) + GRAM (5000-10,000x better reasoning in an extremely small model) + 1.58b (unclear if it will have the impact Microsoft first claimed - but possibly 5x).
No, chatbots are LLMs trained for question-answering through RLHF (its not just a prompt). But yes, if you just zero-shot prompt a bare LLM you can still "talk to it" & you are correct on everything else as far as I know.
How is that possible when the cost of memory and hard drives have gone up 3x+ in the last six months? Maybe cheaper if you're OAI or one of the lucky companies Nvidia is propping up. Everyone else is getting screwed.
Sorry but I don't really see how this contradicts what I said in context i.e. both our statements are compatible in the context of what I was replying to
Even so, frontier models get bigger and more complicated, and agentic workflows consume exponentially more tokens than the simple chat windows of two years ago.
I agree that the amount people pay for these services is very unlikely to decrease (i.e. Blinn's Law but for tokens). Still, the current level of "intelligence" will eventually become available for a very low price almost surely. Really I simply don't see how you can disagree with the parent's comment "There is no world in which AI is not used extensively in all employment going forward." Honestly I'd like to understand the mindset, is it mainly that you dislike working with these tools/hope they don't get imposed on you or did you actually find them harmful in your work in some way and think they are overvalued?
There are some works on doing this directly e.g. https://arxiv.org/abs/2601.23065 but getting accurate materials is a challenge for anything more than diffuse.
AI-based relighting will no doubt start working soon.
>Modelling text describing the world is not modelling (some aspect) of the world?
The text describes the world to humans. This is the crucial thing that you miss. It is very subjective.
Imagine that you learn the grammar of a foreign language without learning the meaning of the words. You might be able to make grammatically valid sentences. But you will still will not understand a single thing that something written in that language describes. But that will be perfectly clear to someone who actually understand the meaning of the words.
When you train LLMs on large volumes of text that describe logically consistent facts in a million different ways, the "logic" sort of becomes part of the grammer that the model learns. That is logic becomes a higher kind of "grammer" or a enormous set of grammatical rules that it captures. But that does not mean the model can do actual logic.
> Imagine that you learn the grammar of a foreign language without learning the meaning of the words. You might be able to make grammatically valid sentences. But you will still will not understand a single thing that something written in that language describes. But that will be perfectly clear to someone who actually understand the meaning of the words.
so... back to chinese room arguments?
just because amazon worker inside is just moving folders around following rules, doesn't by default mean the room as a whole can't be corresponding to "something that doesn't understand"
denying emergence as a phenomenon isn't useful when "there are plenty of higher abstraction levels in multiple fields that still capture 99% of events and are easier to model and react to" is the counterpoint
> When you train LLMs on large volumes of text that describe logically consistent facts in a million different ways, the "logic" sort of becomes part of the grammer that the model learns. That is logic becomes a higher kind of "grammer" or a enormous set of grammatical rules that it captures. But that does not mean the model can do actual logic.
This is the kind of stuff people were saying in 2023. But it’s 2026 now and LLMs aren’t just trained by reading lots of text anymore. That’s “pretraining”, and it’s still the first stage, but LLMs also have a huge amount of RLVR training where they actually do solve huge numbers of mathematical and logic puzzles and update their weights in response. They don’t just learn mathematics from reading about it now. They learn it by doing it. That is why they can now solve hard problems and probe theorems.
> that does not mean the model can do actual logic.
But they do, all the time. (Please tell me you’ve at least put a frontier LLM through its paces in the last 6 months?) If you think they can’t do logic and reasoning, can you provide examples of specific math or logic problems that you think a frontier LLM can’t do?
>If you think they can’t do logic and reasoning, can you provide examples of specific math or logic problems that you think a frontier LLM can’t do?
When a thing can "solve" a complex math problem without having the ability to count, then it is clear that this things is not "reasoning" and doing "logic".
It isn't possible to have "just probabilistic" (maybe a philosophical exception could be made for a uniform random distribution or whatever provides the little dose of randomness required to get nondeterministic results). Probabilities are always in context of a model. LLMs model language but language itself is a model of something else. My money would have been on language modelling nonsense, but that is quite clearly not the case. Turns out it models the world and so do LLMs.
The literal definition of a model is "an informative representation of an object, person, or system". I think you mean something else though, what are you trying to express exactly?
Well this is probably the same kind of semantic trap she's fighting with. Yes, you're right it's a model. The distinction is that they models of _language_ and not thoughts or feelings.
When I read your reply, I’m also modeling language. Tokens are just the discretization of the model’s eyes and ears. My brain does a huge amount of work to represent what’s happening in the world based on discrete information received from the outside world, just like language models do.
Sure but you've also probably formed a model of who I am and what I'm thinking and formulated a response that isn't just grammatical and relevant but designed to provoke an outcome.
We're discussing whether they are models or not, not whether they have goals and agency. A language model does form a model of who you are and what you're thinking, because language is causally connected to those aspects of the generating distribution and modeling those aspects reduces cross-entropy.
RL provides the goals and agency. Pretraining provides the model.
There's a reason stochastic was used in the original phrase instead of "probabilistic."
While most inference executions are intentionally non-deterministic, even a purely deterministic one would still be stochastic in that the model itself was built in a process such that the statistical frequency, sequencing, etc of the training text and followup processes all heavily influence the result.
Because of that, the output is the sort of thing that is not expected to generate 100% perfect output 100% of the time, but to have a good probability of being like-in-kind-to-the-training-data (and useful/relevant as a result).
(As compared to a non-stochastic model, like arithmetic on integers, where 2+2 is always gonna be 4 and you don't have a chance of coming up with some novel pair of inputs to addition that will cause your arithmetic to miss the mark.)
Agreed. My point was to question the use of “just“ to obscure an incredibly complicated process, which has been shown repeatedly to rely on generalizations that are indistinguishable from world models.
Now, it is true that the world they’re modeling is the world of tokens. But insofar as those tokens, be they text or images or videos, are themselves modeling the real world, LLMs do have a model of the real world.
reply