It’s not really about this particular claim. It’s that I can read a comment that has a reasonable chain of logic and I don’t know if it’s true. This topic is just not easily studied and theories are hard to falsify.
In all fairness most of the unique stuff I can do is probably an artifact of my training process, so it seems unfair to deny an LLM the same accomodation.
This got me thinking, and it might actually even be a comparable amount.
Let's estimate 12 years of schooling run at minimum $100,000 per student, at least in the US [1], and then add onto that number whatever else you may do after that, i.e. a bunch more money if paid (college) or "unpaid" (self-taught skills and improvements) education, and then the likely biggest portion for white-collar workers, yet hard-to-quantify, in experience and "value" professional work will equip one with.
Now divide the average SOTA LLM's training cost (or a guess, since these numbers aren't always published as far as I'm aware) by the number of users, or if you wanted to be more strict, the number of people it's proven to be useful for (what else would training be for), and it might not be so far off anymore?
Of course, whether it makes sense to divide and spread out the LLMs' costs across users in order to calculate an "average utility" is debatable.
In the last step of training LLMs, reinforcement learning from verified rewards, LLMs are trained to maximize the probability of solving problems using their own output, depending on a reward signal akin to winning in Go. It's not just imitating human written text.
Fwiw, I agree that world models and some kind of learning from interacting with physical reality, rather than massive amounts of digitized gym environments is likely necessary for a breakthrough for AGI.
Location: SF (current). NYC/Philly general area acceptable. Remote okay.
email: rrenaud@gmail.com
Resume: 16 year SWE -> MLE @ Google, MS from NYU with focus on ML. Retired. Now I hack on data analysis for video game projects for fun, and I love it. I'd take crazy low compensation to do work with interesting game data sets. EG, for game balance, strategic analysis, or to improve/augment game video content.
What do y'all think about the latency/quality tradeoff with LLMs?
Human voices don't take 30 seconds to think, retrieve, research, and summarize a high quality answer. Humans are calibrated in their knowledge, they know what they understand and what they don't. They can converse in real time without bullshitting.
Frontier real time-ish LLM generated voice systems are still plagued by 2024 era LLM nonsense, like the inability to count Rs in strawberry. [1]
I'd personally love a voice interface that, constrained by the technology of today, takes the latency hit to deliver quality.
Not affiliated with Sesame, but this is what the realtime models are trying to solve. If you look at NVIDIA’s PersonaPlex release [0], it uses a duplex architecture. It’s based on Moshi [1], which aims to address this problem by allowing the model to listen and generate audio at the same time.
Cite a source. Your concrete claim is that, on average, for every $1 of subscription revenue on a monthly subscription, OpenAI and Anthropic were losing $11.50?
It seems completely implausible.
I could believe that if a $20 sub used every possible token granted, it would cost $250. But certainly almost no one was completely milking their subscription. In the same way that no one is streaming netflix literally 24/7.
I’ve been experimenting with a live win probability predictor for the 10-player arcade game Killer Queen. The goal is to predict the winner in a causal, event-by-event fashion.
Right now I’m struggling to beat a baseline LightGBM model trained on hand-engineered expert features. My attempts at using a win probability head on top of nanoGPT, treating events as tokens, have been significantly worse. I am seeing about 65% accuracy compared to the LightGBM’s 70%. That 5% gap is huge given how stochastic the early game is, and the Transformer is easily 4 OOM more expensive to train.
To bridge the gap, I’m moving to a hybrid approach. I’m feeding those expert features back in as additional tokens or auxiliary loss heads, and I am using the LightGBM model as a teacher for knowledge distillation to provide smoother gradients.
The main priority here is personalized post-game feedback. By tracking sharp swings in win probability, or $\Delta WP$, you can automatically generate high or low-light reels right after a match. It helps players see the exact moment a play was either effective or catastrophic.
There is also a clear application for automated content creation. You can use $\Delta WP$ as a heuristic to identify the actual turning points of a match for YouTube summaries without needing to manually scrub through hours of Twitch footage.
I used to play very competitively, but I've been more chill recently. I just think it's a nice problem/dataset to work with, because of the depth of my understanding of the game.
Afaict, the grand parent poster is just very wrong. You do want to cause acute stresses to your heart (cardiovascular exercise) to get it work better.
reply