They want it, sure. Customers want everything if it's free, but this is about what they value with their money. In this thought experiment, you're Anthropic, not the customer. You're making a choice that's best for Anthropic. Will Anthropic lose customers because the latency is higher? No way. Customers want low cost and lots of usage more than they want low latency. In a cutthroat race to the bottom, there's no room to "give away" massively expensive freebies like a data center near every population center when the customer doesn't value those extras with actual money. It's the same reason we all tolerate the relatively slow batched token generation rate--the batching dramatically lowers the cost, and we need low cost inference more than we want fast generation. If the cost goes up we'll actually leave, for real.
After the initial announcement of "fast mode" in Claude Code, did you ever hear about anyone using it for real? I didn't. Vanishingly few people are willing to pay extra for faster inference.
Remember that the time-to-first-token is dominated by the time to process the prompt. It's orders of magnitude more latency than the network route is adding. An extra 200 milliseconds of network delay on a 5-10 second time-to-first-token is not even noticeable; it's within the normal TTFT jitter. It would be foolish to spend billions of dollars to drop data centers around the world to reduce the 200 milliseconds when it's not going to reduce the 5-10 seconds. Skip the exotic locales and put your data centers in Cheap Power Tax Haven County, USA. Perhaps run the numbers and see if Free Cooling City, Sweden is cheaper.
They’re unwilling to pay for fast mode because of the current step function price increase once you hit your quota. It’s a psychological effect. Because most shops I know in the US currently paying $125/mo per seat for Claude would happily - HAPPILY - pay 2x, and begrudgingly pay 10x that amount for the same service. If fast mode was priced 25% or 50% more they’d happily pay for that too. But it’s just not priced that way currently with weird growth subsidization & psychology.
The only AI use case that cares about latency is interactive voice agents, where you ideally want <200ms response time, and 100ms of network latency kills that. For coding and batch job agents anything under 1s isn't going to matter to the user.
tbh, that's a good point about the voice agents that I hadn't considered. I guess there are some latency-sensitive inference workloads. Thanks for pointing that out.
A customer service chatbot can require more than one LLM call per response to the point that latency anywhere in the system starts to show up as a degraded end-user experience.
Easy solution - use hyperscalers with super expensive API charge only when latency really matters. Otherwise build your own DC. Easy to expect customers don't care latency that much over money.
Black macbooks are anodized aluminum which are thin coatings that would be removed when filing. It might look cool but it’d be the silvery color of raw aluminum
This looks really interesting. I'm curious to learn more about security around this project. There's a small section, but I wonder if there's more to be aware of like prompt injection
I'm happy you brought this up. I've been thinking about this and working on a plan to make it as solid as possible. For now, the best way would be to run each agent in a docker container (there is an example Dockerfile in the repo) so any destructive actions will be contained to the container.
However, this does not help if a person gives access to something like Google Calendar and a prompt tells the LLM to be destructive against that account.
You can make it free but still require a person to travel to the county seat or some other distant location to get the ID. That requirement disproportionately hinders minority and poor voters. It’s also easy to “forget” their registrations.
Yeah I remember the discourse around that acquisition as being a really smart play to shore up the new frontier in social media as Facebook grew stale and uncool.
Tons of critical comments on HN at the time, for one: https://news.ycombinator.com/item?id=3817840. And most of the positive ones viewed it as a defensive measure rather than another Google + YouTube story.
I strongly disagree. I think he might be a joke as an individual, and I hate a lot about his impact on the world, but as a business leader, he's probably at the top 1% of all CEOs, which isn't saying that much, but it's very much not a joke if your metric is shareholder value.
I mean I also think this move doesn’t make sense, but I always find these type of comments interesting. Do people think they could do better in Mark’s shoes?
Cost is not the first thing I care about in war, but I felt like this is a useful site for tracking the money we're lighting on fire in order to pursue this conflict
Civilian costs are real, unjustified, and incalculable.
That’s good. But it seems that the American anti-war discourse is slanted towards the cost of it. Maybe because the whole political spectrum can relate to “our tax dollars”, while (1) the cost for the military personell might not be a concern for all because it is all-volunteer, and (2) some Americans don’t care what happens to people in other countries.
Certainly: American progressives can use this to counter the “fiscally conservatives” (for domestic spending) who are also hawkish.
Remember: The opinions of people that either didn't vote or voted for Trump are all that really matter this November (unless the Democrats somehow lose voters, but the polls suggest that is unlikely).
Those are the votes that need to be won over to make any sort of difference during the second half of the Trump administration.
This makes it make way more sense. That is a huge amount of growth really fast. I've worked in those companies, it's really hard on the work culture and organization when things grow that quickly.
I think the potential for productivity is there with AI, but this size of a cut based on speculation made no sense. This is actually reasonable in this light and is probably for the best. I'll be curious to see if any employees, former or otherwise talk about it
It is not sarcasm he is fleshing out this sentence earlier in the paragraph, "One of the pervasive new doctrines of Silicon Valley is that we’re in the early stages of a bifurcation event"
Right, but in the context of this article about these wretched enfents terribles, and later when we get to the rationalist termite colony, it's clearly something to chuckle at. Like, the fact that people think this "bifurcation event" idea is real is legitimately funny.
I see your point, but I don't think he's being sarcastic in this paragraph. To me this paragraph isn't sarcasm rather he's presently a serious factual recounting of the logic driving AI evangelists that he then undermines by contrasting it with the callousness, messiness, and illogic of the people pushing this narrative. (I too had a good chuckle at the termite description)
But this is veering into lit crit territory, so agree to disagree
reply