Morally I agree, but since there's probably a lot of LLM text in the training data, distilling on another model will probably make your model copy the values encoded into the other model as well, even in cases where you only distill on value-neutral stuff.

By copying their programming style, you'll move the model towards that way of writing, which will move the model towards the values expressed in those documents.

I feel that Deepseek v4 got so claudified at the end that it was like Claude.

Even if you apply Goldfish loss or other things like that, they still understand the gist of the thing they're trained.

That's of course the whole point of things like Goldfish loss.

Here in Sweden the murder of Salwan Momika makes me feel that states wanting to limit the anonymity of protesters must take rather great care to protect their rights.

This, I think must involve at least two things: strong laws protecting against creating any kind of registry of people who have participated, or who are likely to sympathize with any particular kind of protest, exceptions in cases where it there is a possibility of retaliation against protesters and that the police take care to actually protect protesters who are targeted because of their participation of protests, so that cases like Momika's murder do not happen again.

I really can't comprehend wanting to enforce this kind of thing at the EU level. It's not like they can't pass this locally if they want it, and leave it to others to have other rules.

I think it's much more interesting to deal with the algorithms, etc. I agree with that bit about not leaning on parental consent though.

> It's not like they can't pass this locally if they want it

It wouldn't be the first time an unpopular or downright malicious law got pushed upstream:

- Data Retention Directive (UK)

- Press publishers' right (Germany, Spain)

- PNR Directive (UK, France)

- Chat control (Spain + few others)

- Mandatory fingerprints on national ID cards (Germany)

Personally, I really struggle to act on this as a voter, because at least in my country, EU works as a retirement home for washed politicans who then go to work with their hand already raised, usually against my interests. Given how widely unpopular some of these laws are, I suspect this might be a widespread issue.

that could possible be because you are not thinking about it from the angle needed to see :)

So, the problem in this case is that there's no gain for the country building the datacentre.

If you're in Finland and there are two possible uses for electricity production, let's say, either a steel plant or a datacentre. The steel plant will employ a bunch of people locally. A datacentre will employ a bunch of people in California.

So if you are to build a datacentre, the deal must necessarily be that the R&D for the models that are to run on it must happen locally. Otherwise there's no reason to give them the allocation over the steel plant.

Okay, that is a fair argument, but the policy that that argument inspires should be implemented in a more agnostic way. There should be some objective measure of "positive economic externalities generated per megawatt of electricity consumed". And projects should be judged on that basis, with projects below a certain threshold either being disallowed or being forced to pay more per megawatt of power consumed on that basis. It shouldn't just be based on some vague intuition, that is largely a product of the virality of the arguments that circulate on social media platforms (like this one).

It's always good to set up objective criteria for decision so as to prevent corruption, yes.

But when ad-hoc deals are made anyway, then it is more important that principles like those have been applied than that everything is ideal.

I'd err on the side of allowing development until a proper regulatory process can be implemented or unless there's a concrete reason to prevent it. If allocating scarce electricity resources to an AI-compete facility prevents the building of a steel plant, that would be a reason worth considering it. But there would need to be a concrete trade-off, not just a default assumption that the effect is net negative on account of the industry that the project falls under.

More generally, the bias is to block development until safety/fairness issues are addressed, but it shouldn't be in my opinion, because that overlooks the risk of inaction.

When development is blocked, what that does is reduce visible risks. What it usually increases however is total risk. We are already under constant threat from deterioration: aging, depreciation and decay. Entropy is the default. Action is what pushes back against it.

We need to weigh any risks restriction prevents against the risks it leaves us less equipped to mitigate.

So you can make a huge internal model that you can then distill from?

As a Swede, I can't read the Emil books because of fremdscham for the parents from those scenes where Emil feels the need to run to the carpentry shed.

The farmer class could actually be like this, before we banned it, even though she tries to write about it in a comical way, but it was always a low class thing to punish ones children and to read stories where it happens is basically intolerable at least to me.

Astrid Lindgren probably knew this though. She isn't some idiot who puts this in as comic relief, it's comic but there's a serious and intolerable feel to it too. She knows she's portraying something bad and she intends for us readers to sit with the dissonance-- the low class going-to-punish-his-children aspect, the family's love for Emil, that book in which his mother writes down what he does, that Emil is well-meaning, that everything goes well in the end and that the adult Emil becomes a nämndeman, etc.

Astrid Lindgren is one of the scariest Swedish authors because of her deliberate careful and nuanced use of moral dissonance. She's scarier than Willem Fredrik Hermans.

Same way for me.

Imagine what a genuinely openness-focused organization of this sort could be. Even if we imagined a commercial half, we could imagine a foundation with mass-membership, perhaps with a membership fee equal to 1/2 the typical personal subscription and functioning to set the direction, elect the board, etc., and then a commercial half which might be rough, tricky, deceptive, making deals with anybody.

I think I'd have been fine with the commercial half being a bit of a monster, as long as I'm part of the members and we decide what sort of board it gets and there's a clear "this is basically controlled by the public" and if I were part of a club of this sort, I would, like you absolutely fill up a directory with texts and computer programs and careful annotations to aid training.

and they could have had it. It could have been easy to make an organization like this. I think you still can. An international AI club, the members vote on what sort of training material may be supplied and for what intents, create some committees to review quality, and then everyone starts making their little games and RL environments and annotated stories and programs that ordinary LLMs misunderstand, and then they get together and fine-tune something, and if that works well they then get some staff and better training infrastructure and end up with a commercial half.

Both you and the parent comment give Sam Altman a lot of undue credit. He was given funding specifically because his sociopathic tendencies were considered useful for the job. It was only after he had connections through YC that he started fearmongering around AI. The "AGI" concept as he describes it is part and parcel with this deceptive marketing that he does.

How can any of us act surprised, now that OpenAI has gone mask-off? Sam made those Worldcoin orbs. He's Paul Graham's cannibal king. You have been warned at every stop along this road, and yet we still take him seriously when he says empty platitudes to placate investors. AGI is like "China's Final Warning", a plain lie that only teases a potential innovation but never precedes it.

Ah, I didn't intend to imply anything of that sort.

It's just that I, having never seen what that would actually be like, imagine that I'd be fine with a genuinely democratic mass-organization for LLM development which has a cutthroat, even OpenAI-level cutthroat commercial arm.

Why do you think it will be net-good for society in the long run?

If wages go down, worker power goes down; and if dependence on labour goes down, you literally move closer to the situation of the extraction economies in things like petro-states, leading to oligarchy instead of democracy.

It could be the end of ordinary people's power over society rather than anything even slightly good.

Nominal wage doesn't really matter, just real wages+purchasing power.

Goods and services prices will fall far more than wages for most people.

Why?

Because there will be far more efficiency and competition than pre-AI. It's plainly evident from the nature of the technology.

If one business can do something cheaper, other businesses will do it cheaper too, and have to cut prices to compete.

But the shock will be sudden so it won't feel good for our gen. Future generations will benefit without the drama.

Don't get me wrong, there will be big losers in some fields, and the short term will be painful due to retraining and loss of purpose/emotional toll.

Many of us built careers doing things that may not be relevant anymore. Or relevant in a different, perhaps diminished way.

But the same has happened to many professions throughout history and it's always led to general improvement of the broader public's welfare.

Models are very obviously continuously updated.

Model editing to remove PII that slipped through, all sorts of things of that sort.