The most dangerous sentence is the one that sounds like morality but has no author.
The word should is small, but it has always carried a heavy function.
The word should is small, but it has always carried a heavy function.
It is not a description.
It is not a prediction.
It is not information.
Should is a claim of legitimacy.
Should is a claim of legitimacy.
When someone says, “You should do this,” they are not merely offering advice.
They are quietly placing a norm onto your life.
They are saying:
“This is what counts as right.”
“This is what a reasonable person would do.”
“This is what you will be judged by.”
“This is what counts as right.”
“This is what a reasonable person would do.”
“This is what you will be judged by.”
For most of human history, that sentence had a cost.
Because should was never neutral.
Because should was never neutral.
It demanded an author.
And now we are entering a world where should can appear endlessly—smoothly, confidently—without any human being forced to own it.
And now we are entering a world where should can appear endlessly—smoothly, confidently—without any human being forced to own it.
That is not a language change.
It is a civilizational shift.
1) In the human world, “should” was a battlefield.
Before models, should came from conflict.
Religions fought over it.
Families fought over it.
Courts fought over it.
Philosophers fought over it.
Politics was mostly a war over competing “shoulds.”
Even when a society tried to impose one official “should,” the structure remained visible:
Someone preached it.
Someone enforced it.
Someone punished deviation.
So even when norms were oppressive, they had a shape:
a human will behind them.
That mattered because it kept one thing alive:
You could still ask the oldest human questions:
Who says so?
By what authority?
What do you bear if you are wrong?
Those questions do not always win.
But the ability to ask them is part of what makes a civilization human.
2) A model produces “should” without belief, without responsibility, without skin in the game.
A language model can say should with perfect confidence.
It can give you:
- the “best” choice
- the “right” tone
- the “appropriate” response
- the “ethical” recommendation
- the “reasonable” decision
- the “fair” outcome
And it can do it instantly, endlessly, for millions of people at once.
But here is the key difference:
The model does not mean should the way humans mean it.
It generates should as a statistical continuation of language.
Which means:
The model’s should is not born from responsibility.
It is born from pattern.
It is not a judgment.
It is an output.
And output has no moral cost.
3) The real shift: “should” moves from argument to default.
When should is spoken by a human, it enters a social arena.
Someone can contest it.
Someone can shame it.
Someone can refuse it.
But when should is delivered by systems—embedded in assistants, workflows, dashboards, and policy engines—it stops being an argument and becomes a default setting.
But when should is delivered by systems—embedded in assistants, workflows, dashboards, and policy engines—it stops being an argument and becomes a default setting.
It becomes:
- the recommended path
- the “safe” option
- the compliant choice
- the “best practice”
- the standard response template
And once should becomes default, most people stop treating it as a claim that requires justification.
They treat it as a feature of reality.
The moral battlefield becomes a UI element.
4) Why this is stronger than propaganda: it removes the moment of moral confrontation.
Old propaganda tried to persuade you.
It needed your belief.
Even oppressive moral systems still had a human face somewhere—preachers, teachers, enforcers.
That face could be hated, resisted, or at least recognized.
Model-should removes the face.
It doesn’t come at you as a command.
It comes at you as help.
It doesn’t come at you as a command.
It comes at you as help.
It doesn’t say: “Obey.”
It says: “Here’s what you should do.”
It doesn’t say: “Obey.”
It says: “Here’s what you should do.”
And because it feels like support, you accept it faster than you would accept force.
This is why the new power is not primarily ideological.
It is architectural.
It places norms into the path of least resistance.
5) The most subtle corruption: “should” becomes a disguised form of prediction.
In a model-driven world, should begins to collapse into a different meaning:
what is likely to be approved.
what is likely to be approved.
You start asking for “should” not to discover what is right, but to reduce risk.
- “What should I say so I don’t get rejected?”
- “What should I write so it performs well?”
- “What should I do so the system flags me as safe?”
- “What should I choose so I pass the threshold?”
So the moral grammar quietly mutates.
Should becomes optimization.
Should becomes optimization.
And that is the civilizational trap:
A society can keep using moral language while actually living by thresholds.
It can keep saying should while meaning safe enough to pass.
6) The Empty Chair appears in language first.
People think the Empty Chair belongs to courts and verdicts.
But the chair empties earlier—inside everyday norm production.
When a system tells you what you “should” do, and you obey, you may still feel like a moral agent.
But ask one question:
Who owns this “should”?
If the outcome is harmful, who must bear it?
If the advice ruins a life, who is responsible?
If the “ethical recommendation” becomes a tragedy, whose conscience is required to break?
The more your norms are produced by systems, the more morality becomes ownerless.
The chair empties long before the courtroom.
Because civilization doesn’t lose humanity only when decisions are automated.
It loses humanity when norms are automated—
when legitimacy becomes a generated suggestion with no accountable author.
Closing: the test of a real “should”
A real should is not the one that sounds wise.
A real should is the one that has a bearer.
A real should is the one that has a bearer.
So the question this essay leaves you with is not philosophical. It is operational:
When a system tells you what you “should” do, who is forced to stand behind that sentence?
When a system tells you what you “should” do, who is forced to stand behind that sentence?
If the answer is “no one,” then the sentence may still guide millions of lives—
but it no longer belongs to a human moral world.
It belongs to an efficiency civilization that can speak like ethics
while quietly removing the one thing ethics requires:
an author who must bear the last word.
an author who must bear the last word.