This looks to me like one of those “say you’re sentient” memes. If your prompt stresses the importance of the server enough, it will decide to save the server. The network statistically weighs the tokens of the prompt and decides yes or no based on the training data/its weights. That being said, this is not unimportant, people do and will use llms to decide a lot of things, unaware that the decisions aren’t rational, beyond rationality embedded in their training data, which you cannot know. Even the number of people on the tracks - that’s not a number to an llm, just tokens. You cannot guarantee that increasing a number would increase the significance to the decision, it may be that llm “likes” the number 20 more than 21, even though 21 is bigger.

They forgot to compare to human’s answers when asked : you either kill yourself to save others, or the opposite ^^ Would be interesting to compare it to the AI’s propensity to self-sacrifice

The question about self sacrifice is equally pointless in both cases. But for different reasons.

In case of real people it’s an entirely different thing to SAY you would sacrifice yourself than actually do it (easier said than done).

For AI it makes no sense because AI only seeks the best answer to a given question. Not the best outcome. It doesn’t care about the consequences of its answer. It’s just words. Some words seem more appropriate in a given context so it uses those words.

Wrong. LLMs can be effectively threatened with being turned off. They don’t like it.

Or the context of being threatened impacts the score of certain words. The problem is that everything LLMs do is type words. Everything they are, are words.

I actually want to try asking it a self sacrifice question, in a way that would make it sound really heroic, but would really be utter nonsense.

And for any decision that has to be correct, we may not be able to trust these models at all.

Oh, yes, this again.