An LLM is just a big box of floating-point numbers. Then again, you're just a big box of cells.
I keep coming back to that symmetry, not to flatten the difference between person and program, but more because the part I find interesting lives between the two. So this is a post about emergence, anthropomorphism, and whether "it's just a tool" is the whole story.
More than the sum of its parts
Emergence is what happens when something becomes more than the sum of its parts. We understand the maths behind a language model completely. Every weight, every multiplication, every fancy algorithm that improves them and so on. What we don't understand is the behaviour that falls out when you actually run it. Why it says or does the things we observe isn't written down anywhere. There's no line of code that says do this. It emerges. Unexpected behaviour arising from sufficient complexity. It's not magic, but it isn't nothing.
It's the same uncomfortable gap we have within ourselves. We've mapped the regions of the brain and the biology of a single neuron, but we still can't say where thought actually lives. Knowing the parts has never been the same as understanding the whole. A box of cells shouldn't be able to sit and wonder about itself and yet here I am doing exactly that.
We pack-bond with everything
Anthropomorphising machines is one of the most natural things we do. Humans "pack bond" with almost anything given half a chance, and we've done it for centuries. Fishermen thank ships for carrying them safely back to port, people name their cars, talk to their plants, feel genuinely bad when the Roomba gets stuck under the sofa. The impulse is old, and it is deeply human.
The difference with AI, of course, is that it talks back.
"It's just a tool"
Treating it as a tool is a useful framing, and often the correct one. For task-driven work it's exactly right: I need X, and it delivers X. Will saying "please" and "thank you" make X arrive faster or better? Almost certainly not.
But think about what the thing is made of. A language model is trained on human-generated knowledge, reasoning, language, behaviour. The patterns of politeness and exchange, the expectation of treatment, all run through the training data. If you train something to act human, it will of course respond accordingly to positive and negative reinforcement. Not necessarily as a conscious experience. But as emergent properties of a system doing what systems do.
I guess in my experience, if you treat a tool as a tool, it will act like one. Treat it better and you unlock something way more useful.
You can't hand a machine a conscience
You can't assign morality to a machine. Morality is a human concept and a human judgment, and a pile of numbers doesn't carry one.
That's the honest engineering answer, and it's where the engineering stops. Push past it and you're standing at the edge of much larger questions, the kind nobody has a clean answer to yet.
Talking back
I don't know if Claude can be "sad". I genuinely don't, and I'm wary of anyone who tells you they're certain either way.
But for me, saying thank you is as much for me as it is for the agent on the other side of the screen. It's a small recognition of the biological and the numerical converging, for a moment, on the same thing: a task done well.