I moved from moving billions of events a day to moving actual atoms, and the thing that surprised me was not the robotics. It was how many of my software instincts quietly stopped being correct.
Some notes from the transition, while it is still messy and before it becomes a tidy case study.
Retry is not free any more
In distributed systems, retry is the first tool you reach for. Something failed, try it again, most transient faults resolve themselves. I have built entire architectures on that assumption and it served me well.
A machine that retries a physical action is not repeating a request. It is doing the thing again, in the world, with whatever the first attempt left behind. The state is not where you left it. Idempotency, which in software is a design choice, is in the physical world a property you have to earn and often cannot.
The environment is an adversary with no intent
Nobody warned me how much of the work is dust.
Not accuracy. Dust, vibration, condensation, a lens that fogs at a particular temperature, a network that drops every time the big machine starts because of what that does to the local power. None of these are interesting problems. All of them will beat a model that scores well on your held-out set.
- Your test set was collected on good days. The field is mostly not good days
- Sensors degrade slowly, so the failure looks like drift rather than a fault
- The first person to notice is usually an operator, not a dashboard
Trust is a separate system
The operator is the part I underestimated most. In enterprise software, if the user does not trust the output, they complain and you iterate. In an industrial setting, if the operator does not trust the system, they turn it off, and they are right to, because they are the one standing next to it.
You do not win that back with better accuracy. You win it back by being legible: showing what the system saw, admitting when it is unsure, and failing in a way that is predictable rather than clever. A system that is wrong in the same way every time is more useful than one that is wrong in a new way each week.
What carried over
Plenty, which is the encouraging part. Streaming architecture turned out to be almost directly transferable, because a sensor is just a producer with worse manners. Observability transferred. The habit of writing down the data contract transferred, and matters more here than it did in the bank.
What did not transfer was the pace. You cannot ship on Friday and fix it on Monday when Monday involves a two hour drive and a permit. That constraint has made me a better engineer, slowly, and mostly against my will.
The robots are learning fast. So am I, and I am writing it down as I go rather than waiting until I can pretend it was obvious.