The cost of a fast answer

A junior engineer asked me last week why a particular endpoint was slow. It took four hundred milliseconds, and he wanted it under fifty. Reasonable. But the question he was really asking — the one underneath — was whether four hundred milliseconds meant somebody had done something wrong. Usually it does not. Usually it means the system is doing the honest thing, and the honest thing takes time. Where speed comes from There are only a handful of ways to make a computation return faster, and every one of them is a trade you have to be willing to name out loud. Do less work. The only genuinely free option, and the one people reach for last. Do the work earlier. Caches, materialised views, precomputed aggregates. You pay in staleness and in the cost of knowing when to throw it away. Do the work elsewhere. Background jobs and queues. You pay in complexity and in every failure mode that comes with a second moving part. Do the work closer. Replicas, edge caches, denormalised copies. You pay in consistency and in money. Do the work approximately. Sampling, estimates, bloom filters. You pay in correctness, which is sometimes fine and sometimes catastrophic. That is the whole list. When someone shows you a system that got ten times faster without giving anything up, either they were doing something genuinely stupid before, or you have not found the invoice yet. The invoice always arrives The cache that made the dashboard instant is the reason a customer saw yesterday’s balance and called support. The queue that made checkout snappy is the reason an order sat unprocessed for six hours when a worker died quietly at three in the morning. The read replica that took load off the primary is the reason a user updated their profile, refreshed, and watched their old name come back. None of these are arguments against caches, queues or replicas. They are arguments for saying, at the moment you add one, exactly what you are buying and exactly what you are paying. Write it in the pull request. Write it in a comment above the code. In two years the person reading it will not be you, and the trade will look like an accident. A performance optimisation you cannot explain the cost of is not an optimisation. It is a bet you have forgotten you placed. Back to the four hundred milliseconds We looked at the endpoint together. It was joining across three tables, none of them large, and then doing a permission check per row in application code. The permission check was the whole cost — a hundred and forty round trips to fetch data we already had. So we did the only free thing on the list. We did less work: one query for the permissions, a map in memory, no round trips. It came down to thirty-one milliseconds, and we gave up nothing at all. That is the case worth looking for first, every time, before you reach for the clever machinery. It is also the case people skip, because doing less work is unglamorous and adding a cache feels like engineering. The four hundred milliseconds was not a law of physics. It was a hundred and forty questions we did not need to ask.

Responses