Observability at tastytrade, and a datacenter move
OpenTelemetry across a brokerage platform, then using it to move the whole thing to a new datacenter without dropping a trade.
I did not expect to end up in fintech. A brokerage runs its own datacenters and its own fiber because order routing punishes latency and downtime in dollars, and that turned out to be exactly the kind of complexity I enjoy digging into.
The brief was roughly "we can't see what's happening in production, figure it out." So the first job was making the platform observable: shared libraries for standard metrics and traces, rolled out across 30+ Ruby microservices, and the brokerage systems onboarded to Honeycomb. Then the brief got bigger: the Chicago facility was out of space and power, and everything had to move to a new one in Aurora.
I was the application and telemetry lead for that migration. Running both datacenters at once and comparing them side by side, down to the millisecond, is how we found the problems before customers did. Along the way the platform went from Chef, VMs, and bash to containers on Nomad, and I upgraded the core Elixir/Erlang messaging service everything depends on.
I gave a talk on it at Honeycomb's O11yDay in New York: De-risking datacenter migrations with observability.