Background

tastytrade is a brokerage. It runs its own datacenters and its own fiber, because when you're routing orders, latency and downtime get expensive very fast. The original Chicago facility ran out of space and power, so the plan was to move everything to a new one in Aurora, Illinois. Twice the space at a third of the cost. The catch: you can't just turn the old one off and hope.

What I did

I was the application and telemetry lead for the move. Before we could migrate anything, we needed to be able to see it. So the first job was OpenTelemetry: I built shared libraries for collecting standard service metrics and traces, tested them, and rolled them out across 30+ Ruby microservices, then worked with the engineering leads to get the brokerage systems into Honeycomb.

That paid off during the migration. Running two datacenters at once meant two streams of telemetry for every service. We tagged everything by datacenter and compared them side by side, down to the millisecond of latency between the two sites. When something looked off, we could ask the question instead of guessing.

Along the way the platform got modernized too: Chef, VMs, and bash scripts gave way to containers on Nomad running on bare metal. I also upgraded the core brokerage messaging service, which is written in Elixir/Erlang, and ran telemetry talks and sharing sessions for the engineering team.

Things I learned

1. Instrument first, migrate second. The monsters live in the gap between what you think is running and what actually is.
2. Comparing two live environments is a much better test than any staging environment.
3. An Elixir upgrade on a service everyone depends on is a migration of its own.

I gave a talk about the project at Honeycomb's O11yDay in New York: De-risking datacenter migrations with observability. Honeycomb also wrote up a case study.