Background

tastytrade is a brokerage. It runs its own datacenters and its own fiber, because when you're routing orders, latency and downtime get expensive very fast. The original Chicago facility ran out of space and power, so the plan was to move everything to a new one in Aurora, Illinois. Twice the space at a third of the cost. The catch: you can't just turn the old one off and hope. In addition to the move to new hardware, we migrated to a Nomad/docker based orchestration platform. This added to the complexity of the project because in addition to a physical move we were adding an entire re-architecting of the application stack as well.

What I did

I was the application and telemetry lead for the move. Before we could migrate anything, we needed to be able to see what we had and how it was currently performing. So the first job was OpenTelemetry: I built shared libraries for collecting standard service metrics and traces, tested them, and rolled them out across 30+ Ruby microservices. Once that was in place, I worked with the engineering leads to get the brokerage systems into Honeycomb. Everything needed to be measured and labeled correctly.

That paid off during the migration. Running two datacenters at once meant two streams of telemetry for every service. We tagged everything by datacenter and compared them side by side. We could even see down to the millisecond of latency between the two sites. When something looked off, we could ask the question instead of guessing, or worse waiting for our customers to tell us.

Along the way the platform got modernized too: Chef, VMs, and bash scripts gave way to containers on Nomad running on bare metal. I also upgraded the core brokerage messaging service, which is written in Elixir/Erlang, and ran telemetry talks and sharing sessions for the engineering team who were new to observability standards and frameworks.

Things I learned

1. Instrument first, migrate second. The monsters live in the gap between what you think is running and what actually is.
2. Comparing two live environments is a much better test than any staging environment.
3. An Elixir upgrade on a service everyone depends on is a migration of its own.

I gave a talk about the project at Honeycomb's O11yDay in New York: De-risking datacenter migrations with observability. Honeycomb also wrote up a case study.

Photo by Kevin Ache on Unsplash