I did not expect to end up in fintech. A brokerage runs its own datacenters and its own fiber because order routing punishes latency and downtime in dollars, and that turned out to be exactly the kind of complexity I enjoy digging into.

The brief was roughly "we can't see what's happening in production, figure it out." So the first job was making the platform observable: shared libraries for standard metrics and traces, rolled out across 30+ Ruby microservices, and the brokerage systems onboarded to Honeycomb. Then the brief got bigger: the Chicago facility was out of space and power, and everything had to move to a new one in Aurora.

I was the application and telemetry lead for that migration. Running both datacenters at once and comparing them side by side, down to the millisecond, is how we found the problems before customers did. Along the way the platform went from Chef, VMs, and bash to containers on Nomad, and I upgraded the core Elixir/Erlang messaging service everything depends on.

I gave a talk on it at Honeycomb's O11yDay in New York: De-risking datacenter migrations with observability.

Background

tastytrade is a brokerage. It runs its own datacenters and its own fiber, because when you're routing orders, latency and downtime get expensive very fast. The original Chicago facility ran out of space and power, so the plan was to move everything to a new one in Aurora, Illinois. Twice the space at a third of the cost. The catch: you can't just turn the old one off and hope. In addition to the move to new hardware, we migrated to a Nomad/docker based orchestration platform. This added to the complexity of the project because in addition to a physical move we were adding an entire re-architecting of the application stack as well.

What I did

I was the application and telemetry lead for the move. Before we could migrate anything, we needed to be able to see what we had and how it was currently performing. So the first job was OpenTelemetry: I built shared libraries for collecting standard service metrics and traces, tested them, and rolled them out across 30+ Ruby microservices. Once that was in place, I worked with the engineering leads to get the brokerage systems into Honeycomb. Everything needed to be measured and labeled correctly.

That paid off during the migration. Running two datacenters at once meant two streams of telemetry for every service. We tagged everything by datacenter and compared them side by side. We could even see down to the millisecond of latency between the two sites. When something looked off, we could ask the question instead of guessing, or worse waiting for our customers to tell us.

Along the way the platform got modernized too: Chef, VMs, and bash scripts gave way to containers on Nomad running on bare metal. I also upgraded the core brokerage messaging service, which is written in Elixir/Erlang, and ran telemetry talks and sharing sessions for the engineering team who were new to observability standards and frameworks.

Things I learned

1. Instrument first, migrate second. The monsters live in the gap between what you think is running and what actually is.
2. Comparing two live environments is a much better test than any staging environment.
3. An Elixir upgrade on a service everyone depends on is a migration of its own.

I gave a talk about the project at Honeycomb's O11yDay in New York: De-risking datacenter migrations with observability. Honeycomb also wrote up a case study.

Photo by Kevin Ache on Unsplash