HardikMurdia

Mira was a new product whose landing page took seven seconds to load, and the brief was to at least halve it. There was no tracing anywhere in the stack — no spans, no instrumentation, nothing to ask a question of. So the first job wasn’t the latency, it was building something that could see it.

The network waterfall in devtools ruled out the front end immediately: the browser was waiting, not working. The time sat in the backend calls, GraphQL resolvers running on Lambda, and the same call varied wildly between requests — which is the signature of something other than the query being slow.

You cannot see inside a Lambda invocation from the outside, so I built the trace flow from zero — OpenTelemetry collectors through to Honeycomb — and instrumented the handler down to individual operations rather than stopping at the request boundary. That resolution is what made the next step possible.

Two spans owned almost the whole request. DynamoDB resource initialisation burned 2.66 s before a single query was constructed, and query execution took a further 8.24 s. Query construction itself — the part you’d suspect — took 0.27 ms.

Our working assumption was cold starts, and the reflex fix for that is warming or provisioned concurrency. Both would have been wasted money. The cost was memory. boto3 assembles its entire API surface from JSON files on first use, and Lambda scales CPU with allocated memory — so an under-provisioned function spends seconds parsing service models before it has talked to anything. tecRacer measured this at more than 10× slower on 128 MB than on 1408 MB, which matched what our spans were showing. Adding concurrency would have made it worse: more containers, each paying the full initialisation.

Right-sized memory on the affected functions. Landing page load dropped from 7 s to roughly 500 ms; the actions endpoint went from 7.92 s to 1.29 s.

The instrumentation was the real deliverable. Without spans inside the handler, “the page is slow” and “probably cold starts” would both have stayed true, unfalsifiable, and useless — and the fix would have been more Lambda concurrency, which would have cost money and changed nothing.

The Kafka clusters had been sized for a worst case that never arrived, and there was one in nearly every region we ran. Oversized Kafka is expensive in a quiet way — it never pages anyone, it just bills every hour of every month. The savings were obvious on a dashboard. Collecting them was not.

Scaling a stateless app down is a number in a manifest. A Kafka broker is not stateless — it holds partition replicas, and some of those replicas are the in-sync ones other brokers and producers are depending on right now. Terminate the instance and you are not removing capacity, you are removing data and quite possibly breaking every producer configured to require acknowledgement from the in-sync replica set.

Deployment was managed through BOSH, with the cluster topology described in manifest files. BOSH decides which instance goes when the count drops, based on deployment order rather than on which broker happens to be convenient — so the first thing to establish was not how many nodes to remove but which specific node was going to disappear, and then to target it deliberately.

Before the instance could go, its partitions had to move. That meant going into the cluster and running the reassignment from the Kafka console to shift replicas onto the remaining brokers, then waiting — and it is genuinely a wait, because the new replicas have to pull the existing log from scratch and catch all the way up before they join the in-sync set. Only once under-replicated partitions were back to zero was the broker actually carrying nothing, and only then could BOSH take the node.

All of this ran against live production clusters with real ingestion flowing through them. The sequencing is the whole job: drain, verify, remove, confirm the cluster is healthy, then start again on the next broker. Rush any step and the failure is silent data loss rather than a clean error.

Right-sized Kafka across nearly every region we operated, worth $100–150K a year, with no downtime and no lost messages at any point in the exercise.

I cut the BOSH release for Storm 2.3 and promoted the package to S3, so every Storm cluster in production would pull the new version and upgrade its base cluster on the next deployment. One artefact, one promotion, the whole estate follows.

Kryo registration exceptions in the ingestion topologies. Storm 2.2 still fell back to Java serialisation for tuples crossing a worker boundary; 2.3 deprecates that path in favour of Kryo, which refuses to serialise any class it has not been told about. Those topologies passed custom objects between bolts and had never needed to register them, because until 2.3 nothing forced the question.

The first topologies to go were the ones on local field grouping — Supervision, eDiscovery, the S3 marker path. Not everything failed, and the failures did not line up with anything obvious about the topologies themselves. The pattern only came out once we stopped sorting the exceptions by topology and started sorting them by how tuples were moving between workers.

Caught at the start of the rollout, so nothing was lost. I cut a fresh BOSH 2.2 release, promoted it the same way, and redeployed the clusters — the rollback was a cluster deployment and nothing more, because the upgrade path had been built to run in both directions. No ingestion failures, no compliance records dropped.

Registered the tuple classes with Kryo in the affected topologies, then re-promoted 2.3 and took the estate through the upgrade again. Second time it held.

A deprecation in the release notes can be invisible right up until a specific runtime condition makes it mandatory — and the condition here was a design choice made years earlier in a different file. The thing that made this safe was not spotting it in review; it was that rolling back cost one deployment.

A JPMorgan Chase deployment needed sustained ingestion of 1,000 documents per second. The pipeline was doing 30–40. At an average message size of ~220 KB that target is roughly 220 MB a second of continuous ingest — and because this is a compliance archive, none of it may be dropped, reordered into uselessness, or left unsearchable.

Nothing could be tuned in isolation, because a document crosses eight systems between arriving as an email and becoming searchable. The first job was writing that path down honestly.

Every stage had its own. RabbitMQ and Kafka were configured for a fraction of the target. Storm’s in-memory queue and topology parallelism throttled everything behind it. The PCF routers and the apps behind them needed scaling out. Hazelcast needed rework. Queries against the metadata stores needed indexes they had never needed at 40 docs/sec. Removing one ceiling only ever revealed the next.

Horizontal scaling where the work partitioned cleanly and vertical where it did not, PCF router and app scaling, a Storm topology rework, Kafka and RabbitMQ resizing, Hazelcast changes, database indexing, and a pass over the per-document code path to remove work that at 40 docs/sec was invisible and at 1,000 was structural. Each stage was benchmarked on its own before and after, because a pipeline this long will otherwise happily hide a regression behind an improvement.

The system reached 1,000 docs/sec and held it, throughput up roughly 25× while cost went down. It brought recognition across the org and my promotion to Delivery Engineer 3.