By: Arun Mishra, Enterprise Architect
For a long time, one of our most important financial data pipelines took 18 hours to run. Everyone treated that as normal. It ran overnight, it finished before the business day started, and nobody had a reason to question it. That’s usually how the most dangerous problems in enterprise systems hide: not as outages, but as things everyone has quietly agreed to live with.
We eventually rebuilt that pipeline so it ran in about 6 minutes. The story of how, and why it took as long as it did for anyone to treat it as a real problem, taught me more about risk than any incident postmortem ever has.
The Problem Nobody Was Calling a Problem
The pipeline processed financial transaction data in large overnight batches. It worked. It had worked for years. But 18 hours is a long time for data to sit unprocessed, and that gap quietly shaped everything downstream. Reports were always a day behind. Errors weren’t caught until the next business day, sometimes later. If something went wrong mid-run, there often wasn’t enough time left in the window to rerun it before people needed the output.
None of that showed up as an incident. It showed up as small workarounds: a report that always ran late, a reconciliation step that always happened the next morning, a team that had learned to plan around the delay instead of questioning it. That’s the part that took me a while to see clearly. The system wasn’t broken. It was just slow in a way that had become invisible because everyone had adapted to it.
Why We Finally Looked Closer
The trigger wasn’t a failure. It was a new business requirement that needed same-day visibility into transaction data, something the 18-hour batch process couldn’t support no matter how we tuned it. That’s often how these things get uncovered. Nobody goes looking for a redesign when the current system technically works. It usually takes a new requirement to expose how much of the old approach was built around assumptions nobody had reexamined in years.
When we actually opened up the pipeline, the batch design turned out to be the whole problem. Data moved through a single sequential process: extract everything, transform everything, load everything, in that order, as one long chain. Every stage waited for the one before it to finish completely. There was no reason two independent pieces of that chain couldn’t run at the same time. Nobody had ever asked that question, because the system had never needed to be fast. It had only ever needed to finish by morning.
What Changed
The rebuild came down to a few decisions, none of them exotic on their own.
We moved from a single overnight batch to an event-driven design, where transactions get processed as they arrive instead of piling up for one big nightly run. That alone removed most of the 18 hours, because the system was no longer waiting for a full day’s worth of data to accumulate before doing anything with it.
We also broke the pipeline into independent stages that could run in parallel instead of one strict sequence. A lot of what had been treated as one long chain was actually several unrelated processes that had simply been built to run one after another, mostly because that was how the original system was designed a decade earlier, not because the work genuinely depended on it.
And we added monitoring that could catch a stuck or failing stage within minutes instead of the next morning. That last part matters more than it sounds. A fast pipeline that fails silently isn’t actually safer than a slow one. Speed only helps if you can trust what it’s telling you.
The Real Lesson Wasn’t About Speed
It’s tempting to tell this story as a performance win: 18 hours down to 6 minutes, a good number for a slide. But the number isn’t really the point. The point is what that 18-hour window had been quietly costing us the whole time, in delayed error detection, in reports nobody trusted until the next day, in a team that had built its entire workflow around waiting.
That’s the part I’d want any other architect or technology leader to sit with. Slow systems don’t usually announce themselves as risks. They get absorbed into normal operations until they feel like just how things are. The real work isn’t rebuilding the pipeline once someone finally complains. It’s noticing the workarounds your team has quietly built around a system’s limitations, and asking whether those workarounds are actually masking a bigger problem.
In financial services especially, a delay is never just a delay. It’s a window where errors go unnoticed, where fraud has more time to move before anyone sees it, where a bad number sits in a report before anyone catches it. Eighteen hours doesn’t sound dangerous. It just sounds normal. That’s exactly why it’s worth questioning.
What I’d Tell Other Leaders
If there’s a system in your organization that everyone describes as slow but stable, don’t take stable at face value. Ask what the team has built around that slowness to make it livable. Those workarounds are usually the clearest map of where the actual risk is hiding, long before it turns into an incident anyone has to explain after the fact.
About the Author
Arun Mishra is an Enterprise Architect and Senior Manager with 16 years of experience in financial services and healthcare technology, spanning event-driven architecture, real-time payments systems, cloud-native platform design, and ML-based fraud detection. He holds 8 AWS certifications, Google Cloud Professional Architect, and Google Generative AI Leader credentials.
