Every year, state governments process over $600 billion in federal assistance programs, from food stamps to unemployment insurance. But behind those transactions lies a problem most citizens never see: outdated data systems that can take days to refresh, leaving caseworkers without current information and auditors struggling to verify compliance. When errors cascade through fragmented databases, families wait longer for benefits they qualify for, and agencies lose visibility into where taxpayer dollars flow.
State governments move hundreds of billions of dollars in federal assistance every year, but the systems tracking those transactions often run on technology older than the smartphones in caseworkers’ pockets. Data sits in silos. Reports take hours to generate. When something breaks, it can take days to figure out where the problem started. The people waiting for benefits don’t see any of this, but they feel it when approvals take longer than they should or when paperwork gets lost between systems.
Federal agencies reported $236 billion in improper payments across programs in fiscal year 2023. A chunk of that comes from outdated verification systems and data that doesn’t sync correctly between state and federal databases. When information moves slowly or inaccurately, real families wait longer for the help they’ve already qualified for.
One state decided to rebuild the whole thing from scratch. Not with a flashy vendor contract or a big announcement, just a steady migration from legacy batch processing to real-time data pipelines that actually worked.
From reports to infrastructure
Srinubabu Kilaru started as a BI Developer doing what most people in that role do: build ETL jobs, maintain dashboards, and troubleshoot when someone’s report doesn’t load. The work was technical but narrow. Design a pipeline, hand it off, and move to the next ticket.
But he kept running into the same problems. Data would refresh overnight, but by morning, it was already stale. Analysts would build their own workarounds because the official reports didn’t update fast enough. Every agency had a slightly different definition of basic terms like “pending application” or “household income,” so numbers never matched across departments. Audits turned into archaeological digs through a spreadsheet, trying to reconcile conflicting totals.
Kilaru saw an opportunity that most people in the data team didn’t: the problem wasn’t individual pipelines, it was the entire architecture. So he started pushing for a modernisation of the state’s eligibility data ecosystem.
The first step was moving everything off on-premises servers and onto Azure Databricks. The old system ran batch jobs overnight because that was the only way it could handle the processing load. He replaced that with streaming pipelines that updated continuously. Analysts stopped waiting until morning for fresh data because the data was already fresh.
He introduced DBT to bring version control and automated testing to transformations. In the old setup, if someone changed a calculation in one report, it could quietly break something else downstream. Nobody would notice until an auditor asked why two dashboards showed different numbers for the same metric. With DBT, every transformation got tested before it went live, and teams could trace exactly how each number was derived.
Power BI dashboards fed by a unified semantic layer gave everyone in the state a single source of truth. Caseworkers, compliance officers, and agency directors all looked at the same data, defined the same way, updated in near real-time.
“We weren’t just moving data faster,” Srinubabu Kilaru said. “We were making it possible to trust the data in the first place. If an auditor pulls a number and a caseworker pulls the same number, they should match. That sounds basic, but it wasn’t happening before.”
Intelligent automation handled the quality checks that used to require manual review. The system flagged duplicate applications, income mismatches, missing documentation, and anything that didn’t pass validation rules embedded directly in the pipeline. Compliance officers got a real-time audit trail instead of spending weeks before federal reviews trying to piece together what happened.
What changed on the ground
The impact wasn’t abstract. Caseworkers who used to toggle between multiple disconnected systems to verify a single applicant’s eligibility now pull consolidated profiles from one interface. Processing times dropped. Error rates in federal reporting fell because agencies were working from validated datasets instead of conflicting spreadsheets.
Audit preparation stopped being a crisis. Compliance teams that once spent weeks reconciling numbers before federal reviews could generate certified reports on demand, complete with lineage tracking showing exactly how each figure was calculated. That transparency didn’t just satisfy regulators; it gave staff time to focus on improving policies instead of firefighting data problems.
The unified semantic layer solved a problem most people outside government IT never think about: what counts as “household income”? When is an application officially “pending”? How do you calculate eligibility windows consistently across programs? Before standardisation, every agency answered those questions slightly differently, which meant self-service analytics produced conflicting metrics. After his work, agencies could build their own dashboards without waiting for the BI team, confident that the numbers would align.
Refresh delays dropped 60 per cent. That’s the difference between decisions made on yesterday’s data and decisions made on data from an hour ago.
The bigger picture
States across the country are wrestling with the same problem. Systems built in the 1990s, patched repeatedly over decades, now buckling under modern demands. Federal oversight has gotten stricter. Caseloads have grown. The expectation that data should update in real time, not overnight, has become standard everywhere except government IT.
An estimated $140 billion in federal benefits goes unclaimed each year, partly because outdated systems make it harder for eligible people to navigate the application process. When data is fragmented and slow, caseworkers can’t easily verify eligibility, and applicants get stuck in bureaucratic loops that shouldn’t exist.
The shift from batch to streaming isn’t just a technical upgrade. It’s what makes the next generation of government services possible. Predictive analytics that could identify applicants at risk of falling through the cracks or flag suspicious claim patterns require clean, current data. Machine learning trained on stale or inconsistent inputs produces stale or inconsistent results. By building a foundation of reliable, accessible data, he positioned the state to pilot AI-driven tools that would have failed under the old architecture.
Other states are watching. The model he built (cloud infrastructure, automated validation, a single source of truth) offers a template for jurisdictions still running legacy systems. The technical choices matter, but so does the shift in mindset: treating data as infrastructure, not a byproduct.
What comes next
The eligibility system he rebuilt now processes millions of transactions with a governance framework that supports both compliance and innovation. But the work isn’t static. As agencies begin testing AI for case prioritisation and fraud detection, the data layer becomes critical. Models are only as reliable as the data they consume.
“Data modernisation isn’t a project with an end date,” Srinubabu Kilaru said. “It’s about building systems that adapt when policies change, when caseloads shift, when new technology becomes available. If the foundation is solid, everything built on top has a chance to work.”
For states still patching legacy systems and hoping they hold together, the lesson is straightforward: incremental fixes don’t solve structural problems. Real modernisation means rethinking how data moves, who’s responsible for quality, and how technology serves the people, depending on public assistance. In one state, that transformation already happened. The blueprint exists for anyone willing to use it.
Be First to Comment