Summary
This is the most ambitious use of AI we have attempted at rhino.fi, on the biggest single piece of engineering work we have ever shipped. The numbers tell the story.
5,822 files changed, 88 pull requests, roughly 380,000 lines touched
Two weeks of the whole engineering team's time, from first commit to production
Usage of Fable 5.1, Opus 5.5 and GPT Astra, all within plan subscriptions to control costs
Zero production downtime for 4.5M Smart Deposit Addresses, zero customer impact
Estimated infrastructure cost saving of at least 25%. Memory usage cut in half
How we pulled off the biggest codebase change we've ever shipped
Last week rhino.fi shipped one of the largest changes we have ever made to our codebase. We upgraded Effect, the TypeScript library that underpins most of our platform, from v3 to v4. The change touched 5,822 files across 88 pull requests, with 192,562 lines added and 187,582 removed. The first commit was on 11 September. The rollout reached production on 15 October.
There was no downtime for any of our 4.5M Smart Deposit Addresses, no customer impact, and cross-chain stablecoin payments kept running throughout.
The whole engineering team spent two weeks on it. Without AI, our honest view is that we would have given up. The scale of the change and the opportunity cost of taking the team off everything else for months would have made it impossible to justify. This post is about how we used Claude and Codex, where it worked, where it did not, and what we learned.
Why we had to make the leap
Effect is the missing standard library for TypeScript. We first introduced it in early 2024 and it has since spread through the codebase. Today it sits in 4,740 of our 7,863 TypeScript files (60%), 87 of 116 shared packages and 32 of 38 services. Effect Cluster and Workflow power our collateral rebalancing and many of the customer-facing flows our clients depend on, including Smart Deposit Addresses.
The trigger for the upgrade was load. Under heavy traffic, work was not being distributed evenly across runners and there was no back pressure, so runners were being OOM-killed. Effect v4 fixes this and brings a 50% smaller memory footprint, lower CPU usage, a much smaller bundle size for the frontend and a more mature API. The v3 line had also stopped receiving bug fixes, so staying put was not an option.
A library upgrade sounds routine. This one was not. Effect v4 changed the semantics of many APIs, so code that looked identical behaved differently. Every one of those 5,822 files needed to be read, changed and checked.
First attempt: let the agents loose
We started with the shared packages. A coordinator agent spawned a subagent for each package, working through them in dependency order so packages with no dependencies went first. Learnings from each package were passed to the next session. All 87 packages were upgraded in a single 12-hour run.
Review was split between the team based on who knew which packages, balanced by line count, and combined manual review with agentic review.
Then we tried to migrate the services on top of the upgraded packages, and it failed.
Why it failed
The shared packages had low test coverage. Every check passed, but the agents had introduced a large number of breaking changes that nothing caught. Many of Effect's v4 changes altered behaviour without altering shape, which is exactly the kind of change a type check will not flag. On top of that, our spec-based approach to building APIs and their clients keeps them internally consistent, which makes it easy to introduce a breaking change on both sides at once without noticing.
The lesson was not that the agents were bad at the refactor. It was that we had given them no way to know when they were wrong.
Second attempt: build the safety net first
Two ideas from the team changed the approach. One of our engineers suggested we write tests for the shared packages before refactoring them, rather than after. Another engineer noticed we already had automated regression tests that used copied API specs and schemas to protect against breaking changes, and proposed extending that idea into contract regression tests that run against specs pulled from old commits.
We built a regression suite that checks the API contract on every edge of the system - client to server, server to persistence - against the pre-upgrade specs. Then we let the agents loose again, this time with a test suite that would tell them immediately if they had broken a contract.
That is the pattern we will reuse. AI makes the code change cheap. The expensive part is knowing whether the change is correct, and that is where the engineering effort should go.
Checking the work
Once the code was migrated, we used Claude across the verification stages as well.
Security. We ran Claude's security review across every service to scan for newly introduced vulnerabilities. We did not expect to find much in a refactor of this kind, but the review has caught critical issues for us before and we are now building it into our delivery pipeline.
Automated tests. We ran our automation suite on staging against both the v3 and v4 codebases, in both directions, to make sure clients on the old behaviour would see no difference on the new. Along the way we fixed configuration drift between environments and introduced parallel test execution, which cut a run from around 18 minutes to around 13.
Load testing. We ran a full load test on staging covering spikes of valid SDA transfers, spikes of spam SDA transfers, webhook bursts with unresponsive URLs, bridge execution and bridge quotes under load. Bridge results matched the previous run. The runner distribution problem that started the project was gone.
What the upgrade delivered
Memory usage across the platform has halved since the rollout. We have not yet measured the cost impact directly, but based on the memory reduction alone we estimate the saving at 25% or more on the affected infrastructure. The OOM-killed runners that triggered the project are no longer an issue, and the platform is now on a version of Effect that is actively maintained.
Deployment
Before rolling out we scaled up Postgres and Mongo. The Postgres increase turned out to be vital - we saw a sharp memory spike on the database right after deployment.
During and after the rollout we used sandboxed Claude agents with read-only access to parse logs. They first established a baseline from pre-deployment logs, then scanned for anything new, and checked pod and service health. Non-blocking issues with the new v4 shard transport were found straight away and patched shortly after. There was a brief gap in processing and a spike in delivery times immediately after deployment, which cleared quickly.
The team paused all new development for the duration, prepared rollback options ahead of time, agreed timings internally, and had every developer on a huddle during rollout ready to fix anything that came up.
What we missed
Two things got through. First, limit-reached errors were being returned as 500s instead of the correct error code. The follow-up agent scans did identify this, but it was buried in a large text result and we missed it, and it was also lost in the 500 spikes on our internal dashboards. A client reported it and we fixed it the day after the upgrade. Second, the Postgres memory spike, which was patched the next day.
Both are small and both are instructive. The agent found the error. A human did not read far enough into the output to act on it. The fix is in how we surface agent findings, not in whether to use agents.
What this tells us about AI in engineering
A few things we are confident about after this project.
AI collapsed the mechanical work. 5,822 files were changed in two weeks of team time, by a team that was also running the business. Before AI, we would not have started this. The upgrade would have sat on the backlog for several quarters as a known risk we could not afford to address, or we would have started it and given up part way through.
AI is only as good as the feedback loop around it. The first run produced plausible-looking code that was wrong. The second run, with contract tests in place, produced code we could trust. The investment that paid off was the test suite.
Humans still own the review. Every PR was reviewed by a developer who knew the package. Agentic review ran alongside that, not instead of it.
The monitoring role is real. Read-only agents scanning logs for deviations from a baseline caught issues faster than people watching dashboards, with the caveat that someone still has to read what they report.
This was a full-team effort, and the credit goes to the engineers who planned, built, tested and shipped it. Claude turned a project we would have shelved into one we finished in two weeks, with the platform running on half the memory it did before. The shape of the work changed too. Less time typing the change, more time deciding how to prove it was safe.
