tl;dr
- Smart contract upgrade with 28 day upgrade period leads to temporary inability to deposit or withdraw two non standard ERC20 tokens
- No funds compromised or lost
- Users still able to access, trade and withdraw normally
- Sleep lost
- CFO missing, presumed frightened
Late last night I was engaging in what passes for social interactions this year (note to self go one conversation / blog post without referring to Coronavirus) when my phone lit up with unusual messages from some of my colleagues. Seeing words like ‘hyperventilating’ and ‘In theory a fix takes 28 days’ is a good alternative to caffeine late at night.
It seemed we had a problem. That is, we did.. and still do have a problem, but at least now we have had time to mull it over and come up with Plan A, B, C and possibly D if we so need it.
This post will be a slightly rambling summary of what happened and how we are resolving it.
Before I say anything else, given the sensitivity of this type of post I want to make it clear that at no point were any funds compromised or lost. All user funds are safe, although in some cases an extra hoop may have to be jumped through to successfully withdraw.
Whilst the last few weeks have been exceptionally busy for us here at Deversifi with many new features going live behind the scenes (we will talk about them shortly when we can start enabling them for everyone), the main goal was to have all of these upgrades occur with no obvious change to our users.
On Wednesday our partner StarkWare, after a long period of testing, upgraded the smart contract that forms the basis of our layer 2 scaling solution to the latest and greatest version. These contracts were naturally audited and heavily tested on Ropsten prior to the upgrade. The deployment process completed and all seemed well initially.
Part of how we are able to offer such deep liquidity on our platform is that we leverage some liquidity from other sources blending our order books to provide a seamless experience for our users. In order to accomplish this, we periodically need to rebalance our internal collateral. Tonight this began failing and was the first sign that something was amiss.
After previously posting on Slack that he’d be going ‘offline for the evening’ (a seemingly innocent statement I half heartedly thought the universe was sure to punish) our smart contract mastermind (and also CEO) Will discovered that deposits and withdrawals of Tether were failing.
So, this was a problem.
DeversiFi’s mission is to take the best parts of a centralised exchange experience, with fast liquid markets, instant executions, privacy, and no gas fees, but augment it with the self custody and trustless-ness of DeFi. Suddenly faced with the nightmare scenario that we might have customer funds locked for 28 days and may have to suspend deposits and withdrawals, it suddenly seemed like our attempt to bring the CEX experience to the DeFi space might have worked a bit too well.
The root cause was quickly identified to be the way that some non-standard tokens (USDt and OMG) handle transfers was not compatible with the upgraded contracts. Worse still, because of safeguards specifically put in place to prevent rogue or malicious upgrades, there is a programatically enforced 28 day period before a contract upgrade can go live. That meant whilst (as is often the case) the fix was likely a one-liner, it was not going to quickly solve the problem we now found ourselves in.
For those of you interested in the specifics, I present to you the offending line(s).
v2:
token.transferFrom(msg.sender, address(this), amount);
v1:
address(tokenAddress).call(callData);
So, now this was a larger problem.
Luckily there was some good news, the issue was restricted to only two tokens which meant that anyone could trade out of the situation freeing their collateral to be withdrawn. New deposits were failing, preventing more tokens from becoming trapped, at least preventing the situation from becoming worse whilst we were engaged in animated conversations to work out how to solve the problem.
So now we come to what happens next, we have three options for users with funds temporarily locked on Deversifi.
Option the first. Do nothing.
Whilst this might seem like wishful thinking on our part, many of our traders do not deposit and withdraw their tokens at a high rate, simply trading with available balances.
For users who wish to keep their balances on Deversifi, their USDt and OMG will be available for withdrawal 28 days after the new version of the Starkware contract is checked and deployed. As we realise this is a sign of trust in a space where barely a week passes without some sort of drama, to compensate and thank these users we will be depositing additional tokens for withdrawal alongside their USDt and OMG when the new contracts are deployed. We will announce the exact APR tomorrow, but it will be competitive compared to staking for the equivalent duration in a token lending platform.
Option the second. Trade it out
All other markets are operating normally and any user can opt to trade their impacted tokens for others which can be freely withdrawn and further traded as per normal.
Option the third. Bear with us, something juicy is coming
Whilst this has been a stressful few hours for us, sometimes exciting things can come from unexpected directions.
Anyone who has been following along in our Discord or blog posts will have seen we have a fast withdrawals mechanism that we’ve been developing over the last couple of months. The StarkWare smart contract upgrade was one of the final pieces of this particular puzzle.. and we are evaluating how we can make this available more quickly.
Alongside this, another tempting solution is providing a wrapper to allow stuck tokens to be withdrawn.
Since it’s pretty late we will pick this part up tomorrow and hope to have more information available shortly, but we’re pretty excited for what might be possible in the next few days.
This update slash incident report slash ill advised stream of late night consciousness is already much longer than I anticipated, but in the interests of transparency I really wanted to share what has happened, why and what we are doing about it.
To summarise: no-one will be adversely impacted by this and no user funds have been lost or put at risk. More to come tomorrow.
