A 23.7 million dollar Ethereum transaction fee post mortem…

Yesterday morning a hardware wallet deposit to DeversiFi attracted a lot more attention than such a deposit would normally warrant. Attached to the transaction was a very generous and unanticipated bonus fee dwarfing and by far upstaging the intended transfer itself. We’ve seen so called ‘fat-finger’ errors in transfers before, with incorrectly entered values or just plain errors made setting fees. This time the cause was far more interesting and in this post mortem I cover not only the technical elements of how it happened, but also how the amazing cryptocurrency community came together to return the misspent funds.

Summary

Here’s a quick summary of what happened, what we did to resolve it and what we’re doing to ensure this does not ever happen again, either to users of DeversiFi but also other DeFi apps.

What happened?

• At 11.10.08 AM UTC +1 A user  made a 100,000 USDT deposit to DeversiFi.
• This was confirmed within 1 minute… but the tx fee for this transaction was 7,676 ETH (or ~ $23million dollars) https://etherscan.io/tx/0x2c9931793876db33b1a9aad123ad4921dfb9cd5e59dbb78ce78f277759587115

Why did it happen?

• Underlying issues in the EthereumJS library coinciding with gas fee changes associated with the EIP-1559 upgrade in some circumstances can lead to transactions with extremely high fees
• Combine this with the fact that Ledger hardware wallets may sometimes display fees in a non-human readable manner, could have removed a key human safety check
• Only wallets with very large quantity of funds would be impacted, all other users would see a failed transaction

What did we do about it?

• By 12:30:00 PM UTC+1 the team at DeversiFi were aware of the issue and began our investigation.
• We quickly identified two primary areas of concern which we began actively testing in an attempt to reproduce and explain how the erroneous transaction was created
• We then shared an explanation with our community and the blockchain world at large who had started to notice this transaction https://twitter.com/deversifi/status/1442487743922286594
• By 16:45 UTC + 1 we had disabled deposits from Ledger users to enable us to identify the issue without putting further users at risk
• By the evening we had found the likely culprits in the gas fee functions and set about implementing an improvement to prevent the edge case
• Additional safety and sanity checks were also added to ensure gas fees associated with transactions could not exceed unrealistic thresholds to protect against user error, extreme network fee spikes and as an additional layer of protection against any future coding error
• We have raised an issue with the EthereumJs maintainers describing the defect in the library
• Lastly we communicated with the Ledger team about anomalies discovered during testing which could obfuscate abnormally high fees for any Ethereum transaction
• Safety improvements rolled out and ledger deposits re-enabled by 15:30 on 28/09/21

Fund recovery

• After seeing that the miner of block 13307440 who had received the fee was periodically depositing mined ETH to Binance we made contact with Binance
• Binance agreed to pass our email addresses to their customer so that they might be able to reach out to us
• By 20:36 UTC + 1 we had received an email from the miner who had reached a process for safely returning funds
• Within an hour they had made the first return transaction, with a total of 7626 returned: https://etherscan.io/tx/0x85294effd53126b3bfa9e7f655267e00ac1ae2ef76f4569644670bf5403637d6
• DeversiFi offered for the miner to keep 50 ETH as a return fee

What are we doing to ensure it cannot happen again?

• DeversiFi are actively engaging with both the Ethereum community and Ledger to patch issues that may have contributed to this occurrence
• On our platform we’ll be implementing stronger defensive measures when interfacing with external libraries, reviewing how we treat failed transactions and also enforcing a ceiling value for any max transaction fees as additional protection

Background

Before we dig in further, let’s go over how EIP-1559 changed the way in which Ethereum transaction fees are handled.

In response to high and often unpredictable transaction fees a new concept was introduced – namely a base cost of gas which would scale more smoothly depending on network load. The gas consumed by a transaction would be specified by the product of this base fee and the complexity of the interaction (i.e. the gas cost). On the face of it, this would result in a much more predictable mechanism eliminating the possibility of accidentally overspending on fees. These fees would then be burned during execution, potentially pushing Ethereum into deflationary economic territory.

As has been well documented previously, this conflicted with the interests of miners who historically enjoyed inflated fees paid for transaction inclusion. Not only would they lose out on the bidding feeding frenzy during times of increased activity, but they would miss out on the transaction fees only being eligible for the block reward.

An additional field that acted as a tip was added to EIP-1559 transactions, intended to incentivise prioritised inclusion by miners who were able to collect this additional fee.

In summary,rather than a single Gas Price which is paid entirely to the miner as a bid for inclusion into a block, there are now different components.

• Base Fee – determined by the network and burned
• Max Fee Per Gas – the maximum amount that will be paid per unit of gas to get a transaction included in a block
• Max Priority Fee – an optional user specified tip paid directly to miners

EIP-1559 transactions include these new fields and are known as Type 2, whilst legacy transactions supplying the original Gas Price field remain supported and are known as Type 0. We don’t talk about what happened to Type 1.

A common misconception is that EIP-1559 transactions entirely eliminate the possibility of someone overpaying for a transaction. Whilst this may be true for the max fee which specifies a ceiling cost, not the final cost – the priority fee behaves like a legacy transaction in that it is all taken by the miner. In a situation where both the priority fee and max fee are set too high there is no protection from accidental overpayment.

So with this frame of reference, let’s dig into what happened.

Investigation

DeversiFi is a layer 2 protocol on Ethereum for Decentralised Finance. Our team hosts a front-end which provides an easy interface to access the protocol from a variety of wallets, including Metamask and Ledger. About a month ago we updated the front-end for DeversiFi to make use of EIP-1559 transactions provided by the London hardfork activation. We switched to the latest version of the Ethereum library and implemented the new functionality as documented.

The reliability of the platform and safety of user funds is of paramount importance to us at DeversiFi. As such we have multiple stages of validation and testing before any changes to the platform or app are deployed to production.

Prior to going live with the new transaction type, the changes were extensively tested on our test and staging environments. Additionally, we actively monitor on mainnet any on-chain events  which enabled us to see that for the vast majority of users the updated transactions were working correctly. On-chain transactions can sometimes fail due to network activity bumping up gas prices suddenly or through insufficient fees being manually set. As such when we saw occasional layer 1 transactions failing to be broadcast or confirmed on-chain, it was not always easy to understand why.

As chance would have it, one of our dev teams was already in the middle of a week-long deep dive into all reported transaction anomalies. The universe, it seems, has a sense of humour.

In this space, being sent an etherscan link without context or explanation can evoke a range of responses from curiosity to, in this case, bewilderment. When we were alerted to the transaction our starting point was to look at any dynamic inputs to the fee calculation mechanism.

We aggregate gas prices from different providers and use these as the basis of setting suggested gas parameters for on-chain transactions. The first concern was that perhaps something had gone wrong in this estimation causing a huge spike – this seemed unlikely as we have alerting which would have triggered notifications, but it was the most obvious starting point.

Reported gas fees in the duration were well within normal ranges.

The situation was puzzling as not only had a complete platform regression test been completed just a few days prior, there had been several successful deposits from other users since the EIP-1559 update. On top of this, this wasn’t the first deposit from the user using this wallet – a previous deposit for a slightly lesser amount (https://etherscan.io/tx/0x4ee1843405c87656acd9c9230dfd3505b98042d00f0653e7589183328f1c540d) was made a few days prior without incident.

We then set about trying to replicate the conditions of the deposit with several members of the team, performing mock deposits using Ledgers in order to replicate the problem. It was discovered that in a small number of cases the transactions would fail with a somewhat unhelpful error message.

We had our thread to pull on.

Root cause

So what happened..

Metamask performs a lot of the heavy lifting when generating messages and signatures, however for other wallets like Ledger, we generate the transactions ourselves using the @ethereumjs/tx npm package (https://github.com/ethereumjs/ethereumjs-monorepo).

Specifically we create an EIP1559 transaction body, generate the message imbuing parameters and fees before interfacing with the Ledger wallet library to prompt the user to sign on their hardware device.

Libraries that handle fixed precision and expanded numerical range are important in the Ethereum ecosystem as smart contracts can return numbers with up to 256 bits. JavaScript on it’s own can’t handle that precision leading to truncation or floating point errors. Not all of the big number libraries support floating point values and unfortunately the ethereumjs library uses BN (https://github.com/indutny/bn.js/) which also does not. On the face of it this somewhat makes sense since Solidity doesn’t directly support anything other than integers, however it does push the responsibility on to anyone integrating their libraries to also not use decimal values.

The first part of this process is where a problem occurred, specifically when the gas and priority fees were calculated and then converted into a big number object. Since the last few blocks are used to predict priority fees, the calculation could result in a decimal figure (this is something that Tay at MyCrypto has been warning about for a while)

When the gas values generated were integers, the underlying Ethereum library code (https://github.com/ethereumjs/ethereumjs-monorepo/blob/cf95e04c6a2769865999a4262705a7adf6db416c/packages/tx/src/eip1559Transaction.ts#L201) worked perfectly, however when passing a decimal value things quickly became strange. The BN library used by the Ethereum library code throws an error to indicate an invalid value has been passed, however since the value was converted to a buffer first no error handling was triggered.

For example, passing a value of 33974230439.550003 would set an integer 35624562649959629 – potentially six orders of magnitude higher than intended.

When this mangled numeric interpretation occurred it would either fail due to the priority gas amount being higher than the max fee per gas or, more insidiously, because the amount of ETH a user had in the wallet was almost always unlikely to be high enough to cover this enormous overspend of gas fees.

This means that for the minority of hardware wallet users who experience this issue, almost all will never understand why their transaction (sometimes silently) failed. They will then simply retry and in all likelihood it will work on the second attempt when the base fee predictions for the next block had updated to return a non-decimal value.

When signing a transaction on a Ledger, the max fee is displayed to the user for them to verify the terms of the transaction they’re about to authorise. An exacerbating factor is that sometimes Ledger currently displays very large fees as a hex value.

In trying to reproduce the issue, we encountered fee prompts as shown above. In this example transaction showing the issue, a hex value of B526167CF91FECE4 equals 13053145295991336164 which equates to an astronomical fee of 13053145295991.336164 Gwei or ~13.05 ETH.

If this transaction were accepted (and the funds to cover it present in the wallet) the user would be signing a maximum fee of 216,564 ETH. The actual amount might be lower depending on the priority fee which is not shown.

We wondered whether this might have been the case with the now infamous deposit (https://etherscan.io/tx/0x2c9931793876db33b1a9aad123ad4921dfb9cd5e59dbb78ce78f277759587115) where the max fee authorised could have resulted in as much as 2x more ETH being paid in fees than was already paid.

Whilst these may be outrageous numbers for the majority of wallets, resulting in these transactions not normally being accepted by the network, for wallets who have the  funds to cover these eye-watering sums there was no other safety mechanism to prevent the broadcast of such an expensive transaction.

Takeaways

For everyone:

• Ethereumjs (https://github.com/ethereumjs/ethereumjs-monorepo) library does not support decimal values. This in itself won’t be a surprise to most Ethereum developers, however rather than throwing an error, decimal values are mangled silently into something potentially dangerous
• Explicit validation and sanity checking of max fees before signing and broadcasting strongly encouraged by all dApps
• Strongly recommend to enforce platform maximum gas fee when generating transactions to guard users against spikes or platform bugs
• EIP-1559 does not protect against accidental overspending

For DeversiFi:

• Stronger defensive measures when interfacing with external libraries
• Treat all failed transactions as a potential defect and investigate rather than assuming user error
• Enforce a ceiling value for any max transaction fees as additional protection

Fund recovery

In parallel with the technical investigation, we immediately began an effort to reach the miner who had unknowingly swept the transaction along with others into block 13307440.

Whilst we knew we couldn’t count on anything, there are several examples in the past of miners returning large gas fees sent as the result of critical bugs and mistakes. The miner of the block was ranked in the top 10 according to etherscan for recent blocks, and appeared to have automated deposits set up to send their mined ETH to a Binance account.

Our first port of call was therefore to immediately contact Binance to see if they might be able to put us in contact with the miner. Binance’s reaction was incredibly fast, rallying immediately to help return funds by passing on our contact details to their customer so that they could contact us if they wished. Simultaneously we explored other ways of reaching out to the miner, but assumed that they could be based in any timezone and therefore might not yet be aware of the transaction.

Early in the evening the miner established contact via email, and after a few emails back and forth quickly agreed to return the funds involved to their origin address!

Transaction returning bulk of funds:

https://etherscan.io/tx/0x85294effd53126b3bfa9e7f655267e00ac1ae2ef76f4569644670bf5403637d6

Aftermath

Fortunately, this is a story which had a happy ending and speaks volumes about the amazing Ethereum and cryptocurrency community in general. The community rallied together: from the teams who supported us through the issue and investigation (Bitfinex, Binance, Alchemy, Ledger), to the incredible Ethereum miner who returned the funds, to our valuable community of users who carried out their own investigations to try and get to the root of the issue.

We have become used to treating failed transactions as a mild irritation, the root cause of which is often blamed on network factors and dismissed. As a platform operator in this space supporting many wallets, it’s extremely important that we don’t make the assumption that failed transactions ‘happen’ but instead drill down into the causes.

The ultimate aim of DeFi and the crypto space is mass adoption, but until we are able to provide a safe and accessible environment for everyone (without a seven paragraph introduction explaining how something works when it goes wrong) we are not going to reach that goal. It’s our mission to work with the wider community to make this a reality, to collaborate where we can, to make improvements across the blockchain space, to innovate and improve our user experience (and confidence) and to achieve our goal of making DeFi accessible for all.

About DeversiFi

DeversiFi makes DeFi easy. Swap, Invest and Send in your own DeFi control centre.

Website: https://rhino.fi/

Twitter: https://twitter.com/deversifi

Discord: https://discord.gg/bfNDxZqPSvf