Sepolia Incident | Ethereum Foundation Blog

This blog post discloses a threat against the Ethereum network that was present from the Merge up until the Dencun hard fork.

Background

Prior to the merge, different message size limits for RPC communication were set to protect clients from denial-of-service (DOS) attacks. These limits, applied to messages received via HTTP endpoints, were carried over to the engine API, which plays a crucial role in connecting Execution and Consensus Layer clients during block production. Due to the engine API’s involvement in block production, it became possible for blocks to be produced that surpassed the RPC size limits of some clients but remained within the acceptable range for others.

If an attacker creates a message that exceeds the size limit of the client with the lowest setting, while still adhering to the gas limit requirements, and then waits for a block to be produced, it could result in a situation where some clients regard the block as valid, while others reject it, issuing a HTTP error code “413: Content Too Large.”

Impact

An attacker that could craft these messages would be able to force the majority of nodes (=geth) to reject blocks that a minority would accept. These blocks would be forked away and the proposer would miss out on rewards.

In the beginning we thought that it was only possible to create these blocks by using builders or a modified version of a client. Geth has a builtin limit of 128KB for transactions, which means that a big transaction like the one under discussion would not end up in the transaction pools of any geth node. It was however possible to still trigger the limit by having a client with a higher limit propose the block and the CL requesting validation of this proposed bigger block.

We proposed a solution in temporarily lowering the RPC limit on all clients to the lowest value (5MB). This would make the block invalid and an attacker would be very limited in the chaos they can cause in the network since the majority of the nodes would reject their blocks.

However on February 7th we discovered that it was possible to create a block that would hit the 5MB limit with a bunch of transactions that are below the 128KB limit and not exceed 30 million gas.

This is a bigger issue because we realized an attacker could create a bunch of high paying transactions and send them to the network. Since he outpays everyone else in the mempool, every node (even geth nodes) would include the attack transactions in their block thus creating a block that would not be accepted by the majority of the network, resulting in a lot of forks (all being deemed valid by the minority nodes) and the chain keeps reorging over and over again.

Later on February 7th, we came to the conclusion that everyone raising their RPC limits would be the safer alternative.

Timeline

• 2024-02-06 13:00: Toni (EF), Pari (EF) and Justin (Besu) try to submit a specificly grinded transaction to the network. The transaction contributes to up to 2.7 MB blocks when snappy compressed.
• 2024-02-06 13:25: Pari receives errors from his local Geth node although the transaction should be valid.
• 2024-02-06 15:14: Justin managed to put the transaction in a block and submitted it through the Besu client.
• 2024-02-06 20:46: Sam (EF) alerts Pari (special thanks to mysticryuujin on X), Toni and Alex about certain Sepolia nodes struggeling.
• 2024-02-06 21:05: Team double checks with Marius from Geth and confirms the bug.
• 2024-02-06 21:10: The gang gets together to debug it
• 2024-02-07 23:40: We decided for all clients to limit their RPC request limit to 5MB
• 2024-02-07 6:40: We discovered that there might be a bigger issue and the attack can be executed with transactions less than 128KB size.
• 2024-02-07 10:00: We decided for all clients to increase the RPC request limit.
• 2024-02-07 21:00: The fix was merged in geth.
• 2024-02-09: Geth was released

While Geth was the only client affected by this bug, other clients have also updated their defaults to be safe of this attack even if gas limits are increased.
The client teams indicated that the following updates have the safe rpc limits:

Geth: v1.13.12

Nethermind: v1.25.4

Besu: 24.1.2

Erigon: v2.58.0

Reth: v0.1.0-alpha.18