By Vlad Estoup and Pedro Aisenson

The week starting on December 4th, 2023 was marked by the discovery of a widespread, critical smart contract vulnerability affecting thousands of contracts using meta transactions and Multicall. A small, committed group of blockchain security experts devoted their efforts around the clock to ensure the ecosystem's security. Here is a recount of their efforts and learnings in scanning and triaging through millions of verified contracts on 6 different chains to find vulnerable instances and assist the affected teams.

Table of Contents

The Response

1. 1. Code Red

   2. Initial Panic

   3. Monitoring and Triaging

   4. The First Incidents and Going Public

   Lessons Learned

   1. Libraries and secure standards aren’t plug-and-play

   2. Timely, responsible communication

   3. Ability to Respond to Vulnerabilities and Incidents

   4. Ecosystem-Wide Efforts

The Response

I. Code Red 

On Monday 12/4 at 4pm ET, the OpenZeppelin team was informed by Thirdweb about a vulnerability stemming from the combination meta transactions (ERC-2771) with a self-delegate call using user-controlled data, often seen in the Multicall standard. For a full technical explanation, check out our in-depth technical disclosure. 

At the time of the initial private disclosure, we realized the vulnerability had been out in the wild for some time, and several projects were at risk.

II. Initial Panic

The initial public disclosure by Thirdweb incorrectly inferred the vulnerability originated  “in a commonly used open-source library in the web3 industry”, which erroneously implicated the OpenZeppelin Contracts library. This caused widespread, unjustified panic in the community. 

We had to be careful about how we explained the vulnerability and reassured the community about the safety of the OpenZeppelin Contracts library - the last thing our team wanted was to arm hackers with information they could use to exploit projects. On Tuesday, our team worked deep into the morning hours on a reassuring message. It had to be informative and solution-oriented while remaining vague about the specific nature of the issue. 

While OpenZeppelin libraries are battle-tested and remain the safest for smart contract development, all software can be prone to bugs and it’s impossible to guarantee safe interoperability when users make integrations. As such, the initial claims about a bug in a widely used library were based on a misunderstanding of the library's security scope. Both ERC-2771 and Multicall do not require or ensure mutual compatibility. The vulnerability, observed in many different implementations of these standards, highlights the risks of incorrectly integrating individually secure components, emphasizing the need for careful architecture design and thorough security audits.

That said, our commitment to safeguarding the blockchain community and ecosystem compelled us to lead the effort: assess who in the community was affected, provide them with mitigation strategies, and perform a responsible disclosure at the appropriate time. 

III. Monitoring and Triaging

The playbook was to identify who could be at risk, inform their teams, and help them implement mitigation strategies. 

Detecting which contracts were affected was a challenge: neither ERC-2771 nor Multicall specify any events, requiring a novel approach to our existing incident response monitoring tools focused on event identification. To overcome this hurdle, we successfully employed an alternative method, relying on the querying of code patterns within verified contracts.

For additional support and community involvement, we created a war room with some of the most reputable white hats in the space, including Samczsun, David Benchimol from Ironblocks and Yannis Smaragdakis from Debaub. The whitehat team quickly began reaching out to major protocols that we thought could be at risk. Yannis from Debaub and the Ironblocks team took the lead in running queries to identify instances affected by the combination of ERC-2771 and self-delegate calls. Simultaneously, our Contracts and Incident Response team also did queries, enabling OpenZeppelin to generate a list of addresses for urgent triage.

First, we reviewed all of our Audit clients’ codebases, filtering for those implementing ERC-2771. We also monitored all OpenZeppelin Contracts users that were verified on Etherscan – highlighting the importance of verifying your contracts. Whenever we got a hit, our team conducted a deeper investigation to determine whether the particular protocol was affected. Affected teams were then contacted through private channels, and given specific steps to protect themselves. We felt it was still early to go public and kept a close look around the ecosystem for any potential exploits that could leak information.

In parallel, the OpenZeppelin Contracts development team worked on a new release of the library that would ensure safe interoperability between ERC-2771 and Multicall.

To assist projects reaching out or asking on Twitter if they were affected, we provided them with a self-service triaging solution to ascertain if they were vulnerable. The team released a detector within Defender’s Code Inspector tool within just 3 hours, used by 1000+ protocols. Those that tested positive had a direct line of communication with our team.

Urgent undisclosed vulnerability notification within Contract Inspector reports

Thanks to all these efforts, teams like Velodrome Finance acted swiftly to protect their protocol, making a public announcement shortly after. 

IV. The First Incidents and Going Public

On Wednesday – 40 hours after the first alert – the next step before public disclosure was to notify key community members, particularly security representatives from EVM chains and Layer 2 networks. We conducted the outreach in private, ensuring they could monitor the issue within their communities. Some of them made public statements afterward.

Having already informed the EVM chains, most of our Audit clients, and the list of identified vulnerable contacts, we were informed that several pools holding almost 90 ETH had been hacked. Time was running out. The first hack observed on-chain was that of a liquidity pool involving the Time contract, a token using Thirdweb’s code. Luckily, the attack was front-run by an MEV bot. This information was then leaked on Twitter and shortly deleted due to backlash from the whitehat community.

By now we faced the quintessential dilemma when racing against hackers. Public disclosure of the security issue could warn unalerted protocols, but it would certainly inform black-hats at the same time. Recognizing that skilled hackers could likely infer the issue from recent mainnet attacks, and considering our exhaustive private outreach efforts and the temporary public exposure of certain tweets, we concluded that alerting protocols in public was a net positive. Our decision was reinforced after consulting with leading whitehat experts, who concurred with our assessment.

After the public disclosure, the response from the security community was swift and collaborative. The widespread sharing of the findings by multiple security companies and experts significantly amplified the reach of the message, ensuring that a vast number of protocols were alerted to the potential risks. 

This experience serves as a testament to the effectiveness of proactive and coordinated action from multiple parties in safeguarding the ecosystem. Looking forward, it is clear that the lessons learned would be invaluable in shaping future security practices and vulnerability management strategies in Web3.

Lessons Learned

I. Libraries and secure standards aren’t plug-and-play

Even well-audited, secure code can become a liability when combined in unforeseen ways. When importing dependencies, be sure to know how they work on the inside and think about how you’re combining them with other functionality. 

II. Timely, responsible communication

Managing a crisis like this one requires precise, measured communication that informs without inducing undue alarm. To ensure timely and private security notifications, make sure to verify your contracts and add a security contact as a comment at the beginning of each one. For some reachouts, we lost 24 hours due to Twitter DMs being the best line of contact. 

III. Ability to respond to vulnerabilities and incidents

For effective incident response, make sure that your contracts are upgradable and you have circuit breaker mechanisms in place.

IV- Ecosystem-wide efforts

Effective, proactive and coordinated action from multiple parties are key to identify, react, and mitigate ecosystem-wide vulnerabilities.