Every day, hundreds of Web3 project founders and builders hold their breath, hoping that they’re not the latest victim in what feels like an endless stream of hacks. Developers lacking a robust incident response plan are often powerless to intervene or implement remediation strategies. In this article, we will explore strategies of how Web3 protocols and developers can equip themselves to respond to security incidents.

Incident response means preparing for unknown threats and responding to those threats when they occur. Although many developers are familiar with these tools from the Web2 world, many Web3 companies are unaware that security solutions exist which harmonize with the Web3 ethos and are effective against the unique challenges posed by the blockchain ecosystem.

Security in Emerging Technologies

As new technologies emerge, they invariably attract hackers eager to exploit vulnerabilities in protocols that are still in their infancy. For many startups, the relentless pace of innovation often relegates security to the sidelines, seen as a resource-intensive hurdle.

Security audits are crucial at weeding out potential security issues. However, the Web3 ecosystem, which is constantly evolving, demands a more dynamic approach to security.

The shift in Web3 security best practices requires developers and protocol teams to not only perform thorough security audits but to also take steps beyond that. Developers and protocol teams need to incorporate the post-audit security measures of security hardening via monitoring and incident response.

Lessons from Web2

Hacks have been around since the invention of telephone systems. Even as recently as 5 years ago, a hacker could jump around a Fortune 5 company undetected with nothing more than a little PowerShell and some basic JavaScript obfuscation. However, over time, Web2 security has evolved with companies like Microsoft and CrowdStrike developing AI-driven threat detection tools to detect and block threats before or as they happen. Companies like Mandiant provide organizations with the ability to rapidly respond to and recover from security incidents. 

Web2 security is critical to Web3 security. Web2 concepts such as secure key management and phishing prevention ensure that Web3 systems remain available and maintain their integrity. OWASP provides a best practices guide for key management which has a Key Management Lifecycle Best Practices section that is of particular interest for managing wallet private keys. Phishing prevention requires a defense-in-depth approach with a combination of technical controls and user awareness. This article from Microsoft highlights key strategies organizations can take to defend against phishing attacks. Phishing isn’t a matter of if, but when, so it is critical to have a detailed incident response plan to deal with successful phishing attacks and the technical capabilities to respond to phishing incidents. This blog post by Wallet Guard highlights some of the common trends for phishing attacks in Web3.

Security Parallels between Web2 and Web3

Threat Detection

Threat detection in corporate networks has evolved over time but today relies heavily on the identification of known malicious techniques and anomaly detection. A common example of a known malicious technique is the use of the CreateRemoteThread Windows API to access the memory space of another process that contains an image not backed on disk, a common indicator of DLL injection. 

The Web2 space has had the benefit of decades to analyze and identify these techniques, whereas Web3 hasn’t had nearly as long to identify emerging threats. Forta, Web3’s security network, has already taken this concept and applied it to the blockchain space by supplying security teams with common indicators of malicious activity, such as an EOA being funded by Tornado Cash. Other indicators, like Suspicious Contract Creation, trigger alerts based on common bytecode patterns. This is akin to Web2 detections that rely on static analysis of shellcodes to identify common remote access trojans.

Anomaly Detection

Traditional threat detection tools gather data from a variety of sources across the network. When an attacker does something out of the ordinary, like accessing a file share they have never accessed in the past, or logging in from a distant location compared to their usual spot, the system flags the behavior as atypical. It then integrates this information with other data to make a determination on whether a threat is present. The cornerstone of anomaly detection is establishing a baseline for normal activity and employing various heuristics to pinpoint deviations. In Web3, this task is more arduous due to the limited data available for making decisions on abnormality. Nonetheless, Forta bots such as the forta-ice-phishing-forta-kit bot are designed to detect irregularities in token approval processes. As more data accumulates, anomaly detection in Web3 will continue to advance.

Forta is heavily utilizing anomaly detection with AI to detect threats on-chain, before, during, and after they happen. On many occasions, such as the Euler and Yearn Finance exploits, they have detected threats before they happen with enough time for the protocol to take action, but these alerts often go unheeded or the protocol is unprepared to execute a mitigation or prevention mechanism.

In the case of Euler, three Forta bots generated alerts related to the exploit. With the last, and most high fidelity, one generated about three and a half minutes before the exploit transaction was executed.

The first alert indicates that the attacker’s EOA was funded by Tornado Cash, which is suspicious but insufficient to take action on. As in Web2, the use of a known malicious technique is not always a clear indicator that something nefarious is going on.

The second alert is high severity and shows that a suspicious contract was created, flagging one of the primary exploiter contracts. This is a stronger potential indicator, although this alert is prone to generate false positives. As more data becomes available and more indicators are identified, the false positive rate for these types of alerts declines.

The third alert is an attack detector that combines a set of heuristics and anomaly detection models to further reduce false positives.

This alert is actionable and correctly identifies the Euler protocol address as one of the primary addresses involved. 

Allowlisting

Allowlisting and blocklisting concepts exist in both realms. In Web2, allowlisting is most commonly used in the form of application allowlisting, where only specific applications are allowed to run on an endpoint and other applications are blocked from execution. In Web3, this concept most commonly exists as a mechanism for projects to give a subset of users early access to a protocol, but it can also be a useful tool for restricting feature access to a trusted set of addresses.

Blocklisting is the opposite of allowlisting and assumes all addresses are allowed until specified otherwise. It is most commonly used to prevent known malicious addresses from interacting with a protocol or transferring tokens.

Both of these techniques are fairly easy to implement in a smart contract and can be automated using OpenZeppelin Defender. By adding a blocklist mapping and a require statement that checks the blocklist to a central entrypoint, or a specific function for more granularity, it’s simple to prevent an address from interacting with specific functionality within a protocol. Whether or not these measures are suitable for a particular company is a decision that each entity must make on its own.

mapping(address => bool) public blocklist;

function mint(address to, uint256 amount) public {

    require(!blocklist[msg.sender], 'address-blocklisted');

    _mint(to, amount);

}

function addToBlocklist(address[] _threats) external onlyAdmin {

    for (uint i=0; i<_threats.length; i++) {

        blocklist[_threats[i]] = true;

    }

}

Once the code is added to a contract, automation can be set up in Defender to respond to a certain event, such as an alert from the attack-detector Forta bot, and take an automated action on your contract, such as adding the flagged address to the blocklist. Doing so will prevent the exploiter contract from interacting with the contract, potentially preventing a loss of funds.

This comes with some drawbacks discussed below. Additionally, there are some limitations on the effectiveness of this technique, such as if the primary exploiter contract creates another malicious contract at runtime that is the contract actually interacting with the protocol, but those specifics are beyond the scope of this post.

Bug Bounties

Bug bounties are essentially public penetration tests and this concept exists 1:1 in the Web2 space on platforms such as HackerOne. OpenZeppelin works with Immunefi for bug bounties on OpenZeppelin Contracts, and there are several similar options to help with post-deployment security.

Transaction Simulations

Transaction simulations are the Web3 equivalent of sandboxing. The goal of sandboxing is to analyze the output of allowing a program to run in a controlled environment and making a decision on the maliciousness based on the outputs instead of just static indicators. In Web2, the outputs may be making network connections, running exploits, or searching for sensitive files. In Web3, these outputs are typically state changes such as the transfer of funds or a change in the ownership of a contract. Transaction simulations are a much more high fidelity way to identify malicious activity because the end state is known and can be used to evaluate maliciousness. Forta’s attack-simulation-bot even has “sandbox environment” in the description. The following screenshot is a transaction simulation of the Euler exploit using Tenderly. Note the state changes following the simulation, which can allow for more informed decisions on the maliciousness of transactions.

Pausing

A discussion on incident response in Web3 wouldn’t be complete without a discussion on pause functionality, the most commonly used incident response technique today. Pausing has become the default response and can occur at a broad level with the whole protocol being halted or at a more granular level where individual pieces of functionality are paused. The Web2 equivalent of this is bringing down an entire service or network. While it’s certainly an effective measure if executed before the protocol is exploited, it creates a denial of service condition for all other users and trades off integrity for availability. 

Recovery

Recovery in both Web2 and Web3 is a challenge, requiring root cause analysis, patches and upgrades, and many stressful hours to ensure that the exploit is mitigated. Oftentimes, one exploit is fixed just for another one to be created or a workaround identified. Private companies and individuals assist in identifying the threat actors and depending on the jurisdiction, law enforcement gets involved. Some protocols and organizations never recover successfully and lose the trust of their customers. Others build back stronger and take the lessons learned to prevent it from happening again.

Takeaways

With so many similarities there are some key lessons that can be learned from the evolution of traditional Web2 security.

1. Human behavior will always be a weak spot: The easiest thing for any attacker to exploit is human behavior and phishing will always be a favorite method of compromise. As an industry, Web3 users must be aware and informed of common phishing techniques and the industry must adopt best practices, such as cold hardware wallets and asset segregation to see continued user growth. On the tooling side, tools like revoke.cash and WalletGuard are critical in making the end user more secure and aware of potential social engineering attacks.
2. Force attackers down a narrow path: Security is a relentless cycle: attackers discover a successful method, use it until defenders adapt, and then switch tactics, leaving defenders scrambling to keep up. A significant shift in traditional security happened when defenders shifted focus to memory analysis instead of solely analyzing binaries on disk. This change countered attackers who had been using memory-based techniques with basic obfuscation to evade detection. As defense tools started terminating processes based on memory signatures, it posed new challenges and forced attackers to adapt, narrowing their options. 
3. Web3 can embrace a comparable strategy to enhance security and progressively restrict options for attackers. The handling of reentrancy in Web3 serves as an excellent example. Due to its prevalence, the check-effects-interactions pattern has become a staple among Solidity developers, compelling attackers to seek alternative avenues for easy exploits. 

• Refine anomaly scope: What looks normal across an entire industry is very different from what looks normal within a smaller scope, such as a protocol or a traditional corporate network. Take for instance the use of Microsoft Word, which is widespread across global corporate networks. However, if an organization primarily uses Google Docs, observing a user frequently launching Microsoft Word would be quite unusual within that specific context. This principle also holds true in Web3: behavior typical for one protocol might be atypical for another. The more tailored the anomaly detection, the more effective it is. For example, there might be distinct bytecode patterns common in contracts interacting with Uniswap as opposed to those interacting with CowSwap. Analyzing such differences is key to understanding how malicious bytecode patterns may be particular to certain protocols.

There are many other examples just like this that we can learn from traditional security but despite the various similarities with Web2 incident response concepts, blockchain technology and Web3’s ethos present a unique set of challenges for responding to security incidents.

Challenges of Web3 Incident Response

Two properties of blockchain technology that make it so incredible also make it challenging to respond to security incidents: speed and values.

Exploit Frontrunning

In traditional cybersecurity incidents, attackers may sit in a network for days, weeks, months, or even years before accomplishing their overall objective. It takes time from an attacker to learn a network, blend in with their environment, identify their target, escalate their privileges, move around the network, and accomplish their objective. Browsing a single drive for sensitive files can easily take days or weeks in a large organization if done carefully. 

In Web3, attackers don’t have to worry about the hurdles of network penetration and attacks often happen in seconds or minutes with millions of dollars gone in a single transaction. All the code is public and crafting an exploit can be done long before any need to interact with the protocol on a public network. This makes incident response extremely challenging because attackers often accomplish their entire objective, taking funds, before there is an opportunity to respond. 

Techniques like frontrunning are possible and often effective but have a very short time window in which they need to be executed. This requires proactive threat detection and automation to ensure that the mitigation action is executed before the exploit transaction, which can be accomplished with Forta and OpenZeppelin Defender. Protocols may want to be even more proactive by doing mempool monitoring to identity potentially malicious transactions before they are executed 

Incident Response Frontrunning Protection

When executing incident response actions it may be beneficial to protocols to prevent users and bots from seeing the transaction before it is fully validated. If an attack is in progress and the protocol tries to frontrun the malicious transaction with an incident response action but submits the transaction to a public mempool, it’s possible for another frontrunner to cause a significant loss of funds by frontrunning the incident response action with another malicious transaction. Using a service like flashbots prevents your transaction from being submitted to a public mempool so that frontrunning bots are unable to see the transaction before it is validated.

Designating Security Roles

Many times administrative actions, including IR actions, are executed by a trusted group of users on a multisig (more about this shortly) but the average time to execute a transaction on a multisig is 24 hours, at which point the attack has long been successful and over. Automation can help solve this problem by alerting multisig signers of pending transactions to provide a more immediate alert system. 

Another potential solution is to have the multisig designate specific incident response actions to a Relayer through access control. This helps to maintain the decentralized aspect of protocol management while allowing the flexibility for quick response. Due to the rapid pace that blockchain technology moves at, early proactive detection, prevention, and mitigation are crucial in Web3 incident response. 

Security Monitoring of Privileged Accounts

Web3’s value system can also pose significant challenges for attaining security. People love Web3 for a variety of reasons, but a significant number of us are here for the promises of an “open economy” that promotes transparency and decentralization. This open economy imparts a sense of fairness and equality where users can be free of censorship and use the technology in a permissionless way. Web3 ideals motivate developers to enable the user to remain in control.

Protocols vs End Users

This article has focused almost exclusively on addressing how protocols can implement incident response mechanisms, however, users can be just as proactive. Forta is an open network where users can subscribe to alerts and users can leverage Defender’s free tier without any financial requirements to help protect themselves where individual protocols may fail. Whether it's by creating automated systems to withdraw funds from protocols that might be on the verge of exploitation, or by identifying unfavorable market conditions, end users have just as much capability to protect themselves as the protocols do.

Transparent Blocklisting

The concept of blocklisting clashes with the fundamental ethos of some Web3 users. Blocklisting, a prevalent method today, involves protocols labeling an address as malicious and subsequently freezing its funds or barring its interaction with the protocol. This represents not only censorship but also reflects central authority intervention. While some may argue that censoring malicious actors is beneficial, this can be a contentious issue. Many community members are averse to protocols employing blocklists.

As a resolution, protocols can provide transparency to the allowlisting process by taking steps to decentralize. For instance, while a select group of trusted protocol holders could have the authority to add addresses to the blocklist, the wider community could vote on the retention of these addresses on the blocklist. Misuse of this power by the holders could result in their exclusion from the multisig. This system would, therefore, revolve around community-backed governance, striking a balance between rapid response and community involvement in blocklisting decisions.

Progressive Decentralization

Web2 organizations, having full control over infrastructure, code, personnel, and processes, can easily implement multifaceted security and threat prevention measures. Decentralization tends to slow things down, and in incident response, agility is key. DAOs face an even more pronounced version of this challenge. However, by initially adopting a more centralized structure and gradually transitioning towards decentralization as they grow, companies can find a middle ground that harmonizes the principles of decentralization with the imperative of swift incident response.

Rug Pull or Incident Response?

Certain incident response strategies resemble centralized rug pulls. For instance, transferring funds to a backup wallet, revoking access, and blocklisting might be incident response tactics but can also be indicative of centralized projects or contracts facilitating malicious activities. Upholding the “code is law” principle is crucial, but protocols need to incorporate safeguards against the inherent imperfections of code. Ultimately, the community will shape the landscape by supporting protocols with balanced mechanisms, and the stakeholders will recognize the long-term value in sustaining protocols rather than exploiting them.

Security Response Automation

Automation emerges as a recurring theme and often stands as the linchpin for effective incident response strategies in blockchain. During the audit process, it’s essential for auditors and protocols to flesh out specific risk parameters for which they would want to execute an incident response mechanism. These risk parameters can then be converted into alerting and automation that triggers an incident response process when these risk parameters are met. Timely detection forms the cornerstone of prevention, and automation fuels this process.

With all this in mind, incident response in the Web3 world requires a unique approach.

OpenZeppelin’s Approach to Incident Response

At OpenZeppelin, we collaborate with our clients to tailor an incident response strategy by helping them pinpoint threats within their contracts and the broader Web3 landscape. Together, we craft a comprehensive plan for identifying and defending against these threats and rigorously test it to ensure that they're well-prepared to take action when needed.

Organizations are diverse in their approaches. Some uphold the principle of decentralization without any compromise, while others consider it essential to implement a degree of control to maintain the security of the protocol and its users. Nonetheless, one undeniable constant in Web3 incident response is that speed is vital.

OpenZeppelin Defender offers features that empower organizations to transform their incident response playbook into automated sequences of actions. This ensures that when a threat is detected, the processes of mitigation, response, and recovery are rapidly initiated. As we dedicate ourselves to protecting the open economy in a manner consistent with Web3’s core values, the ecosystem’s proficiency in threat detection, prevention, and incident response within Web3 will only escalate.