The Challenges and Breakthroughs of Public Goods from the Perspective of Ethereum Infrastructure Iteration

Author: shew

Overview

As a public goods endowment fund, GCC is dedicated to advancing Ethereum infrastructure research. The most commonly used and public component of Ethereum's infrastructure is the Ethereum testnet. A testnet is a testing environment similar to the Ethereum mainnet, providing crucial infrastructure consistent with the mainnet. Because transactions within the testnet are virtually feeless, testnets benefit all developers, researchers, and users.

This report focuses on the history and current status of testnets. This research contributes to a broader understanding of the public nature of testnets, highlighting how collaborative testing infrastructure supports innovation while remaining accessible to the entire community. By documenting the evolution and functionality of these networks, we aim to inform future testnet initiatives and demonstrate the value of community-supported infrastructure.

Public TestNet and DevNet

Public testnets are the most commonly used testnets by developers and users. These testnets can be used to deploy smart contracts, and users can also obtain ETH from faucets to interact with contracts within the testnet. The public testnet deployment timeline is as follows:

The latest Ethereum testnet plans are missing from the above chart. The latest testnet plans are as follows:

The names of these testnets are particularly interesting, all named after subway stations or train stations. Ropsten and Rinkeby are both subway stations in Stockholm, Sweden, Goerli is a train station in Berlin, and Sepolia is a subway station in Athens, Greece. The latest Hoodi testnet is based on a train station in Bangalore, India.

In addition to the public testnets mentioned above, Ethereum also has a series of internal testnets for specialized testing. For example, the YOLO testnet is used to test BLS signatures and other features required for the Berlin hard fork upgrade. We generally refer to these testnets as DevNets. They are sometimes also referred to as Client Integration Testnets. These testnets are typically not open to the public and are typically used only by client developers and auditors to conduct internal testing. These tests aim to verify that different client implementations function properly and verify interoperability between different clients.

At All Core Devs Meeting 109, core developers discussed using geological fault lines as a naming convention for hard fork testnets. Following this principle, core developers launched the Aleut (representing the Aleutian Trench) and Baikal (representing Lake Baikal) testnets for testing the London upgrade. However, subsequent testnet naming practices have deviated from this convention. For example, after the Baikal testnet, another testnet used for testing the London upgrade was named the Calaveras testnet. However, core developers haven't always adhered to this naming convention. For example, the recent Fusaka hard fork upgrade opted for simple and clear naming, such as devnet-1 and devnet-2.

History of Public Testnets

Pioneering Era: Olympic and Morden

The Olympic testnet launched in early 2015 as Ethereum's first public testbed. This groundbreaking network, launched even before the official launch of the Ethereum mainnet, served as the final validation of the protocol's functionality. Its Chain ID is 0, reflecting its long-standing status.

The Ethereum development team established a bounty system where anyone who successfully stress-tested the network and created numerous forks would receive a 25,000 ETH reward. This bounty aimed to test Ethereum's limits through high transaction volume and extreme usage patterns. The chart below shows the EF airdrop to testnet participants after the Olympic testnet event concluded.

The Olympic testnet played such a crucial role in the early development of Ethereum that the genesis block of the Ethereum mainnet used block 1028201 from the Olympic testnet as a template. However, the testnet also suffered from numerous issues, including an excessively large state set and potential private key vulnerabilities, leading to its deprecation after the successful launch of the Ethereum mainnet in July 2015. The large state set issue stemmed from the Olympic testnet's inherent incentive to spam transactions, with users sending large numbers of junk transactions in pursuit of rewards, contributing to its enormous size. The potential private key vulnerability issue stemmed primarily from a vulnerability in the early Olympic testnet code that could have enabled replay attacks.

The Morden testnet was launched shortly after the Olympic testnet's retirement and launched simultaneously with the Ethereum mainnet in July 2015. Since the Ethereum mainnet's chain ID is 1, the Morden testnet chose chain ID 2. For over a year after its launch, Morden served as the sole testing environment and established itself as the primary development platform for early Ethereum applications. However, the network developed serious consensus issues. To mitigate potential replay attacks, all transactions within Morden began with nonce values starting at 2^20. EIP-161 modified some of the nonce rules, causing a conflict between Morden's existing nonce rules and those specified in EIP-161. This ultimately led to Geth and Parity creating incompatible blocks at block 1885074, leading to the deprecation of the Morden testnet. Notably, the ETC community embraced this abandoned network, renaming it "Morden Classic" and continuing it within the ecosystem. However, in 2019, the ETC community replaced Morden Classic with the Mordor testnet as its latest testnet.

The Ropsten Era and Security Challenges

Ropsten, the third iteration of the Ethereum testnet, launched in November 2016. Named after a Stockholm metro station and assigned Chain ID 3, this proof-of-work network boasted a more robust design than its predecessor and successfully supported all major Ethereum clients, making it a pillar of Ethereum's testing infrastructure.

Ropsten's most interesting moment came in February 2017, when it suffered a devastating denial-of-service attack. A malicious attacker exploited the network's proof-of-work mechanism to gradually increase the block gas limit from a reasonable 4.7 million to an astronomical 9 billion. The attack generated a flood of junk blocks, consumed significant disk space, made client syncing nearly impossible, and caused etherscan to display data incorrectly.

But the Ethereum community wasn't ready to abandon the Ropsten testnet outright. Supported by community-donated GPU computing power, the Ropsten team successfully restored the network in March 2017, clearing the accumulated junk blocks and restoring normal operations. The recovery process was simple: Ethereum developers used the community's donated GPU computing power to mine blocks from the period before the attack, resulting in a heavier chain that replaced the attacked chain. We can simply assume that Ethereum developers used this computing power to launch a 51% attack, successfully tampering with the chain's history to clear junk transactions.

This incident marked a watershed moment in the network's development, prompting the development of alternative consensus mechanisms and more robust testing environments. On June 8, 2022, Ropsten became the first major testnet to successfully complete a merge, marking the successful evolution of Ethereum testnets from proof-of-work (PoW) to proof-of-stake (PoS).

The Ropsten merge also encountered some issues. Simply put, Ropsten was scheduled to initiate the merge process when the block difficulty reached 43531756765713534, a value known as TTD (Terminal Total Difficulty). However, Ropsten's TTD was reached prematurely due to a malicious attack, before the Beacon chain, used for consensus in PoS, had yet to launch. To prevent further issues, core developers instructed Ropsten nodes to manually set the TTD to 10000000000000000000000000, pending further revisions by core developers. The details of this event can be found in the Ropsten TTD Postmortem and Ropsten testnet Merge. Ultimately, Ethereum core developers resolved the TTD issue, and Ropsten successfully completed the merge.

The Proof of Authority (PoA) Revolution

The Ropsten attack in February 2017 spurred the development of Proof of Authority (PoA) testnets, culminating in the Kovan testnet in March 2017. Created by the Parity team, Kovan represented a fundamental shift in testnet philosophy, sacrificing pure decentralization for security and stability. Named after a subway station in Singapore, the network adopted the chain ID 42 and maintained a 4-second block time.

Specifically, Kovan used the Aura algorithm, a simple algorithm that defines multiple trusted block producers that can produce blocks and rotates them based on time, with only one block producer producing blocks during each time period. Clearly, under the PoA algorithm, attackers cannot launch arbitrary DDoS attacks. In extreme cases, PoA block producers may simply reject any transactions from the attacker.

Despite its innovations, Kovan remains limited in client support, being compatible only with Parity and not with the broader Ethereum client ecosystem. This limitation meant that developers using other clients, such as Geth, were unable to fully utilize Kovan for testing, leading to a fragmented testing environment.

Rinkeby was launched in April 2017 in response to the Ethereum team's demand for a more general Proof-of-Authority (PoA) solution. Named after another Stockholm metro station, Rinkeby implements the Clique Proof-of-Authority (PoA) consensus engine, designed to minimize disruption to existing client codebases. With a chain ID of 4 and a block time of 15 seconds, Rinkeby provides a stable testing environment, making it easier to implement across different Ethereum clients.

Because Clique consensus is a general-purpose solution, Ethereum developers specifically described the consensus algorithm in EIP-225. This consensus algorithm reuses existing block header fields and introduces innovative governance mechanisms. The extra-data field was expanded to accommodate secp256k1 signatures, while the miner and nonce fields were converted to a voting protocol for managing the list of authorized block producers. This elegant solution allows for dynamic validator management via referendums while maintaining compatibility with existing technologies.

Rinkeby achieved remarkable stability, processing approximately 164 million transactions across 11 million blocks during its operational life. The final outcome of the Rinkeby chain was the decision by the Rinkeby maintenance team to no longer migrate to Rinkeby via a merge. The reasons for this can be found in this tweet. Simply put, Rinkeby was primarily maintained by Geth developers, and the network's long operation had accumulated a large amount of data, making a merge difficult.

Multi-client Breakthrough: Goerli

Goerli originated at the ETHBerlin Hackathon in September 2018. At the time, the ChainSafe team had initiated an interesting project to implement Parity's Aura Proof-of-Authority (PoA) consensus mechanism in Go for compatibility with Geth. This project gradually evolved into a formal initiative when Afri Schoedon collaborated with ChainSafe to create their envisioned "next-generation" public PoA testnet.

A core goal of Goerli is multi-client support. To achieve this, the development team first attempted to implement Parity's Aura algorithm within Geth. This was achieved at the ETHBerlin hackathon, but it proved too intrusive into the existing codebase. Ultimately, the Goerli team chose the Clique consensus engine for multi-client compatibility, having proven its worth on the Rinkeby testnet. However, the Goerli development team wrote a significant amount of code to support the consensus protocol across all major Ethereum clients.

Goerli officially launched on January 31, 2019, with Chain ID 5 and a block time of 15 seconds. The network achieved remarkable client diversity, supporting mainstream platforms such as Geth, Parity, and Nethermind. This broad compatibility made Goerli the first truly universal PoA testnet, addressing the fragmentation issues that plagued earlier networks.

The network's stability and reliability made it a top choice for application developers, infrastructure providers, and protocol researchers. As Ethereum prepared for its merger, Goerli was selected to merge with the Prater beacon chain testnet on August 11, 2022, successfully transitioning from PoA to PoS consensus.

Goerli's most serious issue was its limited supply of ETH, which led to severe ETH supply issues near the end of its life. Most developers and users developing on Goerli have to obtain test tokens from multiple faucets, or even pay to purchase them.

Modern Testnets: Sepolia/Holešky and Hoodi

Sepolia is also a PoA testnet, primarily maintained by ETHPandaOps. ETHPandaOps is a team focused on monitoring and optimizing Ethereum infrastructure. They are currently the core maintenance team for Ethereum infrastructure, primarily providing the following capabilities:

1. Network monitoring tools, providing data on the health, performance, and behavior of the Ethereum network. The group also maintains Xatu, a dataset on Ethereum network performance.
2. Testing tools, primarily for Ethereum core developers, help them quickly build DevNets containing multiple different clients to test the correct implementation of certain EIPs within their node software. A representative example of this tool is the ethereum-package.
3. Other tools. For example, ETHPandaOps has also developed the Funding-vault testnet token distribution tool and the Snapshotter node snapshot tool.

As a PoA testnet, Sepolia cannot conduct full PoS-level testing. Therefore, it focuses on execution-layer testing. Simply put, it's designed for smart contract engineers and users. Compared to the Goerli testnet, Sepolia's biggest advantage is that the supply of test ETH is uncapped, making it relatively easy for developers to obtain test tokens from the water and short positions.

Unlike Sepolia, Holešky and Hoodi are both public testnets, focusing on protocol-layer testing. Protocol-layer testing primarily involves testing PoS functionality, such as whether ETH stakers can exit normally. Holešky was once the preferred testnet for protocol-layer testing, but it suffered significant damage during the Pectra upgrade. Simply put, during the Pectra upgrade, the Holešky testnet was misconfigured, preventing three node clients from participating in consensus. This consensus issue resulted in a large number of stakers being slashed, leading to an extremely congested exit queue for PoS stakers. Ethereum core developers implemented a series of recovery plans, including client software updates. This was accompanied by a massive slashing campaign over the course of approximately two weeks, resulting in numerous faulty nodes losing all their funds and being unable to participate in PoS consensus. After the correct node software and faulty stakers were removed, the Holešky testnet eventually returned to normal operation. However, the Holešky testnet incident prevented it from participating in other tests of the Pectra upgrade until the end of February 2025. The most significant impacts include:

• Due to previous consensus issues, the Holešky testnet was in a state of disarray. Testers had to deal with consensus issues caused by previous misconfigurations, which further increased the difficulty of testing.
• Due to the large number of stakers being fined and exited, the exit queue was extremely large, making it difficult for testers to test staking exits. Staking exits are part of the Pectra upgrade.

To prevent disruption to the smooth launch of the Pectra upgrade, ETHPanOps launched the Hoodi testnet. Essentially, the Hoodi testnet and the Holešky testnet share the same responsibilities: verifying the proper functioning of protocol layers like PoS. However, the Hoodi testnet is cleaner than the Holešky testnet.

Currently, for smart contract engineers, the Sepolia testnet is the best choice for relevant testing. For protocol-level testing, the Hoodi testnet is the best choice, while the Holešky testnet is largely abandoned. Recently, ETHPanOps used only the Sepolia and Hoodi testnets to verify the impact of the 60MB block gas limit on Ethereum.

The Value Orientation of Bitcoin Testnets

Above, we introduced the basic history of Ethereum testnets. Each change in Ethereum testnets was due to technical reasons. The transition from Olympic to Morden was due to excessive spam transactions within Olympic; the transition from Morden to Ropsten was due to a split consensus issue within Morden; the transition from Ropsten to Rinkeby was due to Ropsten's vulnerability to denial of service attacks as a public testnet; the transition from Rinkeby to Goerli was due to Ropsten's completion of its historical mission after the Merge upgrade, resulting in the accumulation of a large amount of data that made it unsuitable for further operation; and the transition from Goerli to Sepolia was due to ETH supply issues and Goerli's operational lifecycle.

The transition of BTC testnets, on the other hand, is quite volatile. Bitcoin has historically had four major testnets, each simply named Testnet1 or Testnet2. While it's difficult to find relevant history for Testnet1 and Testnet2, the updates between Testnet3 and Testnet4 offer a number of interesting details.

Testnet3, introduced in Bitcoin Core v0.7, primarily addressed the issues of high difficulty and prolonged transaction confirmation delays on Testnet2. However, there are issues with the testnet3 code. For a detailed explanation of these issues, see lopp's article "Bitcoin Testnet Block Storms," published in April 2024. Simply put, the testnet3 testnet has a vulnerability that causes a block difficulty reset. The image below illustrates a block difficulty reset:

We can see that the testnet3 difficulty fluctuates greatly, occasionally causing a block difficulty reset to cause the difficulty to drop by seven orders of magnitude. Once this abnormal drop in difficulty occurs, testnet3 miners could theoretically mine a large number of blocks in a short period of time using ASICs or GPUs.

In April 2024, Lopp published a new article titled "Griefing Bitcoin's Testnet." This article describes how Lopp exploited a previously discovered vulnerability in Testnet3 to launch a fatal attack on the network. In the article, Lopp also expressed his intentions, specifically:

1. Testnet3 should be abandoned
2. BTC within the Testnet network should have no value

Of course, the direct cause of Lopp's attack on Testnet3 was his claim that the SatoshiVM project, which was building a ZK rollup for Bitcoin, wanted widespread user participation in the testing. Obviously, these users would need to obtain Testnet Bitcoin for gas and other purposes. This led to the sale of Bitcoin on Testnet3.

Lopp believes that testnet BTC should have no value, that any developer can obtain testnet BTC for free, and that testnet BTC should be used solely for development purposes. The SatoshiVM project clearly does not align with Lopp's values. To express his dissatisfaction and promote updates to the BTC testnet, Lopp launched an attack to mine a large number of testnet3 blocks.

As shown in the chart above, after Lopp participated in mining, the number of testnet3 blocks increased by 300% per day. After this operation began, Lopp observed protests from some projects selling testnet tokens and the Motoswap project. After investigation, Lopp discovered that Motoswap was operated by the BSV community. Lopp was not entirely satisfied with these user statements, so he increased the hash rate for mining testnet3 blocks, achieving the astonishing block production speeds shown in the image below:

This means that testnet3 is producing blocks at a rate 150 to 200 times faster than before, significantly impacting the infrastructure that relies heavily on testnet3. For example, after Lopp began attacking testnet3, the well-known BTC sidechain project Stack's testnet, which relied on testnet3, ceased to function properly. In the article "The Challenges of Building on Bitcoin Testnet," Stack stated that engineers spent significant time modifying the current testnet code to maintain the Stack testnet's functionality on testnet3, but were still unable to get the Stack testnet to function properly. Stack ultimately decided to abandon its testnet on testnet3 and use Bitcoin Regtest, a sovereign testnet maintained by the Stack team.

lopp wrote in his article:

Simply put, the fundamental reason for lopp's attack on testnet3 is that the BTC within testnet3 has begun to have value. Of course, as a Bitcoin Core developer, lopp also stated that the attack on testnet3 only used his own GPU and locally deployed nodes. At the same time, Lopp also stated that the testnet4 code is gradually being finalized.

At the end of the article, Lopp stated that the only way to prevent the accumulation of testnet token value may be to promote a culture of regular resets. Ideally, testnets should be reset regularly, which both reduces testnet state occupancy and prevents the accumulation of BTC value within the testnet. From this perspective, Ethereum's pre-planned lifecycle for each testnet, waiting for the testnet's lifecycle to expire and then automatically deprecate it, is a forward-thinking and effective design.

Finally, in Bitcoin Core version 28.0, released in October 2024, core developers added support for testnet4.

Similar to Ethereum, Bitcoin also has several other testnet types. In addition to the public testnets supported by Bitcoin Core, testnet3 and testnet4, as mentioned above, Bitcoin also has the following testnet types:

1. Bitcoin Regtest is also a testnet provided by Bitcoin Core, but it is a local testnet. Testnet owners can directly generate blocks using RPCs.
2. Bitcoin Signet is a hybrid testnet. This testnet is public but requires blocks to have specific signatures, similar to Proof-of-Authority (PoA). All blocks can only be approved with administrator approval.

As mentioned earlier, the Bitcoin testnet upgrade was, in some ways, a "brute force" attack. Lopp used the most extreme attack, mining testnet3 blocks, to force the Bitcoin testnet upgrade and express his personal wishes.

If an issue similar to the one on testnet3 were to arise within Ethereum, the core developers would likely invite stakeholders to discuss and resolve it in a core developer meeting. From some perspectives, this approach is more elegant and does not impact projects running tests on the testnet. However, from another perspective, this approach may be less decentralized. Lopp's direct attack on testnet3, however, expressed his personal will in the most decentralized way possible. Of course, the resulting disruption can only be considered transitional pains.

Public Goods and Isolation Theory

Both the Ethereum and Bitcoin testnets are public goods. In this section, we briefly discuss the economic properties of public goods and provide an economic explanation for some testnet behaviors. First, we need to define the nature of a public good: a public good is something that can be shared by multiple people without interfering with others' enjoyment. The definition of a public good is simple, but identifying which products are public goods requires a specific perspective. In many cases, public goods possess both public and private attributes. For example, NFTs can be appreciated by everyone for their artistic value, and the appreciation of some users does not affect the appreciation of others. From this perspective, NFTs can be considered public goods. However, NFT ownership is vested in a single on-chain address, so from this perspective, NFTs are not public goods.

Why are public goods often not subject to public fees? The answer to this question may not align with popular perceptions. Take the Ethereum testnet as an example. The value it brings to Alice might be 10, while the value it brings to Bob might be 1. As a public good, the Ethereum testnet can be shared by Alice and Bob. So, if the Ethereum testnet charges no fees, Alice receives a benefit of 10 and Bob receives a benefit of 1, for a total benefit of 11. However, if the testnet charges a fee of 2, Bob, who would have otherwise used the testnet, will abandon it because the fee is greater than its value. Therefore, only Alice receives a total benefit of 10. In the simple example described above, we observed that once the testnet imposed fees, some users who would otherwise benefit from the public good would withdraw, resulting in a decrease in overall benefits.

One possible solution is price discrimination, charging Alice $10 and Bob $1. However, within a large-scale network, this would incur significant costs to implement different fees for different users. Therefore, to achieve economic optimality, public goods must be free. Therefore, public goods are not inherently free; it's simply because differential pricing is costly to achieve global economic optimality that public goods must be free. The core issue here is the cost of price discrimination for public goods. If the cost is sufficiently low, public goods can be priced.

In economics, there's the concept of segregation theory, which addresses how to charge fees while maintaining the public nature of public goods. Segregation involves isolating some users of a public good from others at a cost and then imposing a fee on these users. First, for users who haven't been quarantined, public goods remain free to use. However, quarantined users must pay for them. Note that this payment doesn't necessarily mean direct monetary costs; it could also mean indirect costs like time.

Take the testnet, for example, as an example. The Ethereum testnet implements quarantine. Regular users can obtain test tokens from a public faucet. However, for users who need to consume large quantities of test tokens, the Ethereum testnet quarantines these users, requiring them to pay an additional fee to obtain a large amount of test tokens all at once. Currently, Ethereum uses the Funding Vault, a repository maintained by ethpandaops. Users seeking to obtain large amounts of test tokens for project testing can submit a set of supporting documentation to demonstrate their need for the tokens, and then receive them directly from ethpandaops.

The BTC testnet series also utilizes quarantine. Users can obtain small amounts of test BTC directly from a public faucet, but users with large test token needs are quarantined. These users can only obtain a significant amount of test tokens by running testnet nodes and expending mining resources.

We believe that providers of public goods may consciously or unconsciously employ segregation theory to ensure the proper use of public goods. While segregation theory sometimes appears disguised as "preventing the abuse of public goods," it essentially establishes a separate isolation mechanism. Of course, the specific users isolated and the methods used reflect the values of each testnet. Currently, both the BTC and Ethereum testnets generally agree that test tokens should not have value.

Summary

This article summarizes the evolution of all Ethereum public testnets and introduces the dramatic events of BTC testnet 3. Finally, we discuss the economics of public goods, emphasizing that the fundamental reason public goods are not subject to public charges is that any such charges would negatively impact the overall social good. Finally, we introduce segregation theory.