The Solana network has gone down eight times throughout its two years of existence. In 2022 alone, the network experienced five outages that repeatedly sent chills down the spines of investors, developers, and the community.
Solana’s degraded performance isn’t something new. In fact, the first Solana outage occurred back in December 2020, when there were barely any users on the chain. You can only imagine how ‘heavier’ its network load is in 2022, as it peaked at 1.1M daily active users in the second quarter.
The fear of another potential network freeze remains high for builders developing platforms on the chain, as well as end users who utilize Solana-based decentralized apps (dapps) on a daily basis. The next mandatory day off, so to speak, could happen at any moment, it seems, as the network becomes more unpredictable.
So how bad is the outage situation? When will this be fixed? Is it something that users need to accept as the norm? We will address all of that in this Solana outage full report.
This guide aims to get to the bottom of Solana’s downtime issues, learn what caused these failures, and uncover the ultimate solutions that are being proposed.
Why is Solana Having Outages?
Solana is experiencing multiple outages because of problems that emerge from high network demand coupled with relatively few validators to handle the load. That, and the bugs in its blockchain system.
The network suffers every time transaction inflows become abnormally high, which normally happens whenever someone deliberately attacks Solana (DDoS attack) or utilizes bots to gain an advantage in non-fungible token (NFT) minting events.
Unlike Ethereum, Solana has ultra-low fees (roughly $0.00025) that make it really affordable to launch an attack on the network or flood it with bots.
But this issue can be potentially mitigated if the network has more validators, as each node would not be overwhelmed by the overflow. The common reason why Solana has few node operators is because of its expensive validator hardware requirements, which have pros and cons to it.
To run a Solana validator node, you need a device with at least 12 cores/24 threads CPU, 128GB RAM, and 500 GB disk space. RPC node recommendations are even more demanding. Check out the full hardware requirements here if you wish.
The pricey requirement repels bad actors, but, as a consequence, also severely limits the number of participants that can help the network handle its massive transaction demand. This is a huge predicament since more validators translate to not only better performance and less probability of downtime, but also better decentralization.
Solana’s longest outage incident lasted for 17-long hours, which occurred last September 2021. The shortest Solana downtime happened on January 2022, which interrupted the network for four hours.
List of Solana’s Outage Incidents
We tracked the full history of Solana’s downtime events from the first occurrence down to the latest.
| Incident | Date | Reason |
| 1st Outage | December 4, 2020 | Mainnet cluster failure to produce blocks |
| 2nd Outage | September 14, 2021 | DDoS attack or bots |
| 3rd Outage | December 2021 | Alleged DDoS attack |
| 4th Outage | January 2022 | DDoS attack or bots |
| 5th Outage | March 2022 | Bug causing RPC nodes to fork |
| 6th Outage | April 30, 2022 | Bots |
| 7th Outage | June 2022 | Bug |
| 8th Outage | September 30, 2022 | Consensus Bug |
Let’s look at each downtime occurrence in greater detail.
Solana Outage 1st Occurrence
Date of Occurrence: December 4, 2020
Barely nine months after its launch, the so-called ‘Ethereum killer’ suffered a six-hour outage incident due to the failure of its Mainnet Beta cluster to produce new blocks. With its operations halted, the blockchain’s ability to confirm new transactions was paralyzed.
The incident, which happened on December 4, 2020, was immediately resolved the next day through Solana’s 200+ validators, who restarted the network.
Solana Outage 2nd Occurrence
Date of Occurrence: September 14, 2021
On September 14, 2021, Solana experienced a 17-hour outage on its network due to distributed denial-of-service (DDoS) attack. This type of cyber attack involves flooding a network with overwhelming traffic to paralyze its operations.
In Solana’s case, Grape Protocol conducted an initial DEX offering (IDO) on Raydium, in which 400,000 transactions per second (tps) flooded that operation. The incident was so severe that it became the network’s longest outage so far.
The protocol’s validators opted to restart the network (again) to mitigate the problem in the most convenient way possible.
Note: Grape is a web3 social networking protocol, while Raydium is an automated market maker (AMM) and liquidity provider on the Solana blockchain.
Solana Outage 3rd Occurrence
Date of Occurrence: December 2021
Solana hasn’t logged this event on its Incident History, but on December 2021, there were reports that Solana had suffered again from a DDoS attack, temporarily clogging the platform.
But unlike the previous incident, the network appeared to have remained online and hasn’t shown any severe downtime.
Blockasset, a Solana-based NFT project, detected the incident and alarmed the community about the supposed DDoS attack.
There were conflicting claims about whether or not the network has suffered from a global outage again. It’s almost a year since this alleged downtime occurred, but it seems that the network doesn’t recognize this as an ‘official’ outage.
Solana Outage 4th Occurrence
Date of Occurrence: January 2022
Barely a month after the last reported outage, Solana hogged the headlines again when it suffered from protocol problems causing network instability and degraded performance.
There were conflicting updates during this time, with Wu Blockchain, a web3 news outlet reporting a 4-hour network outage, while Solana Status maintained its stance of a 100% operational platform.
Anatoly Yakovenko, one of Solana’s co-founders, denied that another DDoS attack caused the incident. Solana Status has logged this issue two days later on its Incident History as a mere ‘performance degradation’.
Solana Outage 5th Occurrence
Date of Occurrence: March 2022
There weren’t many reports about this particular network problem, but the protocol logged this issue on its incident records. According to the data, the platform suffered from a degraded performance last March 10 and 21. It also registered that on March 28, some of its remote procedure call (RPC) nodes were down.
While not as serious as the other occurrences, this was caused (partly) by a faulty update that caused some RPC nodes to fork off.
Solana Outage 6th Occurrence
Date of Occurrence: April 2022
On April 30, 2022, Solana reported that its Mainnet Beta cluster had frozen, preventing the network from creating new blocks and reaching consensus.
Validators later discovered that it was caused by the flooding of transactions in the network, clocking at 6 million tps, equivalent to over 100 Gbps of traffic. As a result, the Solana community suffered a seven-hour outage.
But Solana quickly clarified that there was ‘no evidence’ of a DDoS attack. Instead, it blamed the bots for the incident, which were programmed to win an NFT mint at Metaplex’s Candy Machine (an NFT launch program). The network was later restarted to resume the platform’s operation.
The downtime occurred a couple of days after the Okay Bears launch, which broke records as the most hyped (and botted) Solana mint at the time. Coincidence? Probably not.
Solana Outage 7th Occurrence
Date of Occurrence: June 2022
This time, the outage lasted for four and a half hours due to a bug that generated different outputs that caused a major consensus failure.
Yakovenko explained that the incident stemmed from the ‘durable nonce intersection’, a feature that prompted a part of the network to consider a specific block invalid. To prevent this incident from happening again, this feature was disabled as the network restarted.
Solana Outage 8th Occurrence
Date of Occurrence: September 30, 2022
Last September 30, 2022, Solana continued its ‘Outage Saga’ with its latest downtime episode, which lasted for six hours. This time, the network halt was caused by a misconfigured node, freezing the protocol’s entire operations.
Solana Status initially reported that the platform was experiencing degraded performance and later updated it as a full-blown outage due to a consensus algorithm bug.
Will Solana Be Able to Fix Its Outage Problem?
The Solana founder himself (Yakovenko ) assured the community that fixing the outage issues is the “number one priority” of Solana Labs. It has introduced three major upgrades to help contain DDoS attacks and bots and proposed easier ways for users to run validator nodes.
As of press time, there are two solutions (stake-weighted QoS and Fee markets) that are “in progress” and one (QUIC) launched on testnet.
If there is one thing that can significantly reduce SOL’s outage problems noticeably in the short term, it would be the speedy increase of its validators. No one can deny the power of numbers, as Ethereum, which currently has 8,373 nodes in its network, remains to be one of the most robust blockchain networks in the field. In contrast, Solana merely has 2,065 validator nodes despite handling 30-40x the transaction volume of ETH!
As a response, the Solana Foundation opened its server rental program to bring down the cost for would-be validators. While this isn’t the best compromise since the servers will still be highly centralized (controlled by fewer entities), it will help the performance massively if more people can become independent validators.
In the long run, this issue will still sort itself out as the cost of running a server is trending down thanks to Moore’s Law. But patience isn’t the primary virtue of most web3 users.
As for the systemic bugs, the current solution is to have independent teams building separate validator software, such as Firedancer by JumpCrypto. This is because the possibility of having the same bug in different validator software is nearly zero, which prevents network-wide contagions that lead to downtimes.
