Bootstrap Hell

In epoch 28, we received a schedule to create 12 blocks (3 times more than epoch 27). With three times more blocks (more potential rewards), the DApp360 stake pool team was upbeat. All we had to do was create all 12 blocks. During this epoch, it did not take long before our team realized it was going to be a rocky one. The problem started with bootstrapping our node. We restarted our node because it was stuck. When we restarted the node, it took almost 1 hour to bootstrap the node. In the past epochs, bootstrapping a node normally took 3 to 10 minutes. The trusted nodes of the Incentivized Testnet (ITN) were not bootstrapping the DApp360 stake pool node. It seems other nodes, in the Incentivized Test, quarantined our node. While bootstrapping for an excessively long time, our node missed its scheduled block. It seems like we were back to babysitting the node again because the network was becoming more and more unstable.

In this epoch, I stayed awake till 2:30 am eastern time, trying to make sure the DApp360 node did not miss any more blocks. By 2:30 am, our node had missed two blocks and created 5. The remainder of the epoch got worse as a result of the many adversarial forks. PoolTools.io displayed some of the adversarial forks from several stake pools, including Lion, Lion4, etc. These adversarial forks caused our node to miss 4 of the five remaining blocks.

DApp360 Block Real-Time Block Height

Again, for the second consecutive epoch, our node performance dropped from 75% in the last epoch to 50% percent in this epoch. Our team was now on edge that we might lose more delegates.