In today’s article, we’ll review some past discussions on the topic of network optimization, which resulted in improvements now lying at the core of the new index-based block synchronization method. In the approach, nodes can request missing block data in a single, compact message containing a range of desired block indexes.

Proposals to improve Neo’s network protocol date back as far as 2018, with various inefficiencies being identified. A number of suggestions from community developer Kevaundray noted several of these issues, focused on improving response times to node disconnects and reducing delays in block synchronization.

These suggestions lead to implemented improvements, such as the addition of ping/pong heartbeat messages for checking node height, but also inspired another solution. A proposal by COZ’s neo-python maintainer Ixje in December 2018 provided the first example of index-based synchronization. In the proposal, Ixje noted the current inefficiencies and semantic issues with the getblocks command:

“The current getblocks command is inefficient and also has a misleading name. The name suggests you will receive blocks. In reality, you get hashes (via an inv message). You then basically repackage the same information of this inv message but now with the getdata command to finally receive the blocks messages.”

Ixje proposed a more efficient command, getfullblocks, which would reduce network traffic and business logic for retrieving blocks by moving the process to be a two-step process. Rather than requesting headers, receiving headers, then using hashes to request and retrieve block data, nodes would be able to request the desired blocks and receive the corresponding data directly.

Taking the solution a step further, it was here that the proposal to use block indexes was first floated, which offers the advantages of reduced message size and removal of the need to know hashes (including the genesis block hash) in order to begin block retrieval.

Index synchronization

Ixje’s proposed method foregoes the header-first approach that is ubiquitous in the blockchain industry which we discussed in the previous article. By leveraging the unforkable nature of Neo provided by the dBFT consensus mechanism, nodes on the network can request block data by index alone.

This is because dBFT ensures that every node will share the same view of the blockchain as the consensus nodes, meaning they can be assured that the data at each block index is always identical. In the event that nodes have incorrect block data or state, they can quickly become aware of the issue by referencing the signed state shared through P2P messages.

Coupled with the simplifications in block retrieval logic, the proposal theoretically offered significant improvements to the synchronization process. Despite this, discussions were put on hold following the announcement of Neo3. Later in the year, as the core developer team began to evaluate the changes to be implemented as part of Neo3, inefficiencies stemming from the header-first synchronization approach re-emerged.

This prompted Ixje to reiterate his prior proposal, formalizing the use of block indexes rather than hashes for synchronization:

“This PR introduces a new getfullblocks command. It has IndexStart and Count parameters that indicate the starting Block height and how many blocks from that point forward to return. It re-uses the same inv type payloads that are used in a response to getdata messages, meaning there’s no need to change any Block receiving logic. It just reduces a lot of the overhead and complexity of syncing blocks via headers.”

Alongside the proposal, Ixje provided a working example of the code from neo3-python, and also highlighted the efficiency of the approach with a simple example of the message size reduction for requesting blocks with the index approach.

Despite the advantages, attempts to implement support for index synchronization showed that changes would be needed in other affected areas, such as the TaskManager. In the next episode, we’ll explore the implementation of index synchronization and required changes in more detail, and consider other related improvements.

Tags: