Neo St. Petersburg Competence Center (Neo SPCC) has released an article introducing its performance test bench for Neo blockchain nodes. Alongside the introduction, the team shared its initial test results from benchmark comparisons between the neo-cli and neo-go nodes, noting significant throughput improvements for its Go node.

The project was created in order to determine the ability of nodes to cope with a high load in both isolated and networked environments. This assists in the detection of bottlenecks, and since the infrastructure is designed to work with any Neo node implementation, it can also be used to compare metrics such as TPS and resource consumption between different node types.

According to Neo SPCC, future improvements to the test bench will include Neo3 compatibility, more efficient hardware, and potentially support for heterogeneous networks. The test bench code will be published on the team’s GitHub in the near future.

In the performance test bench article, Neo SPCC said:

“We hope that this new test environment will simplify the work of the Neo community and all implementations of Neo 3.0 will move together to new performance achievements, surpassing all other existing blockchain solutions.”

Test environment

Neo SPCC has conducted its initial comparisons between the core C# node and NeoGo in several scenarios split between two test cases, single-node consensus and four-node consensus with an optional RPC node.

Two different transaction generation processes are used for the benchmark. The generator can be set to send a fixed number of requests per second or use multiple worker threads that send requests at their maximum rate. Requests, each taking the form of an invocation transaction with a simple “PUSH1” script, are redistributed to consensus nodes via the P2P protocol.

Four-node and single-node test cases (Source: Neo SPCC)

Each node and transaction generator was run in its own docker container on a single machine:

  • Quad-core i7 @ 2.9 GHz
  • 16 GB DDR3 @ 2133MHz
  • Neo-cli 2.10.3 and neo-go 0.71.1-pre (release 0.72).
  • Docker 19.03.5, Golang 1.13.5 Alpine, Dotnet 3.0-runtime-stretch-slim

The team observed TPS, memory consumption, and CPU usage between nodes by using RPC calls to check new blocks. Monitoring continues until all sent transactions are included in blocks, or when a timeout occurs.

Initial results

Single node tests showed improved performance and reliability in the Go node. Tests with 10 and 30 active workers showed 1,320 and 2,046 TPS respectively in the Go node, compared to 968 and 934 TPS respectively for the C# node.

TPS comparison with single-node consensus, 30 workers, fixed number of requests (Source: Neo SPCC)

Neo SPCC detected that the C# node’s RPC subsystem could not handle more than 30 simultaneous worker threads pushing requests, however, the Go implementation showed an average 1,931 TPS with 100 workers.

This bottleneck was also apparent in the four-node consensus test network. When using a fixed number of transactions, the Go node averaged 344 TPS compared to the 44 TPS processed by the C# node, with empty blocks and unprocessed transactions observed at the end of the test.

Overall, the results showed similar CPU consumption between the two nodes however the Go node was more memory efficient when under load. Neo SPCC notes that if the bottleneck in the C# node’s RPC component is resolved, a significant (2x) TPS increase and node stability improvements under high load can be expected. In this case, it recommends that the maximum load is limited by the resources consumed.

Full context and a more detailed per-case breakdown for the tests can be found in the original article at the link below:
https://medium.com/@neospcc/the-first-step-in-stress-testing-infrastructure-for-neo-nodes-4f325ca2097

Tags: ,