Road to Neo3: Oracle request execution and consensus

In the previous article on Neo3’s oracle solution we showed an example oracle request then looked at the structure of transactions and how they are propagated through the network. This time we’ll look at how oracle requests are executed in NeoVM, how consensus can be achieved on the results, and outline the protocols, filters, and response formats that will be initially supported on Neo3.

Executing oracle requests

Although normal Neo nodes will assist in relaying oracle transactions, the only action they will take after receiving one is checking the transaction hash against a known hash list and updating it in its memory pool. When an oracle node receives an oracle transaction, it can instead add the hash to the known hash list and begin fulfilling the request.

Requests to the oracle service are carried by the oracle system calls, using the URL and filter as inputs as shown in the previous article. Oracle nodes will execute oracle syscalls in NeoVM, first checking to see if there are any existing cached results. If no result exists, the oracle node will download the content from the provided URL, apply the filter, and then cache the results.

At this point, the oracle node has determined a result to the request, however this result cannot be trusted by users without verification by multiple oracle nodes. The OracleAgreement section of the original oracle transaction is updated with the results cache, and it is forwarded through the network to begin the consensus process.

Oracle consensus

Consensus is a very important part of any oracle network as it is the process by which the validity of requested data is confirmed. By having multiple nodes all executing the same query, the response received by each node can be compared in order to determine the most consistent outcome.

This is intended to prevent tampering and create an environment where developers can trust the off-chain data being pulled in for use in smart contracts. In the current proposal, an approval threshold has been suggested as the most appropriate consensus mechanism.

Approval thresholds are a simple approach to finding majority agreement between oracle nodes, where a specific number/percentage of nodes that must agree on the correct response to a given oracle request. Other conditions can also be attached, as is shown in the examples provided by Neo core developer Belane:

1. At least 10 signatures to move to the verified pool. 50% of oracles agree and no more than a 10% against, moving with positive result.
2. At least 5 signatures to move to the verified pool. Twice the amount of oracle nodes agree than those against, and no more than 10 signatures against, moving with positive result.

As an example, consider an oracle network of 10 nodes reaching consensus using the first formula shown above. An unverified oracle transaction is sent to the network, requesting the current temperature in Shanghai from a weather monitoring API.

If 9 oracle nodes call the API and return a temperature of 20 degrees, and 1 node returns the value 19, then the correct result would be determined to be 20 degrees. Each of the 9 nodes would sign agreement for the value “20,” and now with enough signatures to satisfy the approval conditions, the transaction can be moved to the verified pool for inclusion in a block.

The final design for this consensus process has yet to be finalized due to the different approaches that could be taken. For example, the approval threshold could be a fixed value defined in the oracle policy, or each smart contract could specify its own approval threshold. Alternatively, a completely different consensus process could be explored.

Supported protocols, filters, and responses

Oracle requests can take different forms depending on the needs of developers. Different resources need to be accessible, and standardized responses and filters are required to optimize results before oracle consensus.

The main protocol that needs to be supported by the oracle service is HTTP (and its secure counterpart HTTPS), the standard for data exchange on the Internet. The initial implementation will be built with support for both HTTP and HTTPS, however discussion is ongoing on the appropriate methods to support.

Neo co-founder and core developer Erik Zhang proposed that only GET should be supported due to its simplicity and suitability for oracle requests, however support for POST has also been suggested due to its widespread usage in REST APIs.

Additionally, the oracle service will also be able to process requests to the NeoFS decentralized object storage network, allowing contract developers to access off-chain data. A proposed set of NeoFS operations has been discussed here.

Support for other protocols or methods may be explored in the future, although it should be noted that the solution has been designed with a focus on simplicity and so more complex operations or custom protocols are not prioritized at this time.

Filters were proposed by Shargon as an extension to the oracle system call, aiming to provide a way for responses to be optimized before consensus by the oracle nodes. As we touched on in the previous article, this offers a few benefits such as conserving space on chain, aiding determinism in consensus, and ensuring that third party APIs are easy to use.

One issue with this approach is that each language would need a different implementation for the same filter, possibly leading to differences in responses (after filtering) between nodes running different software.

A possible solution for this was proposed by Igor Coelho, suggesting that oracle filters could be implemented as smart contracts. This would allow developers to create custom filters for use in calls, or repurpose them for use outside of the oracle system itself.

For the initial oracle implementation, the JSONPath standard will be used to provide filtering to responses. The responses themselves are also an important point to be clarified, referring to the data format in which an API may provide an answer to a request. Following discussion between Neo core developers on the GitHub, it has been proposed that users should define the format or Content-Type they expect beforehand.

Matching the filter implementation, only JSON responses will be supported initially, however in the future support will be added for other formats such as XML or HTML to meet the potential needs of contract developers.

In the next article we will cover the remaining components for the Neo3 oracle solution, such as the oracle policy native contract and discussion around oracle node election.