Preface
There is a lot of talk about scalability, sharding, and how to get factomd to the next level. What I want to talk about in this blog is skipping past all those steps in between and start right at the end: a fully customizable, modularized, shardable factom node.
This is not meant to be a proposal of things we should implement in factomd right now, it is an idealistic vision of the future that doesn’t account for hardware limits or optimizations. The haute couture of programming — not something meant to be implemented but rather to inspire goals and trends.
Factom EDA
The foundation of extendable modularization is a unified message bus. All modules should be able to react to any message, rather than demultiplexing the message stream. The basic idea would be an event-driven architecture (EDA) pattern:
The core of Factom EDA is the message bus, which exists just to deliver messages to the modules. All functionality is handled by the modules themselves, which are consumers, emitters, or both. The modules themselves are independent, plug-and-play pieces of software that can be swapped out at any time, allowing for easy extendability of core functionality.
Communication between modules is carried out over the message bus using protobuf messages, making it both module-agnostic and asynchronous. If a node does not have the necessary module installed, requests can go to the network and be answered by other nodes. A node with no modules will act as a repeater in the P2P network, propagating messages.
Network
The network is a UDP based P2P network with the ability to message any node directly. Requests are made to neighboring* nodes. If the node knows the answer, there will be an asynchronous answer that returns to the message bus ( Scenario 1). If the node does not know the answer, it will ask its own neighbors until a node that knows the answer is found. That node will then send the answer directly to the originating node. ( Scenario 2)
* Determined via a routing table
Modules
A module is an independent piece of code that registers itself on the message bus for certain messages, ranging from all to just a specific type. Modules have the ability to (1) read messages, (2) stop messages from being propagated further, (3) add new messages to the message bus. How they work internally is up to each module, meaning that this is an opportunity to shard locally. Each module should react to an “info” message, which returns information about how it’s currently performing.
Modules have a pre-determined order of execution to allow for deterministic program flow. It is possible to run multiple modules of the same type with different parameters.
In-Filter Module
This module sits at the first order of execution and checks if a message are duplicates. Repeated messages within a given timeframe are stopped.
Admin Module
The admin module is responsible for keeping track of which servers are leader and audit nodes for a given height. Incoming messages that should have a valid signature but don’t are stopped. Admin block messages that add/remove servers will update the list.
Clock Module (Leader Set)
Sends a periodic message to the message bus based on a configurable setting: the “Minute”.
Process List Module (Leader Set)
This is the module that is the “process list” functionality but only runs on nodes in the leader set. Chains, entries, and transactions from the API/Network arrive and are read by this module. To check the current balance of an address to verify if a transaction is valid, it will send a request to the message bus for a balance check. If the state is acceptable, it will send a signed message that it has been accepted (the “ACK”).
When a Minute message arrives, the module will start the end-of-minute and block signing processes. Once a block or minute is finished, it will send a notification.
Consensus Module (Leader Set)
This module keeps track of the finished blocks from the process list and the signatures arriving from other nodes. If there are enough signatures, it sends out a signal that the node is moving to a new height.
Election Module (Leader Set)
This module keeps track of clock messages and height messages. If there is a network pause, it will kick off the process to replace the unresponsive leaders.
Anchoring Module (Leader Set)
This module would listen to finished block events and then, if the node has responsibility, anchor it in the bitcoin blockchain, with another module for Ethereum anchoring.
Oracle Module
This module answer requests of the current FCT to EC conversion rate
Out-Filter Module
This module sits as the last order of execution and determines which messages should be stopped from going out. This can filter out local messages like the clock or module information requests. Messages that pass this filter are then sent to the network.
Balance Module
The module keeps track of all addresses and balances. It reads messages for finished blocks and updates its internal state to reflect that. It will also intercept requests for balance checks, stopping the request and then sending a balance notification to the bus.
Persistence Module
The module reads notifications of blocks being finished and then persists them. It also reads requests for full blocks and if it has those stored, will stop the message and send the block to the message bus.
API Module
The module runs a web server with a JSON-RPC endpoint that lets people send transactions/chains/entries as well as request information. For example, a request for a balance will send a balance check to the message bus. The eventual reply with the balance is then returned to the caller.
Control Panel Module
The module runs a web server that displays statistics, gleaned from sending information requests over the bus and having each module respond.
Log Module
The log keeps a record of all messages that arrive. It could also send messages from a log file for debugging purposes.
Intermission
The modules listed above are enough to re-create the current functionality of factomd inside of Factom EDA. Let’s look at some of the benefits of this system.
- Modules are (mostly) language independent and can run as a standalone application. You only need a small interfacer that sends the protobuf message from the message bus to your application and back.
- A module does not have to run entirely on the same system as the message bus. The same small interfacer mentioned in the point above could establish a connection a remote system, like a database that’s connected on another machine in the local network. Potentially every single module can run on its own machine or cluster of machines sitting behind an optimizer.
- Modules are not required. For example, if the balance module is missing, the request for a balance will go out to the network and then be returned from the network. If the database module is missing, the node will be ephemeral, only keeping track of the current state without caring what happens once a block is signed.
- Multiple modules of the same type allow for internal sharding. You could run a database module for just entries or specific chains and all requests it can’t handle will be sent further down the line, to be answered by another database module responsible for transactions. Or you could run redundant databases.
- External modules are hot-swappable and could update without Factom EDA even being aware, requiring nodes to only go offline for low-level changes.
Additional Modules
Now that we have the basics covered, let’s introduce a few modules that currently have no equivalent.
Pub-Sub Module
This module is an externally facing API that just listens to messages on the message bus. All events like changing to a new height, blocks being finished, etc are read by the module and subscribers can be immediately notified. This would even allow fine-grained subbing to things like specific chain ids or even specific extids without requiring anything special from other modules.
Identity Module
Similar to the balance module, this module could listen to all incoming identity chains and their entries, then keep an up-to-date record to answer any requests for identity existence, addresses, or public keys.
Universal Connector Module
Instead of each module writing its own interface to connect to an external application, this module could standardize it. It would be configured with an endpoint that it connects to and the messages it wants via protobuf. The module then is able to maintain the connection and re-establish it should it ever be severed, such as during hot-swaps. These connections could be network or OS Channels.
Since this module is already loaded into memory when Factom EDA starts, it’s also possible to provide an API for Factom EDA itself to allow adding and removing of UCMs during runtime.
Proprietary Modules
It is possible for companies to release proprietary, pre-compiled modules that connect via the universal connector module. The primary benefit would be to incentivize outside companies to get involved in blockchain development without having to sacrifice their intellectual property under an open-source license.
FAT Module(s)
Instead of FAT being a separate application that polls factomd, it could connect directly to the message bus and interact directly. And in a very similar vein…
“Smart Contracts” / Automation
I put the term “Smart Contracts” into quotations because what exactly constitutes a smart contract is debatable, especially in a proof-of-consensus system like factom. These could be complex state machines like FAT and On-Chain Voting that will take any input and then end up at the same state but those are not implemented in the core. A core smart contract would have to be a module that runs on all of the leaders and is executed before the block signing process to write data into the block itself with the resulting transactions and entries but that would likely have to be built into the consensus module, not able to be its own module.
But there could be modules that perform smart-contract like behavior, for example a module running on Factomize’s node that integrates with Factomize’s canonical ledgers forum addon. Every time a post is added (and signed by Factomize), the module could take that post’s identity, and send a few Factoshi from Factomize to them. That would fall more under automation than a smart contract since the module is only required on a single node and the behavior is centralized.
This area is unfortunately not one that is easily tackled by Factom EDA but I wanted to include it anyway.
Conclusion
In the preface, I said that the goal was “a fully customizable, modularized, shardable factom node.” Can Factom EDA do these things?
Customizable
Which modules a node operator runs will be up to them. They can either run nothing at all, just an API, or in the case of federated nodes, the “Leader Set”. They can also choose different versions of modules that fit their platform, such as a database connected via a universal connector module, or modules from competing ANOs. To me, that fulfills the goal of being customizable.
Modularized
This is the core philosophy of Factom EDA and I believe that this goal is achieved. Each module runs in its own memory space, unable to communicate with other modules outside of the message bus and only responsible for a small fragment of the whole. They can be independently audited to and tested to make sure they work. Other applications and non-Golang modules can be tied in with a connector module.
Sharding
A Factom EDA node is shardable in many different ways. The modules can all run in the same application, as a set of applications on the same server, with modules on different VMs, and on physically separate machines. Modules can also be internally sharded. The Persistence Module, for example, could have a cluster of servers with a load balancer behind it. Modules can be redundant or complementary as I mentioned earlier.
It’s even possible to run two separate Factom EDAs, with one node having the leader set and the other node having the persistence module. Messages not handled by the first node could be routed specifically to the second.
Reality
So far everything has been theoretical, would it even be possible to build such a system? Yes, but…
The biggest drawback to Factom EDA is the reliance on slow communication channels. A message going from one node to the network and eventually returning works in theory but in practice, the few milliseconds it takes in a best-case scenario are still orders of magnitudes slower than a local system. In a similar vein, it would be hard to determine if a request is just slow to answer or if no answer is coming at all.
Inside of Factom EDA, asynchronous communication is going to increase overhead for individual modules and increase the complexity of the program flow. At times it will not be possible to determine whether a message should be stopped by a module at the time it arrives and you’d have to wait for an asynchronous request for more data arrives from other modules.
As I said in the preface, this blog was not about creating a realistic implementation of Factom EDA. It’s about creating a leitmotif to inspire goals.
Thank you for reading
Originally published at https://factomize.com on May 28, 2019.
