Recently, there has been a lot of discussion around the topic of ANO and server management. That was a stumbling block for me when I started tinkering with Factomd since the code responsible for on-boarding ANOs happens to spread across different repos. This blog aims to shine some light on the whole situation, detailing what exactly Inc. does when they “onboard” and “deboard” servers, how interested parties could launch their own custom network, and a few other maintenance utilities.
This blog is aimed at developers and other technically inclined people. I tried to keep it as simple as I could so everyone can follow along but fully detailing all the concepts like cryptographic signatures would result in a novel, not a blog post.
All encryption used in this blog is ed25519, which has a 32-byte private key and a 32-byte public key component. In most cases, they are just written as a hexadecimal string. Identity chains, also written as 32-byte hexadecimal strings, are chain id hashes. Which is which should be apparent from the context.
The genesis block needs to have a valid signature and for that purpose, Factom uses a “Bootstrap Identity” and “Boostrap Key” for the server creating the genesis block which does not exist on-chain and accepts arbitrary data. Once the genesis block is made, new nodes can be configured to use an identity chain, which lets them sign messages in their name. These identities are created on-chain and added to the identity registration chain.
To add a new node to the authority set, the message needs to be authenticated by the “Skeleton Identity”. If the Skeleton Identity doesn’t exist, it will fall back to the “Bootstrap Key”.
Tools
- AddServerMessage: The tool to manage the authority set. You need the private skeleton key in order to use it.
- ServerIdentity: The tool that ANOs use to create, register, and manage their server identities.
- Sign With Ed25519: A helper utility to sign arbitrary data with a key
- FER Entry Creator: The tool to adjust the EC rate
This blog is not a guide on how to create or manage your identity, there are existing guides for this already.
Configuring a Node/Network
All the important settings are part of factomd.conf, though not all of them are listed in the default config. Some are not configurable at all. They are split into two sections: what the node uses and what the network uses.
What the Node uses:
* Not necessary when bootstrapping a network
Sidenote: If a node has no config file or a non-hex value for the identity chain, it will default to SHA256("FNode0") (38bab1455b7bd7e5efd15c53c777c79d0c988e9210f1da49a99d95b3a6417be9). This is the default BootstrapIdentity for LOCAL networks.
What the Network uses:
You can specify a custom BootstrapIdentity and BootstrapKey for CUSTOM networks only. Please note that the SkeletonIdentity is hardcoded and not a config setting.
With the following defaults:
* TestNet is a deprecated network type, not the factom protocol’s testnet, which is a CUSTOM network with the id “fct_community_test”.
** The private key for this is 4c38c72fc5cdad68f13b74674d3ffb1f3d63a112710868c9b08946553448d26d
*** The private key for this is 0000000000000000000000000000000000000000000000000000000000000000
The process of onboarding is rather simple.
- The server operator has to create and register a valid identity
- The block the identity is in is completed
- Using the “addservermessage” tool, you type “addservermessage -host=<address of a factom node> send a <identity chain id of the server to be promoted> <network skeleton key>”
That will add the identity as an authority server. If you use “f” instead of “a”, it will onboard them directly as a federated note and increase the “fed slots” by one. To remove them, you can use “sendR” instead of “send”. You can also use this to manage the servers, by “promoting” them. You can promote a fed to an audit, or an audit to a fed. Before a node is promoted to fed, you should promote it to an audit first and make sure that it’s configured correctly and sending heartbeats.
So that’s it? It’s that simple? Pretty much, yeah.
However, let’s take a look under the hood. If you run “addservermessage” with “show”, it will print out a curl command you can execute anywhere. For example:addservermessage show a 8888881570f89283f3a516b6e5ed240f43f5ad7cb05132378c4a006abe7c2b93 4bded4f3620f6c598d710c0f97b4d83c9743d6a4ac28d24612d6c9d1dc6d06ab
results in
curl -X POST -data ‘{“jsonrpc”: “2.0”, “id”: 0, “params”: {“message”:”1601721232ac208888881570f89283f3a516b6e5ed240f43f5ad7cb05132378c4a006abe7c2b9301a997bdaf2a8519fdc227ae8084afe77f24402275f6e3fc9a48ce0bf3760a4f756e408fd774e06042e1d15f1ccf9725480a09fb8c32a873e9354e25a00f3454c907924c2744515acbc73cbb43ef1f3ce9592fde2c554868b6fab7b9991956560b”}, “method”: “send-raw-message”}’ -H ‘content-type:text/plain;’ http://localhost:8088/v2
If you read my blog on decentralizing ANO management, this message should look familiar. The bytes “0x16” are the message type for “AddServerMsg”. “0x01721232ac20” is the timestamp. Following that is the identity we provided: 8888881570f89283f3a516b6e5ed240f43f5ad7cb05132378c4a006abe7c2b93. The next byte 0x01 specifies that it’s supposed to be an audit server. The rest is the cryptographic public key followed by the signature.
( Verify this example on the golang playground)
Like onboarding, the tool does most of the job for us.
- Make sure the factoid exchange rate chain exists (111111118d918a8be684e0dac725493a75862ef96d2d3f43f84b26969329bf03)
- Configure the utility and specify the private key counterpart to “ExchangeRateAuthorityPublicKey” and an EC private key
- Run the tool
- Expiration Height: the height the entry becomes invalid if it was never activated
- Activation Height: the height the price activates (needs to be +/- 6 blocks of expiration height and cannot be more than 12 blocks ahead)
- Priority: The default priority after a price activates is 0, so this is typically always 1. If you mess up, you can use a higher priority to override a pending change
- Factoshi per EC: the EC rate
After that, it prints out two curl commands and also calculates the assumed price of FCT so you can doublecheck it before submitting the message. The data is a simple chain entry like all other entries and the content is encoded in JSON, so there’s not much interesting under the hood; you could write these by hand and sign the JSON data. You can look at existing entries in the MainNet chain for examples of how the activation height and expiration height are used.
Let’s assume we want to start a new network that we control. The first step is generating a secure key to use as our network bootstrap identity and skeleton key. We also need to pick an identity chain to use as bootstrap chain, which can be anything we want:
- Private key: 4bded4f3620f6c598d710c0f97b4d83c9743d6a4ac28d24612d6c9d1dc6d06ab
- Public key: a997bdaf2a8519fdc227ae8084afe77f24402275f6e3fc9a48ce0bf3760a4f75
- Bootstrap Identity: 8888881111111111111111111111111111111111111111111111111111111111
We also need identities for our test servers but fortunately, factomd already has 100 test identities built in with known private keys that we can use. You can order factomd to create these by typing gX in the console, where X is a number between 1 and 100, which will also create the registration chain. The entries are paid for by the initial sandbox coinbase, so this wouldn't work for a random node on MainNet.
For our example, we’ll need two additional identities:
I ran this test on one machine using temporary RAM-only instances of factomd (“db=Map”), using config files serverX.conf. In server0.conf, we’re setting the server to use the private key we generated above. The other two config files use the first two test identities for their local settings, but the same bootstrap identity and key.
The commands I used to start them were: (the .conf files were in $HOME/.factom/m2/)
factomd -db=Map -customnet=blogpost -blktime=60 -config=server0.conf
factomd -db=Map -customnet=blogpost -blktime=60 -config=server1.conf
factomd -db=Map -customnet=blogpost -blktime=60 -config=server2.conf
That launched a new chain, with server0 as the only federated node, creating blocks every 60 seconds.
Next, I added one server as a fed, one as an audit:
addservermessage send f 8888881570f89283f3a516b6e5ed240f43f5ad7cb05132378c4a006abe7c2b93 4bded4f3620f6c598d710c0f97b4d83c9743d6a4ac28d24612d6c9d1dc6d06ab
addservermessage send a 888888aeaac80d825ac9675cf3a6591916883bd9947e16ab752d39164d80a608 4bded4f3620f6c598d710c0f97b4d83c9743d6a4ac28d24612d6c9d1dc6d06ab
The messages were submitted and can be seen in the processlist:
The result? A shiny new fed node and audit node.
Demoting
Let’s say we want to demote a fed to an audit. Let’s run:
addservermessage send a 8888881570f89283f3a516b6e5ed240f43f5ad7cb05132378c4a006abe7c2b93 4bded4f3620f6c598d710c0f97b4d83c9743d6a4ac28d24612d6c9d1dc6d06ab
It shows up in the processlist just like we were onboarding it:
The result: two audit nodes.
I wanted to change the EC price to 2 Factoshi per EC. At the time I ran the utility, the block height was 50. Before I could submit anything, I had to fund an EC address and create the FER chain:
factom-cli buyec FA2jK2HcLnRdS94dEcU27rF3meoJfpUcZPSinpb7AwQvPRY6RL1Q EC2DKSYyRcNWf7RS963VFYgMExoHRYLHVeCfQ9PGPmNzwrcmgm2r 1000
factom-cli addchain -n “FCT EC Conversion Rate Chain” -n “1950454129” EC2DKSYyRcNWf7RS963VFYgMExoHRYLHVeCfQ9PGPmNzwrcmgm2r
(The FA address is one of the sandbox coinbase addresses, the EC key is the address with a private key of all zeros)
I configured my FactomFER.conf thusly:
PaymentPrivateKey = “0000000000000000000000000000000000000000000000000000000000000000”
SigningPrivateKey = “4bded4f3620f6c598d710c0f97b4d83c9743d6a4ac28d24612d6c9d1dc6d06ab”
Version = “1.0”FactomFER.conf
And ran the following:
After executing those commands and waiting until block 54 rolled around, the EC rate was applied.
While going pouring over the code to write this post, I stumbled across something odd: As mentioned above, there is a Skeleton Identity but it’s hardcoded and doesn’t exist anywhere on chain. If the Skeleton Identity is not found, it uses the Bootstrap key as fallback. That’s the one we used in the example and the one that currently exists in factomd.
What I wanted to know: what happens if you create that skeleton identity at a later stage? Does it start using that?
The answer is “yes, but”.
I had to recompile factomd to accept a new Skeleton Identity for CUSTOM networks, for which I used the third built-in identity of “8888887020255b631764fc0fd524dac89ae96db264c572391a3f19fcf0f8991e”. I started it up just like in the example, adding id1 and id2 as audit servers.
After creating id3, nothing happens. Factomd only loads identities that are part of the auth set, which doesn’t happen upon creation.
Next step, I added id3 as an audit server. Once again, nothing happened on the surface. However, in the background, factomd tried to add the identity into its identity manager in the place of the hardcoded skeleton identity. At that point, the original bootstrap key was no longer valid to sign messages, and hence it did not add the server. From this point on, the skeleton identity has control of the network.
After sending another “add audit server” message, this time signed with 3838383838383730323032353562363331373634666330666435323464616338 instead of 4bded4f3620f6c598d710c0f97b4d83c9743d6a4ac28d24612d6c9d1dc6d06ab, I was able to add the node into the auth set.
If the skeleton identity removes itself from the auth set, the skeleton key will revert back to the bootstrap key.
To summarise, someone who has both a) the current bootstrap key and b) the skeleton identity can backdoor the network and override the bootstrap key. I don’t know why this exists but it could be a safeguard to ensure the network can be recovered in case the bootstrap key is compromised. For this to work, Factom Inc. would have to have the identity already created but kept safe in a vault somewhere.
The skeleton identities are:
- MainNet: 8888882690706d0d45d49538e64e7c76571d9a9b331256b5b69d9fd2d7f1f14a
- All Custom Networks: 88888816d408cd0d7b1b28760f3371a40e98dc2e985c28410e781935954afdf3
Neither of these chains currently exists and it is not possible for someone else to create these identities (until SHA256 is broken). These keys used to be placeholders, like Local/Test, but they were specifically added January 1st, 2017, so I assume they have been created for this purpose.
Managing a Factom network isn’t that daunting once you know what you’re doing as long as you are aware of the pitfalls. It also highlights just how powerful the skeleton key really is. There are no checks and balances; whoever has knowledge of the key has complete control over the network’s authority set.
Originally published at https://factomize.com on May 14, 2020.
