Factom Quickstart for Developers who Don’t Know or Don’t Care what a Blockchain is — Factomize

16 min readApr 25, 2019

Like the title says, this is an explanation of Factom for developers who already have solid programming knowledge and just want a no-buzzword explanation of how to write applications and what is possible. What it is not is a guide to programming or to the underlying Factom protocol itself. If you are interested in the latter, check out some of my other blog posts or the factomprotocol.org website.

Requirements

Working knowledge in a programming language
Knowing what JSON and JSON-RPC are OR using one of the many Factom APIs

That’s pretty much it. While the main network has costs associated with it, you can run a sandboxed local server or use the test network that lets you generate free resources. The sandbox, in particular, will let you start quickly and operate without any worries of accidentally messing anything up.

What can I do with Factom?

The Factom protocol is more or less a key-value database that supports three basic functionalities:

Transfer account balance (“Factoids”) from one address to another
Add arbitrary data using specific keywords
Read data

Notably absent is the ability to delete or update data, and the ability to do something when data is added. The latter is something that will have to be implemented on the application side through polling*. Some typical applications that lend itself to this format:

Logging events
Publishing content
Maintaining records
Proof of existence

To create more complex applications, you’ll likely need some sort of interpreter that will read data and determine the validity of it, maintaining its own state. Some examples of these are listed at the bottom.

* Some third party APIs have built-in support, see the API section at the bottom

Components

When looking at the official documentation and other APIs, there are going to be several components mentioned:

factomd: This is the important application (short for "Factom Daemon"). It connects to the network, syncs with the other nodes, and provides a JSON-RPC endpoint at :8088.
walletd: This is a less important application ("Wallet Daemon"). The wallet is a small database that stores private keys and provides a different JSON-RPC endpoint at :8089 that will let you perform factoid-related actions using only your public key. It's possible to do everything on factomd without a wallet but the wallet needs a factomd endpoint.
factom-cli: This is a command line utility to interact with factomd and walletd instead of having to use JSON-RPC.

Running

There are three different networks: the main network, the test network, and your own networks.

To join the main network, run factomd without any parameters or changes to the config file. This will take a long time to synchronize all the data (expect 24 hours or more) and is typically not needed for developers.

To join the test network, run: factomd -network=CUSTOM -customnet=fct_community_test and use the custom config file that includes necessary keys. You can use the faucet to get tokens.

To run your own sandbox, run: factomd -network=LOCAL -blktime=600 -exclusive=true with a default config. The blktime flag determines how long it takes to create one directory block / increase height (see below), with a default of 10 minutes. Setting it as low as 6 seconds should be fine. exclusive means the node will refuse to connect to other nodes.

Please note that running a public node on a dev server is insecure and there is a guide on how to install and secure a public node. The same goes for walletd. By default it uses a non-authenticated connection with an api open on port 8089. If that port is open to the outside, anyone with knowledge of your IP could connect to it and have access to your private/public keys.

Concepts (Basic)

The short version is this:

You can buy Factoids on an Exchange to send to your Address, increasing its balance. You can either transfer part of your balance to other addresses or turn it into entry credits to write data.
All data is stored in entries belonging to a particular chain. In order to store data, you perform a two-step process of committing first and paying the required entry credits, and then revealing the entry. After that, you wait for the entry to be confirmed, which can take up to 10 minutes. To read data you need either the hash of a specific entry, or traverse through all entries of a chain by knowing the chain id.

Below is a more detailed explanation of the bolded terms.

Two-Token System

There are two types of tokens: Factoids and Entry Credits. If you are building data-driven applications, you’ll spend the majority of your time with Entry Credits.

Factoids & Factoshis

Factoids (abbreviated as FCT) and Factoshis are the monetary tokens associated with a Factoid Address. 1 FCT = 10^8 Factoshi and vice versa, 1 Factoshi = 0.00000001 FCT. You can either transfer them to other addresses or turn them into Entry Credits. At the time of writing, 1 FCT is worth around $9 on an exchange but this will vary.

Entry Credit

Entry Credits (abbreviated as EC) are the “data tokens” associated with an EC Address. Every write action costs a certain amount of EC. The conversion rate is adjusted routinely to keep the price of 1 EC = $0.001. At the time of writing, 1 EC = 11000 Factoshi or 1 EC = 0.0011 FCT.

Factoids can be converted to Entry Credits, but Entry Credits cannot be changed back.

Addresses

There are two types of addresses, a Factoid Address for Factoids and an EC Address for Entry Credits. Both types consist of a “ private:public " keypair, each a 52 character string. You need a Factoid private key in order to transfer FCT from one address to another or turn them into EC. You need an EC private key in order to write data.

These can be generated on-demand via walletd api calls " generate-ec-address" and " generate-factoid-address ". How they work and how to manually generate them will be covered in a future blog.

If you are using walletd, you can avoid having to use the private keys in your application and only use the public keys for keypairs you have stored in the wallet. This only works for methods available via walletd API calls.

Factoid Address private keys start with the letters Fs... ("Factoid secret"), public keys start with FA... ("Factoid Address)

Example:

Private: Fs2LB8Hi5bwseUFUgFuTcFsEZ2thS4psAoWLHN66vTNYB7kRcBE7
Public: FA36Xc9Yjz1gsSinGh6rNiKzhhZnoVdQLkTsUhVYUMeLmVn1aoZS

Entry Credit Address private keys start with the letters Es... ("EC secret"), public keys start with EC... ("Entry Credit")

Transactions

This section could be its own blog, so I will keep it limited to the functionality I typically need as a data-driven developer. To send FCT from one address (that is stored in walletd) to another, use
factom-cli sendfct FA1zT4aFpEvcnPqPCigB3fvGu4Q4mTXY22iiuV69DqE1pNhdF2MC FA28PitepUziaDrLeVAcioNfzHdBc7mvyJJHvag2vyhWm7JR3t8S 1.234.

To turn FCT (from an address in your walletd) into EC, use
factom-cli buyec FA28PitepUziaDrLeVAcioNfzHdBc7mvyJJHvag2vyhWm7JR3t8S EC27kDNpFcJQwvdpFXaXjPqhtDSf6VK8kRN8Fv7EkhvS9tVkuAfX 30

Exchange

An exchange is a third party site where people can trade Factoids for USD or other currencies. How they manage the association between your account and the factoids depends on their implementation but all the funds are controlled by the exchange until you request them to be transferred to an address only you know the private key of.

Chains & Entries

All data lives in a system of Chains and Entries, with each Entry associated with exactly one Chain, and Entries being traversable from the head of the chain. It’s easier to portray this in a diagram than with words:

ExtIds are the keywords responsible for differentiating chains from each other. The chain id is a 64-byte hash generated using extids, which means that you can’t create two different chains with the same extids. It also means the chain is generated deterministically and you can find out what the chain id before you submit anything. The first entry of a chain always has the same extids as the chain but with an added data component. “Creating a chain” is just the idiomatic way of creating an entry with an empty “chain id”.

Entries are the actual records of data. They can be accessed directly by their entry hash, which is generated as a combination of the chain id, the extids, and the data.

The easy way to create chains and entries is via the walletd api calls " compose-chain" and " compose-entry", which will not create them right away but instead return two fully-fledged RPC calls, one for "reveal" and one for "commit" (see below). You will have to send both of these to factomd.

It is possible to create these directly and skip walletd but doing so requires manual binary encoding and correct signing. Some of the API clients mentioned at the bottom have this implemented while others such as factom-cli require you to run both factomd and walletd to create chains or add entries.

The data in both extids and data is completely arbitrary binary data with a combined limit of 10KiB. There's no limit on what you can put into extids or how many you add apart from that, though it adds an additional 2 byte per extid. The formula to calculate the size of an entry is: (data bytes) + SUM(bytes of extids) + 2*COUNT(number of extids)

The cost of creating a new chain is always 11 EC. The cost of an entry is determined based on the payload at the rate of 1 EC / KiB for a maximum of 10 EC, with a minimum of 1 EC.

To calculate a chain id:
<chain id> = SHA256(SHA256(ExtId[0]) | SHA256(ExtId[1]) | ... | SHA256(ExtId[n]))

The entry hash calculation is much more complex.

Chain Head & Entry Block

The API has calls for “ Chain Head”, “ Entry Block”, and “ Entry “. The chain head takes the chain id and points to the most recent entry block. If new entries are added to the chain, the chain head will change.

Entry Blocks will hold a list of at least one or more entries and a reference ( prevkeymr) to the previous entry block for that chain. All entries written to the same chain in the same directory block will be grouped into a single entry block.

A very important feature to point out here is that anyone can write to any chain (apart from the three special chains in the advanced section) and it is also possible to create the same entry multiple times in the same chain. More on this below.

Note: There’s no API call for just “Chain”. If you know the extids, you can calculate the chain id directly. If you need to find out the extids of a chain id, you have to get the extids of the first element. There is also no way to go from an entry to the next entry, or even the entry-block.

Commit & Reveal

Chains and Entries are created in a two-step process. The first step is the Commit which is paying EC to allocate either a Chain or an Entry, and requires the use of an EC Address you control. It does not transmit any actual data, only hashes of the byte data. The second step is Reveal, which is sending the actual byte data. Only after both are sent will the entry be saved. There’s a 24-hour window during which you can send the reveal and the commit doesn’t have to come from the same source as the reveal.

If a commit isn’t revealed, the commit expires and the entry credits are not returned. It is possible to send another commit with a higher EC amount and the new one will be accepted over the old one, though the EC are not refunded. The maximum EC for a commit is 20.

Ack

In order to check the status of a transaction, entry, or chain you can use Ack. This will give you a status update on whether or not your commit/reveal/transaction have been saved yet or whether one exists.

Unknown: Nothing was found
NotConfirmed: The data exists in the factomd instance that hosts your API
TransactionACK: The data has been sent over the network and the important nodes are planning to save it
DBlockConfirmed: The data is saved (this typically takes between 1 and 10 minutes for new data) in a directory block (“DBlock Confirmed”), which means it’s officially saved.

Note: As of writing this, it is not necessary to fill in the “chain id” parameter if you are searching for an entry.

Concepts (Advanced)

The above is enough to work with your own data that you save and retrieve. If you want to build an application that also has to deal with data showing up from other sources, you’ll likely need the directory block.

Directory Block

Similar to Chains and Entries, the entire factom database consists of directory blocks. The “directory block head” points to the most recent directory block and each directory block has a reference to the previous directory block. Inside of dbentries is a list of Entry Blocks that have been updated. There will always be an element for chains c (paying for commits), f (factoid transactions), and a (administrative records).

When new entries are written to chains, there will be additional elements for every chain that was updated.

A simplified diagram of the directory block structure looks like this:

You can get the hash of the most recent directory block with the “ directory-block-head” method, and then retrieve the directory block itself with “ directory block”. That block contains references to the other blocks, which you can retrieve with “ admin block”, “ factoid block”, and “ entrycredit block” methods. All four of these can also be retrieved by their height, with the corresponding methods of “dblock-by-height”, “ablock-by-height”, “fblock-by-height”, and “ecblock-by-height” which takes the height argument instead of hash.

The entry blocks listed in the directory block are the same as the ones listed by the chain head covered above.

Height

Every time another directory block is saved, which happens every ten minutes, the height increases by one.

Summary

Putting all of this together, the structure of the data looks like this:

(Black arrows are references included in the directory block. Green arrows are back-references included in the individual structures. Entries themselves are not included in this diagram)

That should be enough to allow you to find your way around.

Building Applications

The above describes only the technical aspects of data entry and retrieval but there are a few more very important things.

Chains and Entries are public

You can’t create private chains/entries. You can’t create chains that only you can write to. You can’t stop someone from adding an entry to a chain. Even checking which address paid for an entry can be unreliable (see Commits).

When reading any data from a chain, you have to treat it as potentially malicious user input. This adds some overhead to your data, depending on what you are doing. The typical way to ensure that an entry was actually written by yourself is to include a digital signature of the data inside of the entry, which lets you verify that the data was signed properly and the data has not been altered.

At the time of writing, the identities system is not yet finished. In the future, the ability to create identities (public/private key combo) inside of the system itself will be possible and you’ll be able to sign and verify using those. At the moment, you’ll have to manage your own signing process including storing keys.

Entries are repeatable

It is possible to submit the same entry twice to the same chain. There is a built-in time-restriction that prevents the same entry inside of an hour (60 minutes), so you never have duplicate entries inside of a single entry block, but in general, you have to be aware that someone else could submit an identical entry to one you submitted (including the identical signature).

The typical way of dealing with this is to include an additional nonce (such as a timestamp) in the extids. If the same nonce shows up twice (or the timestamp is outdated), you can filter those messages out.

Namespace

If you’re planning on having a lot of chains in your system, especially ones not created by you, you can use extids to build namespaces. If the first extid of any new chain is always “MyApp” then you can later poll new data to specifically search for new chains including the keyword. For larger applications, you can use extids as a multi-tiered path.

Examples

Some very basic examples of what a program flow could look like

Logging (Proof of Existence)

You want to prove that your log files are valid by recording their file hash in a chain. The chain is committed and revealed with the following parameters:

ExtId1: MyCompany ExtId2: Log File Hashes

and no data, which creates a chain with the chain id 3d28d16c4831f8be794128bd63b0c0f1fd94898458521d43ebb0b6a24c028c7b.

Every 60 minutes, you take the SHA-256 hash of your log file and commit & reveal an entry that looks like this:

Chain Id: 3d28d16c4831f8be794128bd63b0c0f1fd94898458521d43ebb0b6a24c028c7b ExtId1: 01-01-2019 00:00 ExtId2: filename.log Data: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 <placeholder for a cryptographic signature here that signs the chain id, extid1, extid2, and the file hash>

That is it. It will cost 11 EC (~1 US cent) to set up and operate at 24 EC / 2.4 cent a day. Only entries in the chain signed with your key would be considered valid and you will have to store the log files that you hashed yourself. Even without the signature, it is enough to prove in court that the log file existed exactly like that (your hash) at that specific date (backed up by Factom’s date).

With the addition of a signature over chain id, both extids, and the hash itself, you can also prove that it is the *only* log file that you claim is accurate for that time.

Reading Unknown Data (Polling)

You just released an application that writes data to Factom and you want to know how many customers are actually using it. Every app has its own chain based on data unknown to you, but you know that the first extid is always “MyApp”.

You create another app that calls “directory-block-head” every ten minutes (and if it hasn’t changed at that point, you check again every 15 seconds until it does) to get the latest directory block.

Loop through all the elements of “entryblocklist”, skipping past the first three.
For every element, call “entry-block” with the element’s “keymr”
Loop through the entry block’s “entrylist” array
For every element, call “entry” for the given “entryhash”
Check the Entry’s first extid to see if it matches “MyApp”
If it does, add the chain to your local database with the current time. If it does not, move on to the next element in 2.

Using that data, you can track how many applications are running by seeing which chain ids are still being updated.

Tic-Tac-Toe (Interactivity)

You want to create a game of Tic-Tac-Toe that you can play against a crowd, where the crowd’s move is determined by voting, with ties going to the first mention. Players have the current and following 3 heights to respond.

You start a new game by creating a new chain with ExtId1 = TicTacToe and ExtId2 = <gameid> and your public key as data
You add an entry for your first move (X) with ExtId1 = <gameid>, Extid2 = <signature> and coordinates in the form (x,y) as data. The signature covers both gameid and coordinates.
You wait until the height has increased by 4. Players vote by adding entries with coordinates and randomized nonces in the extids.
Get the chainhead and loop through all the new entries. Discard all entries with invalid (nonexistent, already placed) coordinates and add up the rest. Whichever coordinate has the most entries contains the next “O”. In case of a tie, the one that came first is used
Repeat steps 2–4 until one side has won or it ends in a tie

At this point, the data is complete but who is actually determined as the winner depends entirely on how the data is interpreted. Nothing stops you from using the same coordinates twice or writing an “X” over an “O” but if everyone knows the rules of the game, they can tell that this would be an invalid move and have proof that someone attempted to cheat.

That is how a more complex factom based application would work. The data written to chains could be anything but everyone who follows the same rules will be in agreement with each other.

Please note that in this variant, one person could represent multiple people in step 3 and overrule a group by spending more EC than them on entries.

Existing Infrastructure

If you’re going into production, you don’t necessarily have to run your own factomd node. The recommended node is the Factom Open Node, which runs a factomd API endpoint backed up by a cluster. If you are planning on using walletd in combination with either of these, you will need to run that on your own.

API Clients

If you don’t want to code the JSON-RPC calls yourself, you can use one of the existing open source clients or one of the commercial ones. Some are simple API wrappers and some have more functionality.

One of the ones I want to highlight is Paul Bernier’s node.js client particularly because it has a built-in “ Event Emitter “. This lets you skip having to keep implementing polling in your own apps and delegate it to the API, only dealing with the actual events. This API also supports the ability to not run an instance of walletd if you are comfortable with (or don’t need) loading private keys in your application.

Existing Projects

There are a lot of different applications all running on the mainnet, some public and some private, but here are a few examples that hopefully cover the whole gamut.

Canonical Ledgers’ Factomize

This is an application running on the Factomize.com forum and will log a signature of every post and edit made. This lets people check whether or not a post has been changed from the original, including a history of changes. Every entry is signed using Factomize’s public key.

This is an example of a fairly simple logging application.

Factom Robot

Factom Robot is an application to hash a file and then save that hash to a specific chain as proof of existence, proof of ownership, and timestamping.

Voting

The Voting Protocol is a work in progress to be able to create votes inside Factom. If you look at the specifications, you can see that it’s nothing more than a very complex set of guidelines for the ExtIds and data of chains and entries but correctly entering data (voting) and reading (showing results) is left up to third parties to implement themselves.

Factom Asset Tokens

Factom Asset Tokens (FAT) is a very ambitious project that is essentially a system of tokens running as a second layer on top of factom to create tokens. Explaining FAT is beyond the scope of this article but I wanted to mention it because it works similar to the Tic-Tac-Toe example but on a much, much larger scale, requiring anyone who wants to participate to run fatd, which is an interpreter that has its own clients and APIs. If the restrictions of Factom are unsuitable for your application, you should look into developing it for FAT instead.

Where to go from here

This blog should be enough to get you started but if something goes wrong or isn’t covered, the best place to ask questions would be the channel #developer-discussion in the official Factom discord.

Originally published at https://factomize.com on April 25, 2019.