Factom Core Dev Quickstart

  • Knowledge of golang
  • Experience working with factomd

Community

The Code

Repos

factomd

Project Structure

Core:

  • /common/interface: This is the central location that holds all interfaces widely used throughout factomd. You’ll end up having to modify these whenever you want to add new functions used by other packages. For more information, see the Interfaces section.
    Note: Despite the name, there are some structs mixed in with the interfaces
  • /engine: The heart of factomd. The engine handles starting up the node, from parsing command line parameters to initiating the FNodes, and the internal P2P network. Important files here are:
    - factomParams.go: parsing command line parameters
    - NetStart.go: initializing the FNodes/States with the config and starting all processes
    - NetworkProcessorNet.go: Routes incoming application messages from the network to various queues in State (and vice versa).
    - simControl.go: A monstrous 1,400 line switch that processes keyboard input on the factomd console for a variety of debug functions.
    - timer.go: Easy to overlook but this is what triggers end-of-minutes for nodes
  • /state: The heart of the core code that contains the brunt of the logic. Each FNode has exactly one state. Functionality is more or less just scattered around here, with most files contributing to the State struct in some way. Some important files:
    - authority.go: verify authority signatures
    - createEOM.go: send out internal EOMs
    - dbStateCatchup.go: if a node falls too far behind (or is syncing), “Catchup” will retrieve DBStates from peers
    - dbStateManager.go: The DBStateList contains downloaded or created DBStates and is responsible for applying the contents therein to State (“ProcessBlocks()”), as well as saving and signing them
    - entrysync.go: Downloads all missing entries from the network (usually referred to as “2nd pass”)
    - factoidstate.go: Holds the balances of all addresses
    - grants.go: Factom Grants are hardcoded here. Each grant round, the forum generates a template that core devs integrate and release in a new version. Every ANO must update in time as this is technically a hardfork.
    - HoldingList.go: the “holding list” contain messages that can’t yet be processed, like those that require Acks from Leaders before they go into the process list
    - identity.go: the State counterpart to the identity system.
    - loadDatabase.go: load DBStates from the database and also contains the code to generate the genesis block
    - MMR.go: Retrieves missing messages in the ProcessList
    - processList.go: The process list that is used to build new blocks. Each PL has a “VM” for each Leader. Each VM is a collection of messages added for that block. When messages have “Process()” called, this is what they work with. The PL will then update the State with its own “Process()” function
    - replay.go: The Replay filter that prevents duplicate messages
    - saveAndRestore.go: This is what’s used to save the current State to disk and generate the FastBoot file. If new features are added to State that contain information not saved in the database, it needs to be added here.
    - state.go: Has the State struct definition and an overwhelming amount of functionality for one file. Of particular note are: LoadConfig(), Init(), UpdateState() (this is what processes ProcessLists), GetNetworkBootStrapKey()/BootStrapIdentity
    - stateConsensus.go: Has code to validate, execute, and process incoming messages, and most of the functionality of messages that isn’t contained in the messages themselves, making it one of the most important files in the project for the consensus algorithm
    - stateDisplay.go: This is the “DisplayState” copy that gets sent to the control panel
    - stateFER.go: Controls the EC price
    - validation.go: Routes messages between queues and also creates the EOM when the timer ticks. Also has the node shutdown functionality
  • /common/[admin,directory,entry,entryCredit]Block: The very important structs for the internal node representation of those blocks. These are the objects that are serialized and stored in the database. This is the place where all blocks have their serialization code to for their binary formats and calculate hashes from. They are used throughout the codebase.
    (Note: the factoid block is in /common/factoid)
  • /common/factoid: Contains the FBlock code as well as functionality related to transactions, such as addresses and RCD definitions. Transactions are validated (signature and input/output checks) here.
  • /common/messages: Functionality to decode application-level messages and to execute their various tasks. All messages have “MsgBase” as a base, which is in /common/messages/msgbase. The subfolder “electionMsgs” contains similar functionality for election messages. For more information, see the Messages section.

Useful:

  • /wsapi: The API (:8088) that can be used to interact with factomd. Only V2 is used these days. This is an amazing reference point you can use as an index for where many features are located. Want to find out where entries are stored? Just look at the code for the “entry” and you get the State’s entry lookup method as well as the database entry lookup.
    Also interesting is “wsapi.go” which contains middleware for the server.
  • /common/primitives: A dumping ground for basic shared functionality, which you’ll end up using a lot. Things like the marshaling Buffer, converting between human-readable keys and binary, the ubiquitous Hash definition, and Timestamp. Worth taking a look around in.
  • /util: Apart from miscellaneous functionality, this contains the default config file, parsing code, and the struct it parses into. If you need to add config file entries, this is the place to go.
  • /common/globals: This is mostly a globally accessible struct that holds the result of command-line parameter parsing (globals.Params). You’ll need this if you add more parameters.
  • /: The root folder contains a few interesting ReadMes. For actual core dev work, the files of note are:
    - Docker/Dockerfile.alpine: the docker files that are used to create the docker images that ANO pull from dockerhub (everyone uses alpine)
    - VERSION: contains the node’s version and should be updated with every release. There are no hard rules on numbering but the convention is major numbers for big hard forks, medium numbers for grants and backward compatible releases, minor numbers for small changes.
  • /.circleci: This is the configuration for Circle, which runs integration tests on pull requests, specifically build, go fmt, go vet, and go test.
  • /common/constants: An accumulation of constants in a central location. Important ones are: a list of valid application-level messages, ANO payout amounts and frequency, default network definitions, default chain definitions, and version number for the FastBoot file.

Relevant:

  • /activations: Very small folder that only contains the functionality to activate specific features at specific heights for different networks. You will likely need this for feature-based hard forks.
  • /anchor: Contains the code to parse the anchor entries JSON format and verify the signature against the config setting. Submission of anchors is not part of factomd.
  • /common/identity: Functionality to decode the Factom Identity Chains and related functionality, like the identity manager. The identity manager is fed entries by the State and parses them.
  • /common/identityEntries: This deals with decoding the entries that ANOs submit for individual entities.
  • /controlPanel: Contains all of the code related to the node control panel (:8090). The control panel is a go web server running a custom html/js front-end with the “Foundation” css framework. It works by being passed a copy of State (“DisplayState”) every second, which is then polled via Ajax.
    The templates are compiled into a go binary via “staticfiles”, which you’ll need to do to make changes.
  • /database: Internally, factomd can use a variety of different databases, which are wrapped by the “database overlay” (/database/databaseOverlay). The overlay provides high-level functions for use throughout factomd. Data is stored as key-value pairs in “buckets” which are essentially prefixes for the keys.
    By default, factomd will use LevelDB to store the blockchain data and a BoltDB for a replay filter. You can specify the blockchain data db type with the “-db” command line parameter. It is very useful when developing to use “-db=Map” to use a temporary memory-only database for a clean chain.
  • /p2p: My rewrite of the P2P package, frequently dubbed P2P2. It’s a standalone package (also available separately) responsible for building the gossip network that connects nodes to each other. It can be treated as a black box where messages go in and come out. This package is extensively documented in the README.
    (Note: depending on when you read this, this folder may hold the outdated, old p2p package. At the time of writing, 6.12 is in the process of being rolled out to MainNet, which will contain P2P2)
  • /elections and /electionsCore: The code responsible for handling elections. Even after two years of working on factomd, this code is really hard to understand. It’s a mess of messages piggybacked onto other messages in multiple levels. The only person who really knows this code is Steven M, who no longer works on the Factom Protocol.

Other:

  • /events: The factomd side of the live-feed-api. It’s hooked into a few spots in factomd and (not enabled by default) will keep trying to connect via tcp to :8040 (default) and broadcast events via protobuf (see the “eventmessages” subfolder).
    It’s intended that you run the live-feed-api to interpret the messages but I wrote a debug utility that you can use to just print them out.
  • /modules: One of the reorganization attempts introduced the concept of “modules” which is supposed to be independent packages that can be included in factomd. At the moment, only the rewritten ChainHeadFix is in there.
  • /modules/chainheadfix: This is integrated into factomd (the -checkheads parameter, enabled by default) and its job is to parse all DBlocks to ensure that each “chain head” in the database points to the latest eblock of said chain.
  • /receipts: The code to generate and verify entry receipts, which is essentially just pulling all the relevant hashes to prove existence. The subfolder contains a standalone command-line app to calculate this without needing to run factomd.
  • /log: One of the custom logging features. This one will print messages to specific files that can be selectively enabled or disabled. For more information, see my post on logging in factomd.
  • /testHelper: Contains various support functions for unit tests, including the unit test node simulator.
  • /simTest: A collection of unit tests that use the built-in node simulator to run tests requiring multiple nodes.
  • /longTest: A couple of unit tests that take a long time to run.
  • /netTest: A collection of unit tests that only run against an already existing dev network.
  • /peerTest: A collection of unit tests that use the network. Tests are split into filename_A and filename_B, where both need to be started simultaneously in order for the test to work.

Useless:

  • /Utilities: A collection of random utilities, most of them outdated, that should really be in their own repo. I don’t know if they’re still useful in any way.
  • /scripts: A file dump of random scripts. There’s no documentation for what any of these do and most are probably outdated. Try not to add anything to this folder.
  • /support/dev: An outdated ELK-Stack docker-compose setup that no one uses anymore. Inside the “factom” subfolder, there are three config files that are useful as crib sheets on how to configure running your own local network with multiple nodes.

State of the Code

Interfaces

Marshalling

Messages (IMsg)

  • Validate: Defines if a message should be routed. Has three possible results: valid (1), can’t process yet (0), invalid (-1). Invalid messages are rejected outright, but ones that can’t be processed yet go into “Holding” and are continuously re-assessed until they become either valid or invalid.
  • LeaderExecute: The functionality that’s performed if the node is a Leader (Federated) and the message is meant for the VM assigned to that node for that minute. For most messages, this is just a wrapper for “FollowerExecute” (don’t do anything special) or “state.LeaderExecute”. The latter is a generic function which adds the message to the process list and sends out a signed Ack.
  • FollowerExecute: The functionality that’s performed by all nodes, including Leaders to whom the message is not assigned that minute. Most are just a wrapper for one of the state functions to handle functionality. The generic behavior (state.FollowerExecuteMsg) is to add it to Holding until both the message and its Ack have arrived. When both are present, the message is added to the process list at the right spot.
  • Process: The functionality of what happens when a message is inside a PL and the PL is being executed. This is the point where messages actually do work (adding entries, chains, transactions, etc). Messages that do not go into the PL will just have a call to panic() instead of implementing this.

The Future

Resources

--

--

--

Developer

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

From lean-startup via design thinking to implementation

Image result for minimum viable product

2021. 04. 01

Excel advanced built-in functions — ordering and dynamic references

Configurable data and parity drives on Minio server

Event Sourcing, CQRS and DDD, the way I see it.

SymfonyCon Lisbon summary

.NET Core vs Node.js: What Should You Choose?

Cassandra Myth Busters: Is Coding on Cassandra Too Complicated?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Who

Who

Developer

More from Medium

Footnote

Hi, I’m Transgender

Detailed explanation about how to create new contact in Odoo V15 Contact Module