Dynamically On-boarding New Assets — Factomize

At the time of writing, adding a new asset to PegNet is possible but it is a laborious manual process:

1. Find at least two APIs that provide free and reliable* data for the market price
2. Create a polling adapter that is able to read the API endpoint and transform the data to use in pegnetd
3. Add a new OPR version that indicates the addition of a new asset, updating the OPR and Grading package
4. Determine an activation height for this change
5. Implement the activation height change in both pegnetd and the reference miner
6. Deploy the new code and work with exchanges, mining pools, and discord to coordinate the hard fork

* Free means there is a way to make 144 requests a day and 4,464 requests per month from the API without paying. Free registration is okay. Reliable means the update rate of the price is at the very least 10 minutes, with minutely updates being recommended.

While theoretically this could be done for every asset, reality is that we’d likely collect a bunch of changes first, then bundle them for an update. It is also unlikely that anyone but core developers will be able to convince the PegNet community to update, limiting open source contributions to steps 1 and 2.

So what would a dynamic system of onboarding and removing assets look like?

The Two Components

There are two distinct problems that each require its own system to solve:

1. Dynamic API: The ability for a miner to mine a new asset and submit it
2. Dynamic OPR: Deciding when an asset is active and required for submissions to be valid

The first system corresponds to steps 1–3, the second to steps 4–6.

First System: Dynamic API

The non-standardization of APIs is an age-old problem and unfortunately, nobody has come up with a universal solution. That necessitates the creation of a layer between pegnetd and the API which translates requests from one to the other. This can be accomplished with descriptor files that allow a varying set of different inputs.

With the existence of descriptor files, we will also need a way for miners to find and add them. That means there has to be a marketplace of sorts that is both responsible for vetting the files as well as distributing them.

Descriptor Files

Each API endpoint likely has a different way of accessing it, sometimes with authentication required. The responses vary, with JSON being one of the most common formats, though other than that there is very little similarity between two separate APIs. To facilitate multiple assets requiring the same API, descriptor files are split into API Descriptor Files ( API-DF) and Asset Descriptor Files ( ASSET-DF). Descriptor files would need to specify everything necessary so that a general pegnetd interpreter can access the data. The format of the descriptor file would be JSON, which is easily writable by humans and also easily machine-readable and importable during runtime in golang.

API-DF:

• Protocol: The method of connecting, which likely will be just HTTP(S)
• URL / Endpoint: The address to contact in a format specific to the protocol
• Authentication Scheme: Can be “none” or a variety of pre-configured methods like www-authentication or OAuth
• Response format: JSON, XML, CSV, etc.

ASSET-DF:

• API-DF: The entry hash of an API-DF
• Assets: A list of assets that this API can read, i.e. PEG & EUR
• Paths: Each asset would have its own path for both price and the time it was updated, which depends on the format. For CSV this could be expressed as a (row, column) tuple, whereas JSON could be a list of nested keys
• Data type: How the asset price is encoded, i.e. float vs integer

Each parameter specified would have to be supported by pegnetd, which would fall on the core developers to maintain and expand over time to support more methods. The actual implementation would likely be more complex; this is a foundation to build on. The goal isn’t to make it easy for anyone to submit any API but rather a way to dramatically reduce the amount of skill and knowledge required and enable automation.

Descriptor File Examples

PegNetMarketCap runs an API that provides a price for PEG, the endpoint of which you can access via the following URL: https://pegnetmarketcap.com/api/asset/all. The response consists of a very simple JSON format.

API-DF:

{ "version": 1, "protocol": "HTTP", "endpoint": "https://pegnetmarketcap.com/api/asset/all?columns=ticker_symbol,exchange_price,exchange_price_dateline", "authentication": "none", "response": "json" }

ASSET-DF:

{ "api-df": "abcdef...", "assets": ["PEG"], "path": { "PEG": { "price": "root/10/exchange_price", "price_type": "float64", "time": "root/10/exchange_price_dateline", "time_format": "unix_integer" } } }

That was a fairly straightforward example since the PNMC API was coded specifically for this use and kept simple without authentication. It gets more interesting if we attempt to get the price for bitcoin and factoids from Coinmarketcap, which requires you to sign up for a key. The endpoint is “https://pro-api.coinmarketcap.com/v1/cryptocurrency/quotes/latest" but we require both additional URL parameters and HTTP headers in order to access it. More specifically, we need to include the ID of every asset as GET parameters.

API-DF:

{ "version": 1, "protocol": "HTTP", "protocol_options": { "header": { "X-CMC_PRO_API_KEY":"%s", "Accepts": "application/json" }, "GET": [ {"type": "commalist", "key": "id", "value": "ids"}, {"type": "raw", "key": "convert", "value": "USD"} ] }, "endpoint": "https://pro-api.coinmarketcap.com/v1/cryptocurrency/quotes/latest", "authentication": "header:X-CMC_PRO_API_KEY", "response": "json" }

ASSET-DF:

{ "api-df": "abcdef...", "assets": ["XBT", "FCT"], "protocol_options": {"ids": ["1","1807"]}, "path": { "XBT": { "price": "root/data/1/quote/USD/price", "price_type": "float64", "time": "root/data/1/quote/USD/last_updated", "time_format": "ISO-8601" }, "FCT": { "price": "root/data/1807/quote/USD/price", "price_type": "float64", "time": "root/data/1807/quote/USD/last_updated", "time_format": "ISO-8601" } } }

The big difference is the presence of the protocol options. In this case, the “header” authentication is one that only exists for the “HTTP” protocol, and indicates that we insert the API key into the header “X-CMC_PRO_API_KEY” that is specified in the protocol_options.

The “GET” is responsible for narrowing down which assets we request from CMC. In this case, there is one GET parameter that specifies “convert=USD”, and another that uses a “comma list” to grab the values “ids” from the path. In this instance, the resulting address would look like:

https://pro-api.coinmarketcap.com/v1/cryptocurrency/quotes/latest?id=1,1807&convert=USD

If someone decided that more tokens from CMC are needed, we only need to add another ASSET-DF:

ASSET-DF:

{ "api-df": "abcdef...", "assets": ["ETH"], "protocol_options": {"ids": ["1027"]}, "path": { "ETH": { "price": "root/data/1027/quote/USD/price", "price_type": "float64", "time": "root/data/1027/quote/USD/last_updated", "time_format": "ISO-8601" } } }

When combined, these two ASSET-DFs would result in:

https://pro-api.coinmarketcap.com/v1/cryptocurrency/quotes/latest?id=1,1807,1027&convert=USD

This is mostly for demonstration purposes and the final format will support a number of idiosyncratic options in addition to “commalist”. Fortunately, APIs actually want users to be able to easily read their data, so it should be feasible to have a set of rules that is able to deal with the majority of common API formats. If it is impossible to describe an API with the existing ruleset, it falls to the core developers to implement it and enable API-DFs with higher versions.

Using a Descriptor File

Apart from having a common list of APIs, the order and priority they’re used is currently defined by the miners manually via the config file. With the addition of descriptor files, this would quickly grow too complex. There could be a section for this in the control panel, where miners can upload/enable new descriptor files as well as change their priorities.

This would also enable miners to write custom Descriptor Files for already existing coins, allowing them to specify a greater variety of APIs than what is currently available.

Marketplace

The marketplace would be the place where one can upload and download Descriptor Files. PegNet itself is decentralized and here is yet another decision to make. There are possibilities for both centralized and decentralized marketplaces, each with their own pros and cons.

Centralized Marketplace

This is by far the easier option. It’s a simple website run by a team that is likely to include core developers. Anyone wishing to submit a new Descriptor File can do it on the website. The team will verify the format, audit the API itself to make sure it’s reliable, and then approve it. The miners will be able to download the descriptor files on-demand or the software can automatically list the available files.

The downside is obvious: it’s a centralized option. This could be eased by having the team selection be decentralized. The people in charge of approving submissions could be selected by the community in order to somewhat mitigate the centralization.

Decentralized Marketplace

This would essentially be a Factom chain where anyone is free to submit Descriptor Files. This opens up the possibility of abuse, putting the work of verifying descriptor files into the hands of miners who may or may not have the technical know-how to verify it.

To aid this, we can introduce “auditors”, which are people that go through the submissions and make sure they work as expected. They do this by signing entries and submitting “audit” entries into the chain. Miners can then configure a list of public keys they trust and the mining software only uses Descriptor Files that have been signed by an approved auditor. Anyone could be an auditor, it would be their job to convince miners to trust them.

Second System: Dynamic OPR

Price data is secured by PoW and since there are unlimited conversions between almost all of the PegNet Assets, all assets must be mined equally. If one asset can be artificially manipulated because it has a lower hashrate than other assets, attacks can be routed through this “weaker” asset and affect “stronger” assets.

For this reason, all miners have to mine the same set of assets to ensure maximum security.

Therein lies the challenge of the dynamic OPR: coordinate the requirement of a new asset in a deterministic manner so there is a certain block height where a miner knows they have to start mining this asset in order to be considered a valid OPR.

Consensus

In a previous blog post, I introduced a system of voting via PoW. The TL;DR is that miners can include voting metadata in OPRs and it is possible to estimate what percentage of overall hashpower has this metadata.

This system would be ideal for dynamic OPRs. I propose that an asset is required starting with the first block where the following is true: the SMA-144 is over 67% for a period of 144 blocks.

This provides a substantial, but not too difficult, hurdle to asset adoption. The theoretical minimum timeframe of adding an asset (if 100% of miners decide to include it from one block to the next) is 241 blocks, or 1 day, 16 hours, and 10 minutes. It’s much more reasonable to expect asset adoption to run over the course of a few weeks, where the proponents lobby the mining pools to vote for the asset. However, the minimum timeframe still allows for potential intervention should there be problems.

If the threshold described above is met in Block N, all miners are required to submit valid price data for this asset starting in Block N+1. OPRs that do not contain the new asset are counted as invalid.

Data Structure

The current OPR format is a protocol buffer, which is amended to include the ability to vote:

syntax = "proto3"; package opr; message DynamicAssetOPR { string Address = 1; string ID = 2; int32 Height = 3; repeated bytes Winners = 4; repeated uint64 Assets = 5; repeated Vote Votes = 6; } message Vote { string Asset = 1; enum Action { ADD = 0; REMOVE = 1; } }

This allows for the inclusion of an indefinite amount of votes every OPR. The Asset field of a Vote entry would be used to look up matching Descriptor Files. The Action field would be used to determine whether this is a vote for the inclusion of an asset or the removal of an existing asset. If a miner sees an OPR with a vote for an asset that they do not have a Descriptor File for, they can start sending out notifications, which would let the operator also choose to support it at that point.

There are only “approval” votes but not negative votes. No vote is the same thing as disapproval in this system.

API Requirements

Among core developers there exists an informal requirement of at least two APIs for every asset, the more the better. If there were an asset required that did not have an API, miners would be unable to submit OPRs, which would stall the network. This is an opportunity for PegNet to be exploited. In the case of a centralized Descriptor File Marketplace, it is easy to add a restriction that says there have to be at least two vetted APIs available before a vote is considered valid.

In the case of a decentralized marketplace, this is much more challenging. In that case, it would fall on the miners to validate that there is sufficient redundancy to allow mining of an asset. Core developers can provide testing frameworks that aid in auditing descriptor files, but ultimately it would be up to the community to decide if adding an asset with only flaky APIs is worth using. This works along the same principle as I elaborated on in the consensus-by-PoW blog:

If the majority of the hashpower (in this case, 67%+) decides to add an asset or API with malicious intent, it is much easier and more profitable to simply provide false currency data directly in the OPR. As long as the majority of the hashpower checks that they have two trustworthy sources before voting to add an asset, the risk is minimized.

Removing Assets (Optional)

Since the destruction of an asset has a much more significant impact on the economy, the threshold for success should be higher than adding an asset. I propose: an SMA-144 of over 85% for a period of 144 blocks. For technical reasons, the following assets would not be possible to remove: PEG, pUSD, pFCT (as long as burning FCT->pFCT exists).

If the threshold for removing asset pABC is met in block N, all existing balances are converted to pUSD using the rate of pABC in block N.

Final Thoughts

Even with this framework, there are a lot of challenges that have to be solved. By far the hardest task will be managing the APIs and Descriptor Files. Doing it in a decentralized manner will result in a significant increase in complexity and increase the appeal of mining pools. Another challenge is that there just aren’t that many good and free APIs out there. While this system does leave open the potential for paid APIs, this would be a death knell for individual miners.

However, this system would make PegNet much more flexible as well as decentralized.

Originally published at https://factomize.com on February 4, 2020.