blog:2022:0529_crypto_collecting_tokens_pairs [NervTech's Wiki]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Crypto: Collecting tokens and pairs from DEX ======

{{tag>dev python crypto nervhome finance bsc blockchain}}

So lately, I've been working on restoring more of my blockchain management system (with the ultimate goal to get back to some arbitrage experiments 😎), and right now, I have a first version of a "Binance Smart Chain" blockchain, "ERC20" compatible Token and Pair classes, and "PancakeSwap" DEX based on a unified "UniswapBase" exchange representation.

And from the **PanscakeSwap** DEX (ie. //Decentralized exchange//) object I can request to collect all the pairs available on the exchange and all the tokens used in those pairs, to store information about them in a database. It's working pretty well already, but I still have a few exceptions from time to time due to very specific errors, and I'm still in the process of collecting the pairs **only** for PancakeSwap (version 1 I mean 😅): there are about 133K pairs already there!

What I will do here is to provide some improvements/extensions around this concept to try to improve it further, so let's get to work!

====== ======

===== Example of strange issue: nul character in symbol =====

  * Just as an example, the latest exception I got to date while collecting those pairs was due to apparently "nul" characters in the token symbols I'm retrieving from the blockchain, and then I cannot push those strings in my PostgresSQL table (will produce a **ValueError**), so I'm now removing any potential nul character before storing the token data: <sxh python; highlight: []>    def token_row_from_desc(self, desc, validate_symbol=True, sym_names=None):
        """Convert a token desc dict to a token row"""
        min_slippage = desc.get('min_slippage')
        swap_fees = desc.get('swap_fees')

        minslip = int(min_slippage*1000) if min_slippage is not None else None
        sfp = int(swap_fees*1000) if swap_fees is not None else None

        cls = 0
        clval = desc.get('classification')
        if clval is not None:
            cls = int(TokenClassification[clval.upper()]) if isinstance(clval, str) else int(clval)

        sym = desc['symbol']
        name = desc['name']
        addr = desc['address']

        # Leave some room for duplicate values
        if len(sym) > 30:
            logger.warning("Detected too long token symbol: %s", sym)
            sym = sym[:30]

        # Remove the null characters in the symbols and names if any:
        if '\x00' in sym:
            sym = sym.replace('\x00', '')
            logger.warning("Removed null characters in token %s symbol %s", addr, sym)

        # Note: if the name or symbol are too long then we should cut them:
        if len(name) > 64:
            logger.warning("Detected too long token name: %s", name)
            name = name[:64]

        if '\x00' in name:
            name = name.replace('\x00', '')
            logger.warning("Removed null characters in token %s name %s", addr, name)

</sxh>

  * But another thing I noticed while doing this is that when I fix one of those exception, and then re-run my command to update the pairs, I don't seem to get the failing pair/token handled again, which is unexpected... 🤔 So I need to have a look at that next.

===== Finding token by address or symbol =====

  * To figure out if a pair/token is indeed in the database I should add a command line mechanism to search for token by address/symbol 👍!
  * In this process I also created a new **BlockchainManager** class to unify the command line handling for BSC or ETH chains already: <sxh python; highlight: []>"""blockchain manager class"""

import logging

from nvp.nvp_component import NVPComponent
from nvp.nvp_context import NVPContext

from nvh.crypto.blockchain.evm_blockchain import EVMBlockchain
from nvh.crypto.blockchain.uniswap_base import UniswapBase

logger = logging.getLogger(__name__)


class BlockchainManager(NVPComponent):
    """BlockchainManager component class"""

    def __init__(self, ctx):
        """BlockchainManager base constructor"""
        NVPComponent.__init__(self, ctx)

    def process_command(self, cmd):
        """Check if this component can process the given command"""

        if cmd == 'collect-gas-price':
            chain_name = self.get_param("chain")
            chain: EVMBlockchain = self.get_component(f"{chain_name}_chain")
            chain.collect_gas_price_statistics()
            return True

        if cmd == 'update-pairs':
            # Retrieve the exchange name:
            ex_name = self.get_param("exchange")

            # Retrieve that exchange:
            dex: UniswapBase = self.get_component(ex_name)

            # Update the pairs:
            dex.update_pairs()
            return True

        return False


if __name__ == "__main__":
    # Create the context:
    context = NVPContext()

    # Add our component:
    comp = context.register_component("chain_man", BlockchainManager(context))

    context.define_subparsers("main", {
        "collect-gas-price": None,
        "update-pairs": None
    })

    psr = context.get_parser('main.collect-gas-price')
    psr.add_argument("-c", "--chain", dest="chain", type=str, default="bsc",
                     help="Blockchain from where to collect the gas prices.")

    psr = context.get_parser('main.update-pairs')
    psr.add_argument("exchange", type=str,
                     help="exchange name where to update the pairs")

    comp.run()
</sxh>

  * So now to collect the gas price I will use instead: <sxh bash; highlight: []>$ nvp bchain collect-gas-price -c bsc
$ nvp bchain collect-gas-price -c eth</sxh>

  * And to update the pairs from one dex I will use: <sxh bash; highlight: []>$ nvp bchain update-pairs pancakeswap</sxh>

<note>Each DEX is already aware of its parent blockchain, so we don't have to specify it on the command line above.</note>

  * So here is a first extension to find a specific token either by address or by symbol in our database: <sxh python; highlight: []>        if cmd == 'find-token':
            chain_name = self.get_param("chain")
            chain: EVMBlockchain = self.get_component(f"{chain_name}_chain")

            pat = self.get_param("pattern")

            # We check if we have a token for that address:
            token = chain.get_token_desc(symbol=pat)

            if token is not None:
                logger.info("Found token: %s", self.pretty_print(token))
            else:
                logger.info("Token not found.")

            return True</sxh>

  * And this works as expected: <sxh bash; highlight: []>$ nvp bchain find-token -c bsc WBNB
2022/05/27 08:11:16 [__main__] INFO: Found token: { 'address': '0xbb4CdB9CBd36B01bD1cBaEBF2De08d9173bc095c',
  'classification': 0,
  'decimals': 18,
  'id': 3,
  'min_slippage': None,
  'name': 'Wrapped BNB',
  'swap_fees': None,
  'symbol': 'WBNB'}

$ nvp bchain find-token -c bsc PANDA
2022/05/27 08:11:25 [__main__] INFO: Found token: { 'address': '0xc757025cFcC34a279BD56Ae70F00b4aA601648ed',
  'classification': 0,
  'decimals': 18,
  'id': 5781,
  'min_slippage': None,
  'name': 'PandaSwap',
  'swap_fees': None,
  'symbol': 'PANDA'}
</sxh>

  * Okay, now, last time the process failed we were adding a new "PANDA" token, but probably not the one above: <code>2022/05/27 03:44:37 [nvh.crypto.blockchain.evm_blockchain] INFO: Adding new token 0x0b2534c22bb60a51C1AB87c2266b14182e3b063E (PANDA<nul><nul<nul>)
2022/05/27 03:45:18 [nvp.communication.email_handler] INFO: Should send the email message <p style="color: #fd0202;">**WARNING:** an exception occured in the following command:</p><p><em>['/mnt/data1/dev/projects/NervProj/.pyenvs/bsc_env/bin/python3', '/mnt/data1/dev/projects/NervHome/nvh/crypto/bsc/binance_smart_chain.py', 'update-pairs', 'pancakeswap']</em></p><p>cwd=None</p><p >=> Check the logs for details.</p>
2022/05/27 03:45:18 [nvp.components.runner] ERROR: Error occured in script command: ['/mnt/data1/dev/projects/NervProj/.pyenvs/bsc_env/bin/python3', '/mnt/data1/dev/projects/NervHome/nvh/crypto/bsc/binance_smart_chain.py', 'update-pairs', 'pancakeswap'] (cwd=None)
</code>

  * So let's see if that token was added: <sxh bash; highlight: []>$ nvp bchain find-token -c bsc 0x0b2534c22bb60a51C1AB87c2266b14182e3b063E
2022/05/27 08:36:22 [__main__] INFO: Token not found.</sxh>

  * Hmmm... if the token is not in the database, I would expect that there is no pair registered either on that token: let's add a method to search for pairs on a given token.
  * So I added this command line processing in the **BlockchainManager**: <sxh python; highlight: []>        if cmd == 'search-pair':
            chain_name = self.get_param("chain")
            chain: EVMBlockchain = self.get_component(f"{chain_name}_chain")

            pat = self.get_param("pattern")

            # Search for a pair containing the given tokens:
            # Not that we may have only 1 token provided here:
            parts = pat.split("/")
            ntokens = len(parts)
            self.check(ntokens == 1 or ntokens == 2, "Invalid number of tokens in %s", pat)

            chain_db = chain.get_db()
            t0 = chain.get_token_address(parts[0])
            t1 = chain.get_token_address(parts[1]) if ntokens == 2 else None

            pairs = chain_db.find_pairs_with(token0=t0, token1=t1)

            if pairs is not None:
                logger.info("Found %d pairs: %s", len(pairs), self.pretty_print(pairs))
            else:
                logger.info("No token pair found.")

            return True</sxh>

  * Then I also added the support method ''find_pairs_with'' in the **ChainDB**: <sxh python; highlight: []>    def find_pairs_with(self, token0, token1=None):
        """Find the pairs with the given token0 and token1"""
        if token1 is None:
            # Single token search:
            sql = "SELECT * FROM pairs WHERE (token0=%s) OR (token1=%s)"
            cur = self.execute(sql, (token0, token0))

        else:
            sql = "SELECT * FROM pairs WHERE ((token0=%s AND token1=%s) OR (token0=%s AND token1=%s))"
            cur = self.execute(sql, (token0, token1, token1, token0))

        rows = cur.fetchall()
        return [self.pair_row_to_desc(row) for row in rows]
</sxh>

  * And with that we can find pairs given 1 or 2 tokens (address or unique symbol): <sxh bash; highlight: []>$ nvp bchain search-pair -c bsc ADA/WBNB
2022/05/27 09:09:57 [__main__] INFO: Found 1 pairs: [ { 'address': '0xBA51D1AB95756ca4eaB8737eCD450cd8F05384cF',
    'decimals': 18,
    'exchange_id': 1,
    'id': 7,
    'name': 'Pancake LPs',
    'pool_id': 6,
    'symbol': 'Cake-LP',
    'token0': '0x3EE2200Efb3400fAbB9AacF31297cBdD1d435D47',
    'token1': '0xbb4CdB9CBd36B01bD1cBaEBF2De08d9173bc095c'}]
</sxh>

  * Or: <sxh bash; highlight: []>$ nvp bchain search-pair -c bsc ADA
2022/05/27 09:11:46 [__main__] INFO: Found 79 pairs: [ { 'address': '0xBA51D1AB95756ca4eaB8737eCD450cd8F05384cF',
    'decimals': 18,
    'exchange_id': 1,
    'id': 7,
    'name': 'Pancake LPs',
    'pool_id': 6,
    'symbol': 'Cake-LP',
    'token0': '0x3EE2200Efb3400fAbB9AacF31297cBdD1d435D47',
    'token1': '0xbb4CdB9CBd36B01bD1cBaEBF2De08d9173bc095c'},
  { 'address': '0xF35145e28e1dd67055221739D8554E0b5C7Adcd5',
    'decimals': 18,
    'exchange_id': 1,
    'id': 162,
    'name': 'Pancake LPs',
    'pool_id': 161,
    'symbol': 'Cake-LP',
    'token0': '0x3EE2200Efb3400fAbB9AacF31297cBdD1d435D47',
    'token1': '0xe9e7CEA3DedcA5984780Bafc599bD69ADd087D56'},
  { 'address': '0x4D673a471d442fFF0dF4f9e71Dc406A75DE30Eb4',
    'decimals': 18,
    'exchange_id': 1,
    'id': 163,
    'name': 'Pancake LPs',
    'pool_id': 162,
    'symbol': 'Cake-LP',
    'token0': '0x3EE2200Efb3400fAbB9AacF31297cBdD1d435D47',
    'token1': '0x4B0F1812e5Df2A09796481Ff14017e6005508003'},
   (...more pairs here...)
   ]</sxh>
  
  * So now let's search for pairs on our missing token: <sxh bash; highlight: []>$ nvp bchain search-pair -c bsc 0x0b2534c22bb60a51C1AB87c2266b14182e3b063E
2022/05/27 09:14:50 [__main__] INFO: Found 1 pairs: [ { 'address': '0x3863Dd0b3E7b1b88498007c11dcc77f5ec0e70e8',
    'decimals': 18,
    'exchange_id': 1,
    'id': 46679,
    'name': 'Pancake LPs',
    'pool_id': 46678,
    'symbol': 'Cake-LP',
    'token0': '0x0b2534c22bb60a51C1AB87c2266b14182e3b063E',
    'token1': '0xbb4CdB9CBd36B01bD1cBaEBF2De08d9173bc095c'}]</sxh>

  * Hmmm, interesting... 🤔 It seems we do have the pair added there already, even if the token itself could not be saved in the database correctly: is that legit/correct ? I don't know: I need to check the code. Ohh okay, I see now:the token themself are only registered **after** the pair is registered when we try to retrieve them at the end of the **ERC20Pair** constructor: <sxh python; highlight: []>        # Load the tokens:
        self.t0 = self.chain.get_token(self.desc['token0'])
        self.t1 = self.chain.get_token(self.desc['token1'])</sxh>

  * => So this should not be a big deal at the end: the next time we try to retrieve that pair, the token data will be registered 😉!

===== Using multiple provider urls =====

  * Collecting the pairs from a given DEX is... well, pretty slow unfortunately, and on some exchanges we have tons of them, so the first idea I have to try to speed this up would be to try to use multiple provider Urls when this is possible to update the pairs from 2 exchanges (or more) in parallel.

  * **Note**: Found on [[https://docs.binance.org/smart-chain/developer/rpc.html|this page]] that the BSC endpoints have a rate limit of The rate limit of **10K requests/5min**.
  * **Note2**: Also found on the same page the RPC nodes at https://bscrpc.com and https://bscrpc.com/erigonbsc => maybe I shoudl given these a try too.

  * To classify the provider URLs I need some kind of monitoring of the speed of the requests: let's see if I can come up with soemthing on this point...
  * Okay, so I added a new table where i'm now storing infos about the duration of each RPC call I make on a given endpoint: <sxh python; highlight: []>SQL_CREATE_RPC_REQUESTS_TABLE = """ CREATE TABLE IF NOT EXISTS rpc_requests (
                                id SERIAL PRIMARY KEY,
                                timestamp float,
                                url varchar(128),
                                duration float,
                                function varchar(128)
                            ); """
</sxh>

  * => With that I can see that I'm making a lot of calls already to register a single pair 😅 => I could eventually speed this up by implementation my own contract to retrieve the data I want all at once, but for now, that's not really a priority: let's rather focus on adding the support to specify the provider url.

  * **OK**, now I can specify the provider url when calling update-pairs: <sxh python; highlight: []>        if cmd == 'update-pairs':
            purl = self.get_param("provider_url")
            chain_name = self.get_param("chain")

            if purl is not None:
                logger.info("Using provider url %s for chain %s", purl, chain_name)
                self.config['blockchains'][chain_name]['provider_url'] = purl

            # Retrieve the exchange name:
            ex_name = self.get_param("exchange")

            # Retrieve that exchange:
            dex: UniswapBase = self.get_component(ex_name)

            # Update the pairs:
            dex.update_pairs()
            return True</sxh>

  * And so I tried to call this update-pairs function on another dex (JupSwap): <sxh bash; highlight: []>$ nvp bchain update-pairs julswap -c bsc -p "https://bscrpc.com/erigonbsc"</sxh>
  * And the results are surprising since we find that the url https://bscrpc.com/erigonbsc is significantly slower than https://bsc-dataseed1.binance.org:443: 

{{ blog:2022:0527:provider_url_mean_dur.png }}

  * Let's test with https://bscrpc.com and BakerySwap now: <sxh bash; highlight: []>$ nvp bchain update-pairs bakeryswap -c bsc -p "https://bscrpc.com"</sxh>
  * And for that one the mean duration is 0.29 secs so still not so good...

  * Now trying https://bsc-dataseed1.defibit.io with ApeSwap: <sxh bash; highlight: []>$ nvp bchain update-pairs apeswap -c bsc -p "https://bsc-dataseed1.defibit.io"</sxh>
    * => mean call duration for that one: 0.133 secs
  * Now trying https://bsc-dataseed1.ninicoin.io with Mdex: <sxh bash; highlight: []>$ nvp bchain update-pairs mdexswap -c bsc -p "https://bsc-dataseed1.ninicoin.io"</sxh>
    * => mean call duration for that one: 0.243 secs

  * **Conclusion**: currently, the fastest BSC endpoint seems to be the one I'm using by default: **https://bsc-dataseed1.binance.org:443**

===== Error retrieving token names =====

  * Eventually on pancakeswap I started to get errors like the following: <code>2022/05/28 08:37:48 [nvh.crypto.blockchain.evm_blockchain] ERROR: Exception occured while calling name: Could not transact with/call contract function, is contract deployed correctly and chain synced?
2022/05/28 08:37:50 [nvh.crypto.blockchain.evm_blockchain] ERROR: Exception occured while calling name: Could not transact with/call contract function, is contract deployed correctly and chain synced?
2022/05/28 08:37:52 [nvh.crypto.blockchain.evm_blockchain] ERROR: Exception occured while calling name: Could not transact with/call contract function, is contract deployed correctly and chain synced?
2022/05/28 08:37:54 [nvh.crypto.blockchain.evm_blockchain] ERROR: Exception occured while calling name: Could not transact with/call contract function, is contract deployed correctly and chain synced?
</code>

  * => So, to me it seems that the "name" function is not available on those tokens ? Let's try to clarify that further.
  * Now reporting the contract address to in this error message I get for instance: <code>2022/05/28 08:49:32 [nvh.crypto.blockchain.evm_smart_contract] ERROR: Exception occured while calling 'name' (on 0xecC736C39b6f8ccAEE79a3EdaA31A41fE1f8b914): Could not transact with/call contract function, is contract deployed correctly and chain synced?</code>
  * So let's check that contract... and it cannot be found on bscscan 😳 (giving me an error...) How could that be ? let's check what pair this is...
    * (**Update**: The contract is found now (was a temporary error) but it is **self-destructed**)
  * => Added more outputs to get the pair: <sxh bash; highlight: []>2022/05/28 09:04:15 [nvh.crypto.blockchain.erc20_pair] INFO: Registering pair at 0x83Bf4ccdD9B61B9e4373bfc90739bac035A43d4a
2022/05/28 09:04:16 [nvh.crypto.blockchain.evm_smart_contract] ERROR: Exception occured while calling 'name' (on 0x441388fFC0235944C9Ad3aa5d834C0037EcB684C): Could not transact with/call contract function, is contract deployed correctly and chain synced?
2022/05/28 09:04:19 [nvh.crypto.blockchain.evm_smart_contract] ERROR: Exception occured while calling 'name' (on 0x441388fFC0235944C9Ad3aa5d834C0037EcB684C): Could not transact with/call contract function, is contract deployed correctly and chain synced?
2022/05/28 09:04:21 [nvh.crypto.blockchain.evm_smart_contract] ERROR: Exception occured while calling 'name' (on 0x441388fFC0235944C9Ad3aa5d834C0037EcB684C): Could not transact with/call contract function, is contract deployed correctly and chain synced?
</sxh>
  * Ohhh... 🥴 I just found this transaction: https://bscscan.com/tx/0x1ba791c58c62c6a19f4296ecedf128d2bc6d15337b21a5c4bfd10ad9f9942a1f
  * => It seems the contract 0x441388fFC0235944C9Ad3aa5d834C0037EcB684C gets self-destructed at that point! 

  * Studying **self-destruct**:
    * https://hackernoon.com/how-to-hack-smart-contracts-self-destruct-and-solidity
    * https://slowmist.medium.com/intro-to-smart-contract-exploits-selfdestruct-function-14c10ca00bb6
    * https://dl.acm.org/doi/10.1145/3488245
    * https://solidity-by-example.org/hacks/self-destruct/
  
  * => So I need a way to check if a given contract was self-destructed: and it seems we could use ''web3.eth.getCode(address))'' to check that (cf. https://ethereum.stackexchange.com/questions/66315/check-if-contract-is-destructed)
  * And indeed when we call this on jupyter we get: <code>chain.web3.eth.getCode("0xecC736C39b6f8ccAEE79a3EdaA31A41fE1f8b914")
HexBytes('0x')</code>

<note>When we call the same function with a valid contract we get a lot of bytes in the HexBytes() of course.</note>

  * So I added the ''has_code()'' method in the evm_blockchain class: <sxh python; highlight: []>    def has_code(self, addr):
        """Check if there is some code for a given address (ie. a smart contract)
        If there is no code, then we can consider the contract was self-destructed
        if this was supposed to be a valid contract somehow."""
        return self.web3.eth.getCode(addr).hex() != '0x'</sxh>
  * And then using that method to check for destruction when creating new token objects: <sxh python; highlight: []>        if desc is None:
            # We should register this new token:
            logger.debug("Registering token at %s", address)

            # First we really need to check here if the contract was not self destructed:
            destructed = False
            if not self.chain.has_code(address):
                logger.info("Note: contract at %s was self-destructed.", address)
                destructed = True

            sct = self.contract
            try:
                desc = {
                    'address': address,
                    'name': '<destructed-token>' if destructed else sct.call_function("name"),
                    'symbol': '<destructed>' if destructed else sct.call_function("symbol"),
                    'decimals': 0 if destructed else sct.call_function("decimals"),
                }
            except Exception as err:  # pylint: disable=broad-except
                logger.error("Exception occured while trying to initiate token at %s: %s", address, err)
                desc = {
                    'address': address,
                    'name': "???",
                    'symbol': "???",
                    'decimals': 0
                }

            desc = self.chain.add_token_desc(desc)</sxh>

  * **OK**! And now I could finally get all the current pairs from PancakeSwap collected properly 👍!

===== Consolidating pairs/tokens databases =====

  * Yet, with all the problems/exceptions that occured above, I'm not quite sure anymore my database tables are really in a fully consistent state: 
    * It might be that some tokens for some pairs are not registered
    * It might be that some pool indices were not registered properly
  * => So I should now implement a function to check the database consistency, and fill any missing element in the process.

  * Added a new command **check-db** that I will use to check the database consistency: <sxh bash; highlight: []>$ nvp bchain check-db -c bsc
2022/05/29 07:05:21 [nvh.crypto.blockchain.evm_blockchain] INFO: Collected 147585 pairs and 135759 tokens</sxh>
  * And here is the function I implemented to check the pairs/tokens in the database: <sxh python; highlight: []>    def check_db_consistency(self):
        """Check that the tokens/pairs tables are consistents"""

        # get all pairs:
        allpairs = self.get_db().get_all_pairs()
        npairs = len(allpairs)
        alltokens = self.get_db().get_all_tokens()
        ntokens = len(alltokens)

        logger.info("Collected %d pairs and %d tokens", npairs, ntokens)

        # We should iterate on each pair and collect its pool index:
        # each pair row contains:
        # id,address,token0,token1,name,symbol,decimals,pool_id,exchange_id
        # each token row conotains:
        # id,address,name,symbol,decimals,min_slippage,swap_fees,classification

        # Create a dict of all token addresses:
        taddrs = {}
        for token in alltokens:
            taddr = token[1]
            taddrs[taddr] = True

        pool_ids = {}

        for idx, pair in enumerate(allpairs):
            # pool_id:
            pidx = pair[7]
            # ex_id:
            ex_id = pair[8]
            if not ex_id in pool_ids:
                pool_ids[ex_id] = []

            # add the pool id to the list:
            pool_ids[ex_id].append(pidx)

            t0addr = pair[2]
            t1addr = pair[3]

            if t0addr not in taddrs:
                logger.info("Token at %s is not registered.", t0addr)
                # we register that token now:
                self.get_token(t0addr)
                # And then add the address to the list:
                taddrs[t0addr] = True

            if t1addr not in taddrs:
                logger.info("Token at %s is not registered.", t1addr)
                # we register that token now:
                self.get_token(t1addr)
                # And then add the address to the list:
                taddrs[t1addr] = True

        # check that all the pool ids are registered for each
        # available exchange:
        for ex_id, pids in pool_ids.items():
            pids.sort()
            idx = 0
            tgt = 0
            npids = len(pids)
            while idx < npids:
                if pids[idx] == tgt:
                    # increment the target:
                    tgt += 1
                    idx += 1
                else:
                    logger.warning("Missing pool id %d in exchange %d", tgt, ex_id)
                    # we just increment the target pool id in this case:
                    tgt += 1

        logger.info("Done checking pairs/tokens db consistency on %s", self.short_name)</sxh>

  * Running that code above, we can find some missing tokens in the database, and register them directly: <sxh bash; highlight: []>$ nvp bchain check-db -c bsc
2022/05/29 07:25:51 [nvh.crypto.blockchain.evm_blockchain] INFO: Collected 147585 pairs and 135759 tokens
2022/05/29 07:25:51 [nvh.crypto.blockchain.evm_blockchain] INFO: Token at 0x0b2534c22bb60a51C1AB87c2266b14182e3b063E is not registered.
2022/05/29 07:25:52 [nvh.crypto.blockchain.evm_blockchain] INFO: Adding new token 0x0b2534c22bb60a51C1AB87c2266b14182e3b063E (PANDA  )
2022/05/29 07:25:52 [nvh.crypto.blockchain.chain_db] WARNING: Removed null characters in token 0x0b2534c22bb60a51C1AB87c2266b14182e3b063E symbol PANDA
2022/05/29 07:25:52 [nvh.crypto.blockchain.chain_db] WARNING: Removed null characters in token 0x0b2534c22bb60a51C1AB87c2266b14182e3b063E name Panda
2022/05/29 07:25:52 [nvh.crypto.blockchain.chain_db] INFO: Replacing symbol PANDA with PANDA#40
2022/05/29 07:25:52 [nvh.crypto.blockchain.evm_blockchain] INFO: Token at 0x9a2129134944117e2C16514311E5AF94Ddaa3E05 is not registered.
2022/05/29 07:25:53 [nvh.crypto.blockchain.evm_blockchain] INFO: Adding new token 0x9a2129134944117e2C16514311E5AF94Ddaa3E05 (COMCOIN)
2022/05/29 07:25:53 [nvh.crypto.blockchain.chain_db] INFO: Replacing symbol COMCOIN with COMCOIN#2
2022/05/29 07:25:53 [nvh.crypto.blockchain.evm_blockchain] INFO: Token at 0xD4031a4964d7f7c8587cBda36Dcd7DB3E6B29f4C is not registered.
2022/05/29 07:25:53 [nvh.crypto.blockchain.erc20_token] INFO: Note: contract at 0xD4031a4964d7f7c8587cBda36Dcd7DB3E6B29f4C was self-destructed.
2022/05/29 07:25:53 [nvh.crypto.blockchain.evm_blockchain] INFO: Adding new token 0xD4031a4964d7f7c8587cBda36Dcd7DB3E6B29f4C (<destructed>)
2022/05/29 07:25:55 [nvh.crypto.blockchain.chain_db] INFO: Replacing symbol <destructed> with <destructed>#2302
2022/05/29 07:25:55 [nvh.crypto.blockchain.evm_blockchain] INFO: Token at 0x7eb2E0F97cbEb98cc3F5787bcAE7CF3898E8209A is not registered.
2022/05/29 07:25:55 [nvh.crypto.blockchain.erc20_token] INFO: Note: contract at 0x7eb2E0F97cbEb98cc3F5787bcAE7CF3898E8209A was self-destructed.
2022/05/29 07:25:55 [nvh.crypto.blockchain.evm_blockchain] INFO: Adding new token 0x7eb2E0F97cbEb98cc3F5787bcAE7CF3898E8209A (<destructed>)
2022/05/29 07:25:57 [nvh.crypto.blockchain.chain_db] INFO: Replacing symbol <destructed> with <destructed>#2303
2022/05/29 07:25:57 [nvh.crypto.blockchain.evm_blockchain] INFO: Token at 0x7dD2290E434699aCf9f54bDA414D8f136f5eA121 is not registered.
2022/05/29 07:25:57 [nvh.crypto.blockchain.erc20_token] INFO: Note: contract at 0x7dD2290E434699aCf9f54bDA414D8f136f5eA121 was self-destructed.
2022/05/29 07:25:57 [nvh.crypto.blockchain.evm_blockchain] INFO: Adding new token 0x7dD2290E434699aCf9f54bDA414D8f136f5eA121 (<destructed>)
2022/05/29 07:25:59 [nvh.crypto.blockchain.chain_db] INFO: Replacing symbol <destructed> with <destructed>#2304
2022/05/29 07:25:59 [nvh.crypto.blockchain.evm_blockchain] INFO: Token at 0xecC736C39b6f8ccAEE79a3EdaA31A41fE1f8b914 is not registered.
2022/05/29 07:25:59 [nvh.crypto.blockchain.erc20_token] INFO: Note: contract at 0xecC736C39b6f8ccAEE79a3EdaA31A41fE1f8b914 was self-destructed.
2022/05/29 07:25:59 [nvh.crypto.blockchain.evm_blockchain] INFO: Adding new token 0xecC736C39b6f8ccAEE79a3EdaA31A41fE1f8b914 (<destructed>)
2022/05/29 07:26:01 [nvh.crypto.blockchain.chain_db] INFO: Replacing symbol <destructed> with <destructed>#2305
2022/05/29 07:26:01 [nvh.crypto.blockchain.evm_blockchain] INFO: Done checking pairs/tokens db consistency on bsc</sxh>

  * And naturally on the next run, ther are no more issues: <sxh bash; highlight: []>$ nvp bchain check-db -c bsc
2022/05/29 07:29:11 [nvh.crypto.blockchain.evm_blockchain] INFO: Collected 147585 pairs and 135765 tokens
2022/05/29 07:29:11 [nvh.crypto.blockchain.evm_blockchain] INFO: Done checking pairs/tokens db consistency on bsc</sxh>

===== Conclusion =====

  * So I guess I should stop it here for this time: this is already a way too long article 😂 But I'm really happy I'm getting back to a working system to manage all those pairs/tokens! I'm one step closer to my arbitrage system/experiments now 👍!