NervProj: Finding & adding monitored coins

In one of my previous articles (available here) I introduced support to list/add/remove monitored coins from the command line with my coingecko component. This is very nice and all, but this also assumes that you know what is the ID of the coin you want to add to the list, which is not always absolutely clear…

So, what I want to do here is to add support to search for all coins matching a given pattern and list them directly on the command line so I could then select the proper one in the list and add the corresponding coin id.

  • Before I get to the actual problem I want to fix here, there is another minor thing I want to change too, each time I want to run a script in the NervProj framework I currently have to type nvp run script_name ⇒ I think that, instead, I could try to bypass the “run” keyword here and use instead nvp script_name directly: is that possible ? Let's see…
  • Oki oki, so it seems a bit tricky to handle this kind of setup with argparse only, so I decided to use a simple “workaround” for that:
  • First I added a method in the runner component to check if a given script exists:
        def has_script(self, script_name):
            """Check if a given script name is available."""
            projs = self.ctx.get_projects()
            for proj in projs:
                desc = proj.get_script(script_name)
                if desc is not None:
                    return True
            desc = self.scripts.get(script_name, None)
            return desc is not None
  • Then before parsing the command line arguments I check if the second argument in the list is a script name, and if this is the case, I manually add the “run” command in front of it before parsing the args as usual (note that this operation only makes sense ont the master context for now, so placed in the allow_additionals section):
        def parse_args(self, allow_additionals):
            """Parse the command line arguments"""
            # cf.
            if allow_additionals:
                # before starting the regular parsing, we check if the first argument is a script name,
                # in which case we should caller the runner directly with the remaining args.
                if len(sys.argv) >= 2:
                    script_name = sys.argv[1]
                    self.settings = {}
                    runner = self.get_component('runner')
                    if runner.has_script(script_name):
                        # This is a valid script name, so we should add the "run" command
                        # before this script name:
                        sys.argv.insert(1, "run")
                #"Parsing args: %s", sys.argv)
                self.settings, self.additional_args = self.parsers['main'].parse_known_args()
                self.settings = vars(self.settings)
                #"Got settings: %s", self.settings)
                #"Got additional args: %s", self.additional_args)
                self.settings = vars(self.parsers['main'].parse_args())
  • ⇒ And this works just fine, so now I can run script directly without the “run” command, like this:
    $ nvp coingecko monitored list
  • Arff, just checking the logs from coingecko data retrieval scripts, I noticed I could sometimes get an error like this:
    2022/05/18 21:45:06 [__main__] INFO: Writing 22 new price entries
    2022/05/18 21:45:07 [__main__] INFO: 1/69: Done updating lagging highres data for bitcoin
    2022/05/18 21:45:11 [__main__] INFO: Writing 23 new price entries
    2022/05/18 21:45:11 [__main__] INFO: 2/69: Done updating lagging highres data for ethereum
    Traceback (most recent call last):
      File "/mnt/data1/dev/projects/NervHome/nvh/crypto/", line 596, in <module>
      File "/mnt/data1/dev/projects/NervProj/nvp/", line 69, in run
        res = self.process_command(cmd)
      File "/mnt/data1/dev/projects/NervHome/nvh/crypto/", line 535, in process_command
      File "/mnt/data1/dev/projects/NervHome/nvh/crypto/", line 127, in update_highres_datasets
        self.update_highres_dataset(cid, 300)
      File "/mnt/data1/dev/projects/NervHome/nvh/crypto/", line 269, in update_highres_dataset
        self.update_price_dataset(PriceDB.HIGHRES, cid, start_stamp, range_dur, lambda _: period)
      File "/mnt/data1/dev/projects/NervHome/nvh/crypto/", line 474, in update_price_dataset
        self.fill_price_column(arr, 5, arr2, "eth")
      File "/mnt/data1/dev/projects/NervHome/nvh/crypto/", line 392, in fill_price_column
        assert nn1 < nn0, f"Unexpected number of {hint} prices: {nn1} > {nn0}"
    AssertionError: Unexpected number of eth prices: 24 > 23
  • Not really our concern here, but I cannot leave this as is right ? So let me have a look…
  • Okay, so, I have re-implemented the problematic part from fill_price_column as below:
        def fill_price_column(self, abase, col, arr, hint):
            """Fill the column with index 'col' inside the base array 'abase', with the price data
            coming from the column 1 of the array 'arr'. 'hint' should be a name giving an indication
            on what kind of prices we are trying to add in this function call."""
            assert abase is not None, "Invalid base array"
            if arr is None:
                # Nothing to fill in that case.
            if arr.shape[0] == abase.shape[0]:
                # check that all the timestamps are matching:
                assert np.all(abase[:, 0] == arr[:, 0]), f"Mismatch in {hint} timestamps (all): {abase[:,0]} != {arr[:,0]}"
                # We write all the column data:
                abase[:, col] = arr[:, 1]
                # We only have a partial match, so we expect to have less filling prices than the number of base prices:
                # nn0 = abase.shape[0]
                # nn1 = arr.shape[0]
                # Update: actually here we may also get *more* filling prices than the number of base prices.
                # This is expectable because the size of the abase is computed for the collected USD prices,
                # but we may get different arrays for BTC and ETH prices.
                # assert nn1 < nn0, f"Unexpected number of {hint} prices: {nn1} > {nn0}"
                # Check where we have the first match:
                _, idx1, idx2 = np.intersect1d(abase[:, 0], arr[:, 0], return_indices=True)
                assert np.all(abase[idx1, 0] == arr[idx2, 0]
                              ), f"Mismatch in {hint} timestamps: {abase[idx1,0]} != {arr[idx2,0]}"
                abase[idx1, col] = arr[idx2, 1]
                # If we have NaN values remaining in col, we should perform linear interpolation here:
                count = np.count_nonzero(np.isnan(abase[:, col]))
                if count > 0:
                    logger.warning("Interpolating %d NaN values in %s prices", count, hint)
                    arr = abase[:, col]
                    nans, xfunc = nan_helper(arr)
                    arr[nans] = np.interp(xfunc(nans), xfunc(~nans), arr[~nans])
                    abase[:, col] = arr
                    count = np.count_nonzero(np.isnan(abase[:, col]))
                    # Should not have NaNs anymore:
                    assert count == 0, f"Could still find NaN in array {abase[:, col]}"
  • ⇒ this will now use the “intersection” between the 2 arrays time values to fill the first array, and then it might perform some interpolation to fill the remaining NaN values…
  • Problem is, I cannot really test that for now as the executing conigecko price retrieval script is now working properly again (ie. it is not collecting lagging prices anymore) 😅 So I need to wait until the next time this is failing to check how it behaves lol.
  • Anyway, now finally back to our main concern here: finding coins given a pattern, so how should we do that ?
  • Note: I should onte here that the support to search for coins given a pattern was already implemented at the CoinDB level with the following method, so all I really had to add here was a small frontend to use that method:
        def find_coins(self, content):
            """Find the coin matching a content in the list."""
            # Find coins with a given content text:
            # Will return the coin_id/name/start_timestamp/symbol/platform_id
            sql = f"SELECT coin_id FROM coins WHERE coin_id "
            sql += f"LIKE '%{content}%' OR name LIKE '%{content}%' OR symbol LIKE '%{content}%'; "
            cur = self.sql_db.execute(sql)
            res = cur.fetchall()
            return [row[0] for row in res]
  • ⇒ Here we go! I built a new command called “find-coin” as follow:
        def process_command(self, cmd):
            """Check if this component can process the given command"""
            if cmd == 'update-highres':
                return True
            if cmd == 'update-history':
                return True
            if cmd == "find-coin":
                txt = self.get_param("search_txt")
                cids = self.get_coin_db().find_coins(txt)
                ncoins = len(cids)
                # Get the price data:
                price_data = self.get_prices(cids, ['usd'], include_market_cap=False, include_24hr_vol=False)
                res = []
                idlen = 0
                namelen = 0
                symlen = 0
                pricelen = 10
                coins = []
                for i, cid in enumerate(cids):
                    coin = self.get_coin(cid)
                    idlen = max(len(coin.get_id()), idlen)
                    namelen = max(len(, namelen)
                    symlen = max(len(coin.symbol()), symlen)
                idstr = "Id"
                symstr = "Symbol"
                namestr = "Name"
                pricestr = "Price"
                line = f"Num    {idstr:<{idlen+2}}{symstr:<{symlen+2}}{pricestr:<{pricelen+2}}{namestr:<{namelen}}"
                for i, coin in enumerate(coins):
                    price = price_data[coin.get_id()].get('usd', 0.0)
                    price = f'${price:.4g}'
                    line = f"{i+1:02d}/{ncoins}  {coin.get_id():<{idlen+2}}{coin.symbol():<{symlen+2}}{price:<{pricelen+2}}{<{namelen}}"
      "Found %d coins:\n%s", ncoins, "\n".join(res))
                #"Coin %d/%d: '%s', symbol: '%s', id: '%s'", i+1,
                #                 ncoins,, coin.symbol(), coin.get_id())
                return True
  • And with that I get this kind of result on the command line:
    $ nvp coingecko find-coin ada
    2022/05/18 22:15:53 [__main__] INFO: Found 61 coins:
    Num    Id                       Symbol      Price       Name
    01/61  1x-short-cardano-token   adahedge    $10.26      1X Short Cardano Token
    02/61  3x-long-cardano-token    adabull     $1.34       3X Long Cardano Token
    03/61  3x-short-cardano-token   adabear     $1.001e-08  3X Short Cardano Token
    04/61  aave-dai                 adai        $1.001      Aave DAI
    05/61  adamant-messenger        adm         $0.007166   ADAMANT Messenger
    06/61  cardano                  ada         $0.5266     Cardano
    07/61  derivadao                ddx         $1.36       DerivaDAO
    08/61  mahadao                  maha        $1.42       MahaDAO
    09/61  sada                     sada        $0.6211     sADA
    10/61  yadacoin                 yda         $0.002556   YadaCoin
    11/61  0-5x-long-cardano-token  adahalf     $3.162e+04  0.5X Long Cardano Token
    12/61  matic-aave-dai           madai       $1.007      Matic Aave Interest Bearing DAI
    13/61  adappter-token           adp         $0.01942    Adappter Token
    14/61  dappradar                $radar      $0.00725    DappRadar
    15/61  aave-dai-v1              adai        $1.001      Aave DAI v1
    16/61  instadapp                inst        $0.6535     Instadapp
    17/61  adamant                  addy        $0.3543     Adamant
    18/61  binance-peg-cardano      ada         $0.5257     Binance-Peg Cardano
  • ⇒ Pretty cool, isn't it 😎?
  • And now let's do what I wanted to do from the beginning with this new feature: find the id of LUNA and UST to add them in my monitoring system 🤣, you know, just to keep track of those values in case they would come back to life one day… (yeah… I'm just dreaming, I know that lol)
  • So searching for luna:
    $ nvp coingecko find-coin luna
    2022/05/18 22:20:44 [__main__] INFO: Found 27 coins:
    Num    Id                                      Symbol     Price       Name
    01/27  lunarium                                xln        $0.0006445  Lunarium
    02/27  terra-luna                              luna       $0.0001706  Terra
    03/27  wrapped-terra                           luna       $0.0001698  Wrapped Terra
    04/27  aluna                                   aln        $0.01057    Aluna
    05/27  lunadoge                                loge       $9.25e-16   LunaDoge
    06/27  lunarswap                               lunar      $6.16e-06   LunarSwap
    07/27  lunachow                                luchow     $3.606e-07  LunaChow
    08/27  lunaland                                lln        $0.001261   LunaLand
    09/27  luna-pad                                lunapad    $0.006394   Luna-Pad
    10/27  lunar                                   lnr        $1.479e-08  Lunar
    11/27  lunafox                                 lufx       $2e-18      LunaFox
    12/27  bonded-luna                             bluna      $0.00116    Bonded Luna
    13/27  luna-1x                                 luna1x     $0          LUNA 1x
    14/27  lunarbrain                              lun        $0.00079    LunarBrain
    15/27  luna-rush                               lus        $0.02474    Luna Rush
    16/27  lunaverse                               luv        $0.002186   Lunaverse
    17/27  luna-wormhole                           luna       $0.0001725  LUNA (Wormhole)
    18/27  metaluna                                metaluna   $2.296e-05  MetaLuna
    19/27  solunavax-index                         solunavax  $83.12      SOLUNAVAX Index
    20/27  stader-lunax                            lunax      $0.000192   Stader LunaX
    21/27  nexus-bluna-token-share-representation  nLuna      $0.002256   Nexus bLuna token share representation
    22/27  staked-luna                             stluna     $0.003006   Staked Luna
    23/27  lunafi                                  lfi        $0          Lunafi
    24/27  lunar-token                             lunar      $0.7432     Lunar Token
    25/27  prism-cluna                             cluna      $0.008605   Prism cLUNA
    26/27  prism-pluna                             pluna      $0.0001197  Prism pLUNA
    27/27  prism-yluna                             yluna      $0.008113   Prism yLUNA
  • So I guess this should be terra-luna: let's add that one:
    $ nvp coingecko monitored add terra-luna
    2022/05/18 22:22:21 [__main__] INFO: Added monitored coins: ['terra-luna']
  • OK, now let's search for UST:
    $ nvp coingecko find-coin ust
    2022/05/18 22:23:23 [__main__] INFO: Found 78 coins:
    Num    Id                                  Symbol                                  Price       Name
    01/78  bitcoin-trust                       bct                                     $0.01904    Bitcoin Trust
    02/78  crust-network                       cru                                     $0.97       Crust Network
    03/78  cryptrust                           ctrt                                    $4.07e-06   Cryptrust
    04/78  custom-contract-network             ccn                                     $3.494e-05  Custom contract network
    05/78  dust-token                          dust                                    $0.0004455  DUST Token
    06/78  freight-trust-network               edi                                     $0.0002124  Freight Trust Network
    07/78  global-human-trust                  ght                                     $39.53      Global Human Trust
    08/78  global-trust-coin                   gtc                                     $0.3516     Global Trust Coin
    09/78  globaltrustfund-token               gtf                                     $0.0005818  GLOBALTRUSTFUND TOKEN
    10/78  istardust                           isdt                                    $0.0003652  Istardust
    11/78  just                                jst                                     $0.03982    JUST
    12/78  justbet                             winr                                    $0.0001499  JustBet
    13/78  just-stablecoin                     usdj                                    $1.002      JUST Stablecoin
    14/78  mirrored-invesco-qqq-trust          mqqq                                    $30.49      Mirrored Invesco QQQ Trust
    15/78  mirrored-ishares-gold-trust         miau                                    $3.63       Mirrored iShares Gold Trust
    16/78  mirrored-ishares-silver-trust       mslv                                    $11.49      Mirrored iShares Silver Trust
    17/78  mixtrust                            mxt                                     $0.001199   MixTrust
    18/78  must                                must                                    $20.98      Must
    19/78  terrausd                            ust                                     $0.1052     TerraUSD
    20/78  trust                               trust                                   $0.005176   Harmony Block Capital
    21/78  trust-ether-reorigin                teo                                     $7.585e-05  Trust Ether ReOrigin
    22/78  trustswap                           swap                                    $0.4013     Trustswap
    23/78  trustusd                            trusd                                   $0.0003428  TrustUSD
    24/78  trustverse                          trv                                     $0.009736   TrustVerse
    25/78  trust-wallet-token                  twt                                     $0.6675     Trust Wallet Token
    26/78  utrust                              utk                                     $0.1323     UTRUST
    27/78  wetrust                             trst                                    $0.002013   WeTrust
    28/78  idle-dai-risk-adjusted              idleDAISafe                             $1.075      IdleDAI (Risk Adjusted)
    29/78  idle-usdc-risk-adjusted             idleUSDCSafe                            $1.074      IdleUSDC (Risk Adjusted)
    30/78  idle-usdt-risk-adjusted             IdleUSDTSafe                            $1.081      IdleUSDT (Risk Adjusted)
    31/78  dust-protocol                       dust                                    $2.02       DUST Protocol
    32/78  trustworks                          trust                                   $0.384      Trustworks
    33/78  australian-safe-shepherd            ass                                     $7.135e-10  Australian Safe Shepherd
    34/78  sensitrust                          sets                                    $0.02797    Sensitrust
    35/78  rug-busters                         rugbust                                 $0.004318   Rug Busters
    36/78  greentrust                          gnt                                     $8.86e-10   GreenTrust
    37/78  trustpad                            tpad                                    $0.1246     TrustPad
    38/78  moontrust                           mntt                                    $1.779e-07  MoonTrust
    39/78  sustainable-energy-token            set                                     $8.05e-16   Sustainable Energy Token
    40/78  australian-kelpie                   knockers                                $1.373e-10  Australian Kelpie
    41/78  crust-storage-market                csm                                     $0.01383    Crust Storage Market
    42/78  trustfi-network-token               tfi                                     $0.02015    TrustFi Network Token
    43/78  pist-trust                          pist                                    $0.05005    Pist Trust
    44/78  18433-faust                         realtoken-s-18433-faust-ave-detroit-mi  $42.26      RealT Token - 18433 Faust Ave, Detroit, MI, 48219
    45/78  itrust-governance-token             itg                                     $0.009724   iTrust Governance Token
    46/78  robust-token                        rbt                                     $7.74       Robust Token
    47/78  stargazer-protocol                  stardust                                $5.468e-11  Stargazer Protocol
    48/78  trustbase                           tbe                                     $0.0003444  TrustBase
    49/78  trustercoin                         tsc                                     $0.007037   TrusterCoin
    50/78  wrapped-ust                         ust                                     $0.1056     Wrapped UST
    51/78  justyours                           just                                    $9.615e-05  JustYours
    52/78  braintrust                          btrst                                   $2.65       Braintrust
    53/78  busta                               bust                                    $2.666e-05  BUSTA
    54/78  dust                                dust                                    $0.09771    Dust
    55/78  trustkeys-network                   trustk                                  $0.1087     TrustKeys Network
    56/78  justfarm                            jfm                                     $0.00128    JustFarm
    57/78  upstabletoken                       ustx                                    $0          UpStableToken
    58/78  trustnft                            trustnft                                $0.001545   TrustNFT
    59/78  trusted-node                        tnode                                   $0.01627    Trusted Node
    60/78  hyper-trust                         hptt                                    $0.03958    Hyper Trust
    61/78  anchorust                           aust                                    $0.1335     AnchorUST
    62/78  terrausd-wormhole                   ust                                     $0.1007     TerraUSD (Wormhole)
    63/78  trustrise                           trise                                   $6.43e-06   TrustRise
    64/78  moontrustbsc                        mnttbsc                                 $6.14e-07   MoonTrustBSC
    65/78  moonsdust                           moond                                   $0.08643    MoonsDust
    66/78  scardust                            scard                                   $2.03e-08   SCARDust
    67/78  assangedao                          justice                                 $0.0001822  AssangeDAO
    68/78  australian-crypto-coin-green        accg                                    $0.06471    Australian Crypto Coin Green
    69/78  rfust                               rfust                                   $4.611e-08  rfUST
    70/78  trustpay                            tph                                     $0.01895    Trustpay
    71/78  trust-recruit                       trt                                     $0          Trust Recruit
    72/78  jarvis-synthetic-australian-dollar  jaud                                    $0          Jarvis Synthetic Australian Dollar
    73/78  dumpbuster                          gtfo                                    $1.67e-06   DumpBuster
    74/78  justmoney                           jm                                      $2.314e-05  JustMoney
    75/78  kwiktrust                           ktx                                     $0.07093    KwikTrust
    76/78  qqq-tokenized-stock-defichain       dQQQ                                    $285.4      Invesco QQQ Trust Defichain
    77/78  silver-tokenized-stock-defichain    dSLV                                    $19.5       iShares Silver Trust Defichain
    78/78  spdr-s-p-500-etf-trust-defichain    dspy                                    $388.7      SPDR S&P 500 ETF Trust Defichain
  • Feeeww, there are a lot of them here. But the one I'm looking for is terrausd:
    $ nvp coingecko monitored add terrausd
    2022/05/18 22:25:52 [__main__] INFO: Added monitored coins: ['terrausd']
  • Alright, now let's wait a little for the initial data download to occur, and then we will check the CryptoView gui…
  • Note: Actually I should update the history data manually to avoid waiting 1 full day here:
    * nvp coingecko update-history
  • And now I have the LUNA/UST coins loaded in CryptoView as expected, yeeepeeee:

  • So let's call it a day now lol I need some sleep! 😴
  • blog/2022/0519_nervproj_finding_coingecko_coin_ids.txt
  • Last modified: 2022/05/19 07:05
  • (external edit)