blog:2022:0519_nervproj_finding_coingecko_coin_ids [NervTech's Wiki]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== NervProj: Finding & adding monitored coins ======

{{tag>dev python crypto nervhome finance}}

In one of my previous articles (available [[https://wiki.nervtech.org/doku.php?id=blog:2022:0510_nervproj_crypto_coins_monitoring#support_to_list_add_remove_monitored_coins|here]]) I introduced support to list/add/remove monitored coins from the command line with my coingecko component. This is very nice and all, but this also assumes that you //know// what is the ID of the coin you want to add to the list, which is not always absolutely clear...

So, what I want to do here is to add support to **search** for all coins matching a given pattern and list them directly on the command line so I could then select the proper one in the list and add the corresponding coin id.

====== ======

===== NervProj script commands =====

  * Before I get to the actual problem I want to fix here, there is another minor thing I want to change too, each time I want to run a script in the NervProj framework I currently have to type ''nvp run script_name'' => I think that, instead, I could try to **bypass** the "run" keyword here and use instead ''nvp script_name'' directly: is that possible ? Let's see...
  * Oki oki, so it seems a bit tricky to handle this kind of setup with **argparse** only, so I decided to use a simple "workaround" for that:
  * First I added a method in the runner component to check if a given script exists: <sxh python; highlight: []>    def has_script(self, script_name):
        """Check if a given script name is available."""
        projs = self.ctx.get_projects()
        for proj in projs:
            desc = proj.get_script(script_name)
            if desc is not None:
                return True

        desc = self.scripts.get(script_name, None)
        return desc is not None</sxh>
  * Then **before** parsing the command line arguments I check if the second argument in the list is a script name, and if this is the case, I manually add the "run" command in front of it before parsing the args as usual (note that this operation only makes sense ont the master context for now, so placed in the ''allow_additionals'' section): <sxh python; highlight: []>    def parse_args(self, allow_additionals):
        """Parse the command line arguments"""
        # cf. https://docs.python.org/3.4/library/argparse.html#partial-parsing
        if allow_additionals:
            # before starting the regular parsing, we check if the first argument is a script name,
            # in which case we should caller the runner directly with the remaining args.
            if len(sys.argv) >= 2:
                script_name = sys.argv[1]
                self.settings = {}
                runner = self.get_component('runner')
                if runner.has_script(script_name):
                    # This is a valid script name, so we should add the "run" command
                    # before this script name:
                    sys.argv.insert(1, "run")

            # logger.info("Parsing args: %s", sys.argv)
            self.settings, self.additional_args = self.parsers['main'].parse_known_args()
            self.settings = vars(self.settings)
            # logger.info("Got settings: %s", self.settings)
            # logger.info("Got additional args: %s", self.additional_args)
        else:
            self.settings = vars(self.parsers['main'].parse_args())</sxh>

  * => And this works just fine, so now I can run script directly without the "run" command, like this: <sxh bash; highlight: []>$ nvp coingecko monitored list</sxh>

===== Issue with coingecko highres data retrieval =====

    * Arff, just checking the logs from coingecko data retrieval scripts, I noticed I could sometimes get an error like this: <sxh bash; highlight: []>2022/05/18 21:45:06 [__main__] INFO: Writing 22 new price entries
2022/05/18 21:45:07 [__main__] INFO: 1/69: Done updating lagging highres data for bitcoin
2022/05/18 21:45:11 [__main__] INFO: Writing 23 new price entries
2022/05/18 21:45:11 [__main__] INFO: 2/69: Done updating lagging highres data for ethereum
Traceback (most recent call last):
  File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 596, in <module>
    comp.run()
  File "/mnt/data1/dev/projects/NervProj/nvp/nvp_component.py", line 69, in run
    res = self.process_command(cmd)
  File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 535, in process_command
    self.update_highres_datasets()
  File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 127, in update_highres_datasets
    self.update_highres_dataset(cid, 300)
  File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 269, in update_highres_dataset
    self.update_price_dataset(PriceDB.HIGHRES, cid, start_stamp, range_dur, lambda _: period)
  File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 474, in update_price_dataset
    self.fill_price_column(arr, 5, arr2, "eth")
  File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 392, in fill_price_column
    assert nn1 < nn0, f"Unexpected number of {hint} prices: {nn1} > {nn0}"
AssertionError: Unexpected number of eth prices: 24 > 23</sxh>

  * Not really our concern here, but I cannot leave this as is right ? So let me have a look...

  * Okay, so, I have re-implemented the problematic part from ''fill_price_column'' as below: <sxh python; highlight: []>    def fill_price_column(self, abase, col, arr, hint):
        """Fill the column with index 'col' inside the base array 'abase', with the price data
        coming from the column 1 of the array 'arr'. 'hint' should be a name giving an indication
        on what kind of prices we are trying to add in this function call."""

        assert abase is not None, "Invalid base array"
        if arr is None:
            # Nothing to fill in that case.
            return

        if arr.shape[0] == abase.shape[0]:
            # check that all the timestamps are matching:
            assert np.all(abase[:, 0] == arr[:, 0]), f"Mismatch in {hint} timestamps (all): {abase[:,0]} != {arr[:,0]}"
            # We write all the column data:
            abase[:, col] = arr[:, 1]
        else:
            # We only have a partial match, so we expect to have less filling prices than the number of base prices:
            # nn0 = abase.shape[0]
            # nn1 = arr.shape[0]
            # Update: actually here we may also get *more* filling prices than the number of base prices.
            # This is expectable because the size of the abase is computed for the collected USD prices,
            # but we may get different arrays for BTC and ETH prices.
            # assert nn1 < nn0, f"Unexpected number of {hint} prices: {nn1} > {nn0}"

            # Check where we have the first match:
            _, idx1, idx2 = np.intersect1d(abase[:, 0], arr[:, 0], return_indices=True)
            assert np.all(abase[idx1, 0] == arr[idx2, 0]
                          ), f"Mismatch in {hint} timestamps: {abase[idx1,0]} != {arr[idx2,0]}"
            abase[idx1, col] = arr[idx2, 1]

            # If we have NaN values remaining in col, we should perform linear interpolation here:
            count = np.count_nonzero(np.isnan(abase[:, col]))
            if count > 0:
                logger.warning("Interpolating %d NaN values in %s prices", count, hint)
                arr = abase[:, col]
                nans, xfunc = nan_helper(arr)
                arr[nans] = np.interp(xfunc(nans), xfunc(~nans), arr[~nans])
                abase[:, col] = arr

                count = np.count_nonzero(np.isnan(abase[:, col]))

                # Should not have NaNs anymore:
                assert count == 0, f"Could still find NaN in array {abase[:, col]}"
</sxh>

  * => this will now use the "intersection" between the 2 arrays time values to fill the first array, and then it might perform some interpolation to fill the remaining NaN values...

  * Problem is, I cannot really test that for now as the executing conigecko price retrieval script is now working properly again (ie. it is not collecting lagging prices anymore) 😅 So I need to wait until the next time this is failing to check how it behaves lol.

===== Find coins with a given pattern =====

  * Anyway, now finally back to our main concern here: finding coins given a pattern, so how should we do that ?

  * **Note**: I should onte here that the support to search for coins given a pattern was already implemented at the **CoinDB** level with the following method, so all I really had to add here was a small frontend to use that method: <sxh python; highlight: []>    def find_coins(self, content):
        """Find the coin matching a content in the list."""
        # Find coins with a given content text:
        # Will return the coin_id/name/start_timestamp/symbol/platform_id
        sql = f"SELECT coin_id FROM coins WHERE coin_id "
        sql += f"LIKE '%{content}%' OR name LIKE '%{content}%' OR symbol LIKE '%{content}%'; "
        cur = self.sql_db.execute(sql)
        res = cur.fetchall()
        return [row[0] for row in res]</sxh>

  * => Here we go! I built a new command called "find-coin" as follow: <sxh python; highlight: []>    def process_command(self, cmd):
        """Check if this component can process the given command"""

        if cmd == 'update-highres':
            self.update_highres_datasets()
            return True

        if cmd == 'update-history':
            self.update_history_datasets()
            return True

        if cmd == "find-coin":
            txt = self.get_param("search_txt")
            cids = self.get_coin_db().find_coins(txt)
            ncoins = len(cids)

            # Get the price data:
            price_data = self.get_prices(cids, ['usd'], include_market_cap=False, include_24hr_vol=False)

            res = []

            idlen = 0
            namelen = 0
            symlen = 0
            pricelen = 10
            coins = []

            for i, cid in enumerate(cids):
                coin = self.get_coin(cid)
                idlen = max(len(coin.get_id()), idlen)
                namelen = max(len(coin.name()), namelen)
                symlen = max(len(coin.symbol()), symlen)
                coins.append(coin)

            idstr = "Id"
            symstr = "Symbol"
            namestr = "Name"
            pricestr = "Price"
            line = f"Num    {idstr:<{idlen+2}}{symstr:<{symlen+2}}{pricestr:<{pricelen+2}}{namestr:<{namelen}}"
            res.append(line)
            res.append("")

            for i, coin in enumerate(coins):
                price = price_data[coin.get_id()].get('usd', 0.0)
                price = f'${price:.4g}'
                line = f"{i+1:02d}/{ncoins}  {coin.get_id():<{idlen+2}}{coin.symbol():<{symlen+2}}{price:<{pricelen+2}}{coin.name():<{namelen}}"
                res.append(line)

            logger.info("Found %d coins:\n%s", ncoins, "\n".join(res))
            # logger.info("Coin %d/%d: '%s', symbol: '%s', id: '%s'", i+1,
            #                 ncoins, coin.name(), coin.symbol(), coin.get_id())
            return True</sxh>

  * And with that I get this kind of result on the command line: <sxh bash; highlight: []>$ nvp coingecko find-coin ada
2022/05/18 22:15:53 [__main__] INFO: Found 61 coins:
Num    Id                       Symbol      Price       Name

01/61  1x-short-cardano-token   adahedge    $10.26      1X Short Cardano Token
02/61  3x-long-cardano-token    adabull     $1.34       3X Long Cardano Token
03/61  3x-short-cardano-token   adabear     $1.001e-08  3X Short Cardano Token
04/61  aave-dai                 adai        $1.001      Aave DAI
05/61  adamant-messenger        adm         $0.007166   ADAMANT Messenger
06/61  cardano                  ada         $0.5266     Cardano
07/61  derivadao                ddx         $1.36       DerivaDAO
08/61  mahadao                  maha        $1.42       MahaDAO
09/61  sada                     sada        $0.6211     sADA
10/61  yadacoin                 yda         $0.002556   YadaCoin
11/61  0-5x-long-cardano-token  adahalf     $3.162e+04  0.5X Long Cardano Token
12/61  matic-aave-dai           madai       $1.007      Matic Aave Interest Bearing DAI
13/61  adappter-token           adp         $0.01942    Adappter Token
14/61  dappradar                $radar      $0.00725    DappRadar
15/61  aave-dai-v1              adai        $1.001      Aave DAI v1
16/61  instadapp                inst        $0.6535     Instadapp
17/61  adamant                  addy        $0.3543     Adamant
18/61  binance-peg-cardano      ada         $0.5257     Binance-Peg Cardano
</sxh>

  * => Pretty cool, isn't it 😎?

===== Conclusion =====

  * And now let's do what I wanted to do from the beginning with this new feature: find the id of LUNA and UST to add them in my monitoring system 🤣, you know, just to keep track of those values in case they would come back to life one day... (yeah... I'm just dreaming, I know that lol)
  * So searching for luna: <sxh bash; highlight: []>$ nvp coingecko find-coin luna
2022/05/18 22:20:44 [__main__] INFO: Found 27 coins:
Num    Id                                      Symbol     Price       Name

01/27  lunarium                                xln        $0.0006445  Lunarium
02/27  terra-luna                              luna       $0.0001706  Terra
03/27  wrapped-terra                           luna       $0.0001698  Wrapped Terra
04/27  aluna                                   aln        $0.01057    Aluna
05/27  lunadoge                                loge       $9.25e-16   LunaDoge
06/27  lunarswap                               lunar      $6.16e-06   LunarSwap
07/27  lunachow                                luchow     $3.606e-07  LunaChow
08/27  lunaland                                lln        $0.001261   LunaLand
09/27  luna-pad                                lunapad    $0.006394   Luna-Pad
10/27  lunar                                   lnr        $1.479e-08  Lunar
11/27  lunafox                                 lufx       $2e-18      LunaFox
12/27  bonded-luna                             bluna      $0.00116    Bonded Luna
13/27  luna-1x                                 luna1x     $0          LUNA 1x
14/27  lunarbrain                              lun        $0.00079    LunarBrain
15/27  luna-rush                               lus        $0.02474    Luna Rush
16/27  lunaverse                               luv        $0.002186   Lunaverse
17/27  luna-wormhole                           luna       $0.0001725  LUNA (Wormhole)
18/27  metaluna                                metaluna   $2.296e-05  MetaLuna
19/27  solunavax-index                         solunavax  $83.12      SOLUNAVAX Index
20/27  stader-lunax                            lunax      $0.000192   Stader LunaX
21/27  nexus-bluna-token-share-representation  nLuna      $0.002256   Nexus bLuna token share representation
22/27  staked-luna                             stluna     $0.003006   Staked Luna
23/27  lunafi                                  lfi        $0          Lunafi
24/27  lunar-token                             lunar      $0.7432     Lunar Token
25/27  prism-cluna                             cluna      $0.008605   Prism cLUNA
26/27  prism-pluna                             pluna      $0.0001197  Prism pLUNA
27/27  prism-yluna                             yluna      $0.008113   Prism yLUNA</sxh>

  * So I guess this should be **terra-luna**: let's add that one: <sxh bash; highlight: []>$ nvp coingecko monitored add terra-luna
2022/05/18 22:22:21 [__main__] INFO: Added monitored coins: ['terra-luna']</sxh>

  * OK, now let's search for UST: <sxh bash; highlight: []>$ nvp coingecko find-coin ust
2022/05/18 22:23:23 [__main__] INFO: Found 78 coins:
Num    Id                                  Symbol                                  Price       Name

01/78  bitcoin-trust                       bct                                     $0.01904    Bitcoin Trust
02/78  crust-network                       cru                                     $0.97       Crust Network
03/78  cryptrust                           ctrt                                    $4.07e-06   Cryptrust
04/78  custom-contract-network             ccn                                     $3.494e-05  Custom contract network
05/78  dust-token                          dust                                    $0.0004455  DUST Token
06/78  freight-trust-network               edi                                     $0.0002124  Freight Trust Network
07/78  global-human-trust                  ght                                     $39.53      Global Human Trust
08/78  global-trust-coin                   gtc                                     $0.3516     Global Trust Coin
09/78  globaltrustfund-token               gtf                                     $0.0005818  GLOBALTRUSTFUND TOKEN
10/78  istardust                           isdt                                    $0.0003652  Istardust
11/78  just                                jst                                     $0.03982    JUST
12/78  justbet                             winr                                    $0.0001499  JustBet
13/78  just-stablecoin                     usdj                                    $1.002      JUST Stablecoin
14/78  mirrored-invesco-qqq-trust          mqqq                                    $30.49      Mirrored Invesco QQQ Trust
15/78  mirrored-ishares-gold-trust         miau                                    $3.63       Mirrored iShares Gold Trust
16/78  mirrored-ishares-silver-trust       mslv                                    $11.49      Mirrored iShares Silver Trust
17/78  mixtrust                            mxt                                     $0.001199   MixTrust
18/78  must                                must                                    $20.98      Must
19/78  terrausd                            ust                                     $0.1052     TerraUSD
20/78  trust                               trust                                   $0.005176   Harmony Block Capital
21/78  trust-ether-reorigin                teo                                     $7.585e-05  Trust Ether ReOrigin
22/78  trustswap                           swap                                    $0.4013     Trustswap
23/78  trustusd                            trusd                                   $0.0003428  TrustUSD
24/78  trustverse                          trv                                     $0.009736   TrustVerse
25/78  trust-wallet-token                  twt                                     $0.6675     Trust Wallet Token
26/78  utrust                              utk                                     $0.1323     UTRUST
27/78  wetrust                             trst                                    $0.002013   WeTrust
28/78  idle-dai-risk-adjusted              idleDAISafe                             $1.075      IdleDAI (Risk Adjusted)
29/78  idle-usdc-risk-adjusted             idleUSDCSafe                            $1.074      IdleUSDC (Risk Adjusted)
30/78  idle-usdt-risk-adjusted             IdleUSDTSafe                            $1.081      IdleUSDT (Risk Adjusted)
31/78  dust-protocol                       dust                                    $2.02       DUST Protocol
32/78  trustworks                          trust                                   $0.384      Trustworks
33/78  australian-safe-shepherd            ass                                     $7.135e-10  Australian Safe Shepherd
34/78  sensitrust                          sets                                    $0.02797    Sensitrust
35/78  rug-busters                         rugbust                                 $0.004318   Rug Busters
36/78  greentrust                          gnt                                     $8.86e-10   GreenTrust
37/78  trustpad                            tpad                                    $0.1246     TrustPad
38/78  moontrust                           mntt                                    $1.779e-07  MoonTrust
39/78  sustainable-energy-token            set                                     $8.05e-16   Sustainable Energy Token
40/78  australian-kelpie                   knockers                                $1.373e-10  Australian Kelpie
41/78  crust-storage-market                csm                                     $0.01383    Crust Storage Market
42/78  trustfi-network-token               tfi                                     $0.02015    TrustFi Network Token
43/78  pist-trust                          pist                                    $0.05005    Pist Trust
44/78  18433-faust                         realtoken-s-18433-faust-ave-detroit-mi  $42.26      RealT Token - 18433 Faust Ave, Detroit, MI, 48219
45/78  itrust-governance-token             itg                                     $0.009724   iTrust Governance Token
46/78  robust-token                        rbt                                     $7.74       Robust Token
47/78  stargazer-protocol                  stardust                                $5.468e-11  Stargazer Protocol
48/78  trustbase                           tbe                                     $0.0003444  TrustBase
49/78  trustercoin                         tsc                                     $0.007037   TrusterCoin
50/78  wrapped-ust                         ust                                     $0.1056     Wrapped UST
51/78  justyours                           just                                    $9.615e-05  JustYours
52/78  braintrust                          btrst                                   $2.65       Braintrust
53/78  busta                               bust                                    $2.666e-05  BUSTA
54/78  dust                                dust                                    $0.09771    Dust
55/78  trustkeys-network                   trustk                                  $0.1087     TrustKeys Network
56/78  justfarm                            jfm                                     $0.00128    JustFarm
57/78  upstabletoken                       ustx                                    $0          UpStableToken
58/78  trustnft                            trustnft                                $0.001545   TrustNFT
59/78  trusted-node                        tnode                                   $0.01627    Trusted Node
60/78  hyper-trust                         hptt                                    $0.03958    Hyper Trust
61/78  anchorust                           aust                                    $0.1335     AnchorUST
62/78  terrausd-wormhole                   ust                                     $0.1007     TerraUSD (Wormhole)
63/78  trustrise                           trise                                   $6.43e-06   TrustRise
64/78  moontrustbsc                        mnttbsc                                 $6.14e-07   MoonTrustBSC
65/78  moonsdust                           moond                                   $0.08643    MoonsDust
66/78  scardust                            scard                                   $2.03e-08   SCARDust
67/78  assangedao                          justice                                 $0.0001822  AssangeDAO
68/78  australian-crypto-coin-green        accg                                    $0.06471    Australian Crypto Coin Green
69/78  rfust                               rfust                                   $4.611e-08  rfUST
70/78  trustpay                            tph                                     $0.01895    Trustpay
71/78  trust-recruit                       trt                                     $0          Trust Recruit
72/78  jarvis-synthetic-australian-dollar  jaud                                    $0          Jarvis Synthetic Australian Dollar
73/78  dumpbuster                          gtfo                                    $1.67e-06   DumpBuster
74/78  justmoney                           jm                                      $2.314e-05  JustMoney
75/78  kwiktrust                           ktx                                     $0.07093    KwikTrust
76/78  qqq-tokenized-stock-defichain       dQQQ                                    $285.4      Invesco QQQ Trust Defichain
77/78  silver-tokenized-stock-defichain    dSLV                                    $19.5       iShares Silver Trust Defichain
78/78  spdr-s-p-500-etf-trust-defichain    dspy                                    $388.7      SPDR S&P 500 ETF Trust Defichain</sxh>

  * Feeeww, there are a lot of them here. But the one I'm looking for is **terrausd**: <sxh bash; highlight: []>$ nvp coingecko monitored add terrausd
2022/05/18 22:25:52 [__main__] INFO: Added monitored coins: ['terrausd']</sxh>

  * Alright, now let's wait a little for the initial data download to occur, and then we will check the CryptoView gui...

  * **Note**: Actually I should update the history data manually to avoid waiting 1 full day here: <sxh bash; highlight: []>* nvp coingecko update-history</sxh>

  * And now I have the LUNA/UST coins loaded in **CryptoView** as expected, yeeepeeee: 

{{ blog:2022:0518:luna_coin_added.png }}

  * So let's call it a day now lol I need some sleep! 😴