NervProj: Finding & adding monitored coins
In one of my previous articles (available here) I introduced support to list/add/remove monitored coins from the command line with my coingecko component. This is very nice and all, but this also assumes that you know what is the ID of the coin you want to add to the list, which is not always absolutely clear…
So, what I want to do here is to add support to search for all coins matching a given pattern and list them directly on the command line so I could then select the proper one in the list and add the corresponding coin id.
NervProj script commands
- Before I get to the actual problem I want to fix here, there is another minor thing I want to change too, each time I want to run a script in the NervProj framework I currently have to type
nvp run script_name⇒ I think that, instead, I could try to bypass the “run” keyword here and use insteadnvp script_namedirectly: is that possible ? Let's see… - Oki oki, so it seems a bit tricky to handle this kind of setup with argparse only, so I decided to use a simple “workaround” for that:
- First I added a method in the runner component to check if a given script exists:
def has_script(self, script_name): """Check if a given script name is available.""" projs = self.ctx.get_projects() for proj in projs: desc = proj.get_script(script_name) if desc is not None: return True desc = self.scripts.get(script_name, None) return desc is not None - Then before parsing the command line arguments I check if the second argument in the list is a script name, and if this is the case, I manually add the “run” command in front of it before parsing the args as usual (note that this operation only makes sense ont the master context for now, so placed in the
allow_additionalssection):def parse_args(self, allow_additionals): """Parse the command line arguments""" # cf. https://docs.python.org/3.4/library/argparse.html#partial-parsing if allow_additionals: # before starting the regular parsing, we check if the first argument is a script name, # in which case we should caller the runner directly with the remaining args. if len(sys.argv) >= 2: script_name = sys.argv[1] self.settings = {} runner = self.get_component('runner') if runner.has_script(script_name): # This is a valid script name, so we should add the "run" command # before this script name: sys.argv.insert(1, "run") # logger.info("Parsing args: %s", sys.argv) self.settings, self.additional_args = self.parsers['main'].parse_known_args() self.settings = vars(self.settings) # logger.info("Got settings: %s", self.settings) # logger.info("Got additional args: %s", self.additional_args) else: self.settings = vars(self.parsers['main'].parse_args())
- ⇒ And this works just fine, so now I can run script directly without the “run” command, like this:
$ nvp coingecko monitored list
Issue with coingecko highres data retrieval
- Arff, just checking the logs from coingecko data retrieval scripts, I noticed I could sometimes get an error like this:
2022/05/18 21:45:06 [__main__] INFO: Writing 22 new price entries 2022/05/18 21:45:07 [__main__] INFO: 1/69: Done updating lagging highres data for bitcoin 2022/05/18 21:45:11 [__main__] INFO: Writing 23 new price entries 2022/05/18 21:45:11 [__main__] INFO: 2/69: Done updating lagging highres data for ethereum Traceback (most recent call last): File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 596, in <module> comp.run() File "/mnt/data1/dev/projects/NervProj/nvp/nvp_component.py", line 69, in run res = self.process_command(cmd) File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 535, in process_command self.update_highres_datasets() File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 127, in update_highres_datasets self.update_highres_dataset(cid, 300) File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 269, in update_highres_dataset self.update_price_dataset(PriceDB.HIGHRES, cid, start_stamp, range_dur, lambda _: period) File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 474, in update_price_dataset self.fill_price_column(arr, 5, arr2, "eth") File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 392, in fill_price_column assert nn1 < nn0, f"Unexpected number of {hint} prices: {nn1} > {nn0}" AssertionError: Unexpected number of eth prices: 24 > 23
- Not really our concern here, but I cannot leave this as is right ? So let me have a look…
- Okay, so, I have re-implemented the problematic part from
fill_price_columnas below:def fill_price_column(self, abase, col, arr, hint): """Fill the column with index 'col' inside the base array 'abase', with the price data coming from the column 1 of the array 'arr'. 'hint' should be a name giving an indication on what kind of prices we are trying to add in this function call.""" assert abase is not None, "Invalid base array" if arr is None: # Nothing to fill in that case. return if arr.shape[0] == abase.shape[0]: # check that all the timestamps are matching: assert np.all(abase[:, 0] == arr[:, 0]), f"Mismatch in {hint} timestamps (all): {abase[:,0]} != {arr[:,0]}" # We write all the column data: abase[:, col] = arr[:, 1] else: # We only have a partial match, so we expect to have less filling prices than the number of base prices: # nn0 = abase.shape[0] # nn1 = arr.shape[0] # Update: actually here we may also get *more* filling prices than the number of base prices. # This is expectable because the size of the abase is computed for the collected USD prices, # but we may get different arrays for BTC and ETH prices. # assert nn1 < nn0, f"Unexpected number of {hint} prices: {nn1} > {nn0}" # Check where we have the first match: _, idx1, idx2 = np.intersect1d(abase[:, 0], arr[:, 0], return_indices=True) assert np.all(abase[idx1, 0] == arr[idx2, 0] ), f"Mismatch in {hint} timestamps: {abase[idx1,0]} != {arr[idx2,0]}" abase[idx1, col] = arr[idx2, 1] # If we have NaN values remaining in col, we should perform linear interpolation here: count = np.count_nonzero(np.isnan(abase[:, col])) if count > 0: logger.warning("Interpolating %d NaN values in %s prices", count, hint) arr = abase[:, col] nans, xfunc = nan_helper(arr) arr[nans] = np.interp(xfunc(nans), xfunc(~nans), arr[~nans]) abase[:, col] = arr count = np.count_nonzero(np.isnan(abase[:, col])) # Should not have NaNs anymore: assert count == 0, f"Could still find NaN in array {abase[:, col]}"
- ⇒ this will now use the “intersection” between the 2 arrays time values to fill the first array, and then it might perform some interpolation to fill the remaining NaN values…
- Problem is, I cannot really test that for now as the executing conigecko price retrieval script is now working properly again (ie. it is not collecting lagging prices anymore) 😅 So I need to wait until the next time this is failing to check how it behaves lol.
Find coins with a given pattern
- Anyway, now finally back to our main concern here: finding coins given a pattern, so how should we do that ?
- Note: I should onte here that the support to search for coins given a pattern was already implemented at the CoinDB level with the following method, so all I really had to add here was a small frontend to use that method:
def find_coins(self, content): """Find the coin matching a content in the list.""" # Find coins with a given content text: # Will return the coin_id/name/start_timestamp/symbol/platform_id sql = f"SELECT coin_id FROM coins WHERE coin_id " sql += f"LIKE '%{content}%' OR name LIKE '%{content}%' OR symbol LIKE '%{content}%'; " cur = self.sql_db.execute(sql) res = cur.fetchall() return [row[0] for row in res]
- ⇒ Here we go! I built a new command called “find-coin” as follow:
def process_command(self, cmd): """Check if this component can process the given command""" if cmd == 'update-highres': self.update_highres_datasets() return True if cmd == 'update-history': self.update_history_datasets() return True if cmd == "find-coin": txt = self.get_param("search_txt") cids = self.get_coin_db().find_coins(txt) ncoins = len(cids) # Get the price data: price_data = self.get_prices(cids, ['usd'], include_market_cap=False, include_24hr_vol=False) res = [] idlen = 0 namelen = 0 symlen = 0 pricelen = 10 coins = [] for i, cid in enumerate(cids): coin = self.get_coin(cid) idlen = max(len(coin.get_id()), idlen) namelen = max(len(coin.name()), namelen) symlen = max(len(coin.symbol()), symlen) coins.append(coin) idstr = "Id" symstr = "Symbol" namestr = "Name" pricestr = "Price" line = f"Num {idstr:<{idlen+2}}{symstr:<{symlen+2}}{pricestr:<{pricelen+2}}{namestr:<{namelen}}" res.append(line) res.append("") for i, coin in enumerate(coins): price = price_data[coin.get_id()].get('usd', 0.0) price = f'${price:.4g}' line = f"{i+1:02d}/{ncoins} {coin.get_id():<{idlen+2}}{coin.symbol():<{symlen+2}}{price:<{pricelen+2}}{coin.name():<{namelen}}" res.append(line) logger.info("Found %d coins:\n%s", ncoins, "\n".join(res)) # logger.info("Coin %d/%d: '%s', symbol: '%s', id: '%s'", i+1, # ncoins, coin.name(), coin.symbol(), coin.get_id()) return True
- And with that I get this kind of result on the command line:
$ nvp coingecko find-coin ada 2022/05/18 22:15:53 [__main__] INFO: Found 61 coins: Num Id Symbol Price Name 01/61 1x-short-cardano-token adahedge $10.26 1X Short Cardano Token 02/61 3x-long-cardano-token adabull $1.34 3X Long Cardano Token 03/61 3x-short-cardano-token adabear $1.001e-08 3X Short Cardano Token 04/61 aave-dai adai $1.001 Aave DAI 05/61 adamant-messenger adm $0.007166 ADAMANT Messenger 06/61 cardano ada $0.5266 Cardano 07/61 derivadao ddx $1.36 DerivaDAO 08/61 mahadao maha $1.42 MahaDAO 09/61 sada sada $0.6211 sADA 10/61 yadacoin yda $0.002556 YadaCoin 11/61 0-5x-long-cardano-token adahalf $3.162e+04 0.5X Long Cardano Token 12/61 matic-aave-dai madai $1.007 Matic Aave Interest Bearing DAI 13/61 adappter-token adp $0.01942 Adappter Token 14/61 dappradar $radar $0.00725 DappRadar 15/61 aave-dai-v1 adai $1.001 Aave DAI v1 16/61 instadapp inst $0.6535 Instadapp 17/61 adamant addy $0.3543 Adamant 18/61 binance-peg-cardano ada $0.5257 Binance-Peg Cardano
- ⇒ Pretty cool, isn't it 😎?
Conclusion
- And now let's do what I wanted to do from the beginning with this new feature: find the id of LUNA and UST to add them in my monitoring system 🤣, you know, just to keep track of those values in case they would come back to life one day… (yeah… I'm just dreaming, I know that lol)
- So searching for luna:
$ nvp coingecko find-coin luna 2022/05/18 22:20:44 [__main__] INFO: Found 27 coins: Num Id Symbol Price Name 01/27 lunarium xln $0.0006445 Lunarium 02/27 terra-luna luna $0.0001706 Terra 03/27 wrapped-terra luna $0.0001698 Wrapped Terra 04/27 aluna aln $0.01057 Aluna 05/27 lunadoge loge $9.25e-16 LunaDoge 06/27 lunarswap lunar $6.16e-06 LunarSwap 07/27 lunachow luchow $3.606e-07 LunaChow 08/27 lunaland lln $0.001261 LunaLand 09/27 luna-pad lunapad $0.006394 Luna-Pad 10/27 lunar lnr $1.479e-08 Lunar 11/27 lunafox lufx $2e-18 LunaFox 12/27 bonded-luna bluna $0.00116 Bonded Luna 13/27 luna-1x luna1x $0 LUNA 1x 14/27 lunarbrain lun $0.00079 LunarBrain 15/27 luna-rush lus $0.02474 Luna Rush 16/27 lunaverse luv $0.002186 Lunaverse 17/27 luna-wormhole luna $0.0001725 LUNA (Wormhole) 18/27 metaluna metaluna $2.296e-05 MetaLuna 19/27 solunavax-index solunavax $83.12 SOLUNAVAX Index 20/27 stader-lunax lunax $0.000192 Stader LunaX 21/27 nexus-bluna-token-share-representation nLuna $0.002256 Nexus bLuna token share representation 22/27 staked-luna stluna $0.003006 Staked Luna 23/27 lunafi lfi $0 Lunafi 24/27 lunar-token lunar $0.7432 Lunar Token 25/27 prism-cluna cluna $0.008605 Prism cLUNA 26/27 prism-pluna pluna $0.0001197 Prism pLUNA 27/27 prism-yluna yluna $0.008113 Prism yLUNA
- So I guess this should be terra-luna: let's add that one:
$ nvp coingecko monitored add terra-luna 2022/05/18 22:22:21 [__main__] INFO: Added monitored coins: ['terra-luna']
- OK, now let's search for UST:
$ nvp coingecko find-coin ust 2022/05/18 22:23:23 [__main__] INFO: Found 78 coins: Num Id Symbol Price Name 01/78 bitcoin-trust bct $0.01904 Bitcoin Trust 02/78 crust-network cru $0.97 Crust Network 03/78 cryptrust ctrt $4.07e-06 Cryptrust 04/78 custom-contract-network ccn $3.494e-05 Custom contract network 05/78 dust-token dust $0.0004455 DUST Token 06/78 freight-trust-network edi $0.0002124 Freight Trust Network 07/78 global-human-trust ght $39.53 Global Human Trust 08/78 global-trust-coin gtc $0.3516 Global Trust Coin 09/78 globaltrustfund-token gtf $0.0005818 GLOBALTRUSTFUND TOKEN 10/78 istardust isdt $0.0003652 Istardust 11/78 just jst $0.03982 JUST 12/78 justbet winr $0.0001499 JustBet 13/78 just-stablecoin usdj $1.002 JUST Stablecoin 14/78 mirrored-invesco-qqq-trust mqqq $30.49 Mirrored Invesco QQQ Trust 15/78 mirrored-ishares-gold-trust miau $3.63 Mirrored iShares Gold Trust 16/78 mirrored-ishares-silver-trust mslv $11.49 Mirrored iShares Silver Trust 17/78 mixtrust mxt $0.001199 MixTrust 18/78 must must $20.98 Must 19/78 terrausd ust $0.1052 TerraUSD 20/78 trust trust $0.005176 Harmony Block Capital 21/78 trust-ether-reorigin teo $7.585e-05 Trust Ether ReOrigin 22/78 trustswap swap $0.4013 Trustswap 23/78 trustusd trusd $0.0003428 TrustUSD 24/78 trustverse trv $0.009736 TrustVerse 25/78 trust-wallet-token twt $0.6675 Trust Wallet Token 26/78 utrust utk $0.1323 UTRUST 27/78 wetrust trst $0.002013 WeTrust 28/78 idle-dai-risk-adjusted idleDAISafe $1.075 IdleDAI (Risk Adjusted) 29/78 idle-usdc-risk-adjusted idleUSDCSafe $1.074 IdleUSDC (Risk Adjusted) 30/78 idle-usdt-risk-adjusted IdleUSDTSafe $1.081 IdleUSDT (Risk Adjusted) 31/78 dust-protocol dust $2.02 DUST Protocol 32/78 trustworks trust $0.384 Trustworks 33/78 australian-safe-shepherd ass $7.135e-10 Australian Safe Shepherd 34/78 sensitrust sets $0.02797 Sensitrust 35/78 rug-busters rugbust $0.004318 Rug Busters 36/78 greentrust gnt $8.86e-10 GreenTrust 37/78 trustpad tpad $0.1246 TrustPad 38/78 moontrust mntt $1.779e-07 MoonTrust 39/78 sustainable-energy-token set $8.05e-16 Sustainable Energy Token 40/78 australian-kelpie knockers $1.373e-10 Australian Kelpie 41/78 crust-storage-market csm $0.01383 Crust Storage Market 42/78 trustfi-network-token tfi $0.02015 TrustFi Network Token 43/78 pist-trust pist $0.05005 Pist Trust 44/78 18433-faust realtoken-s-18433-faust-ave-detroit-mi $42.26 RealT Token - 18433 Faust Ave, Detroit, MI, 48219 45/78 itrust-governance-token itg $0.009724 iTrust Governance Token 46/78 robust-token rbt $7.74 Robust Token 47/78 stargazer-protocol stardust $5.468e-11 Stargazer Protocol 48/78 trustbase tbe $0.0003444 TrustBase 49/78 trustercoin tsc $0.007037 TrusterCoin 50/78 wrapped-ust ust $0.1056 Wrapped UST 51/78 justyours just $9.615e-05 JustYours 52/78 braintrust btrst $2.65 Braintrust 53/78 busta bust $2.666e-05 BUSTA 54/78 dust dust $0.09771 Dust 55/78 trustkeys-network trustk $0.1087 TrustKeys Network 56/78 justfarm jfm $0.00128 JustFarm 57/78 upstabletoken ustx $0 UpStableToken 58/78 trustnft trustnft $0.001545 TrustNFT 59/78 trusted-node tnode $0.01627 Trusted Node 60/78 hyper-trust hptt $0.03958 Hyper Trust 61/78 anchorust aust $0.1335 AnchorUST 62/78 terrausd-wormhole ust $0.1007 TerraUSD (Wormhole) 63/78 trustrise trise $6.43e-06 TrustRise 64/78 moontrustbsc mnttbsc $6.14e-07 MoonTrustBSC 65/78 moonsdust moond $0.08643 MoonsDust 66/78 scardust scard $2.03e-08 SCARDust 67/78 assangedao justice $0.0001822 AssangeDAO 68/78 australian-crypto-coin-green accg $0.06471 Australian Crypto Coin Green 69/78 rfust rfust $4.611e-08 rfUST 70/78 trustpay tph $0.01895 Trustpay 71/78 trust-recruit trt $0 Trust Recruit 72/78 jarvis-synthetic-australian-dollar jaud $0 Jarvis Synthetic Australian Dollar 73/78 dumpbuster gtfo $1.67e-06 DumpBuster 74/78 justmoney jm $2.314e-05 JustMoney 75/78 kwiktrust ktx $0.07093 KwikTrust 76/78 qqq-tokenized-stock-defichain dQQQ $285.4 Invesco QQQ Trust Defichain 77/78 silver-tokenized-stock-defichain dSLV $19.5 iShares Silver Trust Defichain 78/78 spdr-s-p-500-etf-trust-defichain dspy $388.7 SPDR S&P 500 ETF Trust Defichain
- Feeeww, there are a lot of them here. But the one I'm looking for is terrausd:
$ nvp coingecko monitored add terrausd 2022/05/18 22:25:52 [__main__] INFO: Added monitored coins: ['terrausd']
- Alright, now let's wait a little for the initial data download to occur, and then we will check the CryptoView gui…
- Note: Actually I should update the history data manually to avoid waiting 1 full day here:
* nvp coingecko update-history
- And now I have the LUNA/UST coins loaded in CryptoView as expected, yeeepeeee:
- So let's call it a day now lol I need some sleep! 😴