====== NervProj: Finding & adding monitored coins ======
{{tag>dev python crypto nervhome finance}}
In one of my previous articles (available [[https://wiki.nervtech.org/doku.php?id=blog:2022:0510_nervproj_crypto_coins_monitoring#support_to_list_add_remove_monitored_coins|here]]) I introduced support to list/add/remove monitored coins from the command line with my coingecko component. This is very nice and all, but this also assumes that you //know// what is the ID of the coin you want to add to the list, which is not always absolutely clear...
So, what I want to do here is to add support to **search** for all coins matching a given pattern and list them directly on the command line so I could then select the proper one in the list and add the corresponding coin id.
====== ======
===== NervProj script commands =====
* Before I get to the actual problem I want to fix here, there is another minor thing I want to change too, each time I want to run a script in the NervProj framework I currently have to type ''nvp run script_name'' => I think that, instead, I could try to **bypass** the "run" keyword here and use instead ''nvp script_name'' directly: is that possible ? Let's see...
* Oki oki, so it seems a bit tricky to handle this kind of setup with **argparse** only, so I decided to use a simple "workaround" for that:
* First I added a method in the runner component to check if a given script exists: def has_script(self, script_name):
"""Check if a given script name is available."""
projs = self.ctx.get_projects()
for proj in projs:
desc = proj.get_script(script_name)
if desc is not None:
return True
desc = self.scripts.get(script_name, None)
return desc is not None
* Then **before** parsing the command line arguments I check if the second argument in the list is a script name, and if this is the case, I manually add the "run" command in front of it before parsing the args as usual (note that this operation only makes sense ont the master context for now, so placed in the ''allow_additionals'' section): def parse_args(self, allow_additionals):
"""Parse the command line arguments"""
# cf. https://docs.python.org/3.4/library/argparse.html#partial-parsing
if allow_additionals:
# before starting the regular parsing, we check if the first argument is a script name,
# in which case we should caller the runner directly with the remaining args.
if len(sys.argv) >= 2:
script_name = sys.argv[1]
self.settings = {}
runner = self.get_component('runner')
if runner.has_script(script_name):
# This is a valid script name, so we should add the "run" command
# before this script name:
sys.argv.insert(1, "run")
# logger.info("Parsing args: %s", sys.argv)
self.settings, self.additional_args = self.parsers['main'].parse_known_args()
self.settings = vars(self.settings)
# logger.info("Got settings: %s", self.settings)
# logger.info("Got additional args: %s", self.additional_args)
else:
self.settings = vars(self.parsers['main'].parse_args())
* => And this works just fine, so now I can run script directly without the "run" command, like this: $ nvp coingecko monitored list
===== Issue with coingecko highres data retrieval =====
* Arff, just checking the logs from coingecko data retrieval scripts, I noticed I could sometimes get an error like this: 2022/05/18 21:45:06 [__main__] INFO: Writing 22 new price entries
2022/05/18 21:45:07 [__main__] INFO: 1/69: Done updating lagging highres data for bitcoin
2022/05/18 21:45:11 [__main__] INFO: Writing 23 new price entries
2022/05/18 21:45:11 [__main__] INFO: 2/69: Done updating lagging highres data for ethereum
Traceback (most recent call last):
File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 596, in
comp.run()
File "/mnt/data1/dev/projects/NervProj/nvp/nvp_component.py", line 69, in run
res = self.process_command(cmd)
File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 535, in process_command
self.update_highres_datasets()
File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 127, in update_highres_datasets
self.update_highres_dataset(cid, 300)
File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 269, in update_highres_dataset
self.update_price_dataset(PriceDB.HIGHRES, cid, start_stamp, range_dur, lambda _: period)
File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 474, in update_price_dataset
self.fill_price_column(arr, 5, arr2, "eth")
File "/mnt/data1/dev/projects/NervHome/nvh/crypto/coingecko.py", line 392, in fill_price_column
assert nn1 < nn0, f"Unexpected number of {hint} prices: {nn1} > {nn0}"
AssertionError: Unexpected number of eth prices: 24 > 23
* Not really our concern here, but I cannot leave this as is right ? So let me have a look...
* Okay, so, I have re-implemented the problematic part from ''fill_price_column'' as below: def fill_price_column(self, abase, col, arr, hint):
"""Fill the column with index 'col' inside the base array 'abase', with the price data
coming from the column 1 of the array 'arr'. 'hint' should be a name giving an indication
on what kind of prices we are trying to add in this function call."""
assert abase is not None, "Invalid base array"
if arr is None:
# Nothing to fill in that case.
return
if arr.shape[0] == abase.shape[0]:
# check that all the timestamps are matching:
assert np.all(abase[:, 0] == arr[:, 0]), f"Mismatch in {hint} timestamps (all): {abase[:,0]} != {arr[:,0]}"
# We write all the column data:
abase[:, col] = arr[:, 1]
else:
# We only have a partial match, so we expect to have less filling prices than the number of base prices:
# nn0 = abase.shape[0]
# nn1 = arr.shape[0]
# Update: actually here we may also get *more* filling prices than the number of base prices.
# This is expectable because the size of the abase is computed for the collected USD prices,
# but we may get different arrays for BTC and ETH prices.
# assert nn1 < nn0, f"Unexpected number of {hint} prices: {nn1} > {nn0}"
# Check where we have the first match:
_, idx1, idx2 = np.intersect1d(abase[:, 0], arr[:, 0], return_indices=True)
assert np.all(abase[idx1, 0] == arr[idx2, 0]
), f"Mismatch in {hint} timestamps: {abase[idx1,0]} != {arr[idx2,0]}"
abase[idx1, col] = arr[idx2, 1]
# If we have NaN values remaining in col, we should perform linear interpolation here:
count = np.count_nonzero(np.isnan(abase[:, col]))
if count > 0:
logger.warning("Interpolating %d NaN values in %s prices", count, hint)
arr = abase[:, col]
nans, xfunc = nan_helper(arr)
arr[nans] = np.interp(xfunc(nans), xfunc(~nans), arr[~nans])
abase[:, col] = arr
count = np.count_nonzero(np.isnan(abase[:, col]))
# Should not have NaNs anymore:
assert count == 0, f"Could still find NaN in array {abase[:, col]}"
* => this will now use the "intersection" between the 2 arrays time values to fill the first array, and then it might perform some interpolation to fill the remaining NaN values...
* Problem is, I cannot really test that for now as the executing conigecko price retrieval script is now working properly again (ie. it is not collecting lagging prices anymore) 😅 So I need to wait until the next time this is failing to check how it behaves lol.
===== Find coins with a given pattern =====
* Anyway, now finally back to our main concern here: finding coins given a pattern, so how should we do that ?
* **Note**: I should onte here that the support to search for coins given a pattern was already implemented at the **CoinDB** level with the following method, so all I really had to add here was a small frontend to use that method: def find_coins(self, content):
"""Find the coin matching a content in the list."""
# Find coins with a given content text:
# Will return the coin_id/name/start_timestamp/symbol/platform_id
sql = f"SELECT coin_id FROM coins WHERE coin_id "
sql += f"LIKE '%{content}%' OR name LIKE '%{content}%' OR symbol LIKE '%{content}%'; "
cur = self.sql_db.execute(sql)
res = cur.fetchall()
return [row[0] for row in res]
* => Here we go! I built a new command called "find-coin" as follow: def process_command(self, cmd):
"""Check if this component can process the given command"""
if cmd == 'update-highres':
self.update_highres_datasets()
return True
if cmd == 'update-history':
self.update_history_datasets()
return True
if cmd == "find-coin":
txt = self.get_param("search_txt")
cids = self.get_coin_db().find_coins(txt)
ncoins = len(cids)
# Get the price data:
price_data = self.get_prices(cids, ['usd'], include_market_cap=False, include_24hr_vol=False)
res = []
idlen = 0
namelen = 0
symlen = 0
pricelen = 10
coins = []
for i, cid in enumerate(cids):
coin = self.get_coin(cid)
idlen = max(len(coin.get_id()), idlen)
namelen = max(len(coin.name()), namelen)
symlen = max(len(coin.symbol()), symlen)
coins.append(coin)
idstr = "Id"
symstr = "Symbol"
namestr = "Name"
pricestr = "Price"
line = f"Num {idstr:<{idlen+2}}{symstr:<{symlen+2}}{pricestr:<{pricelen+2}}{namestr:<{namelen}}"
res.append(line)
res.append("")
for i, coin in enumerate(coins):
price = price_data[coin.get_id()].get('usd', 0.0)
price = f'${price:.4g}'
line = f"{i+1:02d}/{ncoins} {coin.get_id():<{idlen+2}}{coin.symbol():<{symlen+2}}{price:<{pricelen+2}}{coin.name():<{namelen}}"
res.append(line)
logger.info("Found %d coins:\n%s", ncoins, "\n".join(res))
# logger.info("Coin %d/%d: '%s', symbol: '%s', id: '%s'", i+1,
# ncoins, coin.name(), coin.symbol(), coin.get_id())
return True
* And with that I get this kind of result on the command line: $ nvp coingecko find-coin ada
2022/05/18 22:15:53 [__main__] INFO: Found 61 coins:
Num Id Symbol Price Name
01/61 1x-short-cardano-token adahedge $10.26 1X Short Cardano Token
02/61 3x-long-cardano-token adabull $1.34 3X Long Cardano Token
03/61 3x-short-cardano-token adabear $1.001e-08 3X Short Cardano Token
04/61 aave-dai adai $1.001 Aave DAI
05/61 adamant-messenger adm $0.007166 ADAMANT Messenger
06/61 cardano ada $0.5266 Cardano
07/61 derivadao ddx $1.36 DerivaDAO
08/61 mahadao maha $1.42 MahaDAO
09/61 sada sada $0.6211 sADA
10/61 yadacoin yda $0.002556 YadaCoin
11/61 0-5x-long-cardano-token adahalf $3.162e+04 0.5X Long Cardano Token
12/61 matic-aave-dai madai $1.007 Matic Aave Interest Bearing DAI
13/61 adappter-token adp $0.01942 Adappter Token
14/61 dappradar $radar $0.00725 DappRadar
15/61 aave-dai-v1 adai $1.001 Aave DAI v1
16/61 instadapp inst $0.6535 Instadapp
17/61 adamant addy $0.3543 Adamant
18/61 binance-peg-cardano ada $0.5257 Binance-Peg Cardano
* => Pretty cool, isn't it 😎?
===== Conclusion =====
* And now let's do what I wanted to do from the beginning with this new feature: find the id of LUNA and UST to add them in my monitoring system 🤣, you know, just to keep track of those values in case they would come back to life one day... (yeah... I'm just dreaming, I know that lol)
* So searching for luna: $ nvp coingecko find-coin luna
2022/05/18 22:20:44 [__main__] INFO: Found 27 coins:
Num Id Symbol Price Name
01/27 lunarium xln $0.0006445 Lunarium
02/27 terra-luna luna $0.0001706 Terra
03/27 wrapped-terra luna $0.0001698 Wrapped Terra
04/27 aluna aln $0.01057 Aluna
05/27 lunadoge loge $9.25e-16 LunaDoge
06/27 lunarswap lunar $6.16e-06 LunarSwap
07/27 lunachow luchow $3.606e-07 LunaChow
08/27 lunaland lln $0.001261 LunaLand
09/27 luna-pad lunapad $0.006394 Luna-Pad
10/27 lunar lnr $1.479e-08 Lunar
11/27 lunafox lufx $2e-18 LunaFox
12/27 bonded-luna bluna $0.00116 Bonded Luna
13/27 luna-1x luna1x $0 LUNA 1x
14/27 lunarbrain lun $0.00079 LunarBrain
15/27 luna-rush lus $0.02474 Luna Rush
16/27 lunaverse luv $0.002186 Lunaverse
17/27 luna-wormhole luna $0.0001725 LUNA (Wormhole)
18/27 metaluna metaluna $2.296e-05 MetaLuna
19/27 solunavax-index solunavax $83.12 SOLUNAVAX Index
20/27 stader-lunax lunax $0.000192 Stader LunaX
21/27 nexus-bluna-token-share-representation nLuna $0.002256 Nexus bLuna token share representation
22/27 staked-luna stluna $0.003006 Staked Luna
23/27 lunafi lfi $0 Lunafi
24/27 lunar-token lunar $0.7432 Lunar Token
25/27 prism-cluna cluna $0.008605 Prism cLUNA
26/27 prism-pluna pluna $0.0001197 Prism pLUNA
27/27 prism-yluna yluna $0.008113 Prism yLUNA
* So I guess this should be **terra-luna**: let's add that one: $ nvp coingecko monitored add terra-luna
2022/05/18 22:22:21 [__main__] INFO: Added monitored coins: ['terra-luna']
* OK, now let's search for UST: $ nvp coingecko find-coin ust
2022/05/18 22:23:23 [__main__] INFO: Found 78 coins:
Num Id Symbol Price Name
01/78 bitcoin-trust bct $0.01904 Bitcoin Trust
02/78 crust-network cru $0.97 Crust Network
03/78 cryptrust ctrt $4.07e-06 Cryptrust
04/78 custom-contract-network ccn $3.494e-05 Custom contract network
05/78 dust-token dust $0.0004455 DUST Token
06/78 freight-trust-network edi $0.0002124 Freight Trust Network
07/78 global-human-trust ght $39.53 Global Human Trust
08/78 global-trust-coin gtc $0.3516 Global Trust Coin
09/78 globaltrustfund-token gtf $0.0005818 GLOBALTRUSTFUND TOKEN
10/78 istardust isdt $0.0003652 Istardust
11/78 just jst $0.03982 JUST
12/78 justbet winr $0.0001499 JustBet
13/78 just-stablecoin usdj $1.002 JUST Stablecoin
14/78 mirrored-invesco-qqq-trust mqqq $30.49 Mirrored Invesco QQQ Trust
15/78 mirrored-ishares-gold-trust miau $3.63 Mirrored iShares Gold Trust
16/78 mirrored-ishares-silver-trust mslv $11.49 Mirrored iShares Silver Trust
17/78 mixtrust mxt $0.001199 MixTrust
18/78 must must $20.98 Must
19/78 terrausd ust $0.1052 TerraUSD
20/78 trust trust $0.005176 Harmony Block Capital
21/78 trust-ether-reorigin teo $7.585e-05 Trust Ether ReOrigin
22/78 trustswap swap $0.4013 Trustswap
23/78 trustusd trusd $0.0003428 TrustUSD
24/78 trustverse trv $0.009736 TrustVerse
25/78 trust-wallet-token twt $0.6675 Trust Wallet Token
26/78 utrust utk $0.1323 UTRUST
27/78 wetrust trst $0.002013 WeTrust
28/78 idle-dai-risk-adjusted idleDAISafe $1.075 IdleDAI (Risk Adjusted)
29/78 idle-usdc-risk-adjusted idleUSDCSafe $1.074 IdleUSDC (Risk Adjusted)
30/78 idle-usdt-risk-adjusted IdleUSDTSafe $1.081 IdleUSDT (Risk Adjusted)
31/78 dust-protocol dust $2.02 DUST Protocol
32/78 trustworks trust $0.384 Trustworks
33/78 australian-safe-shepherd ass $7.135e-10 Australian Safe Shepherd
34/78 sensitrust sets $0.02797 Sensitrust
35/78 rug-busters rugbust $0.004318 Rug Busters
36/78 greentrust gnt $8.86e-10 GreenTrust
37/78 trustpad tpad $0.1246 TrustPad
38/78 moontrust mntt $1.779e-07 MoonTrust
39/78 sustainable-energy-token set $8.05e-16 Sustainable Energy Token
40/78 australian-kelpie knockers $1.373e-10 Australian Kelpie
41/78 crust-storage-market csm $0.01383 Crust Storage Market
42/78 trustfi-network-token tfi $0.02015 TrustFi Network Token
43/78 pist-trust pist $0.05005 Pist Trust
44/78 18433-faust realtoken-s-18433-faust-ave-detroit-mi $42.26 RealT Token - 18433 Faust Ave, Detroit, MI, 48219
45/78 itrust-governance-token itg $0.009724 iTrust Governance Token
46/78 robust-token rbt $7.74 Robust Token
47/78 stargazer-protocol stardust $5.468e-11 Stargazer Protocol
48/78 trustbase tbe $0.0003444 TrustBase
49/78 trustercoin tsc $0.007037 TrusterCoin
50/78 wrapped-ust ust $0.1056 Wrapped UST
51/78 justyours just $9.615e-05 JustYours
52/78 braintrust btrst $2.65 Braintrust
53/78 busta bust $2.666e-05 BUSTA
54/78 dust dust $0.09771 Dust
55/78 trustkeys-network trustk $0.1087 TrustKeys Network
56/78 justfarm jfm $0.00128 JustFarm
57/78 upstabletoken ustx $0 UpStableToken
58/78 trustnft trustnft $0.001545 TrustNFT
59/78 trusted-node tnode $0.01627 Trusted Node
60/78 hyper-trust hptt $0.03958 Hyper Trust
61/78 anchorust aust $0.1335 AnchorUST
62/78 terrausd-wormhole ust $0.1007 TerraUSD (Wormhole)
63/78 trustrise trise $6.43e-06 TrustRise
64/78 moontrustbsc mnttbsc $6.14e-07 MoonTrustBSC
65/78 moonsdust moond $0.08643 MoonsDust
66/78 scardust scard $2.03e-08 SCARDust
67/78 assangedao justice $0.0001822 AssangeDAO
68/78 australian-crypto-coin-green accg $0.06471 Australian Crypto Coin Green
69/78 rfust rfust $4.611e-08 rfUST
70/78 trustpay tph $0.01895 Trustpay
71/78 trust-recruit trt $0 Trust Recruit
72/78 jarvis-synthetic-australian-dollar jaud $0 Jarvis Synthetic Australian Dollar
73/78 dumpbuster gtfo $1.67e-06 DumpBuster
74/78 justmoney jm $2.314e-05 JustMoney
75/78 kwiktrust ktx $0.07093 KwikTrust
76/78 qqq-tokenized-stock-defichain dQQQ $285.4 Invesco QQQ Trust Defichain
77/78 silver-tokenized-stock-defichain dSLV $19.5 iShares Silver Trust Defichain
78/78 spdr-s-p-500-etf-trust-defichain dspy $388.7 SPDR S&P 500 ETF Trust Defichain
* Feeeww, there are a lot of them here. But the one I'm looking for is **terrausd**: $ nvp coingecko monitored add terrausd
2022/05/18 22:25:52 [__main__] INFO: Added monitored coins: ['terrausd']
* Alright, now let's wait a little for the initial data download to occur, and then we will check the CryptoView gui...
* **Note**: Actually I should update the history data manually to avoid waiting 1 full day here: * nvp coingecko update-history
* And now I have the LUNA/UST coins loaded in **CryptoView** as expected, yeeepeeee:
{{ blog:2022:0518:luna_coin_added.png }}
* So let's call it a day now lol I need some sleep! 😴