====== NVP Framework clean up and consolidation ======

{{tag>dev python nervproj nervhome crypto}}

Lately I've been adding a lot of stuff in my NVP framework and private NervHome sub project. And this framework has been evolving pretty fast so, I have some behaviors/code paths that are obsolete already. Time for some cleanup session I think 😉!

====== ======

===== Pulling multiple git repositories at the same time =====

  * One operation that I end up doing often is to pull my changes from the NVPproject with git, and then also pull the changes from NervHome on my main server:
  * => I think I could simplify that will a "pullall" command in my git component, which I should turn into a dynamic component in the process in fact.
  * Moving nvp/components/git.py as nvp/core/git_manage.py: **OK**
  * Adding the "pullall" command: <sxh python; highlight: []>        if cmd == 'pullall':
            # Pull all the known repositories:
            # We start with the NVP framework itself:
            logger.info("Pulling NVP framework...")
            self.git_pull(self.ctx.get_root_dir())

            # Next we iterate on all the projects:
            for proj in self.ctx.get_projects():
                ppath = proj.get_root_dir()
                if ppath is not None:
                    logger.info("Pulling %s...", proj.get_name())
                    self.git_pull(ppath)
            return True</sxh>
  * **OK**: this works just fine: <sxh bash; highlight: []>$ nvp git pullall
2022/06/19 08:19:44 [nvp.core.git_manager] INFO: Pulling NVP framework...
Already up to date.
2022/06/19 08:19:46 [nvp.core.git_manager] INFO: Pulling nervland...
Already up to date.
2022/06/19 08:19:46 [nvp.core.git_manager] INFO: Pulling packages...
Already up to date.
2022/06/19 08:19:47 [nvp.core.git_manager] INFO: Pulling resp_t7...
Already up to date.
2022/06/19 08:19:47 [nvp.core.git_manager] INFO: Pulling simcore...
From gitlab.gmv-insyen.com:core/simcore
5345f36eb..c6f972e09  master     -> origin/master
Already up to date.
2022/06/19 08:19:52 [nvp.core.git_manager] INFO: Pulling nervhome...
Already up to date.</sxh>


===== Turn other NVP components into dynamic components =====

  * Handle the **GitlabManager** component: **OK**
  * Also added the support to resolve the root dir in the NVPContext: <sxh python; highlight: []>    def resolve_root_dir(self, project=None):
        """Resolve a root directory either from project or CWD"""
        if project is None:
            project = self.get_current_project(True)

        if project:
            return project.get_root_dir()

        # We might still be in the NVP project itself:
        cwd = self.to_absolute_path(self.get_cwd())

        root_dir = self.to_absolute_path(self.get_root_dir())
        if cwd.startswith(root_dir):
            return root_dir

        return None</sxh>
  * Handle the **ToolsManager** component: **OK**

===== Fixing coingecko update issue =====

  * Naturally, while processing the components above, another (totally unrelated) urgent error happened: in the coingecko highres update process I have in place.
  * Took me some time to figure this out, but in this end the error I have seems to be due to the **reef-finance** name not being a valid coin ID anymore.
  * => I should report this kind of error clearly I think, so updated the **coingecko.send_get_request()** method with the content: <sxh python; highlight: []>        while True:
            # logger.info("Sending request to %s (%s) (delay: %f)", url, params or "None", self.delay)
            resp = self.make_get_request(
                url,
                params=params,
                headers=headers,
                timeout=self.connection_timeout,
                max_retries=self.max_retries,
                retry_delay=self.delay * 3,
                status_codes=[200, 429, 404],
            )

            if resp is not None and resp.status_code == 429:
                now = datetime.now()
                now_str = now.strftime("%m/%d/%Y, %H:%M:%S")
                msg = f"[{now_str}] Coingecko: detected rate limit response, cooling down for {self.cool_down_delay} seconds."
                logger.warning(msg)
                utl.send_rocketchat_message(":warning: " + msg, channel="problem-reports")
                time.sleep(self.cool_down_delay)
                continue

            if resp is not None and resp.status_code == 404:
                now = datetime.now()
                now_str = now.strftime("%m/%d/%Y, %H:%M:%S")
                msg = f"[{now_str}] Coingecko: got 404 status error while processing {url}: {resp.text}"
                logger.error(msg)
                utl.send_rocketchat_message(":warning: " + msg, channel="problem-reports")
                resp = None

            break</sxh>
  
  * And now checking the state of our database: <sxh python; highlight: []>from nvp.nvp_context import NVPContext

if NVPContext.instance is None:
    NVPContext()
    
ctx = NVPContext.get()

from nvh.crypto.coingecko import Coingecko

cg = Coingecko(ctx)
cdb = cg.get_coin_db()

cdb.find_coins("reef")
# output: ['reef-finance', 'threefold-token', 'reefer-token']</sxh>

  * => The point is, I'm not currently dynamically updating the list of coins (as I was before), that's why I still have "reef-finance" in this list above and nothing new on reef.

  * => As a quick solution for the moment I'm simply removing the "reef-finance" entry from my monitored coins list: <sxh python; highlight: []>cdb.delete_monitored_coin("reef-finance")</sxh>
  * But I definitely need to update the list of coins regularly, which I will cover next.

===== Updating coins for coingecko =====

  * Here is the handler script I created to update the coins: <sxh python; highlight: []>"""Handler to update the coingecko coins data"""

import logging
from datetime import date

import nvp.core.utils as utl

from nvh.crypto.coingecko import Coingecko

logger = logging.getLogger("update_coins_data")


def handle(self: Coingecko):
    """Handler function entry point"""

    clist = self.get_coins_list()

    cdb = self.get_coin_db()

    # Iterate on each coin:
    ncoins = len(clist)

    def set_infos(obj, cname):
        """Set some infos on a given coin object."""

        infos = self.get_coin_info(cname, market_data=True)
        if infos is None:
            return

        obj["platform_id"] = infos["asset_platform_id"]
        obj["contract_address"] = infos.get("contract_address")
        links = infos["links"]
        obj["explorers"] = [l for l in links["blockchain_site"] if l != ""]
        obj["homepage"] = links["homepage"][0]
        obj["telegram"] = links["telegram_channel_identifier"]
        obj["twitter"] = links["twitter_screen_name"]
        obj["announcements"] = [l for l in links["announcement_url"] if l != ""]
        obj["total_supply"] = infos["market_data"]["total_supply"] if "market_data" in infos else None

    new_coins = []

    # Retrieve all the registered coin Ids:
    all_ids = self.getAllCoinIDs(True)

    for i in range(ncoins):
        cdat = clist[i]
        cid = cdat["id"]

        if cid == "":
            logger.info("Ignoring invalid coin ID for %s", cdat)
            continue

        # Check if we have that coin already in the database:
        if not cid in all_ids:
            res = self.get_market_data(cid, "max")

            if res is None:
                logger.warning("Cannot retrieve any data for coin: %s.", cid)
                nprices = 0
            else:
                nprices = len(res["prices"])

            t0 = None
            tstr = None

            if nprices == 0:
                logger.info("No price data available for coin %s", cid)
                # Use the current time as timestamp:
                # ts = nvGetCurrentTimestamp()
            else:
                t0 = int(res["prices"][0][0] / 1000)
                # logDEBUG("Timestamp: %d, num points: %d" % (ts,np))

                # lastTs = int(res['prices'][-1][0]/1000)

                d0 = date.fromtimestamp(t0)
                tstr = d0.isoformat()
                logger.info("New coin %d/%d: '%s' start date: %s", i + 1, ncoins, cid, tstr)

            desc = {"coin_id": cid, "start_timestamp": t0, "symbol": cdat["symbol"], "name": cdat["name"]}
            set_infos(desc, cid)

            # Add this new coin to the list:
            new_coins.append(desc)

            # Also add the coin id to the all list, to ensure we don't add it twice:
            all_ids.append(cid)

        if len(new_coins) > 20:
            logger.info("Inserting partial coins list..")
            cdb.insert_coins(new_coins)
            new_coins = []

    # If we have new coins then we should submit them to the DB:
    if len(new_coins) > 0:
        cdb.insert_coins(new_coins)

    received_ids = [clist[i]["id"] for i in range(ncoins)]

    # We should also check here if some coins were removed:
    mon_coins = cdb.get_all_monitored_coins()

    for cid in all_ids:
        if cid not in received_ids:
            logger.info("Removing obsolete coin %s", cid)
            cdb.remove_coin(cid)

            # Also remove the coin from the monitored coins list if applicable:
            if cid in mon_coins:
                msg = f"Removing obsolete coin {cid} from list of monitored coins."
                logger.warning(msg)
                utl.send_rocketchat_message(":warning: Coingecko: " + msg)

    logger.info("Done updating coins data.")
</sxh>

  * **Note** before executing the script above, I learnt how to duplicate a postgresql table (=> That could proove very handy in some cases ✌): <sxh python; highlight: []># duplicate the coins table:
sql = "create table coins_orig as (select * from coins);"
cdb.execute(sql, commit=True)</sxh>

  * Now trying to update the coins: <sxh bash; highlight: []>$ nvp coingecko update-coins</sxh>
  * => And this works! Now just adding a dedicated script entry for this: <sxh javascript; highlight: []>    "coingecko_update_coins": {
      "custom_python_env": "defi_env",
      "lock_file": "${PROJECT_ROOT_DIR}/data/locks/coingecko_update.lock",
      "cmd": "${PYTHON} nvh/crypto/coingecko.py update-coins",
      "cwd": "${PROJECT_ROOT_DIR}",
      "python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"]
    },</sxh>
  * And running it as cron daily task: **OK**

===== Reducing blockchain database size =====

  * Next interesting thing I just noticed is that my bsc_chain daily backup file went instantly from about 100MB to 530MB since i'm collecting the blocks/transactions data 😨! 
  * This was somewhat expected of course, but now I'm feeling this might get too big, too quickly, so maybe I should do something about it.
  * My first idea here, is to check for contract creation transactions (ie. "to address" == "0x"): these will usually inject a lot of input data on the blockchain, and really I would not know how to parse that data at all, so maybe I should **not** store the input raw data in the database in that case.

  * Hmmm... okay so, actually I only have about 90MB of data in the registers for the created contracts transactions, as discovered below: <sxh python; highlight: []># Select all the registers from contract creation transations:
sql = "SELECT sig,registers FROM transactions WHERE to_id is null;"
rows = db.execute(sql).fetchall()
nrows = len(rows)
print(f"Got {nrows} rows")

# output: Got 8987 rows

# Compute total length of that data:
bcount = 0
for row in rows:
    bcount += 0 if row[1] is None else len(row[1])

print(f"Total number of bytes: {bcount}")

# output: Total number of bytes: 89376916</sxh>

  * While I have more the 1.9 GB of data in the other 10827212 transactions already in the database after only 1 day 😅, that's not too good...
  * Here is the script I used to classify what are the method signature taking the most space due to the input data: <sxh python; highlight: []>import nvp.core.utils as utl

bcount = 0

size_stats={}

for row in rows:
    sig = row[0]
    if sig is None:
        sig = 0

    if sig not in size_stats:
        size_stats[sig] = {
            'sig': hex(utl.to_uint32(sig)),
            'min_size':100000,
            'max_size':0,
            'mean_size': 0,
            'count': 0
        }
    
    desc = size_stats[sig]
    size = 0 if row[1] is None else len(row[1])
    desc['min_size'] = min(desc['min_size'], size)
    desc['max_size'] = max(desc['max_size'], size)
    desc['mean_size'] += size
    desc['count'] += 1
    
    bcount += size

stats = list(size_stats.values())
for desc in stats:
    desc["share"] = desc["mean_size"]
    desc["mean_size"] /= desc["count"]

stats.sort(key=lambda item: item['share'], reverse=True)

print(f"Total number of bytes: {bcount}")

stats[:20]</sxh>

  * The outputs we get from our +10M transactions with the script above are as follow: <sxh bash; highlight: []>Total number of bytes: 1912614627
[{'sig': '0x38ed1739',
  'min_size': 0,
  'max_size': 416,
  'mean_size': 267.53302217321635,
  'count': 572267,
  'share': 153100320},
 {'sig': '0xa9059cbb',
  'min_size': 63,
  'max_size': 1084,
  'mean_size': 64.36142689845857,
  'count': 1892272,
  'share': 121789326},
 {'sig': '0xc9807539',
  'min_size': 896,
  'max_size': 1408,
  'mean_size': 1240.363025981665,
  'count': 69703,
  'share': 86457024},
 {'sig': '0x7c025200',
  'min_size': 708,
  'max_size': 25924,
  'mean_size': 2497.147587215375,
  'count': 28722,
  'share': 71723073},
 {'sig': '0x7ff36ab5',
  'min_size': 0,
  'max_size': 320,
  'mean_size': 229.34020209492746,
  'count': 297385,
  'share': 68202336},
 {'sig': '0x5c11d795',
  'min_size': 256,
  'max_size': 352,
  'mean_size': 268.7948732170819,
  'count': 234377,
  'share': 62999336},
 {'sig': '0x5f575529',
  'min_size': 640,
  'max_size': 17888,
  'mean_size': 1332.4805035845427,
  'count': 45752,
  'share': 60963648},
 {'sig': '0x791ac947',
  'min_size': 64,
  'max_size': 320,
  'mean_size': 259.65219550223696,
  'count': 201389,
  'share': 52291096},
 {'sig': '0x18cbafe5',
  'min_size': 256,
  'max_size': 352,
  'mean_size': 262.9286742079253,
  'count': 185557,
  'share': 48788256},
 {'sig': '0x95ea7b3',
  'min_size': 44,
  'max_size': 299,
  'mean_size': 65.38600831292992,
  'count': 668597,
  'share': 43716889},
 {'sig': '0x1de9c881',
  'min_size': 224,
  'max_size': 288,
  'mean_size': 244.97880883337052,
  'count': 134490,
  'share': 32947200},
 {'sig': '0x8803dbee',
  'min_size': 224,
  'max_size': 352,
  'mean_size': 264.1658646653026,
  'count': 123649,
  'share': 32663845},
 {'sig': '0xd8169879',
  'min_size': 3872,
  'max_size': 4512,
  'mean_size': 3964.3599106791016,
  'count': 7613,
  'share': 30180672},
 {'sig': '0x84013b6a',
  'min_size': 2400,
  'max_size': 3776,
  'mean_size': 2423.9184163167365,
  'count': 11669,
  'share': 28284704},
 {'sig': '0xfb3bdb41',
  'min_size': 96,
  'max_size': 288,
  'mean_size': 228.33169934640523,
  'count': 117504,
  'share': 26829888},
 {'sig': '0x6326d217',
  'min_size': 256,
  'max_size': 352,
  'mean_size': 270.54684163608846,
  'count': 96474,
  'share': 26100736},
 {'sig': '0xb6f9de95',
  'min_size': 0,
  'max_size': 288,
  'mean_size': 223.66593745077793,
  'count': 114278,
  'share': 25560096},
 {'sig': '0xb7835e4c',
  'min_size': 288,
  'max_size': 288,
  'mean_size': 288.0,
  'count': 86774,
  'share': 24990912},
 {'sig': '0xc5860465',
  'min_size': 352,
  'max_size': 352,
  'mean_size': 352.0,
  'count': 44108,
  'share': 15526016},
 {'sig': '0x5866cc8e',
  'min_size': 640,
  'max_size': 1088,
  'mean_size': 767.4700615846401,
  'count': 19323,
  'share': 14829824}]</sxh>
  * **Note**: Checking https://www.4byte.directory/ we see that **0x38ed1739** corresponds to "swapExactTokensForTokens(uint256,uint256,address[],address,uint256)"

  * => But anyway, I cannot let my blockchain database grow that fast without some kind of control, so let's fix this now.
  * So, the "solution" I have for the moment is to split the transactions into 2 databases: first part in my standard **bsc_chain** database for instance. And then the transactions "heavy data" (ie. input data/value/gas price) in a separated database called **transactions_db**.
  * The idea then is to avoid backing up the full transactions_db database, and instead export the new rows in there regularly with a semi-manual system... At least, this will limit the size of the bsc_chain database itself hopefully!

===== Initial investigations on Avalanche network =====

  * After the CELO blockchain last time, now I would like to also start investigating the arbitrage possibilities on Avalanche, so let's prepare the required elements to start collecting blocks for that network.
  * Built a config for the "avax" blockchain: **OK**
  * And then started collecting blocks with a dedicated new script: <sxh bash; highlight: []>    "collect_avax_blocks": {
      "lock_file": "${PROJECT_ROOT_DIR}/data/locks/collect_avax_blocks.lock",
      "custom_python_env": "bsc_env",
      "cmd": "${PYTHON} ${PROJECT_ROOT_DIR}/nvh/crypto/blockchain/blockchain_manager.py collect blocks -c avax",
      "python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"]
    },</sxh>
  * Also collecting gas prices on AVALANCHE now so I could add a dedicated page in my CryptoView app => ahh, need to wait a little until I have some gas prices for that blockchain before I can display the data 😄

  * One thing I noticed then is that I seem to have quite a few routers on that chain: <sxh python; highlight: []>routers = []
for addid in id_counts.keys():
    routers.append(chain.get_address(addid))

routers</sxh>
  * Output: <sxh bash; highlight: []>['0xC7f372c62238f6a5b79136A9e5D16A2FD7A3f0F5',
 '0x60aE616a2155Ee3d9A68541Ba4544862310933d4',
 '0xE54Ca86531e17Ef3616d22Ca28b0D458b6C89106',
 '0x1b02dA8Cb0d097eB8D57A175b88c7D8b47997506',
 '0x5F1FdcA239362c5b8A8Ada26a256ac5626CC33E0',
 '0xF7b1e993d1b4F7348D64Aa55A294E4B74512F7f2',
 '0xA52aBE4676dbfd04Df42eF7755F01A3c41f28D27',
 '0xb9a9BB6CC39387548BAA7185fbFf51d47eef8771',
 '0x7aE9D51DC773Ea7518509786d2017bb964606663',
 '0x262DcFB36766C88E6A7a2953c16F8defc40c378A',
 '0x9E4AAbd2B3E60Ee1322E94307d0776F2c8e6CFbb',
 '0x763D8D37669bB18064b410e17E3bB46BCF34F487',
 '0x8685aBD6959745E41D64194F42e21bbD25Fd57B3',
 '0x78c18E6BE20df11f1f41b9635F3A18B8AD82dDD1',
 '0xd8aA70F7990Dab4a383a0d8A57dF7a372916575D',
 '0xc2544A32872A91F4A553b404C6950e89De901fdb',
 '0x14f800843F7d70C911d4ec728f2F839c264A9895',
 '0x3f767183E3DFf6726950965EBDE537C5a7C29595',
 '0xb5b2444eDF79b00d40f463f79158D1187a0D0c25',
 '0x0c45FB63001b56a21e29c7dcc1727bfDA273a368',
 '0x0eb4DF266dA7c7BC1629b52a540cc87C8D95Bf24',
 '0x85995d5f8ee9645cA855e92de16FA62D26398060']</sxh>

  * But before pushing this further I think I should write an handler to retrieve those routers/factories.

  * One nice thing I've just been thinking about writing that new handler and calling it **during implementation phase** in jupyter, is that I could even provide some **hot reloading** support here: each time I call an "handler" I could get an hash from the corresponding file, and if it doesn't match the previous hash, then I could reload the handler code from file overriding the previous version (yeah 😎✌! => Sounds realy cool, so let's try it!).

  * Yooohhooo! That's indeed really cool: I don't have to re-start my jupyter kernel each time I change something in my handler now. Here is the updated code providing support for that in ''NVPContext.get_handler()'': <sxh python; highlight: []>    def get_handler(self, hname):
        """Get a handler by name"""

        # Adding support for hot reloading of handlers here:
        # Given a module name, we should try to find the corresponding file:
        filepath = self.resolve_module_file(hname)

        # Compute the hash of that file:
        fhash = self.compute_file_hash(filepath)
        prev_hash = self.handler_hashes.get(filepath, None)
        if prev_hash is not None and prev_hash != fhash:
            logger.debug("Detected change in %s, reloading handler %s", filepath, hname)

            # Remove the already loaded function:
            self.check(hname in self.handlers, "Expected handler to be loader already: %s", hname)
            del self.handlers[hname]

        # Store the new file hash anyway:
        self.handler_hashes[filepath] = fhash

        if hname in self.handlers:
            return self.handlers[hname]

        # otherwise we have to search for that handler:
        comp_module = import_module(hname)
        handler = comp_module.handle
        del sys.modules[hname]

        self.handlers[hname] = handler
        return handler</sxh>

  * Now let's continue with the retrieval of the factories from the router addresses: **OK**

  * Next, trying to collect the pairs on **JoeSwap**: <sxh bash; highlight: []>$ nvp bchain update-pairs joeswap</sxh>
  * => **All good** so continuing with PangolinSwap: <sxh bash; highlight: []>$ nvp bchain update-pairs pangolinswap</sxh>
  * And then with SushiSwap on avalanche: <sxh bash; highlight: []>$ nvp bchain update-pairs avaxsushiswap</sxh>
  * Net I should also ensure that I'm updating all the pairs automatically once per day: **OK**
  * Finally, I also added "SicleSwap" (even if I have no idea which website this is... but it was the fourth router on my ordered list): <sxh bash; highlight: []>$ nvp bchain update-pairs sicleswap</sxh>
  * Now that I have 4 DEXes registered on Avalanche, it's time to start saerching for arb setups there!
  * But to do so at least in dry-run, I need the **GetReservesV2** contract deployed. Yet, this should be done with the evm_arb account, so I should send some avax to that account first (done in jupyter): <sxh bash; highlight: []>chain.transfer(0.2, to_name="evm_arb")</sxh> 
  * Next the contract deployment: <sxh bash; highlight: []>$ nvp bchain deploy -c avax -a evm_arb GetReservesV2</sxh>
  * But now I realize that I would also need the additionnal contract addresses to be able to build the **ArbitrageManager** hmmm... 🤔 Or let's see if I can bypass that.
  * => In fact I could probably do without the FlashArb contract at start, but I will definitely need the PairChecker contract, so deploying that one too: <sxh bash; highlight: []>$ nvp bchain deploy -c avax -a evm_arb PairCheckerV6</sxh>

  * Now trying to monitor the arbs on avalanche (in dry-run mode): <sxh bash; highlight: []>$ nvp arbman monitor-arbs -c avax -n</sxh>

  * Arrff... That's not quite working out of the box yet 😆: <sxh bash; highlight: []>$ nvp arbman monitor-arbs -c avax -n
2022/06/26 20:26:03 [nvh.crypto.blockchain.evm_blockchain] INFO: Keeping 19022/19962 quotable pairs.
2022/06/26 20:26:03 [nvh.crypto.blockchain.evm_blockchain] INFO: Found 689 arb compatible pairs
2022/06/26 20:26:03 [__main__] INFO: wrapped native token address: 0x4cc3Fc374627E9244dc49a258ae717098c2165d0
2022/06/26 20:26:03 [__main__] INFO: Collected 689 arb pairs
2022/06/26 20:26:03 [__main__] INFO: Min profit value: 0.000100
2022/06/26 20:26:03 [__main__] INFO: Quote token WAVAX value: 256326 AVAX
2022/06/26 20:26:03 [__main__] WARNING: Not enough WAVAX funds to send to pair checker contract.
2022/06/26 20:26:03 [__main__] INFO: PairChecker balance: 0 WAVAX
2022/06/26 20:26:04 [nvh.crypto.blockchain.uniswap_base] INFO: Searching for swap path between MIM -> ? -> AVAX
2022/06/26 20:26:04 [nvh.crypto.blockchain.uniswap_base] INFO: Using token swap path: ['MIM', 'WAVAX', 'AVAX']
2022/06/26 20:26:04 [__main__] INFO: Quote token MIM value: 107802 AVAX
2022/06/26 20:26:04 [__main__] WARNING: Not enough MIM funds to send to pair checker contract.
2022/06/26 20:26:04 [__main__] INFO: PairChecker balance: 0 MIM
2022/06/26 20:26:04 [nvh.crypto.blockchain.uniswap_base] INFO: Searching for swap path between USDC.e -> ? -> AVAX
2022/06/26 20:26:04 [nvh.crypto.blockchain.uniswap_base] INFO: Using token swap path: ['USDC.e', 'WAVAX', 'AVAX']
2022/06/26 20:26:05 [__main__] INFO: Quote token USDC.e value: 108200 AVAX
2022/06/26 20:26:05 [__main__] WARNING: Not enough USDC.e funds to send to pair checker contract.
2022/06/26 20:26:05 [__main__] INFO: PairChecker balance: 0 USDC.e
2022/06/26 20:26:05 [nvh.crypto.blockchain.uniswap_base] INFO: Searching for swap path between USDT.e -> ? -> AVAX
2022/06/26 20:26:05 [nvh.crypto.blockchain.uniswap_base] INFO: Using token swap path: ['USDT.e', 'WAVAX', 'AVAX']
2022/06/26 20:26:05 [__main__] INFO: Quote token USDT.e value: 108249 AVAX
2022/06/26 20:26:05 [__main__] WARNING: Not enough USDT.e funds to send to pair checker contract.
2022/06/26 20:26:06 [__main__] INFO: PairChecker balance: 0 USDT.e
2022/06/26 20:26:06 [nvh.crypto.blockchain.uniswap_base] INFO: Searching for swap path between USDC -> ? -> AVAX
2022/06/26 20:26:06 [nvh.crypto.blockchain.uniswap_base] INFO: Using token swap path: ['USDC', 'WAVAX', 'AVAX']
2022/06/26 20:26:06 [__main__] INFO: Quote token USDC value: 108309 AVAX
2022/06/26 20:26:06 [__main__] WARNING: Not enough USDC funds to send to pair checker contract.
2022/06/26 20:26:06 [__main__] INFO: PairChecker balance: 0 USDC
2022/06/26 20:26:06 [nvh.crypto.blockchain.uniswap_base] INFO: Searching for swap path between DAI.e -> ? -> AVAX
2022/06/26 20:26:06 [nvh.crypto.blockchain.uniswap_base] INFO: Using token swap path: ['DAI.e', 'WAVAX', 'AVAX']
2022/06/26 20:26:06 [__main__] INFO: Quote token DAI.e value: 108251 AVAX
2022/06/26 20:26:07 [__main__] WARNING: Not enough DAI.e funds to send to pair checker contract.
2022/06/26 20:26:07 [__main__] INFO: PairChecker balance: 0 DAI.e
2022/06/26 20:26:07 [nvh.crypto.blockchain.uniswap_base] INFO: Searching for swap path between PNG -> ? -> AVAX
2022/06/26 20:26:07 [nvh.crypto.blockchain.uniswap_base] INFO: Using token swap path: ['PNG', 'WAVAX', 'AVAX']
2022/06/26 20:26:07 [__main__] INFO: Quote token PNG value: 8881.42 AVAX
2022/06/26 20:26:07 [__main__] WARNING: Not enough PNG funds to send to pair checker contract.
2022/06/26 20:26:07 [__main__] INFO: PairChecker balance: 0 PNG
2022/06/26 20:26:07 [nvh.crypto.blockchain.uniswap_base] INFO: Searching for swap path between JOE -> ? -> AVAX
2022/06/26 20:26:07 [nvh.crypto.blockchain.uniswap_base] INFO: Using token swap path: ['JOE', 'WAVAX', 'AVAX']
2022/06/26 20:26:08 [__main__] INFO: Quote token JOE value: 43963.4 AVAX
2022/06/26 20:26:08 [__main__] WARNING: Not enough JOE funds to send to pair checker contract.
2022/06/26 20:26:08 [__main__] INFO: PairChecker balance: 0 JOE
2022/06/26 20:26:08 [nvh.crypto.blockchain.uniswap_base] INFO: Searching for swap path between WETH.e -> ? -> AVAX
2022/06/26 20:26:08 [nvh.crypto.blockchain.uniswap_base] INFO: Using token swap path: ['WETH.e', 'WAVAX', 'AVAX']
2022/06/26 20:26:08 [__main__] INFO: Quote token WETH.e value: 275450 AVAX
2022/06/26 20:26:08 [__main__] WARNING: Not enough WETH.e funds to send to pair checker contract.
2022/06/26 20:26:08 [__main__] INFO: PairChecker balance: 0 WETH.e
2022/06/26 20:26:09 [nvh.crypto.blockchain.uniswap_base] INFO: Searching for swap path between USDT -> ? -> AVAX
2022/06/26 20:26:09 [nvh.crypto.blockchain.uniswap_base] INFO: Using token swap path: ['USDT', 'WAVAX', 'AVAX']
2022/06/26 20:26:09 [__main__] INFO: Quote token USDT value: 107662 AVAX
2022/06/26 20:26:09 [__main__] WARNING: Not enough USDT funds to send to pair checker contract.
2022/06/26 20:26:09 [__main__] INFO: PairChecker balance: 0 USDT
2022/06/26 20:26:10 [__main__] WARNING: Unhandled method signature '0x7a42416a' in 0xeb8c29814de99e7999e5493a101018dc1658fae01c021414c4c8a1ea9c9f1300
2022/06/26 20:26:11 [__main__] INFO: 8 pair reserves collected in 0.1369 secs
2022/06/26 20:26:11 [__main__] INFO: Pair reserves block offset: 1
2022/06/26 20:26:11 [__main__] WARNING: Unhandled method signature '0x7a42416a' in 0x510965bca0e7eeac515a84c155a8651f0504dae311cf4711dad576152394ac07
2022/06/26 20:26:11 [__main__] INFO: 4 pair reserves collected in 0.1195 secs
2022/06/26 20:26:12 [__main__] WARNING: Unhandled method signature '0xf91b3f72' in 0x610f19450047f05df550d65c82590e73be11f0d81e2589b165b77750407ef34d
2022/06/26 20:26:18 [__main__] WARNING: Unhandled method signature '0xf91b3f72' in 0x801bdb853b67509d61e3d2f6f375ae8c90e1d96516039ea145903216b2421d21
2022/06/26 20:26:19 [__main__] WARNING: Unhandled method signature '0x2c407024' in 0xdc85d79dc709aef036637b57e922ba391c9da85ebcd8d6fd1a8edda7c44507be
2022/06/26 20:26:19 [__main__] WARNING: Unhandled method signature '0x676528d1' in 0xd56b4db987f3265a385673e0064bbfcd192b3d6db0b6815e91ed9ba59c6659cb
2022/06/26 20:26:19 [__main__] INFO: 2 pair reserves collected in 0.1214 secs
2022/06/26 20:26:31 [__main__] INFO: 2 pair reserves collected in 0.1331 secs
2022/06/26 20:26:34 [__main__] INFO: 3 pair reserves collected in 0.1184 secs</sxh>

  * => Which means I need to perform some debugging here 😉
  * First, let's check what is the "WAVAX" token above: <sxh bash; highlight: []>$ nvp bchain find-token -c avax WAVAX
2022/06/26 20:38:22 [__main__] INFO: Found token: { 'address': '0xB31f66AA3C1e785363F0875A1B74E27b85FD66c7',
'classification': 0,
'decimals': 18,
'id': 1,
'min_slippage': None,
'name': 'Wrapped AVAX',
'swap_fees': None,
'symbol': 'WAVAX'}</sxh>
  * Hmmm, then why do we get a different address for the wapped native token above ? **OK** => was simply the native symbol mapping missing.
  * But then I get a lot of warnings about invalid signatures, simply because we are now using function names such as: ''swapExactAVAXForTokens'', so I need to handle that.

===== Generate method signature from declaration =====

  * To solve the issue reported above with different uniswap function names on differen blockchains, I'm thinking I could build an automated system for this, to generate the expected method signature given a function declaration. So let's see if I can build that.
  * Found this page: https://stackoverflow.com/questions/46279121/how-can-i-find-keccak-256-hash-in-python
  * => To generate a keccak value in python, one can use: <sxh python; highlight: []> from Crypto.Hash import keccak
 k = keccak.new(digest_bits=256)
 k.update('age')
 print k.hexdigest()</sxh>
  * So here are the methods I came up with in a new **UniswapSigmap** class to handle the generation of "signature banks" from function declarations: <sxh python; highlight: []>    def get_keccak256(self, text):
        """Generate the keccak-256 from a given string"""
        kec = keccak.new(digest_bits=256)
        kec.update(text.encode())
        return kec.hexdigest()

    def gen_signatures(self, fnames, params):
        """Generate a list of signatures given a list of function names
        and the parameters that should be apppended to the name"""
        sigs = []

        for fname in fnames:
            # generate the signature:
            full_name = fname + params
            sig = "0x" + self.get_keccak256(full_name)[:8]
            sigs.append(sig)
            logger.info("Registering signature '%s' for '%s'", sig, full_name)

        return sigs

    def setup_signature_banks(self, syms):
        """Prepare the signature banks given a list of acceptable
        native token names in the funtion names."""

        # Swap native for tokens:
        # ex: swapExactETHForTokens(uint256 amountOutMin, address[] path, address to, uint256 deadline)
        params = "(uint256,address[],address,uint256)"
        fnames = [f"swapExact{name}ForTokens" for name in syms]
        fnames += [f"swapExact{name}ForTokensSupportingFeeOnTransferTokens" for name in syms]
        fnames += [f"swap{name}ForExactTokens" for name in syms]

        self.sig_banks["swap_native_for_tokens"] = self.gen_signatures(fnames, params)

        # Swap tokens:
        # ex: swapExactTokensForETH(uint256 amountIn, uint256 amountOutMin, address[] path, address to, uint256 deadline)
        params = "(uint256,uint256,address[],address,uint256)"
        fnames = [f"swapExactTokensFor{name}" for name in syms]
        fnames += [f"swapTokensForExact{name}" for name in syms]
        fnames += [f"swapExactTokensFor{name}SupportingFeeOnTransferTokens" for name in syms]
        fnames += [
            "swapTokensForExactTokens",
            "swapExactTokensForTokens",
            "swapExactTokensForTokensSupportingFeeOnTransferTokens",
        ]

        self.sig_banks["swap_tokens"] = self.gen_signatures(fnames, params)

        # Tokens liquidity:
        # ex: addLiquidity(address tokenA, address tokenB, uint256 amountADesired, uint256 amountBDesired,
        # uint256 amountAMin, uint256 amountBMin, address to, uint256 deadline)
        params = "(address,address,uint256,uint256,uint256,uint256,address,uint256)"
        fnames = ["addLiquidity"]

        self.sig_banks["token_liquidity"] = self.gen_signatures(fnames, params)

        # Remove liquidity:
        # ex: removeLiquidityWithPermit(address tokenA, address tokenB, uint256 liquidity, uint256 amountAMin,
        # uint256 amountBMin, address to, uint256 deadline, bool approveMax, uint8 v, bytes32 r, bytes32 s)
        params = "(address,address,uint256,uint256,uint256,address,uint256,bool,uint8,bytes32,bytes32)"
        fnames = ["removeLiquidityWithPermit"]

        self.sig_banks["token_liquidity"] += self.gen_signatures(fnames, params)

        # removeLiquidity(address tokenA, address tokenB, uint256 liquidity, uint256 amountAMin,
        # uint256 amountBMin, address to, uint256 deadline)
        params = "(address,address,uint256,uint256,uint256,address,uint256)"
        fnames = ["removeLiquidity"]

        self.sig_banks["token_liquidity"] += self.gen_signatures(fnames, params)

        # Native liquidity:
        # ex: addLiquidityETH(address token, uint256 amountTokenDesired, uint256 amountTokenMin,
        # uint256 amountETHMin, address to, uint256 deadline)
        params = "(address,uint256,uint256,uint256,address,uint256)"
        fnames = [f"addLiquidity{name}" for name in syms]
        fnames += [f"removeLiquidity{name}" for name in syms]
        fnames += [f"removeLiquidity{name}SupportingFeeOnTransferTokens" for name in syms]

        self.sig_banks["native_liquidity"] = self.gen_signatures(fnames, params)

        # removeLiquidityETHWithPermit(address token, uint256 liquidity, uint256 amountTokenMin,
        # uint256 amountETHMin, address to, uint256 deadline, bool approveMax, uint8 v, bytes32 r, bytes32 s)
        params = "(address,uint256,uint256,uint256,address,uint256,bool,uint8,bytes32,bytes32)"
        fnames = [f"removeLiquidity{name}WithPermit" for name in syms]
        fnames += [f"removeLiquidity{name}WithPermitSupportingFeeOnTransferTokens" for name in syms]

        self.sig_banks["native_liquidity"] += self.gen_signatures(fnames, params)</sxh>

  * => using this with our regular ArbitrageManager on BSC seems to work as usual, so I guess I didn't break anything 😁.

  * Now time to test on Avalanche 🤞: <sxh bash; highlight: []>$ nvp arbman monitor-arbs -c avax -n</sxh>
  * Arrff... seems better, but we get an error now because we have no funds in the PairChecker contract: <sxh bash; highlight: []>Traceback (most recent call last):
File "D:\Projects\NervHome\nvh\crypto\blockchain\arbitrage_manager.py", line 921, in <module>
comp.run()
File "D:\Projects\NervProj\nvp\nvp_component.py", line 93, in run
res = self.process_command(cmd)
File "D:\Projects\NervHome\nvh\crypto\blockchain\arbitrage_manager.py", line 876, in process_command
self.monitor_arbitrages(chain)
File "D:\Projects\NervHome\nvh\crypto\blockchain\arbitrage_manager.py", line 132, in monitor_arbitrages
result = self.handle_arbitrage_setups()
File "D:\Projects\NervHome\nvh\crypto\blockchain\arbitrage_manager.py", line 297, in handle_arbitrage_setups
self.process_arb_setups(plist, pair_reserves)
File "D:\Projects\NervHome\nvh\crypto\blockchain\arbitrage_manager.py", line 406, in process_arb_setups
if (t0.is_class_unknown() or t1.is_class_unknown()) and not self.check_pair(best_p0, qtoken):
File "D:\Projects\NervHome\nvh\crypto\blockchain\arbitrage_manager.py", line 651, in check_pair
self.check(
File "D:\Projects\NervProj\nvp\nvp_object.py", line 73, in check
raise NVPCheckError(fmt % args)
nvp.nvp_object.NVPCheckError: Initial balance of USDC too low, cannot perform pair check for WAVAX/USDC.</sxh>

  * => So I definitely have to send some tokens there, before pushing it further.
  * First thing I need for that is support to wrap the native token: <sxh python; highlight: []>        if cmd == "wrap":
            chain_name = self.get_param("chain")
            chain: EVMBlockchain = self.get_component(f"{chain_name}_chain")

            account = self.get_param("account")
            if account is not None:
                chain.set_account(account)

            # We can only unwrap the native token:
            token = chain.get_wrapped_native_token()

            value = self.get_param("value")
            self.check(value is not None, "No value specified.")

            amount = token.to_amount(value)

            if amount == 0:
                logger.info("Nothing to wrap.")
                return True

            # Get the initial balance:
            bal0 = chain.get_native_balance()
            self.check(bal0 > value, "Native balance too low.")

            logger.info("Initial wrapper native balance: %f %s", token.get_balance(), token.symbol())
            logger.info("Wrapping %d units of %s", amount, chain.get_native_symbol())
            token.deposit(amount)
            logger.info("Final wrapped native balance: %f %s", token.get_balance(), chain.get_native_symbol())
            return True</sxh>
  * Now I can call: <sxh bash; highlight: []>$ nvp bchain wrap -c avax -a evm_arb -v 0.001</sxh>
  * Then I started to add support to automatically swap some tokens as needed for the PairChecker setup: <sxh python; highlight: []>    def update_quote_token_values(self):
        """Update the value of the quote tokens in the native currency"""
        self.quote_token_values = {}

        dex = self.chain.get_default_exchange()

        # Check that we have some of the wrapped native token available:
        # and if not we send some:
        ntoken = self.chain.get_wrapped_native_token()
        if ntoken.get_balance(as_value=False) == 0:
            logger.info("Wrapping some %s PairChecker setup...", self.native_symbol)
            ntoken.deposit(200000 * 20)

        dex = self.chain.get_default_exchange()

        for addr in self.quote_tokens:
            val = dex.get_quote(1.0, addr, self.native_symbol)
            self.quote_token_values[addr] = val

            token = self.chain.get_token(addr)
            logger.info("Quote token %s value: %.6g %s", token.symbol(), val, self.native_symbol)

            # Get the current balance of qtoken in the pair checker:
            bal = token.get_balance(as_value=False, account=self.pair_checker_address)

            if bal == 0:
                if token.get_balance(as_value=False) < 400000:
                    nval = max(int(203000 * val) + 1, 1)

                    logger.info("Swapping %.6g units of %s for %s", nval, ntoken.symbol(), token.symbol())
                    dex.swap_tokens(nval, [ntoken.symbol(), token.symbol()], is_amount=True)

                nbal = token.get_balance(as_value=False)
                while nbal < 200000:
                    logger.info("Waiting for funds to arrive... (current balance: %d %s)", nbal, token.symbol())
                    time.sleep(1.0)
                    nbal = token.get_balance(as_value=False)

                # # We don't have the funds yet ourself, so we conduct a swap operation first:
                # logger.warning("Not enough %s funds to send to pair checker contract.", token.symbol())

                logger.info("Sending %s funds to pair checker contract...", token.symbol())
                token.transfer(self.pair_checker_address, 200000)

            bal = token.get_balance(as_value=False, account=self.pair_checker_address)
            logger.info("PairChecker balance: %d %s", bal, token.symbol())</sxh>
  * Unfortunately, it seems the swapping operation is failing: <sxh bash; highlight: []>2022/06/28 12:28:05 [__main__] INFO: Quote token MIM value: 0.0501456 WAVAX
2022/06/28 12:28:05 [__main__] INFO: Swapping 10180 units of WAVAX for MIM
2022/06/28 12:28:06 [nvh.crypto.blockchain.uniswap_base] INFO: Reserve for WAVAX: 44787055416129637105101
2022/06/28 12:28:06 [nvh.crypto.blockchain.uniswap_base] INFO: Reserve for MIM: 890459127292300704299446
2022/06/28 12:28:06 [nvh.crypto.blockchain.uniswap_base] INFO: Amount_out_min: 201590
2022/06/28 12:28:06 [nvh.crypto.blockchain.uniswap_base] INFO: Amount_out_min value: 2.016e-13 MIM
2022/06/28 12:28:06 [nvh.crypto.blockchain.evm_blockchain] INFO: Error while trying to perform operation: execution reverted: TransferHelper: TRANSFER_FROM_FAILED
2022/06/28 12:28:06 [__main__] INFO: Waiting for funds to arrive... (current balance: 0 MIM)
2022/06/28 12:28:07 [__main__] INFO: Waiting for funds to arrive... (current balance: 0 MIM)
2022/06/28 12:28:08 [__main__] INFO: Waiting for funds to arrive... (current balance: 0 MIM)
2022/06/28 12:28:09 [__main__] INFO: Waiting for funds to arrive... (current balance: 0 MIM)</sxh>
  * => Actually, this is somewhat expected, because I'm pretty sure I need to approve some contract first before it may swap my WAVAX tokens on my behalf 😅 Question is, which contract should that be exactly 🤔?
  * => OK, that should simply be the router since this is the one that will try to transfer our funds to the first pair on the path at start of the swap process: <sxh cpp; highlight: []>    function swapExactTokensForTokensSupportingFeeOnTransferTokens(
        uint256 amountIn,
        uint256 amountOutMin,
        address[] calldata path,
        address to,
        uint256 deadline
    ) external virtual override ensure(deadline) {
        TransferHelper.safeTransferFrom(
            path[0],
            msg.sender,
            JoeLibrary.pairFor(factory, path[0], path[1]),
            amountIn
        );
        uint256 balanceBefore = IERC20Joe(path[path.length - 1]).balanceOf(to);
        _swapSupportingFeeOnTransferTokens(path, to);
        require(
            IERC20Joe(path[path.length - 1]).balanceOf(to).sub(balanceBefore) >=
                amountOutMin,
            "JoeRouter: INSUFFICIENT_OUTPUT_AMOUNT"
        );
    }</sxh>
  * So let's approve the router on the WAVAX token first... **OK**! This helped: <sxh python; highlight: []>        dex = self.chain.get_default_exchange()
        addr = dex.get_router_address()

        if ntoken.get_allowance(addr) == 0:
            logger.info("Increasing allowance for router (%s) on %s", addr, ntoken.symbol())
            ntoken.approuve(addr, 10000000)</sxh>

  * But I still have another problem when it comes to USDC.e: <sxh bash; highlight: []>2022/06/28 13:35:04 [__main__] INFO: Quote token USDC.e value: 0.0498541 WAVAX
2022/06/28 13:35:04 [__main__] INFO: Swapping 10121 units of WAVAX for USDC.e
2022/06/28 13:35:04 [nvh.crypto.blockchain.uniswap_base] INFO: Reserve for WAVAX: 184375863573191424131829
2022/06/28 13:35:04 [nvh.crypto.blockchain.uniswap_base] INFO: Reserve for USDC.e: 3687209961192
2022/06/28 13:35:04 [nvh.crypto.blockchain.uniswap_base] INFO: Amount_out_min: 0
2022/06/28 13:35:04 [nvh.crypto.blockchain.uniswap_base] INFO: Amount_out_min value: 0 USDC.e
2022/06/28 13:35:04 [nvh.crypto.blockchain.evm_blockchain] INFO: Error while trying to perform operation: execution reverted: Joe: INSUFFICIENT_OUTPUT_AMOUNT
2022/06/28 13:35:04 [__main__] INFO: Waiting for funds to arrive... (current balance: 0 USDC.e)
2022/06/28 13:35:05 [__main__] INFO: Waiting for funds to arrive... (current balance: 0 USDC.e)
2022/06/28 13:35:06 [__main__] INFO: Waiting for funds to arrive... (current balance: 0 USDC.e)
2022/06/28 13:35:07 [__main__] INFO: Waiting for funds to arrive... (current balance: 0 USDC.e)</sxh>

  * => The "Amount_out_min" is incorrect of course. Could that be because the number of decimals are not the same for both tokens ?

  * Arrfff, actually we only have 6 decimals for USDC.e so the amount of WAVAX I'm requesting to swap is going to be too small here. => Let's just do the swap manually. Or let's just ignore that quote token for now: the pair checker would by default need 100000 units of USDC.e to do the test, which is an "high value" already (ie. 0.1$ lol) => We'll get back to this if the arbitrage system works (but it won't work, so no worries).

  * Hmmm, interestingly, I get failing classification for tokens that are really supposed to be valid here: <sxh bash; highlight: []>2022/06/28 14:13:31 [__main__] ERROR: Pair checker test failed for USDT.e: execution reverted: [error] fn=swapToken0FL
2022/06/28 14:13:31 [__main__] ERROR: Pair address: 0xe28984e1EE8D431346D32BeC9Ec800Efb643eef4, exchange: PangolinSwap, router: 0xE54Ca86531e17Ef3616d22Ca28b0D458b6C89106, ifp: 9970
2022/06/28 14:13:31 [__main__] INFO: PairChecker result for USDT.e (against WAVAX) is: FAILED</sxh>
  * => I should really have a closer look at that at some point, but first, I think I could try to deploy the flasharb contract itself ✌: <sxh bash; highlight: []>$ nvp bchain deploy -c avax -a evm_arb FlashArbV5</sxh>
  * Now trying to run the arb monitoring again, but not in dry-run mode: <sxh bash; highlight: []>$ nvp arbman monitor-arbs -c avax</sxh>
  * And crap... 😭 I just get **execution reverted** messages when estimating the gas prices each time I find a valid setup: <sxh bash; highlight: []>2022/06/28 14:47:45 [__main__] INFO: Block 16630211: Found arb setup of 0.001141 WAVAX with pairs on EGG#2/WAVAX
2022/06/28 14:47:46 [__main__] ERROR: Error while estimating operation gas: execution reverted
2022/06/28 14:48:00 [__main__] INFO: 9 pair reserves collected in 0.1179 secs
2022/06/28 14:48:00 [__main__] INFO: Extracted infos: {'r0': 1784880975329859006333, 'r1': 323415904337601654857214, 'r0dt': 0, 'r1dt': 0, 'swap_exp': 18065387, 'swap_rcv': 18065387, 'transfer_exp': 18065387, 'transfer_rcv': 18065387, 'amount_back': 99400}
2022/06/28 14:48:00 [__main__] INFO: PairChecker result for HeC (against WAVAX) is: PASSED
2022/06/28 14:48:00 [__main__] INFO: Saving classification for token HeC
2022/06/28 14:48:00 [__main__] INFO: PairCheck done in 0.2883 secs
2022/06/28 14:48:00 [__main__] INFO: Block 16630218: Found arb setup of 0.0003253 WAVAX with pairs on WAVAX/HeC
2022/06/28 14:48:00 [__main__] ERROR: Error while estimating operation gas: execution reverted</sxh>

  * My initial guess on that error above is that I might have some kind of incompatibility in the assembly code ? Maybe I should consider using a previous version of the compiler ? I don't know...
  * Soooooooo... from there, where do I go ? 😋 Arrfff, I guess I should stop it here for this "session", I need some fresh ideas to work on anyway 🤣...