blog:2022:0524_crypto_gas_price_stats [NervTech's Wiki]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Crypto: Computing EVM blockchain gas price statistics ======

{{tag>dev python crypto nervhome finance bsc blockchain}}

Continuing from my prevvous article on the [[blog:2022:0522_crypto_linear_finance_claims|Linear Finance reward claims]], I think I need to push it a bit further on the gas price I use for my transactions: using an hardcoded value of 7Gwei is certainly not ideal, instead I should build a dedicated system to continuously update the statistics on the gas price, so let's implement that system here.

====== ======

===== Initial experiments =====

  * Actually I already have the required function to compute the gas price from statistics. But I feel I should also prepare a jupyter env to be able to test those jsonrpc functions manually, so I'm setting this up first: <sxh javascript; highlight: []>    "bscjup": {
      "custom_python_env": "bsc_env",
      "cmd": "${PYTHON} -m jupyter lab",
      "cwd": "${PROJECT_ROOT_DIR}",
      "python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"]
    },</sxh>

  * I then tested a minimal script in jupyter: <sxh python; highlight: []>from nvp.nvp_context import NVPContext

if NVPContext.instance is None:
    NVPContext()
    
ctx = NVPContext.get()
chain = ctx.get_component('bsc_chain')

bck = chain.get_block(full_tx=True)</sxh>
  * And indeed, in the resulting block above ('bck') we can retrieve the list of transactions from bck['transactions'], and in the the gas price specified for that transaction
  * So I just added the following methods in the EVMBlockchain class: <sxh python; highlight: []>    def get_gas_prices(self, bnum=None):
        """Collect the gas prices in a given block"""
        # Retrieve the block details with all the transactions:
        bck = self.get_block(block_num=bnum, full_tx=True)
        # logger.info("block content: %s", self.pretty_print(bck))
        # logger.info("block content: %s", bck)

        # We retrieve the list of transactions from the block:
        transactions = bck['transactions']
        tnum = len(transactions)
        bnum = bck['number']
        logger.info("Block %d contains %d transactions", bnum, tnum)

        # We collect all the gas prices from the transactions:
        gprices = []
        for i in range(tnum):
            # We should convert the hex string to a number:
            txn = transactions[i]
            price = txn['gasPrice']/1e9
            if price > 0.0:
                gprices.append(price)

        return {
            'prices': gprices,
            'block_number': bnum,
            'timestamp': bck['timestamp']}

    # Function to collect the mode, mean, stddev statistics from a given list of gas prices
    # from a given block number. If not specified then return the last block details.
    def get_gas_price_statistics(self, bnum=None):
        """"Method used to retrieve the gas price statistics from a given block"""
        data = self.get_gas_prices(bnum)
        gprices = data['prices']
        data['num_prices'] = len(gprices)
        data['mode'] = statistics.mode(gprices)
        data['mean'] = statistics.mean(gprices)
        data['dev'] = statistics.stdev(gprices)

        return data</sxh>

  * Now we have the question of the storage of that data: basically 2 options here: either in a dedicated remote SQL database or some local storage DB or file:
  * => The remote storage option is probably the best option: even if I then need some fast crypto system on an extern computer, "updating the gas price" is not something that would have to be done super quickly, and I could still update the storage mechanism at that time anyway.
  * So let's prepare a database for that: <sxh python; highlight: []>SQL_CREATE_TOKENS_TABLE = """ CREATE TABLE IF NOT EXISTS tokens (
                                id SERIAL PRIMARY KEY,
                                address char(42) UNIQUE,
                                name varchar(64),
                                symbol varchar(32) UNIQUE,
                                decimals SMALLINT,
                                min_slippage INT,
                                swap_fees INT,
                                classification SMALLINT
                            ); """

SQL_INSERT_TOKEN = ''' INSERT INTO tokens(address,name,symbol,decimals,min_slippage,swap_fees,classification)
                       VALUES(%s,%s,%s,%s,%s,%s,%s) '''

SQL_UPDATE_TOKEN = ''' UPDATE tokens SET (name,symbol,decimals,min_slippage,swap_fees,classification)
                       = (%s,%s,%s,%s,%s,%s) WHERE address=%s; '''

SQL_CREATE_GASPRICE_TABLE = """ CREATE TABLE IF NOT EXISTS gasprice (
                                        block_num integer PRIMARY KEY,
                                        timestamp integer,
                                        num_transactions integer,
                                        mode float,
                                        mean float,
                                        std_dev float
                                    ); """

SQL_INSERT_GASPRICE = ''' INSERT INTO gasprice(block_num, timestamp, num_transactions, mode, mean, std_dev)
                          VALUES(%s,%s,%s,%s,%s,%s)'''


class ChainDB(NVPObject):
    """ChainDB component class"""

    def __init__(self, db_name, user, password):
        """ChainDB constructor"""
        self.sql_db = PostgreSQLDB(db_name=db_name, user=user, password=password)
        self.sql_db.execute(SQL_CREATE_TOKENS_TABLE, commit=True)
        self.sql_db.execute(SQL_CREATE_GASPRICE_TABLE, commit=True)

    def execute(self, *args, **kaargs):
        """Forward execute code to the SQL DB"""
        return self.sql_db.execute(*args, **kaargs)

    def insert_gas_price_stats(self, data, check=True):
        """Insert a new gasprice row in the table"""
        # We should insert (block_num,mode,mean,std_dev)
        logger.info("Inserting gasprice data: %s", data)
        keys = ['block_number', 'timestamp', 'num_transactions', 'mode', 'mean', 'std_dev']
        row = [data[key] for key in keys]
        if check:
            # Check if the row already exists:
            bnum = data['block_number']
            cur = self.execute("select exists(select 1 from gasprice where block_num=%s)", (bnum,))
            if cur.fetchone()[0]:
                logger.warning('Gas price entry already provided for block %d', bnum)
                return

        self.execute(SQL_INSERT_GASPRICE, row, commit=True)</sxh>

  * And I now have a dedicate method in the EVMBlockchain to store new rows in the gasprice table: <sxh python; highlight: []>    def collect_gas_price_statistics(self):
        """Collect the latest gas price statistics"""
        data = self.get_gas_price_statistics()
        del data['prices']
        self.get_db().insert_gas_price_stats(data)</sxh>

  * Tested that in jupyter and it works as expected: <sxh python; highlight: []>chain.collect_gas_price_statistics()</sxh>
  * Command above will output: <sxh bash; highlight: []>2022/05/23 08:27:21 [nvh.crypto.blockchain.evm_blockchain] INFO: Block 18045926 contains 150 transactions
2022/05/23 08:27:21 [nvh.crypto.blockchain.chain_db] INFO: Inserting gasprice data: {'block_number': 18045926, 'timestamp': 1653290838, 'num_transactions': 149, 'mode': 5.0, 'mean': 5.84451191375839, 'std_dev': 2.419240396353444}</sxh>
  * => And I can see that a new row is added to the table, all **OK**.

  * Now I just need to add new data every minute, so let's add a command line entry to handle that: <sxh python; highlight: []>class BinanceSmartChain(EVMBlockchain):
    """BinanceSmartChain component class"""

    def __init__(self, ctx):
        """blockchain constructor"""
        cfg = ctx.get_config()['blockchains']['bsc']
        EVMBlockchain.__init__(self, ctx, cfg)

    def get_contract_abi_url(self, address):
        """Retrieve the ABI URL for a given contract"""
        api_key = self.config.get("bscscan_api_key", None)
        url = f"https://api.bscscan.com/api?module=contract&action=getabi&address={address}"

        if api_key is not None:
            url = f"{url}&apikey={api_key}"

        return url, 1.0/5 if api_key is not None else 5.0

    def process_command(self, cmd):
        """Check if this component can process the given command"""

        if cmd == 'collect-gas-price':
            self.collect_gas_price_statistics()
            return True

        return False


if __name__ == "__main__":
    # Create the context:
    context = NVPContext()

    # Add our component:
    comp = context.register_component("bsc_chain", BinanceSmartChain(context))

    context.define_subparsers("main", {
        'collect-gas-price': None,
    })

    comp.run()
</sxh>

  * And we add the corresponding script: <sxh javascript; highlight: []>    "bsc": {
      "custom_python_env": "bsc_env",
      "cmd": "${PYTHON} ${PROJECT_ROOT_DIR}/nvh/crypto/bsc/binance_smart_chain.py",
      "python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"]
    },</sxh>
  * And finally the entry in our **every_1min_operations.sh** file: <sxh bash; highlight: []>log_dir="/mnt/data1/admin/logs"

lfile="${log_dir}/bsc_collect_gas_price.log"
nvp bsc collect-gas-price 2>&1 | tee -a $lfile</sxh>

===== Retrieving the latest statistic values =====

  * I then added the following function in the **ChainDB** class to get back the gasprice data: <sxh python; highlight: []>    def get_gas_price_stats(self, max_num=None, min_ts=None):
        """Function returning the num_transactions, mode, mean, std_dev columns from the gas price table"""
        if max_num is not None:
            sql = f"SELECT * FROM gasprice ORDER BY block_num DESC LIMIT {max_num}"
            sql = f"SELECT * FROM ({sql}) as sub ORDER BY sub.block_num ASC;"
        elif min_ts is not None:
            if min_ts < 0:
                # Assume relative to current timestamp and given in seconds:
                min_ts = self.get_timestamp() + min_ts
                # logger.info("using min_ts: %d", min_ts)
            sql = f"SELECT * FROM gasprice WHERE timestamp > {min_ts} ORDER BY timestamp ASC;"
        else:
            sql = "SELECT * FROM gasprice;"

        cur = self.execute(sql)
        results = cur.fetchall()
        # logger.debug("Retrieved %d gasprice rows", len(results))
        return np.array(results)</sxh>

  * And with that I should be able to compute a more appropriate gas price before I perform a transaction: <sxh python; highlight: []>    def get_current_gas_price_emas(self, period=10):
        """Compute the EMAs for the gas price data"""
        # Retrieve the gas price statistics with a given smoothing EMA period:
        # Note: we should retrieve at least twice as much data as the period length:
        data = self.get_db().get_gas_price_stats(min_ts=-period*2*60)

        # Now compute the ema for each col:
        # Note: we only keep the EMAs for the last 4 columns
        res = [ta.trend.EMAIndicator(pd.Series(data[:, i+2]), window=period).ema_indicator().iloc[-1] for i in range(4)]

        return dict(zip(("num_transactions", "mode", "mean", "std_dev"), res))

    def compute_gas_price_estimate(self, mode="default"):
        """Compute an advanced gas price estimate."""

        # Retrieve the weights from the selected mode:
        weights = self.config["gas_price_weights"][mode]

        # Compute an estimated gas price, using weights for the mode,mean, mean+std_dev price values:
        # the weights should also contain the estimation period:
        self.check(len(weights) == 4, "Invalid number of weights")

        emas = self.get_current_gas_price_emas(period=weights[0])
        weights = weights[1:]
        totw = sum(weights)
        self.check(totw > 0.0, "Invalid total weights.")

        price = (emas['mode'] * weights[0] + emas['mean']
                 * weights[1] + (emas['mean']+emas['std_dev']) * weights[2])/totw
        price = math.floor(price*1000)/1000

        # Clamp if bigger than max gas price:
        maxp = self.max_gas_price
        if price > maxp:
            logger.warning("Clamping estimated gas price of %f to max gas price value of %f", price, maxp)
            price = maxp
        else:
            logger.info("Computed estimated gas price of %f gwei for mode %s", price, mode)

        return price</sxh>

<note>I just discovered the **ta** package above: it's using pandas, but add support for many technical indicators without any fancy dependency build requirement like **TA-lib** so I will go with that one for now!</note>

  * And if I call the ''compute_gas_price_estimate()'' method above in jupyter I get the following kind of result (as expected): <code>2022/05/23 10:26:27 [nvh.crypto.blockchain.evm_blockchain] INFO: Computed estimated gas price of 5.275000 gwei for mode default</code>

===== Displaying the gas price chart in cryptoview =====

  * One last bonus step I would like to take now would be to display the current gas price as a chart in my cryptoview app: could be interesting to have a look at that chart sometimes 😉!
  * So here is my first "BSCPanel" implementation: <sxh python; highlight: []>"""BSCPanel widget module"""

import logging
from PyQt5 import QtCore
from PyQt5.QtCore import Qt
import PyQt5.QtWidgets as qwd
import numpy as np

from nvp.nvp_context import NVPContext
from nvh.gui.app_base import AppBase
from nvh.gui.plot_canvas import PlotCanvas
from nvh.crypto.bsc.binance_smart_chain import BinanceSmartChain

logger = logging.getLogger(__name__)


class BSCPanel(qwd.QWidget):
    """BSCPanel class"""

    def __init__(self):
        """Constructor of the BSCPanel class"""
        super().__init__()

        # Create a CoinGecko() instance:
        ctx = NVPContext.get()
        self.app: AppBase = ctx.get_component("app")
        self.chain: BinanceSmartChain = ctx.get_component("bsc_chain")

        # chart display offset:
        self.offset = 0.04
        self.build_panel()

        setattr(self, "sizeHint", lambda: QtCore.QSize(360, 300))

        self.timer = QtCore.QTimer()
        self.timer.setInterval(1000*60)
        self.timer.timeout.connect(self.update_gas_price_chart)
        self.timer.start()

        self.update_gas_price_chart()

    def build_panel(self):
        """Build the panel for this widget"""

        lyt = qwd.QVBoxLayout()
        lyt.setContentsMargins(5, 5, 5, 5)
        lyt.setSpacing(5)

        self.chart = PlotCanvas(self, width=2, height=2, dpi=100)
        self.toolbar = self.chart.create_toolbar(self)
        self.ax2 = self.chart.axes.twinx()

        lyt.addWidget(self.toolbar)
        lyt.addWidget(self.chart)

        self.setLayout(lyt)

    def update_gas_price_chart(self):
        """Update the gas price chart"""

        logger.info("Updating gas price chart.")

        # Collect the available entries:
        # Get at max 7 days of data:
        data = self.chain.get_db().get_gas_price_stats(min_ts=-7*24*60*60)

        self.chain.check(data.shape[1] == 6, "Invalid number of gas price columns: %d", data.shape[1])

        # we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values:
        now_ts = self.chain.get_timestamp()
        times = now_ts - data[:, 1]

        # convert times to hours:
        times /= 3600
        mean = data[:, 4]
        mode = data[:, 3]
        txn = data[:, 2]
        # mean_up = mean + data[:, 5]

        self.chart.clear()
        self.chart.set_title("BSC Gas Price")

        self.chart.axes.plot(times, mode, color='orange', linewidth=1, label="Mode")
        self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean")
        self.ax2.cla()
        self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx")

        # self.chart.axes.plot(times, mean_up, color='red', linewidth=1, label="Mean + 1sig")
        self.chart.axes.set_xlabel("Time offset (hours)")
        self.chart.axes.set_ylabel("Gas price (Gwei)")
        self.ax2.set_ylabel("Number of tx")

        # cf. https://www.adamsmith.haus/python/answers/how-to-add-secondary-axis-to-a-legend-in-python
        lines_1, labels_1 = self.chart.axes.get_legend_handles_labels()
        lines_2, labels_2 = self.ax2.get_legend_handles_labels()
        lines = lines_1 + lines_2
        labels = labels_1 + labels_2

        self.chart.axes.legend(lines, labels, loc=0)
        xmin = np.amin(times)
        xmax = np.amax(times)
        offset = self.offset * (xmax - xmin)

        self.chart.axes.set_xlim([xmax+offset, xmin-offset])
        self.chart.draw()
</sxh>

  * And here is the kind of display this will produce: 

{{ blog:2022:0523:bsc_gas_price_v1.png }}

  * => yeah... It's a bit messy when we disply raw data directly: let's add a EMA filtering on those values instead 😉
  * So I added a **utility function** ''compute_ema()'' in my **nvh.crypto.utils** module: <sxh python; highlight: []>def compute_ema(arr, period):
    """Compute the EMA of a numpy array"""
    return ta.trend.EMAIndicator(pd.Series(arr), window=period).ema_indicator().values
</sxh>
  * And updated the gas price plot drawing to this: <sxh python; highlight: []>    def update_gas_price_chart(self):
        """Update the gas price chart"""

        logger.info("Updating gas price chart.")

        # Collect the available entries:
        # Get at max 7 days of data:
        period = 20
        data = self.chain.get_db().get_gas_price_stats(max_num=7*24*60+period)

        self.chain.check(data.shape[1] == 6, "Invalid number of gas price columns: %d", data.shape[1])

        # we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values:
        now_ts = self.chain.get_timestamp()
        times = now_ts - data[:, 1]

        # convert times to hours:
        times /= 3600
        mean = utl.compute_ema(data[:, 4], period)
        # logger.info("Mean values: %s", mean)
        mode = data[:, 3]
        # txn = data[:, 2]
        txn = utl.compute_ema(data[:, 2], period)
        # dev = utl.compute_ema(data[:, 5], period)

        # mean_up = mean + dev

        self.chart.clear()
        self.chart.set_title("BSC Gas Price")

        self.chart.axes.plot(times, mode, color='orange', linewidth=1, label="Mode")
        self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean")
        # self.chart.axes.plot(times, mean_up, linewidth=1, label="Mean + 1sig")
        self.ax2.cla()
        self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx")
        # self.ax2.plot(times, txn2, color='black', linewidth=1, label="Num. tx EMA")

        # self.chart.axes.plot(times, mean_up, color='red', linewidth=1, label="Mean + 1sig")
        self.chart.axes.set_xlabel("Time offset (hours)")
        self.chart.axes.set_ylabel("Gas price (Gwei)")
        self.ax2.set_ylabel("Number of tx")

        # cf. https://www.adamsmith.haus/python/answers/how-to-add-secondary-axis-to-a-legend-in-python
        lines_1, labels_1 = self.chart.axes.get_legend_handles_labels()
        lines_2, labels_2 = self.ax2.get_legend_handles_labels()
        lines = lines_1 + lines_2
        labels = labels_1 + labels_2

        self.chart.axes.legend(lines, labels, loc=0)
        xmin = np.amin(times)
        xmax = np.amax(times)
        offset = self.offset * (xmax - xmin)

        self.chart.axes.set_xlim([xmax+offset, xmin-offset])
        self.chart.draw()</sxh>

  * Which gives me the result:

{{ blog:2022:0523:bsc_gas_price_v2.png }}

  * => There isn't much data to display here for the moment, so this still feels a bit noisy, but I think it should get better when I have few days of history data to display instead, so let's just keep it this way fro the moment ✌.

===== Testing with other EVM blockchains =====

  * Actually, in all the steps above we only used some basic JSONRPC block/transaction retrieval: so this should in theory work the same for other EVM compatible blockchains ?
  * => Let's try to implement this for Ethereum:
  * Here is the simple **EthereumChain** class I create as a starting point: <sxh python; highlight: []>"""ETH blockchain."""

import logging

from nvp.nvp_context import NVPContext
from nvh.crypto.blockchain.evm_blockchain import EVMBlockchain

logger = logging.getLogger(__name__)


def create_component(ctx: NVPContext):
    """Create an instance of the component"""
    return EthereumChain(ctx)


class EthereumChain(EVMBlockchain):
    """EthereumChain component class"""

    def __init__(self, ctx):
        """blockchain constructor"""
        cfg = ctx.get_config()['blockchains']['eth']
        EVMBlockchain.__init__(self, ctx, cfg)

    def get_contract_abi_url(self, address):
        """Retrieve the ABI URL for a given contract"""
        api_key = self.config.get("etherscan_api_key", None)
        url = f"https://api.etherscan.io/api?module=contract&action=getabi&address={address}"

        if api_key is not None:
            url = f"{url}&apikey={api_key}"

        return url, 1.0/5 if api_key is not None else 5.0

    def process_command(self, cmd):
        """Check if this component can process the given command"""

        if cmd == 'collect-gas-price':
            self.collect_gas_price_statistics()
            return True

        return False


if __name__ == "__main__":
    # Create the context:
    context = NVPContext()

    # Add our component:
    comp = context.register_component("eth_chain", EthereumChain(context))

    context.define_subparsers("main", {
        'collect-gas-price': None,
    })

    comp.run()
</sxh>

  * Then we add a simple script to use it: <sxh javascript; highlight: []>    "eth": {
      "custom_python_env": "bsc_env",
      "cmd": "${PYTHON} ${PROJECT_ROOT_DIR}/nvh/crypto/eth/ethereum_chain.py",
      "python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"]
    },</sxh>

  * And this works pretty damn well already: <sxh bash; highlight: []>$ nvp eth collect-gas-price
2022/05/23 22:29:06 [nvh.crypto.blockchain.evm_blockchain] INFO: Block 14831913: num_tx=21, mode=13.580, mean=16.079, dev=4.162</sxh>

  * => So also adding that in my 1min operation script: <sxh bash; highlight: []>lfile="${log_dir}/eth_collect_gas_price.log"
nvp eth collect-gas-price 2>&1 | tee -a $lfile</sxh>


===== Investigating gas price collect error on Ethereum =====

  * Remember the system I built in my previous article to [[https://wiki.nervtech.org/doku.php?id=blog:2022:0522_crypto_linear_finance_claims#before_we_start|receive automatic notifications]] in case there is an exception in one of the scripts I run with cron ? Well, I got a few of those notifications over night about the ethereum gas price retrieval 😅:

{{ blog:2022:0523:eth_errors.png }}

  * Let's see what this is about... Ohhh okay, that's interesting: I found at least the following type of error in the logs: 
  * **Error 1**: <code>Traceback (most recent call last):
  File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 55, in <module>
    comp.run()
  File "/mnt/data1/dev/projects/NervProj/nvp/nvp_component.py", line 69, in run
    res = self.process_command(cmd)
  File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 38, in process_command
    self.collect_gas_price_statistics()
  File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 261, in collect_gas_price_statistics
    data = self.get_gas_price_statistics()
  File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 253, in get_gas_price_statistics
    data['mode'] = statistics.mode(gprices)
  File "/mnt/data1/dev/projects/NervProj/.pyenvs/bsc_env/lib/python3.10/statistics.py", line 583, in mode
    raise StatisticsError('no mode for empty data') from None
statistics.StatisticsError: no mode for empty data</code>
  * **Error 2**: <code>Traceback (most recent call last):
  File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 55, in <module>
    comp.run()
  File "/mnt/data1/dev/projects/NervProj/nvp/nvp_component.py", line 69, in run
    res = self.process_command(cmd)
  File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 38, in process_command
    self.collect_gas_price_statistics()
  File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 261, in collect_gas_price_statistics
    data = self.get_gas_price_statistics()
  File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 255, in get_gas_price_statistics
    data['std_dev'] = statistics.stdev(gprices)
  File "/mnt/data1/dev/projects/NervProj/.pyenvs/bsc_env/lib/python3.10/statistics.py", line 828, in stdev
    var = variance(data, xbar)
  File "/mnt/data1/dev/projects/NervProj/.pyenvs/bsc_env/lib/python3.10/statistics.py", line 767, in variance
    raise StatisticsError('variance requires at least two data points')
statistics.StatisticsError: variance requires at least two data points</code>

  * So typically, I think these may happen if we have respectively 0 or 1 transaction only in a given block which is of course totally legit, so let's fix that.
  * **OK** here is the updated code which should take care of those problems: <sxh python; highlight: []>    def get_gas_price_statistics(self, bnum=None):
        """"Method used to retrieve the gas price statistics from a given block"""
        data = self.get_gas_prices(bnum)
        gprices = data['prices']
        num = len(gprices)
        data['num_transactions'] = num
        data['mode'] = statistics.mode(gprices) if num > 0 else math.nan
        data['mean'] = statistics.mean(gprices) if num > 0 else math.nan
        data['std_dev'] = statistics.stdev(gprices) if num > 1 else math.nan

        return data

    def collect_gas_price_statistics(self):
        """Collect the latest gas price statistics"""
        data = self.get_gas_price_statistics()
        del data['prices']
        # Note: we only store the block price data if there are at least 2 transactions:
        if data['num_transactions'] < 2:
            logger.info("Ignoring block %d: not enough transactions.", data['block_number'])
            return

        logger.info("Block %d: num_tx=%d, mode=%.3f, mean=%.3f, dev=%.3f",
                    data['block_number'], data['num_transactions'], data['mode'],
                    data['mean'], data['std_dev'])
        self.get_db().insert_gas_price_stats(data)</sxh>

  * **Side note**: I also get some **warning** message on the ethereum blockchain, because checking every minute, there is sometimes no new block on the chain, so I get this kind of message (but this is not critical and already handled properly internally, so I'm leaving this as is): <code>[nvh.crypto.blockchain.chain_db] WARNING: Gas price entry already provided for block 14834104</code>

===== Using a dedicated rocketchat channel for problem reports =====

  * While I'm at it, I think I should also try to use a dedicated channel to report those exceptions/errors on rochetchat: otherwise this is quickly filling my "general admin" channel which I would prefer to keep clean instead. So let's change that.
    * Created the **problem-reports** channel: **OK**
    * Updated the runner code to notify in that channel: <sxh python; highlight: []>        try:
            self.execute(cmd, cwd=cwd, env=env)
        except subprocess.SubprocessError:
            notify = self.config.get("notify_script_errors", True)
            if notify:
                # And exception occured in the sub process, so we should send a notification:
                msg = ":warning: **WARNING:** an exception occured in the following command:\n"
                msg += f"{cmd}\n"
                msg += f"cwd={cwd}\n\n"
                msg += "=> Check the logs for details."

                rchat = self.get_component("rchat")
                rchat.send_message(msg, channel="problem-reports")</sxh>
    * And also extended the rochetchat command line to support testing the channel: <sxh python; highlight: []>if __name__ == "__main__":
    # Create the context:
    context = NVPContext()

    # Add our component:
    comp = context.register_component("rchat", RocketChat(context))

    context.define_subparsers("main", {
        'send': None,
    })

    psr = context.get_parser('main.send')
    psr.add_argument("message", type=str,
                     help="Simple message that we should send")
    psr.add_argument("-c", "--channel", dest="channel", type=str,
                     help="Channel where to send the message.")

    comp.run()</sxh>

  * And this simple command below works so we are good for now: <sxh bash; highlight: []>$ nvp rchat "hello manu" -c problem-reports</sxh>


===== Setting up backups for the new SQL databases =====

  * In the implementation above I introduced 2 new databases on my postgresql server: **bsc_chain** for the BSC and **eth_chain** for Ethereum (to store tha gas prices only for the moment, but this will be extended soon)
  * => So let's not forget to setup the corresponding backups for these!
  * Which is super easy to add in my custom NervProj config file: <sxh javascript; highlight: []>      // Backup of individual SQL databases:
      "postgresql_crypto_databases": {
        "type": "postgresql",
        "container": "postgresql_server",
        "user": "crypto_user",
        "databases": [
          "crypto_coins",
          "crypto_prices_v2",
          "bsc_chain",
          "eth_chain"
        ],
        "backup_dirs": [
          "${slot1}/sql/crypto",
          "${slot2}/sql/crypto",
          "${slot3}/sql/crypto"
        ]
      },</sxh>/*//*/

===== Adding Ethereum dedicated panel in CryptoView =====

  * And finally, let's also add a dedicated tab in **CryptoView** to report Ethereum specific informations (only the gas price for now, but I expect to be able to extend that soon ;-))
  * => For this I simply convert my previous **BSCPanel** into a more generic **EVMChainPanel** taking the chain component name as input: <sxh python; highlight: []>"""EVMChainPanel widget module"""

import logging
from PyQt5 import QtCore
from PyQt5.QtCore import Qt
import PyQt5.QtWidgets as qwd
import numpy as np


from nvp.nvp_context import NVPContext

import nvh.crypto.utils as utl
from nvh.gui.app_base import AppBase
from nvh.gui.plot_canvas import PlotCanvas
from nvh.crypto.blockchain.evm_blockchain import EVMBlockchain

logger = logging.getLogger(__name__)


class EVMChainPanel(qwd.QWidget):
    """EVMChainPanel class"""

    def __init__(self, comp_name):
        """Constructor of the EVMChainPanel class
        using the comp_name to retrieve the desired chain component"""
        super().__init__()

        # Create a CoinGecko() instance:
        ctx = NVPContext.get()
        self.app: AppBase = ctx.get_component("app")
        self.chain: EVMBlockchain = ctx.get_component(comp_name)

        # chart display offset:
        self.offset = 0.04
        self.build_panel()

        setattr(self, "sizeHint", lambda: QtCore.QSize(360, 300))

        self.timer = QtCore.QTimer()
        self.timer.setInterval(1000*60)
        self.timer.timeout.connect(self.update_gas_price_chart)
        self.timer.start()

        self.update_gas_price_chart()

    def build_panel(self):
        """Build the panel for this widget"""

        lyt = qwd.QVBoxLayout()
        lyt.setContentsMargins(5, 5, 5, 5)
        lyt.setSpacing(5)

        self.chart = PlotCanvas(self, width=2, height=2, dpi=100)
        self.toolbar = self.chart.create_toolbar(self)
        self.ax2 = self.chart.axes.twinx()

        lyt.addWidget(self.toolbar)
        lyt.addWidget(self.chart)

        self.setLayout(lyt)

    def update_gas_price_chart(self):
        """Update the gas price chart"""

        logger.info("Updating gas price chart.")

        # Collect the available entries:
        # Get at max 7 days of data:
        period = 20
        data = self.chain.get_db().get_gas_price_stats(max_num=7*24*60+period)

        self.chain.check(data.shape[1] == 6, "Invalid number of gas price columns: %d", data.shape[1])

        # we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values:
        now_ts = self.chain.get_timestamp()
        times = now_ts - data[:, 1]

        # convert times to hours:
        times /= 3600
        mean = utl.compute_ema(data[:, 4], period)
        # logger.info("Mean values: %s", mean)
        mode = data[:, 3]
        # txn = data[:, 2]
        txn = utl.compute_ema(data[:, 2], period)
        # dev = utl.compute_ema(data[:, 5], period)

        # mean_up = mean + dev

        self.chart.clear()
        sname = self.chain.get_short_name().upper()
        self.chart.set_title(f"{sname} Gas Price")

        self.chart.axes.plot(times, mode, color='orange', linewidth=1, label="Mode price")
        self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean price")
        # self.chart.axes.plot(times, mean_up, linewidth=1, label="Mean + 1sig")
        self.ax2.cla()
        self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx")
        # self.ax2.plot(times, txn2, color='black', linewidth=1, label="Num. tx EMA")

        # self.chart.axes.plot(times, mean_up, color='red', linewidth=1, label="Mean + 1sig")
        self.chart.axes.set_xlabel("Time offset (hours)")
        self.chart.axes.set_ylabel("Gas price (Gwei)")
        self.ax2.set_ylabel("Number of tx")

        # cf. https://www.adamsmith.haus/python/answers/how-to-add-secondary-axis-to-a-legend-in-python
        lines_1, labels_1 = self.chart.axes.get_legend_handles_labels()
        lines_2, labels_2 = self.ax2.get_legend_handles_labels()
        lines = lines_1 + lines_2
        labels = labels_1 + labels_2

        self.chart.axes.legend(lines, labels, loc=0)
        xmin = np.amin(times)
        xmax = np.amax(times)
        offset = self.offset * (xmax - xmin)

        self.chart.axes.set_xlim([xmax+offset, xmin-offset])
        self.chart.draw()
</sxh>

  * Then I create a couple of those panels on my main window: <sxh python; highlight: []>        dock3 = DockWidget("BSC")
        panel = EVMChainPanel("bsc_chain")
        self.widgets["bsc_panel"] = panel
        dock3.setWidget(panel)
        self.widgets["bsc_panel_dock"] = dock3

        dock4 = DockWidget("Ethereum")
        panel = EVMChainPanel("eth_chain")
        self.widgets["eth_panel"] = panel
        dock4.setWidget(panel)
        self.widgets["eth_panel_dock"] = dock4

        dock1.setAllowedAreas(Qt.AllDockWidgetAreas)
        dock2.setAllowedAreas(Qt.AllDockWidgetAreas)
        dock3.setAllowedAreas(Qt.AllDockWidgetAreas)
        dock4.setAllowedAreas(Qt.AllDockWidgetAreas)

        win.addDockWidget(Qt.RightDockWidgetArea, dock1)
        win.addDockWidget(Qt.RightDockWidgetArea, dock2)
        win.addDockWidget(Qt.RightDockWidgetArea, dock3)
        win.addDockWidget(Qt.RightDockWidgetArea, dock4)
        win.tabifyDockWidget(dock1, dock2)
        win.tabifyDockWidget(dock1, dock3)
        win.tabifyDockWidget(dock1, dock4)</sxh>
  
  * And this does the trick 👍!:

{{ blog:2022:0523:eth_gas_price_v1.png }}

  * Except that... it's a pretty dirty graph here lol => I think this is mainly because the **mode price** value is changing significantly on Ethereum (as opposed to on the BSC): So I should maybe applied some EMA filtering to that one also: <sxh python; highlight: []>    def update_gas_price_chart(self):
        """Update the gas price chart"""

        logger.info("Updating gas price chart.")

        # Collect the available entries:
        # Get at max 7 days of data:
        period = 20
        data = self.chain.get_db().get_gas_price_stats(max_num=7*24*60+period)

        self.chain.check(data.shape[1] == 6, "Invalid number of gas price columns: %d", data.shape[1])

        # we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values:
        now_ts = self.chain.get_timestamp()
        times = now_ts - data[:, 1]

        # convert times to hours:
        times /= 3600
        mean = utl.compute_ema(data[:, 4], period)
        # logger.info("Mean values: %s", mean)
        mode = utl.compute_ema(data[:, 3], period)
        # txn = data[:, 2]
        txn = utl.compute_ema(data[:, 2], period)
        # dev = utl.compute_ema(data[:, 5], period)

        # mean_up = mean + dev

        self.chart.clear()
        sname = self.chain.get_short_name().upper()
        self.chart.set_title(f"{sname} Gas Price")

        self.chart.axes.plot(times, mode, color='orange', linewidth=2, label="Mode price")
        self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean price")
        # self.chart.axes.plot(times, mean_up, linewidth=1, label="Mean + 1sig")
        self.ax2.cla()
        self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx")</sxh>

  * **Note**: I also bumped the mode price linewitdh to 2 pixels to try to make it more visible:

{{ blog:2022:0523:eth_gas_price_v2.png }}

  * That's a bit better already, but I feel I should maybe not put the plots of the transactions just mixed with the prices: this is getting confusing, so let's try to separate them: <sxh python; highlight: []>        self.chart.axes.legend(lines, labels, loc=2)
        xmin = np.amin(times)
        xmax = np.amax(times)
        offset = self.offset * (xmax - xmin)
        self.chart.axes.set_xlim([xmax+offset, xmin-offset])

        # update the vertical ranges:
        txn_min = np.nanmin(txn)
        txn_max = np.nanmax(txn)
        txn_range = txn_max - txn_min
        price_min = min(np.nanmin(mean), np.nanmin(mode))
        price_max = max(np.nanmax(mean), np.nanmax(mode))
        price_range = price_max - price_min

        self.chart.axes.set_ylim([price_min-self.offset*price_range,
                                  price_max+price_range*(1.0+3.0*self.offset)])
        self.ax2.set_ylim([txn_min-txn_range*(1.0+3.0*self.offset),
                           txn_max+txn_range*self.offset])

        self.chart.draw()</sxh>

  * And with that change the display feels much better already (from my perspective at least):

{{ blog:2022:0523:eth_gas_price_v3.png }}

  * And here the updated display for the BSC gas prices/transactions:

{{ blog:2022:0523:bsc_gas_price_v3.png }}


===== Conclusion =====

  * Okay, so, with the experiments conducted here we see that it is fairly easy to manually collect gas prices for a given EVM blockchain, and from that, to generate your own "gas price oracle".
  * => Eventually I could extend that to more EVM blockchains that I'm interested in, but for the moment, let's say this will be good enough and stop this long article here ;-)! See ya!