Crypto: Computing EVM blockchain gas price statistics

Continuing from my prevvous article on the Linear Finance reward claims, I think I need to push it a bit further on the gas price I use for my transactions: using an hardcoded value of 7Gwei is certainly not ideal, instead I should build a dedicated system to continuously update the statistics on the gas price, so let's implement that system here.

• Actually I already have the required function to compute the gas price from statistics. But I feel I should also prepare a jupyter env to be able to test those jsonrpc functions manually, so I'm setting this up first:

      "bscjup": {
        "custom_python_env": "bsc_env",
        "cmd": "${PYTHON} -m jupyter lab",
        "cwd": "${PROJECT_ROOT_DIR}",
        "python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"]
      },

• I then tested a minimal script in jupyter:

  from nvp.nvp_context import NVPContext
  
  if NVPContext.instance is None:
      NVPContext()
      
  ctx = NVPContext.get()
  chain = ctx.get_component('bsc_chain')
  
  bck = chain.get_block(full_tx=True)

• And indeed, in the resulting block above ('bck') we can retrieve the list of transactions from bck['transactions'], and in the the gas price specified for that transaction
• So I just added the following methods in the EVMBlockchain class:

      def get_gas_prices(self, bnum=None):
          """Collect the gas prices in a given block"""
          # Retrieve the block details with all the transactions:
          bck = self.get_block(block_num=bnum, full_tx=True)
          # logger.info("block content: %s", self.pretty_print(bck))
          # logger.info("block content: %s", bck)
  
          # We retrieve the list of transactions from the block:
          transactions = bck['transactions']
          tnum = len(transactions)
          bnum = bck['number']
          logger.info("Block %d contains %d transactions", bnum, tnum)
  
          # We collect all the gas prices from the transactions:
          gprices = []
          for i in range(tnum):
              # We should convert the hex string to a number:
              txn = transactions[i]
              price = txn['gasPrice']/1e9
              if price > 0.0:
                  gprices.append(price)
  
          return {
              'prices': gprices,
              'block_number': bnum,
              'timestamp': bck['timestamp']}
  
      # Function to collect the mode, mean, stddev statistics from a given list of gas prices
      # from a given block number. If not specified then return the last block details.
      def get_gas_price_statistics(self, bnum=None):
          """"Method used to retrieve the gas price statistics from a given block"""
          data = self.get_gas_prices(bnum)
          gprices = data['prices']
          data['num_prices'] = len(gprices)
          data['mode'] = statistics.mode(gprices)
          data['mean'] = statistics.mean(gprices)
          data['dev'] = statistics.stdev(gprices)
  
          return data

• Now we have the question of the storage of that data: basically 2 options here: either in a dedicated remote SQL database or some local storage DB or file:
• ⇒ The remote storage option is probably the best option: even if I then need some fast crypto system on an extern computer, “updating the gas price” is not something that would have to be done super quickly, and I could still update the storage mechanism at that time anyway.
• So let's prepare a database for that:

  SQL_CREATE_TOKENS_TABLE = """ CREATE TABLE IF NOT EXISTS tokens (
                                  id SERIAL PRIMARY KEY,
                                  address char(42) UNIQUE,
                                  name varchar(64),
                                  symbol varchar(32) UNIQUE,
                                  decimals SMALLINT,
                                  min_slippage INT,
                                  swap_fees INT,
                                  classification SMALLINT
                              ); """
  
  SQL_INSERT_TOKEN = ''' INSERT INTO tokens(address,name,symbol,decimals,min_slippage,swap_fees,classification)
                         VALUES(%s,%s,%s,%s,%s,%s,%s) '''
  
  SQL_UPDATE_TOKEN = ''' UPDATE tokens SET (name,symbol,decimals,min_slippage,swap_fees,classification)
                         = (%s,%s,%s,%s,%s,%s) WHERE address=%s; '''
  
  SQL_CREATE_GASPRICE_TABLE = """ CREATE TABLE IF NOT EXISTS gasprice (
                                          block_num integer PRIMARY KEY,
                                          timestamp integer,
                                          num_transactions integer,
                                          mode float,
                                          mean float,
                                          std_dev float
                                      ); """
  
  SQL_INSERT_GASPRICE = ''' INSERT INTO gasprice(block_num, timestamp, num_transactions, mode, mean, std_dev)
                            VALUES(%s,%s,%s,%s,%s,%s)'''
  
  
  class ChainDB(NVPObject):
      """ChainDB component class"""
  
      def __init__(self, db_name, user, password):
          """ChainDB constructor"""
          self.sql_db = PostgreSQLDB(db_name=db_name, user=user, password=password)
          self.sql_db.execute(SQL_CREATE_TOKENS_TABLE, commit=True)
          self.sql_db.execute(SQL_CREATE_GASPRICE_TABLE, commit=True)
  
      def execute(self, *args, **kaargs):
          """Forward execute code to the SQL DB"""
          return self.sql_db.execute(*args, **kaargs)
  
      def insert_gas_price_stats(self, data, check=True):
          """Insert a new gasprice row in the table"""
          # We should insert (block_num,mode,mean,std_dev)
          logger.info("Inserting gasprice data: %s", data)
          keys = ['block_number', 'timestamp', 'num_transactions', 'mode', 'mean', 'std_dev']
          row = [data[key] for key in keys]
          if check:
              # Check if the row already exists:
              bnum = data['block_number']
              cur = self.execute("select exists(select 1 from gasprice where block_num=%s)", (bnum,))
              if cur.fetchone()[0]:
                  logger.warning('Gas price entry already provided for block %d', bnum)
                  return
  
          self.execute(SQL_INSERT_GASPRICE, row, commit=True)

• And I now have a dedicate method in the EVMBlockchain to store new rows in the gasprice table:

      def collect_gas_price_statistics(self):
          """Collect the latest gas price statistics"""
          data = self.get_gas_price_statistics()
          del data['prices']
          self.get_db().insert_gas_price_stats(data)

• Tested that in jupyter and it works as expected:

  chain.collect_gas_price_statistics()

• Command above will output:

  2022/05/23 08:27:21 [nvh.crypto.blockchain.evm_blockchain] INFO: Block 18045926 contains 150 transactions
  2022/05/23 08:27:21 [nvh.crypto.blockchain.chain_db] INFO: Inserting gasprice data: {'block_number': 18045926, 'timestamp': 1653290838, 'num_transactions': 149, 'mode': 5.0, 'mean': 5.84451191375839, 'std_dev': 2.419240396353444}

• ⇒ And I can see that a new row is added to the table, all OK.

• Now I just need to add new data every minute, so let's add a command line entry to handle that:

  class BinanceSmartChain(EVMBlockchain):
      """BinanceSmartChain component class"""
  
      def __init__(self, ctx):
          """blockchain constructor"""
          cfg = ctx.get_config()['blockchains']['bsc']
          EVMBlockchain.__init__(self, ctx, cfg)
  
      def get_contract_abi_url(self, address):
          """Retrieve the ABI URL for a given contract"""
          api_key = self.config.get("bscscan_api_key", None)
          url = f"https://api.bscscan.com/api?module=contract&action=getabi&address={address}"
  
          if api_key is not None:
              url = f"{url}&apikey={api_key}"
  
          return url, 1.0/5 if api_key is not None else 5.0
  
      def process_command(self, cmd):
          """Check if this component can process the given command"""
  
          if cmd == 'collect-gas-price':
              self.collect_gas_price_statistics()
              return True
  
          return False
  
  
  if __name__ == "__main__":
      # Create the context:
      context = NVPContext()
  
      # Add our component:
      comp = context.register_component("bsc_chain", BinanceSmartChain(context))
  
      context.define_subparsers("main", {
          'collect-gas-price': None,
      })
  
      comp.run()
  

• And we add the corresponding script:

      "bsc": {
        "custom_python_env": "bsc_env",
        "cmd": "${PYTHON} ${PROJECT_ROOT_DIR}/nvh/crypto/bsc/binance_smart_chain.py",
        "python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"]
      },

• And finally the entry in our every_1min_operations.sh file:

  log_dir="/mnt/data1/admin/logs"
  
  lfile="${log_dir}/bsc_collect_gas_price.log"
  nvp bsc collect-gas-price 2>&1 | tee -a $lfile

• I then added the following function in the ChainDB class to get back the gasprice data:

      def get_gas_price_stats(self, max_num=None, min_ts=None):
          """Function returning the num_transactions, mode, mean, std_dev columns from the gas price table"""
          if max_num is not None:
              sql = f"SELECT * FROM gasprice ORDER BY block_num DESC LIMIT {max_num}"
              sql = f"SELECT * FROM ({sql}) as sub ORDER BY sub.block_num ASC;"
          elif min_ts is not None:
              if min_ts < 0:
                  # Assume relative to current timestamp and given in seconds:
                  min_ts = self.get_timestamp() + min_ts
                  # logger.info("using min_ts: %d", min_ts)
              sql = f"SELECT * FROM gasprice WHERE timestamp > {min_ts} ORDER BY timestamp ASC;"
          else:
              sql = "SELECT * FROM gasprice;"
  
          cur = self.execute(sql)
          results = cur.fetchall()
          # logger.debug("Retrieved %d gasprice rows", len(results))
          return np.array(results)

• And with that I should be able to compute a more appropriate gas price before I perform a transaction:

      def get_current_gas_price_emas(self, period=10):
          """Compute the EMAs for the gas price data"""
          # Retrieve the gas price statistics with a given smoothing EMA period:
          # Note: we should retrieve at least twice as much data as the period length:
          data = self.get_db().get_gas_price_stats(min_ts=-period*2*60)
  
          # Now compute the ema for each col:
          # Note: we only keep the EMAs for the last 4 columns
          res = [ta.trend.EMAIndicator(pd.Series(data[:, i+2]), window=period).ema_indicator().iloc[-1] for i in range(4)]
  
          return dict(zip(("num_transactions", "mode", "mean", "std_dev"), res))
  
      def compute_gas_price_estimate(self, mode="default"):
          """Compute an advanced gas price estimate."""
  
          # Retrieve the weights from the selected mode:
          weights = self.config["gas_price_weights"][mode]
  
          # Compute an estimated gas price, using weights for the mode,mean, mean+std_dev price values:
          # the weights should also contain the estimation period:
          self.check(len(weights) == 4, "Invalid number of weights")
  
          emas = self.get_current_gas_price_emas(period=weights[0])
          weights = weights[1:]
          totw = sum(weights)
          self.check(totw > 0.0, "Invalid total weights.")
  
          price = (emas['mode'] * weights[0] + emas['mean']
                   * weights[1] + (emas['mean']+emas['std_dev']) * weights[2])/totw
          price = math.floor(price*1000)/1000
  
          # Clamp if bigger than max gas price:
          maxp = self.max_gas_price
          if price > maxp:
              logger.warning("Clamping estimated gas price of %f to max gas price value of %f", price, maxp)
              price = maxp
          else:
              logger.info("Computed estimated gas price of %f gwei for mode %s", price, mode)
  
          return price

• And if I call the compute_gas_price_estimate() method above in jupyter I get the following kind of result (as expected):

  2022/05/23 10:26:27 [nvh.crypto.blockchain.evm_blockchain] INFO: Computed estimated gas price of 5.275000 gwei for mode default

• One last bonus step I would like to take now would be to display the current gas price as a chart in my cryptoview app: could be interesting to have a look at that chart sometimes 😉!
• So here is my first “BSCPanel” implementation:

  """BSCPanel widget module"""
  
  import logging
  from PyQt5 import QtCore
  from PyQt5.QtCore import Qt
  import PyQt5.QtWidgets as qwd
  import numpy as np
  
  from nvp.nvp_context import NVPContext
  from nvh.gui.app_base import AppBase
  from nvh.gui.plot_canvas import PlotCanvas
  from nvh.crypto.bsc.binance_smart_chain import BinanceSmartChain
  
  logger = logging.getLogger(__name__)
  
  
  class BSCPanel(qwd.QWidget):
      """BSCPanel class"""
  
      def __init__(self):
          """Constructor of the BSCPanel class"""
          super().__init__()
  
          # Create a CoinGecko() instance:
          ctx = NVPContext.get()
          self.app: AppBase = ctx.get_component("app")
          self.chain: BinanceSmartChain = ctx.get_component("bsc_chain")
  
          # chart display offset:
          self.offset = 0.04
          self.build_panel()
  
          setattr(self, "sizeHint", lambda: QtCore.QSize(360, 300))
  
          self.timer = QtCore.QTimer()
          self.timer.setInterval(1000*60)
          self.timer.timeout.connect(self.update_gas_price_chart)
          self.timer.start()
  
          self.update_gas_price_chart()
  
      def build_panel(self):
          """Build the panel for this widget"""
  
          lyt = qwd.QVBoxLayout()
          lyt.setContentsMargins(5, 5, 5, 5)
          lyt.setSpacing(5)
  
          self.chart = PlotCanvas(self, width=2, height=2, dpi=100)
          self.toolbar = self.chart.create_toolbar(self)
          self.ax2 = self.chart.axes.twinx()
  
          lyt.addWidget(self.toolbar)
          lyt.addWidget(self.chart)
  
          self.setLayout(lyt)
  
      def update_gas_price_chart(self):
          """Update the gas price chart"""
  
          logger.info("Updating gas price chart.")
  
          # Collect the available entries:
          # Get at max 7 days of data:
          data = self.chain.get_db().get_gas_price_stats(min_ts=-7*24*60*60)
  
          self.chain.check(data.shape[1] == 6, "Invalid number of gas price columns: %d", data.shape[1])
  
          # we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values:
          now_ts = self.chain.get_timestamp()
          times = now_ts - data[:, 1]
  
          # convert times to hours:
          times /= 3600
          mean = data[:, 4]
          mode = data[:, 3]
          txn = data[:, 2]
          # mean_up = mean + data[:, 5]
  
          self.chart.clear()
          self.chart.set_title("BSC Gas Price")
  
          self.chart.axes.plot(times, mode, color='orange', linewidth=1, label="Mode")
          self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean")
          self.ax2.cla()
          self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx")
  
          # self.chart.axes.plot(times, mean_up, color='red', linewidth=1, label="Mean + 1sig")
          self.chart.axes.set_xlabel("Time offset (hours)")
          self.chart.axes.set_ylabel("Gas price (Gwei)")
          self.ax2.set_ylabel("Number of tx")
  
          # cf. https://www.adamsmith.haus/python/answers/how-to-add-secondary-axis-to-a-legend-in-python
          lines_1, labels_1 = self.chart.axes.get_legend_handles_labels()
          lines_2, labels_2 = self.ax2.get_legend_handles_labels()
          lines = lines_1 + lines_2
          labels = labels_1 + labels_2
  
          self.chart.axes.legend(lines, labels, loc=0)
          xmin = np.amin(times)
          xmax = np.amax(times)
          offset = self.offset * (xmax - xmin)
  
          self.chart.axes.set_xlim([xmax+offset, xmin-offset])
          self.chart.draw()
  

• And here is the kind of display this will produce:

• ⇒ yeah… It's a bit messy when we disply raw data directly: let's add a EMA filtering on those values instead 😉
• So I added a utility function compute_ema() in my nvh.crypto.utils module:

  def compute_ema(arr, period):
      """Compute the EMA of a numpy array"""
      return ta.trend.EMAIndicator(pd.Series(arr), window=period).ema_indicator().values
  

• And updated the gas price plot drawing to this:

      def update_gas_price_chart(self):
          """Update the gas price chart"""
  
          logger.info("Updating gas price chart.")
  
          # Collect the available entries:
          # Get at max 7 days of data:
          period = 20
          data = self.chain.get_db().get_gas_price_stats(max_num=7*24*60+period)
  
          self.chain.check(data.shape[1] == 6, "Invalid number of gas price columns: %d", data.shape[1])
  
          # we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values:
          now_ts = self.chain.get_timestamp()
          times = now_ts - data[:, 1]
  
          # convert times to hours:
          times /= 3600
          mean = utl.compute_ema(data[:, 4], period)
          # logger.info("Mean values: %s", mean)
          mode = data[:, 3]
          # txn = data[:, 2]
          txn = utl.compute_ema(data[:, 2], period)
          # dev = utl.compute_ema(data[:, 5], period)
  
          # mean_up = mean + dev
  
          self.chart.clear()
          self.chart.set_title("BSC Gas Price")
  
          self.chart.axes.plot(times, mode, color='orange', linewidth=1, label="Mode")
          self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean")
          # self.chart.axes.plot(times, mean_up, linewidth=1, label="Mean + 1sig")
          self.ax2.cla()
          self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx")
          # self.ax2.plot(times, txn2, color='black', linewidth=1, label="Num. tx EMA")
  
          # self.chart.axes.plot(times, mean_up, color='red', linewidth=1, label="Mean + 1sig")
          self.chart.axes.set_xlabel("Time offset (hours)")
          self.chart.axes.set_ylabel("Gas price (Gwei)")
          self.ax2.set_ylabel("Number of tx")
  
          # cf. https://www.adamsmith.haus/python/answers/how-to-add-secondary-axis-to-a-legend-in-python
          lines_1, labels_1 = self.chart.axes.get_legend_handles_labels()
          lines_2, labels_2 = self.ax2.get_legend_handles_labels()
          lines = lines_1 + lines_2
          labels = labels_1 + labels_2
  
          self.chart.axes.legend(lines, labels, loc=0)
          xmin = np.amin(times)
          xmax = np.amax(times)
          offset = self.offset * (xmax - xmin)
  
          self.chart.axes.set_xlim([xmax+offset, xmin-offset])
          self.chart.draw()

• Which gives me the result:

• ⇒ There isn't much data to display here for the moment, so this still feels a bit noisy, but I think it should get better when I have few days of history data to display instead, so let's just keep it this way fro the moment ✌.

• Actually, in all the steps above we only used some basic JSONRPC block/transaction retrieval: so this should in theory work the same for other EVM compatible blockchains ?
• ⇒ Let's try to implement this for Ethereum:
• Here is the simple EthereumChain class I create as a starting point:

  """ETH blockchain."""
  
  import logging
  
  from nvp.nvp_context import NVPContext
  from nvh.crypto.blockchain.evm_blockchain import EVMBlockchain
  
  logger = logging.getLogger(__name__)
  
  
  def create_component(ctx: NVPContext):
      """Create an instance of the component"""
      return EthereumChain(ctx)
  
  
  class EthereumChain(EVMBlockchain):
      """EthereumChain component class"""
  
      def __init__(self, ctx):
          """blockchain constructor"""
          cfg = ctx.get_config()['blockchains']['eth']
          EVMBlockchain.__init__(self, ctx, cfg)
  
      def get_contract_abi_url(self, address):
          """Retrieve the ABI URL for a given contract"""
          api_key = self.config.get("etherscan_api_key", None)
          url = f"https://api.etherscan.io/api?module=contract&action=getabi&address={address}"
  
          if api_key is not None:
              url = f"{url}&apikey={api_key}"
  
          return url, 1.0/5 if api_key is not None else 5.0
  
      def process_command(self, cmd):
          """Check if this component can process the given command"""
  
          if cmd == 'collect-gas-price':
              self.collect_gas_price_statistics()
              return True
  
          return False
  
  
  if __name__ == "__main__":
      # Create the context:
      context = NVPContext()
  
      # Add our component:
      comp = context.register_component("eth_chain", EthereumChain(context))
  
      context.define_subparsers("main", {
          'collect-gas-price': None,
      })
  
      comp.run()
  

• Then we add a simple script to use it:

      "eth": {
        "custom_python_env": "bsc_env",
        "cmd": "${PYTHON} ${PROJECT_ROOT_DIR}/nvh/crypto/eth/ethereum_chain.py",
        "python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"]
      },

• And this works pretty damn well already:

  $ nvp eth collect-gas-price
  2022/05/23 22:29:06 [nvh.crypto.blockchain.evm_blockchain] INFO: Block 14831913: num_tx=21, mode=13.580, mean=16.079, dev=4.162

• ⇒ So also adding that in my 1min operation script:

  lfile="${log_dir}/eth_collect_gas_price.log"
  nvp eth collect-gas-price 2>&1 | tee -a $lfile

• Remember the system I built in my previous article to receive automatic notifications in case there is an exception in one of the scripts I run with cron ? Well, I got a few of those notifications over night about the ethereum gas price retrieval 😅:

• Let's see what this is about… Ohhh okay, that's interesting: I found at least the following type of error in the logs:
• Error 1:

  Traceback (most recent call last):
    File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 55, in <module>
      comp.run()
    File "/mnt/data1/dev/projects/NervProj/nvp/nvp_component.py", line 69, in run
      res = self.process_command(cmd)
    File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 38, in process_command
      self.collect_gas_price_statistics()
    File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 261, in collect_gas_price_statistics
      data = self.get_gas_price_statistics()
    File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 253, in get_gas_price_statistics
      data['mode'] = statistics.mode(gprices)
    File "/mnt/data1/dev/projects/NervProj/.pyenvs/bsc_env/lib/python3.10/statistics.py", line 583, in mode
      raise StatisticsError('no mode for empty data') from None
  statistics.StatisticsError: no mode for empty data

• Error 2:

  Traceback (most recent call last):
    File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 55, in <module>
      comp.run()
    File "/mnt/data1/dev/projects/NervProj/nvp/nvp_component.py", line 69, in run
      res = self.process_command(cmd)
    File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 38, in process_command
      self.collect_gas_price_statistics()
    File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 261, in collect_gas_price_statistics
      data = self.get_gas_price_statistics()
    File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 255, in get_gas_price_statistics
      data['std_dev'] = statistics.stdev(gprices)
    File "/mnt/data1/dev/projects/NervProj/.pyenvs/bsc_env/lib/python3.10/statistics.py", line 828, in stdev
      var = variance(data, xbar)
    File "/mnt/data1/dev/projects/NervProj/.pyenvs/bsc_env/lib/python3.10/statistics.py", line 767, in variance
      raise StatisticsError('variance requires at least two data points')
  statistics.StatisticsError: variance requires at least two data points

• So typically, I think these may happen if we have respectively 0 or 1 transaction only in a given block which is of course totally legit, so let's fix that.
• OK here is the updated code which should take care of those problems:

      def get_gas_price_statistics(self, bnum=None):
          """"Method used to retrieve the gas price statistics from a given block"""
          data = self.get_gas_prices(bnum)
          gprices = data['prices']
          num = len(gprices)
          data['num_transactions'] = num
          data['mode'] = statistics.mode(gprices) if num > 0 else math.nan
          data['mean'] = statistics.mean(gprices) if num > 0 else math.nan
          data['std_dev'] = statistics.stdev(gprices) if num > 1 else math.nan
  
          return data
  
      def collect_gas_price_statistics(self):
          """Collect the latest gas price statistics"""
          data = self.get_gas_price_statistics()
          del data['prices']
          # Note: we only store the block price data if there are at least 2 transactions:
          if data['num_transactions'] < 2:
              logger.info("Ignoring block %d: not enough transactions.", data['block_number'])
              return
  
          logger.info("Block %d: num_tx=%d, mode=%.3f, mean=%.3f, dev=%.3f",
                      data['block_number'], data['num_transactions'], data['mode'],
                      data['mean'], data['std_dev'])
          self.get_db().insert_gas_price_stats(data)

• Side note: I also get some warning message on the ethereum blockchain, because checking every minute, there is sometimes no new block on the chain, so I get this kind of message (but this is not critical and already handled properly internally, so I'm leaving this as is):

  [nvh.crypto.blockchain.chain_db] WARNING: Gas price entry already provided for block 14834104

• While I'm at it, I think I should also try to use a dedicated channel to report those exceptions/errors on rochetchat: otherwise this is quickly filling my “general admin” channel which I would prefer to keep clean instead. So let's change that.
  • Created the problem-reports channel: OK
  • Updated the runner code to notify in that channel:

            try:
                self.execute(cmd, cwd=cwd, env=env)
            except subprocess.SubprocessError:
                notify = self.config.get("notify_script_errors", True)
                if notify:
                    # And exception occured in the sub process, so we should send a notification:
                    msg = ":warning: **WARNING:** an exception occured in the following command:\n"
                    msg += f"{cmd}\n"
                    msg += f"cwd={cwd}\n\n"
                    msg += "=> Check the logs for details."
    
                    rchat = self.get_component("rchat")
                    rchat.send_message(msg, channel="problem-reports")

  • And also extended the rochetchat command line to support testing the channel:

    if __name__ == "__main__":
        # Create the context:
        context = NVPContext()
    
        # Add our component:
        comp = context.register_component("rchat", RocketChat(context))
    
        context.define_subparsers("main", {
            'send': None,
        })
    
        psr = context.get_parser('main.send')
        psr.add_argument("message", type=str,
                         help="Simple message that we should send")
        psr.add_argument("-c", "--channel", dest="channel", type=str,
                         help="Channel where to send the message.")
    
        comp.run()

• And this simple command below works so we are good for now:

  $ nvp rchat "hello manu" -c problem-reports

• In the implementation above I introduced 2 new databases on my postgresql server: bsc_chain for the BSC and eth_chain for Ethereum (to store tha gas prices only for the moment, but this will be extended soon)
• ⇒ So let's not forget to setup the corresponding backups for these!
• Which is super easy to add in my custom NervProj config file:

        // Backup of individual SQL databases:
        "postgresql_crypto_databases": {
          "type": "postgresql",
          "container": "postgresql_server",
          "user": "crypto_user",
          "databases": [
            "crypto_coins",
            "crypto_prices_v2",
            "bsc_chain",
            "eth_chain"
          ],
          "backup_dirs": [
            "${slot1}/sql/crypto",
            "${slot2}/sql/crypto",
            "${slot3}/sql/crypto"
          ]
        },

  /

• And finally, let's also add a dedicated tab in CryptoView to report Ethereum specific informations (only the gas price for now, but I expect to be able to extend that soon )
• ⇒ For this I simply convert my previous BSCPanel into a more generic EVMChainPanel taking the chain component name as input:

  """EVMChainPanel widget module"""
  
  import logging
  from PyQt5 import QtCore
  from PyQt5.QtCore import Qt
  import PyQt5.QtWidgets as qwd
  import numpy as np
  
  
  from nvp.nvp_context import NVPContext
  
  import nvh.crypto.utils as utl
  from nvh.gui.app_base import AppBase
  from nvh.gui.plot_canvas import PlotCanvas
  from nvh.crypto.blockchain.evm_blockchain import EVMBlockchain
  
  logger = logging.getLogger(__name__)
  
  
  class EVMChainPanel(qwd.QWidget):
      """EVMChainPanel class"""
  
      def __init__(self, comp_name):
          """Constructor of the EVMChainPanel class
          using the comp_name to retrieve the desired chain component"""
          super().__init__()
  
          # Create a CoinGecko() instance:
          ctx = NVPContext.get()
          self.app: AppBase = ctx.get_component("app")
          self.chain: EVMBlockchain = ctx.get_component(comp_name)
  
          # chart display offset:
          self.offset = 0.04
          self.build_panel()
  
          setattr(self, "sizeHint", lambda: QtCore.QSize(360, 300))
  
          self.timer = QtCore.QTimer()
          self.timer.setInterval(1000*60)
          self.timer.timeout.connect(self.update_gas_price_chart)
          self.timer.start()
  
          self.update_gas_price_chart()
  
      def build_panel(self):
          """Build the panel for this widget"""
  
          lyt = qwd.QVBoxLayout()
          lyt.setContentsMargins(5, 5, 5, 5)
          lyt.setSpacing(5)
  
          self.chart = PlotCanvas(self, width=2, height=2, dpi=100)
          self.toolbar = self.chart.create_toolbar(self)
          self.ax2 = self.chart.axes.twinx()
  
          lyt.addWidget(self.toolbar)
          lyt.addWidget(self.chart)
  
          self.setLayout(lyt)
  
      def update_gas_price_chart(self):
          """Update the gas price chart"""
  
          logger.info("Updating gas price chart.")
  
          # Collect the available entries:
          # Get at max 7 days of data:
          period = 20
          data = self.chain.get_db().get_gas_price_stats(max_num=7*24*60+period)
  
          self.chain.check(data.shape[1] == 6, "Invalid number of gas price columns: %d", data.shape[1])
  
          # we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values:
          now_ts = self.chain.get_timestamp()
          times = now_ts - data[:, 1]
  
          # convert times to hours:
          times /= 3600
          mean = utl.compute_ema(data[:, 4], period)
          # logger.info("Mean values: %s", mean)
          mode = data[:, 3]
          # txn = data[:, 2]
          txn = utl.compute_ema(data[:, 2], period)
          # dev = utl.compute_ema(data[:, 5], period)
  
          # mean_up = mean + dev
  
          self.chart.clear()
          sname = self.chain.get_short_name().upper()
          self.chart.set_title(f"{sname} Gas Price")
  
          self.chart.axes.plot(times, mode, color='orange', linewidth=1, label="Mode price")
          self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean price")
          # self.chart.axes.plot(times, mean_up, linewidth=1, label="Mean + 1sig")
          self.ax2.cla()
          self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx")
          # self.ax2.plot(times, txn2, color='black', linewidth=1, label="Num. tx EMA")
  
          # self.chart.axes.plot(times, mean_up, color='red', linewidth=1, label="Mean + 1sig")
          self.chart.axes.set_xlabel("Time offset (hours)")
          self.chart.axes.set_ylabel("Gas price (Gwei)")
          self.ax2.set_ylabel("Number of tx")
  
          # cf. https://www.adamsmith.haus/python/answers/how-to-add-secondary-axis-to-a-legend-in-python
          lines_1, labels_1 = self.chart.axes.get_legend_handles_labels()
          lines_2, labels_2 = self.ax2.get_legend_handles_labels()
          lines = lines_1 + lines_2
          labels = labels_1 + labels_2
  
          self.chart.axes.legend(lines, labels, loc=0)
          xmin = np.amin(times)
          xmax = np.amax(times)
          offset = self.offset * (xmax - xmin)
  
          self.chart.axes.set_xlim([xmax+offset, xmin-offset])
          self.chart.draw()
  

• Then I create a couple of those panels on my main window:

          dock3 = DockWidget("BSC")
          panel = EVMChainPanel("bsc_chain")
          self.widgets["bsc_panel"] = panel
          dock3.setWidget(panel)
          self.widgets["bsc_panel_dock"] = dock3
  
          dock4 = DockWidget("Ethereum")
          panel = EVMChainPanel("eth_chain")
          self.widgets["eth_panel"] = panel
          dock4.setWidget(panel)
          self.widgets["eth_panel_dock"] = dock4
  
          dock1.setAllowedAreas(Qt.AllDockWidgetAreas)
          dock2.setAllowedAreas(Qt.AllDockWidgetAreas)
          dock3.setAllowedAreas(Qt.AllDockWidgetAreas)
          dock4.setAllowedAreas(Qt.AllDockWidgetAreas)
  
          win.addDockWidget(Qt.RightDockWidgetArea, dock1)
          win.addDockWidget(Qt.RightDockWidgetArea, dock2)
          win.addDockWidget(Qt.RightDockWidgetArea, dock3)
          win.addDockWidget(Qt.RightDockWidgetArea, dock4)
          win.tabifyDockWidget(dock1, dock2)
          win.tabifyDockWidget(dock1, dock3)
          win.tabifyDockWidget(dock1, dock4)

• And this does the trick 👍!:

• Except that… it's a pretty dirty graph here lol ⇒ I think this is mainly because the mode price value is changing significantly on Ethereum (as opposed to on the BSC): So I should maybe applied some EMA filtering to that one also:

      def update_gas_price_chart(self):
          """Update the gas price chart"""
  
          logger.info("Updating gas price chart.")
  
          # Collect the available entries:
          # Get at max 7 days of data:
          period = 20
          data = self.chain.get_db().get_gas_price_stats(max_num=7*24*60+period)
  
          self.chain.check(data.shape[1] == 6, "Invalid number of gas price columns: %d", data.shape[1])
  
          # we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values:
          now_ts = self.chain.get_timestamp()
          times = now_ts - data[:, 1]
  
          # convert times to hours:
          times /= 3600
          mean = utl.compute_ema(data[:, 4], period)
          # logger.info("Mean values: %s", mean)
          mode = utl.compute_ema(data[:, 3], period)
          # txn = data[:, 2]
          txn = utl.compute_ema(data[:, 2], period)
          # dev = utl.compute_ema(data[:, 5], period)
  
          # mean_up = mean + dev
  
          self.chart.clear()
          sname = self.chain.get_short_name().upper()
          self.chart.set_title(f"{sname} Gas Price")
  
          self.chart.axes.plot(times, mode, color='orange', linewidth=2, label="Mode price")
          self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean price")
          # self.chart.axes.plot(times, mean_up, linewidth=1, label="Mean + 1sig")
          self.ax2.cla()
          self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx")

• Note: I also bumped the mode price linewitdh to 2 pixels to try to make it more visible:

• That's a bit better already, but I feel I should maybe not put the plots of the transactions just mixed with the prices: this is getting confusing, so let's try to separate them:

          self.chart.axes.legend(lines, labels, loc=2)
          xmin = np.amin(times)
          xmax = np.amax(times)
          offset = self.offset * (xmax - xmin)
          self.chart.axes.set_xlim([xmax+offset, xmin-offset])
  
          # update the vertical ranges:
          txn_min = np.nanmin(txn)
          txn_max = np.nanmax(txn)
          txn_range = txn_max - txn_min
          price_min = min(np.nanmin(mean), np.nanmin(mode))
          price_max = max(np.nanmax(mean), np.nanmax(mode))
          price_range = price_max - price_min
  
          self.chart.axes.set_ylim([price_min-self.offset*price_range,
                                    price_max+price_range*(1.0+3.0*self.offset)])
          self.ax2.set_ylim([txn_min-txn_range*(1.0+3.0*self.offset),
                             txn_max+txn_range*self.offset])
  
          self.chart.draw()

• And with that change the display feels much better already (from my perspective at least):

• And here the updated display for the BSC gas prices/transactions:

• Okay, so, with the experiments conducted here we see that it is fairly easy to manually collect gas prices for a given EVM blockchain, and from that, to generate your own “gas price oracle”.
• ⇒ Eventually I could extend that to more EVM blockchains that I'm interested in, but for the moment, let's say this will be good enough and stop this long article here ! See ya!