blog:2022:0524_crypto_gas_price_stats

# Crypto: Computing EVM blockchain gas price statistics

Continuing from my prevvous article on the Linear Finance reward claims, I think I need to push it a bit further on the gas price I use for my transactions: using an hardcoded value of 7Gwei is certainly not ideal, instead I should build a dedicated system to continuously update the statistics on the gas price, so let's implement that system here.

• Actually I already have the required function to compute the gas price from statistics. But I feel I should also prepare a jupyter env to be able to test those jsonrpc functions manually, so I'm setting this up first:
    "bscjup": {
"custom_python_env": "bsc_env",
"cmd": "${PYTHON} -m jupyter lab", "cwd": "${PROJECT_ROOT_DIR}",
"python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"]
},
• I then tested a minimal script in jupyter:
from nvp.nvp_context import NVPContext

if NVPContext.instance is None:
NVPContext()

ctx = NVPContext.get()
chain = ctx.get_component('bsc_chain')

bck = chain.get_block(full_tx=True)
• And indeed, in the resulting block above ('bck') we can retrieve the list of transactions from bck['transactions'], and in the the gas price specified for that transaction
• So I just added the following methods in the EVMBlockchain class:
    def get_gas_prices(self, bnum=None):
"""Collect the gas prices in a given block"""
# Retrieve the block details with all the transactions:
bck = self.get_block(block_num=bnum, full_tx=True)
# logger.info("block content: %s", self.pretty_print(bck))
# logger.info("block content: %s", bck)

# We retrieve the list of transactions from the block:
transactions = bck['transactions']
tnum = len(transactions)
bnum = bck['number']
logger.info("Block %d contains %d transactions", bnum, tnum)

# We collect all the gas prices from the transactions:
gprices = []
for i in range(tnum):
# We should convert the hex string to a number:
txn = transactions[i]
price = txn['gasPrice']/1e9
if price > 0.0:
gprices.append(price)

return {
'prices': gprices,
'block_number': bnum,
'timestamp': bck['timestamp']}

# Function to collect the mode, mean, stddev statistics from a given list of gas prices
# from a given block number. If not specified then return the last block details.
def get_gas_price_statistics(self, bnum=None):
""""Method used to retrieve the gas price statistics from a given block"""
data = self.get_gas_prices(bnum)
gprices = data['prices']
data['num_prices'] = len(gprices)
data['mode'] = statistics.mode(gprices)
data['mean'] = statistics.mean(gprices)
data['dev'] = statistics.stdev(gprices)

return data
• Now we have the question of the storage of that data: basically 2 options here: either in a dedicated remote SQL database or some local storage DB or file:
• ⇒ The remote storage option is probably the best option: even if I then need some fast crypto system on an extern computer, “updating the gas price” is not something that would have to be done super quickly, and I could still update the storage mechanism at that time anyway.
• So let's prepare a database for that:
SQL_CREATE_TOKENS_TABLE = """ CREATE TABLE IF NOT EXISTS tokens (
id SERIAL PRIMARY KEY,
name varchar(64),
symbol varchar(32) UNIQUE,
decimals SMALLINT,
min_slippage INT,
swap_fees INT,
classification SMALLINT
); """

SQL_INSERT_TOKEN = ''' INSERT INTO tokens(address,name,symbol,decimals,min_slippage,swap_fees,classification)
VALUES(%s,%s,%s,%s,%s,%s,%s) '''

SQL_UPDATE_TOKEN = ''' UPDATE tokens SET (name,symbol,decimals,min_slippage,swap_fees,classification)

SQL_CREATE_GASPRICE_TABLE = """ CREATE TABLE IF NOT EXISTS gasprice (
block_num integer PRIMARY KEY,
timestamp integer,
num_transactions integer,
mode float,
mean float,
std_dev float
); """

SQL_INSERT_GASPRICE = ''' INSERT INTO gasprice(block_num, timestamp, num_transactions, mode, mean, std_dev)
VALUES(%s,%s,%s,%s,%s,%s)'''

class ChainDB(NVPObject):
"""ChainDB component class"""

"""ChainDB constructor"""
self.sql_db.execute(SQL_CREATE_TOKENS_TABLE, commit=True)
self.sql_db.execute(SQL_CREATE_GASPRICE_TABLE, commit=True)

def execute(self, *args, **kaargs):
"""Forward execute code to the SQL DB"""
return self.sql_db.execute(*args, **kaargs)

def insert_gas_price_stats(self, data, check=True):
"""Insert a new gasprice row in the table"""
# We should insert (block_num,mode,mean,std_dev)
logger.info("Inserting gasprice data: %s", data)
keys = ['block_number', 'timestamp', 'num_transactions', 'mode', 'mean', 'std_dev']
row = [data[key] for key in keys]
if check:
# Check if the row already exists:
bnum = data['block_number']
cur = self.execute("select exists(select 1 from gasprice where block_num=%s)", (bnum,))
if cur.fetchone():
logger.warning('Gas price entry already provided for block %d', bnum)
return

self.execute(SQL_INSERT_GASPRICE, row, commit=True)
• And I now have a dedicate method in the EVMBlockchain to store new rows in the gasprice table:
    def collect_gas_price_statistics(self):
"""Collect the latest gas price statistics"""
data = self.get_gas_price_statistics()
del data['prices']
self.get_db().insert_gas_price_stats(data)
• Tested that in jupyter and it works as expected:
chain.collect_gas_price_statistics()
• Command above will output:
2022/05/23 08:27:21 [nvh.crypto.blockchain.evm_blockchain] INFO: Block 18045926 contains 150 transactions
2022/05/23 08:27:21 [nvh.crypto.blockchain.chain_db] INFO: Inserting gasprice data: {'block_number': 18045926, 'timestamp': 1653290838, 'num_transactions': 149, 'mode': 5.0, 'mean': 5.84451191375839, 'std_dev': 2.419240396353444}
• ⇒ And I can see that a new row is added to the table, all OK.
• Now I just need to add new data every minute, so let's add a command line entry to handle that:
class BinanceSmartChain(EVMBlockchain):
"""BinanceSmartChain component class"""

def __init__(self, ctx):
"""blockchain constructor"""
cfg = ctx.get_config()['blockchains']['bsc']
EVMBlockchain.__init__(self, ctx, cfg)

"""Retrieve the ABI URL for a given contract"""
api_key = self.config.get("bscscan_api_key", None)

if api_key is not None:
url = f"{url}&apikey={api_key}"

return url, 1.0/5 if api_key is not None else 5.0

def process_command(self, cmd):
"""Check if this component can process the given command"""

if cmd == 'collect-gas-price':
self.collect_gas_price_statistics()
return True

return False

if __name__ == "__main__":
# Create the context:
context = NVPContext()

comp = context.register_component("bsc_chain", BinanceSmartChain(context))

context.define_subparsers("main", {
'collect-gas-price': None,
})

comp.run()

• And we add the corresponding script:
    "bsc": {
"custom_python_env": "bsc_env",
"cmd": "${PYTHON}${PROJECT_ROOT_DIR}/nvh/crypto/bsc/binance_smart_chain.py",
"python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"]
},
• And finally the entry in our every_1min_operations.sh file:
log_dir="/mnt/data1/admin/logs"

lfile="${log_dir}/bsc_collect_gas_price.log" nvp bsc collect-gas-price 2>&1 | tee -a$lfile
• I then added the following function in the ChainDB class to get back the gasprice data:
    def get_gas_price_stats(self, max_num=None, min_ts=None):
"""Function returning the num_transactions, mode, mean, std_dev columns from the gas price table"""
if max_num is not None:
sql = f"SELECT * FROM gasprice ORDER BY block_num DESC LIMIT {max_num}"
sql = f"SELECT * FROM ({sql}) as sub ORDER BY sub.block_num ASC;"
elif min_ts is not None:
if min_ts < 0:
# Assume relative to current timestamp and given in seconds:
min_ts = self.get_timestamp() + min_ts
# logger.info("using min_ts: %d", min_ts)
sql = f"SELECT * FROM gasprice WHERE timestamp > {min_ts} ORDER BY timestamp ASC;"
else:
sql = "SELECT * FROM gasprice;"

cur = self.execute(sql)
results = cur.fetchall()
# logger.debug("Retrieved %d gasprice rows", len(results))
return np.array(results)
• And with that I should be able to compute a more appropriate gas price before I perform a transaction:
    def get_current_gas_price_emas(self, period=10):
"""Compute the EMAs for the gas price data"""
# Retrieve the gas price statistics with a given smoothing EMA period:
# Note: we should retrieve at least twice as much data as the period length:
data = self.get_db().get_gas_price_stats(min_ts=-period*2*60)

# Now compute the ema for each col:
# Note: we only keep the EMAs for the last 4 columns
res = [ta.trend.EMAIndicator(pd.Series(data[:, i+2]), window=period).ema_indicator().iloc[-1] for i in range(4)]

return dict(zip(("num_transactions", "mode", "mean", "std_dev"), res))

def compute_gas_price_estimate(self, mode="default"):
"""Compute an advanced gas price estimate."""

# Retrieve the weights from the selected mode:
weights = self.config["gas_price_weights"][mode]

# Compute an estimated gas price, using weights for the mode,mean, mean+std_dev price values:
# the weights should also contain the estimation period:
self.check(len(weights) == 4, "Invalid number of weights")

emas = self.get_current_gas_price_emas(period=weights)
weights = weights[1:]
totw = sum(weights)
self.check(totw > 0.0, "Invalid total weights.")

price = (emas['mode'] * weights + emas['mean']
* weights + (emas['mean']+emas['std_dev']) * weights)/totw
price = math.floor(price*1000)/1000

# Clamp if bigger than max gas price:
maxp = self.max_gas_price
if price > maxp:
logger.warning("Clamping estimated gas price of %f to max gas price value of %f", price, maxp)
price = maxp
else:
logger.info("Computed estimated gas price of %f gwei for mode %s", price, mode)

return price
I just discovered the ta package above: it's using pandas, but add support for many technical indicators without any fancy dependency build requirement like TA-lib so I will go with that one for now!
• And if I call the compute_gas_price_estimate() method above in jupyter I get the following kind of result (as expected):
2022/05/23 10:26:27 [nvh.crypto.blockchain.evm_blockchain] INFO: Computed estimated gas price of 5.275000 gwei for mode default
• One last bonus step I would like to take now would be to display the current gas price as a chart in my cryptoview app: could be interesting to have a look at that chart sometimes 😉!
• So here is my first “BSCPanel” implementation:
"""BSCPanel widget module"""

import logging
from PyQt5 import QtCore
from PyQt5.QtCore import Qt
import PyQt5.QtWidgets as qwd
import numpy as np

from nvp.nvp_context import NVPContext
from nvh.gui.app_base import AppBase
from nvh.gui.plot_canvas import PlotCanvas
from nvh.crypto.bsc.binance_smart_chain import BinanceSmartChain

logger = logging.getLogger(__name__)

class BSCPanel(qwd.QWidget):
"""BSCPanel class"""

def __init__(self):
"""Constructor of the BSCPanel class"""
super().__init__()

# Create a CoinGecko() instance:
ctx = NVPContext.get()
self.app: AppBase = ctx.get_component("app")
self.chain: BinanceSmartChain = ctx.get_component("bsc_chain")

# chart display offset:
self.offset = 0.04
self.build_panel()

setattr(self, "sizeHint", lambda: QtCore.QSize(360, 300))

self.timer = QtCore.QTimer()
self.timer.setInterval(1000*60)
self.timer.timeout.connect(self.update_gas_price_chart)
self.timer.start()

self.update_gas_price_chart()

def build_panel(self):
"""Build the panel for this widget"""

lyt = qwd.QVBoxLayout()
lyt.setContentsMargins(5, 5, 5, 5)
lyt.setSpacing(5)

self.chart = PlotCanvas(self, width=2, height=2, dpi=100)
self.toolbar = self.chart.create_toolbar(self)
self.ax2 = self.chart.axes.twinx()

self.setLayout(lyt)

def update_gas_price_chart(self):
"""Update the gas price chart"""

logger.info("Updating gas price chart.")

# Collect the available entries:
# Get at max 7 days of data:
data = self.chain.get_db().get_gas_price_stats(min_ts=-7*24*60*60)

self.chain.check(data.shape == 6, "Invalid number of gas price columns: %d", data.shape)

# we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values:
now_ts = self.chain.get_timestamp()
times = now_ts - data[:, 1]

# convert times to hours:
times /= 3600
mean = data[:, 4]
mode = data[:, 3]
txn = data[:, 2]
# mean_up = mean + data[:, 5]

self.chart.clear()
self.chart.set_title("BSC Gas Price")

self.chart.axes.plot(times, mode, color='orange', linewidth=1, label="Mode")
self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean")
self.ax2.cla()
self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx")

# self.chart.axes.plot(times, mean_up, color='red', linewidth=1, label="Mean + 1sig")
self.chart.axes.set_xlabel("Time offset (hours)")
self.chart.axes.set_ylabel("Gas price (Gwei)")
self.ax2.set_ylabel("Number of tx")

lines_1, labels_1 = self.chart.axes.get_legend_handles_labels()
lines_2, labels_2 = self.ax2.get_legend_handles_labels()
lines = lines_1 + lines_2
labels = labels_1 + labels_2

self.chart.axes.legend(lines, labels, loc=0)
xmin = np.amin(times)
xmax = np.amax(times)
offset = self.offset * (xmax - xmin)

self.chart.axes.set_xlim([xmax+offset, xmin-offset])
self.chart.draw()

• And here is the kind of display this will produce:
• ⇒ yeah… It's a bit messy when we disply raw data directly: let's add a EMA filtering on those values instead 😉
• So I added a utility function compute_ema() in my nvh.crypto.utils module:
def compute_ema(arr, period):
"""Compute the EMA of a numpy array"""
return ta.trend.EMAIndicator(pd.Series(arr), window=period).ema_indicator().values

• And updated the gas price plot drawing to this:
    def update_gas_price_chart(self):
"""Update the gas price chart"""

logger.info("Updating gas price chart.")

# Collect the available entries:
# Get at max 7 days of data:
period = 20
data = self.chain.get_db().get_gas_price_stats(max_num=7*24*60+period)

self.chain.check(data.shape == 6, "Invalid number of gas price columns: %d", data.shape)

# we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values:
now_ts = self.chain.get_timestamp()
times = now_ts - data[:, 1]

# convert times to hours:
times /= 3600
mean = utl.compute_ema(data[:, 4], period)
# logger.info("Mean values: %s", mean)
mode = data[:, 3]
# txn = data[:, 2]
txn = utl.compute_ema(data[:, 2], period)
# dev = utl.compute_ema(data[:, 5], period)

# mean_up = mean + dev

self.chart.clear()
self.chart.set_title("BSC Gas Price")

self.chart.axes.plot(times, mode, color='orange', linewidth=1, label="Mode")
self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean")
# self.chart.axes.plot(times, mean_up, linewidth=1, label="Mean + 1sig")
self.ax2.cla()
self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx")
# self.ax2.plot(times, txn2, color='black', linewidth=1, label="Num. tx EMA")

# self.chart.axes.plot(times, mean_up, color='red', linewidth=1, label="Mean + 1sig")
self.chart.axes.set_xlabel("Time offset (hours)")
self.chart.axes.set_ylabel("Gas price (Gwei)")
self.ax2.set_ylabel("Number of tx")

lines_1, labels_1 = self.chart.axes.get_legend_handles_labels()
lines_2, labels_2 = self.ax2.get_legend_handles_labels()
lines = lines_1 + lines_2
labels = labels_1 + labels_2

self.chart.axes.legend(lines, labels, loc=0)
xmin = np.amin(times)
xmax = np.amax(times)
offset = self.offset * (xmax - xmin)

self.chart.axes.set_xlim([xmax+offset, xmin-offset])
self.chart.draw()
• Which gives me the result:
• ⇒ There isn't much data to display here for the moment, so this still feels a bit noisy, but I think it should get better when I have few days of history data to display instead, so let's just keep it this way fro the moment ✌.
• Actually, in all the steps above we only used some basic JSONRPC block/transaction retrieval: so this should in theory work the same for other EVM compatible blockchains ?
• ⇒ Let's try to implement this for Ethereum:
• Here is the simple EthereumChain class I create as a starting point:
"""ETH blockchain."""

import logging

from nvp.nvp_context import NVPContext
from nvh.crypto.blockchain.evm_blockchain import EVMBlockchain

logger = logging.getLogger(__name__)

def create_component(ctx: NVPContext):
"""Create an instance of the component"""
return EthereumChain(ctx)

class EthereumChain(EVMBlockchain):
"""EthereumChain component class"""

def __init__(self, ctx):
"""blockchain constructor"""
cfg = ctx.get_config()['blockchains']['eth']
EVMBlockchain.__init__(self, ctx, cfg)

"""Retrieve the ABI URL for a given contract"""
api_key = self.config.get("etherscan_api_key", None)

if api_key is not None:
url = f"{url}&apikey={api_key}"

return url, 1.0/5 if api_key is not None else 5.0

def process_command(self, cmd):
"""Check if this component can process the given command"""

if cmd == 'collect-gas-price':
self.collect_gas_price_statistics()
return True

return False

if __name__ == "__main__":
# Create the context:
context = NVPContext()

comp = context.register_component("eth_chain", EthereumChain(context))

context.define_subparsers("main", {
'collect-gas-price': None,
})

comp.run()

• Then we add a simple script to use it:
    "eth": {
"custom_python_env": "bsc_env",
"cmd": "${PYTHON}${PROJECT_ROOT_DIR}/nvh/crypto/eth/ethereum_chain.py",
"python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"]
},
• And this works pretty damn well already:
$nvp eth collect-gas-price 2022/05/23 22:29:06 [nvh.crypto.blockchain.evm_blockchain] INFO: Block 14831913: num_tx=21, mode=13.580, mean=16.079, dev=4.162 • ⇒ So also adding that in my 1min operation script: lfile="${log_dir}/eth_collect_gas_price.log"
nvp eth collect-gas-price 2>&1 | tee -a $lfile • Remember the system I built in my previous article to receive automatic notifications in case there is an exception in one of the scripts I run with cron ? Well, I got a few of those notifications over night about the ethereum gas price retrieval 😅: • Let's see what this is about… Ohhh okay, that's interesting: I found at least the following type of error in the logs: • Error 1: Traceback (most recent call last): File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 55, in <module> comp.run() File "/mnt/data1/dev/projects/NervProj/nvp/nvp_component.py", line 69, in run res = self.process_command(cmd) File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 38, in process_command self.collect_gas_price_statistics() File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 261, in collect_gas_price_statistics data = self.get_gas_price_statistics() File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 253, in get_gas_price_statistics data['mode'] = statistics.mode(gprices) File "/mnt/data1/dev/projects/NervProj/.pyenvs/bsc_env/lib/python3.10/statistics.py", line 583, in mode raise StatisticsError('no mode for empty data') from None statistics.StatisticsError: no mode for empty data • Error 2: Traceback (most recent call last): File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 55, in <module> comp.run() File "/mnt/data1/dev/projects/NervProj/nvp/nvp_component.py", line 69, in run res = self.process_command(cmd) File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 38, in process_command self.collect_gas_price_statistics() File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 261, in collect_gas_price_statistics data = self.get_gas_price_statistics() File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 255, in get_gas_price_statistics data['std_dev'] = statistics.stdev(gprices) File "/mnt/data1/dev/projects/NervProj/.pyenvs/bsc_env/lib/python3.10/statistics.py", line 828, in stdev var = variance(data, xbar) File "/mnt/data1/dev/projects/NervProj/.pyenvs/bsc_env/lib/python3.10/statistics.py", line 767, in variance raise StatisticsError('variance requires at least two data points') statistics.StatisticsError: variance requires at least two data points • So typically, I think these may happen if we have respectively 0 or 1 transaction only in a given block which is of course totally legit, so let's fix that. • OK here is the updated code which should take care of those problems:  def get_gas_price_statistics(self, bnum=None): """"Method used to retrieve the gas price statistics from a given block""" data = self.get_gas_prices(bnum) gprices = data['prices'] num = len(gprices) data['num_transactions'] = num data['mode'] = statistics.mode(gprices) if num > 0 else math.nan data['mean'] = statistics.mean(gprices) if num > 0 else math.nan data['std_dev'] = statistics.stdev(gprices) if num > 1 else math.nan return data def collect_gas_price_statistics(self): """Collect the latest gas price statistics""" data = self.get_gas_price_statistics() del data['prices'] # Note: we only store the block price data if there are at least 2 transactions: if data['num_transactions'] < 2: logger.info("Ignoring block %d: not enough transactions.", data['block_number']) return logger.info("Block %d: num_tx=%d, mode=%.3f, mean=%.3f, dev=%.3f", data['block_number'], data['num_transactions'], data['mode'], data['mean'], data['std_dev']) self.get_db().insert_gas_price_stats(data) • Side note: I also get some warning message on the ethereum blockchain, because checking every minute, there is sometimes no new block on the chain, so I get this kind of message (but this is not critical and already handled properly internally, so I'm leaving this as is): [nvh.crypto.blockchain.chain_db] WARNING: Gas price entry already provided for block 14834104 • While I'm at it, I think I should also try to use a dedicated channel to report those exceptions/errors on rochetchat: otherwise this is quickly filling my “general admin” channel which I would prefer to keep clean instead. So let's change that. • Created the problem-reports channel: OK • Updated the runner code to notify in that channel:  try: self.execute(cmd, cwd=cwd, env=env) except subprocess.SubprocessError: notify = self.config.get("notify_script_errors", True) if notify: # And exception occured in the sub process, so we should send a notification: msg = ":warning: **WARNING:** an exception occured in the following command:\n" msg += f"{cmd}\n" msg += f"cwd={cwd}\n\n" msg += "=> Check the logs for details." rchat = self.get_component("rchat") rchat.send_message(msg, channel="problem-reports") • And also extended the rochetchat command line to support testing the channel: if __name__ == "__main__": # Create the context: context = NVPContext() # Add our component: comp = context.register_component("rchat", RocketChat(context)) context.define_subparsers("main", { 'send': None, }) psr = context.get_parser('main.send') psr.add_argument("message", type=str, help="Simple message that we should send") psr.add_argument("-c", "--channel", dest="channel", type=str, help="Channel where to send the message.") comp.run() • And this simple command below works so we are good for now: $ nvp rchat "hello manu" -c problem-reports
• In the implementation above I introduced 2 new databases on my postgresql server: bsc_chain for the BSC and eth_chain for Ethereum (to store tha gas prices only for the moment, but this will be extended soon)
• ⇒ So let's not forget to setup the corresponding backups for these!
• Which is super easy to add in my custom NervProj config file:
      // Backup of individual SQL databases:
"postgresql_crypto_databases": {
"type": "postgresql",
"container": "postgresql_server",
"user": "crypto_user",
"databases": [
"crypto_coins",
"crypto_prices_v2",
"bsc_chain",
"eth_chain"
],
"backup_dirs": [
"${slot1}/sql/crypto", "${slot2}/sql/crypto",
"\${slot3}/sql/crypto"
]
},
/
• And finally, let's also add a dedicated tab in CryptoView to report Ethereum specific informations (only the gas price for now, but I expect to be able to extend that soon )
• ⇒ For this I simply convert my previous BSCPanel into a more generic EVMChainPanel taking the chain component name as input:
"""EVMChainPanel widget module"""

import logging
from PyQt5 import QtCore
from PyQt5.QtCore import Qt
import PyQt5.QtWidgets as qwd
import numpy as np

from nvp.nvp_context import NVPContext

import nvh.crypto.utils as utl
from nvh.gui.app_base import AppBase
from nvh.gui.plot_canvas import PlotCanvas
from nvh.crypto.blockchain.evm_blockchain import EVMBlockchain

logger = logging.getLogger(__name__)

class EVMChainPanel(qwd.QWidget):
"""EVMChainPanel class"""

def __init__(self, comp_name):
"""Constructor of the EVMChainPanel class
using the comp_name to retrieve the desired chain component"""
super().__init__()

# Create a CoinGecko() instance:
ctx = NVPContext.get()
self.app: AppBase = ctx.get_component("app")
self.chain: EVMBlockchain = ctx.get_component(comp_name)

# chart display offset:
self.offset = 0.04
self.build_panel()

setattr(self, "sizeHint", lambda: QtCore.QSize(360, 300))

self.timer = QtCore.QTimer()
self.timer.setInterval(1000*60)
self.timer.timeout.connect(self.update_gas_price_chart)
self.timer.start()

self.update_gas_price_chart()

def build_panel(self):
"""Build the panel for this widget"""

lyt = qwd.QVBoxLayout()
lyt.setContentsMargins(5, 5, 5, 5)
lyt.setSpacing(5)

self.chart = PlotCanvas(self, width=2, height=2, dpi=100)
self.toolbar = self.chart.create_toolbar(self)
self.ax2 = self.chart.axes.twinx()

self.setLayout(lyt)

def update_gas_price_chart(self):
"""Update the gas price chart"""

logger.info("Updating gas price chart.")

# Collect the available entries:
# Get at max 7 days of data:
period = 20
data = self.chain.get_db().get_gas_price_stats(max_num=7*24*60+period)

self.chain.check(data.shape == 6, "Invalid number of gas price columns: %d", data.shape)

# we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values:
now_ts = self.chain.get_timestamp()
times = now_ts - data[:, 1]

# convert times to hours:
times /= 3600
mean = utl.compute_ema(data[:, 4], period)
# logger.info("Mean values: %s", mean)
mode = data[:, 3]
# txn = data[:, 2]
txn = utl.compute_ema(data[:, 2], period)
# dev = utl.compute_ema(data[:, 5], period)

# mean_up = mean + dev

self.chart.clear()
sname = self.chain.get_short_name().upper()
self.chart.set_title(f"{sname} Gas Price")

self.chart.axes.plot(times, mode, color='orange', linewidth=1, label="Mode price")
self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean price")
# self.chart.axes.plot(times, mean_up, linewidth=1, label="Mean + 1sig")
self.ax2.cla()
self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx")
# self.ax2.plot(times, txn2, color='black', linewidth=1, label="Num. tx EMA")

# self.chart.axes.plot(times, mean_up, color='red', linewidth=1, label="Mean + 1sig")
self.chart.axes.set_xlabel("Time offset (hours)")
self.chart.axes.set_ylabel("Gas price (Gwei)")
self.ax2.set_ylabel("Number of tx")

lines_1, labels_1 = self.chart.axes.get_legend_handles_labels()
lines_2, labels_2 = self.ax2.get_legend_handles_labels()
lines = lines_1 + lines_2
labels = labels_1 + labels_2

self.chart.axes.legend(lines, labels, loc=0)
xmin = np.amin(times)
xmax = np.amax(times)
offset = self.offset * (xmax - xmin)

self.chart.axes.set_xlim([xmax+offset, xmin-offset])
self.chart.draw()

• Then I create a couple of those panels on my main window:
        dock3 = DockWidget("BSC")
panel = EVMChainPanel("bsc_chain")
self.widgets["bsc_panel"] = panel
dock3.setWidget(panel)
self.widgets["bsc_panel_dock"] = dock3

dock4 = DockWidget("Ethereum")
panel = EVMChainPanel("eth_chain")
self.widgets["eth_panel"] = panel
dock4.setWidget(panel)
self.widgets["eth_panel_dock"] = dock4

dock1.setAllowedAreas(Qt.AllDockWidgetAreas)
dock2.setAllowedAreas(Qt.AllDockWidgetAreas)
dock3.setAllowedAreas(Qt.AllDockWidgetAreas)
dock4.setAllowedAreas(Qt.AllDockWidgetAreas)

win.tabifyDockWidget(dock1, dock2)
win.tabifyDockWidget(dock1, dock3)
win.tabifyDockWidget(dock1, dock4)
• And this does the trick 👍!:
• Except that… it's a pretty dirty graph here lol ⇒ I think this is mainly because the mode price value is changing significantly on Ethereum (as opposed to on the BSC): So I should maybe applied some EMA filtering to that one also:
    def update_gas_price_chart(self):
"""Update the gas price chart"""

logger.info("Updating gas price chart.")

# Collect the available entries:
# Get at max 7 days of data:
period = 20
data = self.chain.get_db().get_gas_price_stats(max_num=7*24*60+period)

self.chain.check(data.shape == 6, "Invalid number of gas price columns: %d", data.shape)

# we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values:
now_ts = self.chain.get_timestamp()
times = now_ts - data[:, 1]

# convert times to hours:
times /= 3600
mean = utl.compute_ema(data[:, 4], period)
# logger.info("Mean values: %s", mean)
mode = utl.compute_ema(data[:, 3], period)
# txn = data[:, 2]
txn = utl.compute_ema(data[:, 2], period)
# dev = utl.compute_ema(data[:, 5], period)

# mean_up = mean + dev

self.chart.clear()
sname = self.chain.get_short_name().upper()
self.chart.set_title(f"{sname} Gas Price")

self.chart.axes.plot(times, mode, color='orange', linewidth=2, label="Mode price")
self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean price")
# self.chart.axes.plot(times, mean_up, linewidth=1, label="Mean + 1sig")
self.ax2.cla()
self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx")
• Note: I also bumped the mode price linewitdh to 2 pixels to try to make it more visible:
• That's a bit better already, but I feel I should maybe not put the plots of the transactions just mixed with the prices: this is getting confusing, so let's try to separate them:
        self.chart.axes.legend(lines, labels, loc=2)
xmin = np.amin(times)
xmax = np.amax(times)
offset = self.offset * (xmax - xmin)
self.chart.axes.set_xlim([xmax+offset, xmin-offset])

# update the vertical ranges:
txn_min = np.nanmin(txn)
txn_max = np.nanmax(txn)
txn_range = txn_max - txn_min
price_min = min(np.nanmin(mean), np.nanmin(mode))
price_max = max(np.nanmax(mean), np.nanmax(mode))
price_range = price_max - price_min

self.chart.axes.set_ylim([price_min-self.offset*price_range,
price_max+price_range*(1.0+3.0*self.offset)])
self.ax2.set_ylim([txn_min-txn_range*(1.0+3.0*self.offset),
txn_max+txn_range*self.offset])

self.chart.draw()
• And with that change the display feels much better already (from my perspective at least):
• And here the updated display for the BSC gas prices/transactions:
• Okay, so, with the experiments conducted here we see that it is fairly easy to manually collect gas prices for a given EVM blockchain, and from that, to generate your own “gas price oracle”.
• ⇒ Eventually I could extend that to more EVM blockchains that I'm interested in, but for the moment, let's say this will be good enough and stop this long article here ! See ya!
• blog/2022/0524_crypto_gas_price_stats.txt