====== Crypto: Computing EVM blockchain gas price statistics ======
{{tag>dev python crypto nervhome finance bsc blockchain}}
Continuing from my prevvous article on the [[blog:2022:0522_crypto_linear_finance_claims|Linear Finance reward claims]], I think I need to push it a bit further on the gas price I use for my transactions: using an hardcoded value of 7Gwei is certainly not ideal, instead I should build a dedicated system to continuously update the statistics on the gas price, so let's implement that system here.
====== ======
===== Initial experiments =====
* Actually I already have the required function to compute the gas price from statistics. But I feel I should also prepare a jupyter env to be able to test those jsonrpc functions manually, so I'm setting this up first: "bscjup": {
"custom_python_env": "bsc_env",
"cmd": "${PYTHON} -m jupyter lab",
"cwd": "${PROJECT_ROOT_DIR}",
"python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"]
},
* I then tested a minimal script in jupyter: from nvp.nvp_context import NVPContext
if NVPContext.instance is None:
NVPContext()
ctx = NVPContext.get()
chain = ctx.get_component('bsc_chain')
bck = chain.get_block(full_tx=True)
* And indeed, in the resulting block above ('bck') we can retrieve the list of transactions from bck['transactions'], and in the the gas price specified for that transaction
* So I just added the following methods in the EVMBlockchain class: def get_gas_prices(self, bnum=None):
"""Collect the gas prices in a given block"""
# Retrieve the block details with all the transactions:
bck = self.get_block(block_num=bnum, full_tx=True)
# logger.info("block content: %s", self.pretty_print(bck))
# logger.info("block content: %s", bck)
# We retrieve the list of transactions from the block:
transactions = bck['transactions']
tnum = len(transactions)
bnum = bck['number']
logger.info("Block %d contains %d transactions", bnum, tnum)
# We collect all the gas prices from the transactions:
gprices = []
for i in range(tnum):
# We should convert the hex string to a number:
txn = transactions[i]
price = txn['gasPrice']/1e9
if price > 0.0:
gprices.append(price)
return {
'prices': gprices,
'block_number': bnum,
'timestamp': bck['timestamp']}
# Function to collect the mode, mean, stddev statistics from a given list of gas prices
# from a given block number. If not specified then return the last block details.
def get_gas_price_statistics(self, bnum=None):
""""Method used to retrieve the gas price statistics from a given block"""
data = self.get_gas_prices(bnum)
gprices = data['prices']
data['num_prices'] = len(gprices)
data['mode'] = statistics.mode(gprices)
data['mean'] = statistics.mean(gprices)
data['dev'] = statistics.stdev(gprices)
return data
* Now we have the question of the storage of that data: basically 2 options here: either in a dedicated remote SQL database or some local storage DB or file:
* => The remote storage option is probably the best option: even if I then need some fast crypto system on an extern computer, "updating the gas price" is not something that would have to be done super quickly, and I could still update the storage mechanism at that time anyway.
* So let's prepare a database for that: SQL_CREATE_TOKENS_TABLE = """ CREATE TABLE IF NOT EXISTS tokens (
id SERIAL PRIMARY KEY,
address char(42) UNIQUE,
name varchar(64),
symbol varchar(32) UNIQUE,
decimals SMALLINT,
min_slippage INT,
swap_fees INT,
classification SMALLINT
); """
SQL_INSERT_TOKEN = ''' INSERT INTO tokens(address,name,symbol,decimals,min_slippage,swap_fees,classification)
VALUES(%s,%s,%s,%s,%s,%s,%s) '''
SQL_UPDATE_TOKEN = ''' UPDATE tokens SET (name,symbol,decimals,min_slippage,swap_fees,classification)
= (%s,%s,%s,%s,%s,%s) WHERE address=%s; '''
SQL_CREATE_GASPRICE_TABLE = """ CREATE TABLE IF NOT EXISTS gasprice (
block_num integer PRIMARY KEY,
timestamp integer,
num_transactions integer,
mode float,
mean float,
std_dev float
); """
SQL_INSERT_GASPRICE = ''' INSERT INTO gasprice(block_num, timestamp, num_transactions, mode, mean, std_dev)
VALUES(%s,%s,%s,%s,%s,%s)'''
class ChainDB(NVPObject):
"""ChainDB component class"""
def __init__(self, db_name, user, password):
"""ChainDB constructor"""
self.sql_db = PostgreSQLDB(db_name=db_name, user=user, password=password)
self.sql_db.execute(SQL_CREATE_TOKENS_TABLE, commit=True)
self.sql_db.execute(SQL_CREATE_GASPRICE_TABLE, commit=True)
def execute(self, *args, **kaargs):
"""Forward execute code to the SQL DB"""
return self.sql_db.execute(*args, **kaargs)
def insert_gas_price_stats(self, data, check=True):
"""Insert a new gasprice row in the table"""
# We should insert (block_num,mode,mean,std_dev)
logger.info("Inserting gasprice data: %s", data)
keys = ['block_number', 'timestamp', 'num_transactions', 'mode', 'mean', 'std_dev']
row = [data[key] for key in keys]
if check:
# Check if the row already exists:
bnum = data['block_number']
cur = self.execute("select exists(select 1 from gasprice where block_num=%s)", (bnum,))
if cur.fetchone()[0]:
logger.warning('Gas price entry already provided for block %d', bnum)
return
self.execute(SQL_INSERT_GASPRICE, row, commit=True)
* And I now have a dedicate method in the EVMBlockchain to store new rows in the gasprice table: def collect_gas_price_statistics(self):
"""Collect the latest gas price statistics"""
data = self.get_gas_price_statistics()
del data['prices']
self.get_db().insert_gas_price_stats(data)
* Tested that in jupyter and it works as expected: chain.collect_gas_price_statistics()
* Command above will output: 2022/05/23 08:27:21 [nvh.crypto.blockchain.evm_blockchain] INFO: Block 18045926 contains 150 transactions
2022/05/23 08:27:21 [nvh.crypto.blockchain.chain_db] INFO: Inserting gasprice data: {'block_number': 18045926, 'timestamp': 1653290838, 'num_transactions': 149, 'mode': 5.0, 'mean': 5.84451191375839, 'std_dev': 2.419240396353444}
* => And I can see that a new row is added to the table, all **OK**.
* Now I just need to add new data every minute, so let's add a command line entry to handle that: class BinanceSmartChain(EVMBlockchain):
"""BinanceSmartChain component class"""
def __init__(self, ctx):
"""blockchain constructor"""
cfg = ctx.get_config()['blockchains']['bsc']
EVMBlockchain.__init__(self, ctx, cfg)
def get_contract_abi_url(self, address):
"""Retrieve the ABI URL for a given contract"""
api_key = self.config.get("bscscan_api_key", None)
url = f"https://api.bscscan.com/api?module=contract&action=getabi&address={address}"
if api_key is not None:
url = f"{url}&apikey={api_key}"
return url, 1.0/5 if api_key is not None else 5.0
def process_command(self, cmd):
"""Check if this component can process the given command"""
if cmd == 'collect-gas-price':
self.collect_gas_price_statistics()
return True
return False
if __name__ == "__main__":
# Create the context:
context = NVPContext()
# Add our component:
comp = context.register_component("bsc_chain", BinanceSmartChain(context))
context.define_subparsers("main", {
'collect-gas-price': None,
})
comp.run()
* And we add the corresponding script: "bsc": {
"custom_python_env": "bsc_env",
"cmd": "${PYTHON} ${PROJECT_ROOT_DIR}/nvh/crypto/bsc/binance_smart_chain.py",
"python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"]
},
* And finally the entry in our **every_1min_operations.sh** file: log_dir="/mnt/data1/admin/logs"
lfile="${log_dir}/bsc_collect_gas_price.log"
nvp bsc collect-gas-price 2>&1 | tee -a $lfile
===== Retrieving the latest statistic values =====
* I then added the following function in the **ChainDB** class to get back the gasprice data: def get_gas_price_stats(self, max_num=None, min_ts=None):
"""Function returning the num_transactions, mode, mean, std_dev columns from the gas price table"""
if max_num is not None:
sql = f"SELECT * FROM gasprice ORDER BY block_num DESC LIMIT {max_num}"
sql = f"SELECT * FROM ({sql}) as sub ORDER BY sub.block_num ASC;"
elif min_ts is not None:
if min_ts < 0:
# Assume relative to current timestamp and given in seconds:
min_ts = self.get_timestamp() + min_ts
# logger.info("using min_ts: %d", min_ts)
sql = f"SELECT * FROM gasprice WHERE timestamp > {min_ts} ORDER BY timestamp ASC;"
else:
sql = "SELECT * FROM gasprice;"
cur = self.execute(sql)
results = cur.fetchall()
# logger.debug("Retrieved %d gasprice rows", len(results))
return np.array(results)
* And with that I should be able to compute a more appropriate gas price before I perform a transaction: def get_current_gas_price_emas(self, period=10):
"""Compute the EMAs for the gas price data"""
# Retrieve the gas price statistics with a given smoothing EMA period:
# Note: we should retrieve at least twice as much data as the period length:
data = self.get_db().get_gas_price_stats(min_ts=-period*2*60)
# Now compute the ema for each col:
# Note: we only keep the EMAs for the last 4 columns
res = [ta.trend.EMAIndicator(pd.Series(data[:, i+2]), window=period).ema_indicator().iloc[-1] for i in range(4)]
return dict(zip(("num_transactions", "mode", "mean", "std_dev"), res))
def compute_gas_price_estimate(self, mode="default"):
"""Compute an advanced gas price estimate."""
# Retrieve the weights from the selected mode:
weights = self.config["gas_price_weights"][mode]
# Compute an estimated gas price, using weights for the mode,mean, mean+std_dev price values:
# the weights should also contain the estimation period:
self.check(len(weights) == 4, "Invalid number of weights")
emas = self.get_current_gas_price_emas(period=weights[0])
weights = weights[1:]
totw = sum(weights)
self.check(totw > 0.0, "Invalid total weights.")
price = (emas['mode'] * weights[0] + emas['mean']
* weights[1] + (emas['mean']+emas['std_dev']) * weights[2])/totw
price = math.floor(price*1000)/1000
# Clamp if bigger than max gas price:
maxp = self.max_gas_price
if price > maxp:
logger.warning("Clamping estimated gas price of %f to max gas price value of %f", price, maxp)
price = maxp
else:
logger.info("Computed estimated gas price of %f gwei for mode %s", price, mode)
return price
I just discovered the **ta** package above: it's using pandas, but add support for many technical indicators without any fancy dependency build requirement like **TA-lib** so I will go with that one for now!
* And if I call the ''compute_gas_price_estimate()'' method above in jupyter I get the following kind of result (as expected): 2022/05/23 10:26:27 [nvh.crypto.blockchain.evm_blockchain] INFO: Computed estimated gas price of 5.275000 gwei for mode default
===== Displaying the gas price chart in cryptoview =====
* One last bonus step I would like to take now would be to display the current gas price as a chart in my cryptoview app: could be interesting to have a look at that chart sometimes 😉!
* So here is my first "BSCPanel" implementation: """BSCPanel widget module"""
import logging
from PyQt5 import QtCore
from PyQt5.QtCore import Qt
import PyQt5.QtWidgets as qwd
import numpy as np
from nvp.nvp_context import NVPContext
from nvh.gui.app_base import AppBase
from nvh.gui.plot_canvas import PlotCanvas
from nvh.crypto.bsc.binance_smart_chain import BinanceSmartChain
logger = logging.getLogger(__name__)
class BSCPanel(qwd.QWidget):
"""BSCPanel class"""
def __init__(self):
"""Constructor of the BSCPanel class"""
super().__init__()
# Create a CoinGecko() instance:
ctx = NVPContext.get()
self.app: AppBase = ctx.get_component("app")
self.chain: BinanceSmartChain = ctx.get_component("bsc_chain")
# chart display offset:
self.offset = 0.04
self.build_panel()
setattr(self, "sizeHint", lambda: QtCore.QSize(360, 300))
self.timer = QtCore.QTimer()
self.timer.setInterval(1000*60)
self.timer.timeout.connect(self.update_gas_price_chart)
self.timer.start()
self.update_gas_price_chart()
def build_panel(self):
"""Build the panel for this widget"""
lyt = qwd.QVBoxLayout()
lyt.setContentsMargins(5, 5, 5, 5)
lyt.setSpacing(5)
self.chart = PlotCanvas(self, width=2, height=2, dpi=100)
self.toolbar = self.chart.create_toolbar(self)
self.ax2 = self.chart.axes.twinx()
lyt.addWidget(self.toolbar)
lyt.addWidget(self.chart)
self.setLayout(lyt)
def update_gas_price_chart(self):
"""Update the gas price chart"""
logger.info("Updating gas price chart.")
# Collect the available entries:
# Get at max 7 days of data:
data = self.chain.get_db().get_gas_price_stats(min_ts=-7*24*60*60)
self.chain.check(data.shape[1] == 6, "Invalid number of gas price columns: %d", data.shape[1])
# we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values:
now_ts = self.chain.get_timestamp()
times = now_ts - data[:, 1]
# convert times to hours:
times /= 3600
mean = data[:, 4]
mode = data[:, 3]
txn = data[:, 2]
# mean_up = mean + data[:, 5]
self.chart.clear()
self.chart.set_title("BSC Gas Price")
self.chart.axes.plot(times, mode, color='orange', linewidth=1, label="Mode")
self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean")
self.ax2.cla()
self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx")
# self.chart.axes.plot(times, mean_up, color='red', linewidth=1, label="Mean + 1sig")
self.chart.axes.set_xlabel("Time offset (hours)")
self.chart.axes.set_ylabel("Gas price (Gwei)")
self.ax2.set_ylabel("Number of tx")
# cf. https://www.adamsmith.haus/python/answers/how-to-add-secondary-axis-to-a-legend-in-python
lines_1, labels_1 = self.chart.axes.get_legend_handles_labels()
lines_2, labels_2 = self.ax2.get_legend_handles_labels()
lines = lines_1 + lines_2
labels = labels_1 + labels_2
self.chart.axes.legend(lines, labels, loc=0)
xmin = np.amin(times)
xmax = np.amax(times)
offset = self.offset * (xmax - xmin)
self.chart.axes.set_xlim([xmax+offset, xmin-offset])
self.chart.draw()
* And here is the kind of display this will produce:
{{ blog:2022:0523:bsc_gas_price_v1.png }}
* => yeah... It's a bit messy when we disply raw data directly: let's add a EMA filtering on those values instead 😉
* So I added a **utility function** ''compute_ema()'' in my **nvh.crypto.utils** module: def compute_ema(arr, period):
"""Compute the EMA of a numpy array"""
return ta.trend.EMAIndicator(pd.Series(arr), window=period).ema_indicator().values
* And updated the gas price plot drawing to this: def update_gas_price_chart(self):
"""Update the gas price chart"""
logger.info("Updating gas price chart.")
# Collect the available entries:
# Get at max 7 days of data:
period = 20
data = self.chain.get_db().get_gas_price_stats(max_num=7*24*60+period)
self.chain.check(data.shape[1] == 6, "Invalid number of gas price columns: %d", data.shape[1])
# we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values:
now_ts = self.chain.get_timestamp()
times = now_ts - data[:, 1]
# convert times to hours:
times /= 3600
mean = utl.compute_ema(data[:, 4], period)
# logger.info("Mean values: %s", mean)
mode = data[:, 3]
# txn = data[:, 2]
txn = utl.compute_ema(data[:, 2], period)
# dev = utl.compute_ema(data[:, 5], period)
# mean_up = mean + dev
self.chart.clear()
self.chart.set_title("BSC Gas Price")
self.chart.axes.plot(times, mode, color='orange', linewidth=1, label="Mode")
self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean")
# self.chart.axes.plot(times, mean_up, linewidth=1, label="Mean + 1sig")
self.ax2.cla()
self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx")
# self.ax2.plot(times, txn2, color='black', linewidth=1, label="Num. tx EMA")
# self.chart.axes.plot(times, mean_up, color='red', linewidth=1, label="Mean + 1sig")
self.chart.axes.set_xlabel("Time offset (hours)")
self.chart.axes.set_ylabel("Gas price (Gwei)")
self.ax2.set_ylabel("Number of tx")
# cf. https://www.adamsmith.haus/python/answers/how-to-add-secondary-axis-to-a-legend-in-python
lines_1, labels_1 = self.chart.axes.get_legend_handles_labels()
lines_2, labels_2 = self.ax2.get_legend_handles_labels()
lines = lines_1 + lines_2
labels = labels_1 + labels_2
self.chart.axes.legend(lines, labels, loc=0)
xmin = np.amin(times)
xmax = np.amax(times)
offset = self.offset * (xmax - xmin)
self.chart.axes.set_xlim([xmax+offset, xmin-offset])
self.chart.draw()
* Which gives me the result:
{{ blog:2022:0523:bsc_gas_price_v2.png }}
* => There isn't much data to display here for the moment, so this still feels a bit noisy, but I think it should get better when I have few days of history data to display instead, so let's just keep it this way fro the moment ✌.
===== Testing with other EVM blockchains =====
* Actually, in all the steps above we only used some basic JSONRPC block/transaction retrieval: so this should in theory work the same for other EVM compatible blockchains ?
* => Let's try to implement this for Ethereum:
* Here is the simple **EthereumChain** class I create as a starting point: """ETH blockchain."""
import logging
from nvp.nvp_context import NVPContext
from nvh.crypto.blockchain.evm_blockchain import EVMBlockchain
logger = logging.getLogger(__name__)
def create_component(ctx: NVPContext):
"""Create an instance of the component"""
return EthereumChain(ctx)
class EthereumChain(EVMBlockchain):
"""EthereumChain component class"""
def __init__(self, ctx):
"""blockchain constructor"""
cfg = ctx.get_config()['blockchains']['eth']
EVMBlockchain.__init__(self, ctx, cfg)
def get_contract_abi_url(self, address):
"""Retrieve the ABI URL for a given contract"""
api_key = self.config.get("etherscan_api_key", None)
url = f"https://api.etherscan.io/api?module=contract&action=getabi&address={address}"
if api_key is not None:
url = f"{url}&apikey={api_key}"
return url, 1.0/5 if api_key is not None else 5.0
def process_command(self, cmd):
"""Check if this component can process the given command"""
if cmd == 'collect-gas-price':
self.collect_gas_price_statistics()
return True
return False
if __name__ == "__main__":
# Create the context:
context = NVPContext()
# Add our component:
comp = context.register_component("eth_chain", EthereumChain(context))
context.define_subparsers("main", {
'collect-gas-price': None,
})
comp.run()
* Then we add a simple script to use it: "eth": {
"custom_python_env": "bsc_env",
"cmd": "${PYTHON} ${PROJECT_ROOT_DIR}/nvh/crypto/eth/ethereum_chain.py",
"python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"]
},
* And this works pretty damn well already: $ nvp eth collect-gas-price
2022/05/23 22:29:06 [nvh.crypto.blockchain.evm_blockchain] INFO: Block 14831913: num_tx=21, mode=13.580, mean=16.079, dev=4.162
* => So also adding that in my 1min operation script: lfile="${log_dir}/eth_collect_gas_price.log"
nvp eth collect-gas-price 2>&1 | tee -a $lfile
===== Investigating gas price collect error on Ethereum =====
* Remember the system I built in my previous article to [[https://wiki.nervtech.org/doku.php?id=blog:2022:0522_crypto_linear_finance_claims#before_we_start|receive automatic notifications]] in case there is an exception in one of the scripts I run with cron ? Well, I got a few of those notifications over night about the ethereum gas price retrieval 😅:
{{ blog:2022:0523:eth_errors.png }}
* Let's see what this is about... Ohhh okay, that's interesting: I found at least the following type of error in the logs:
* **Error 1**: Traceback (most recent call last):
File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 55, in
comp.run()
File "/mnt/data1/dev/projects/NervProj/nvp/nvp_component.py", line 69, in run
res = self.process_command(cmd)
File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 38, in process_command
self.collect_gas_price_statistics()
File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 261, in collect_gas_price_statistics
data = self.get_gas_price_statistics()
File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 253, in get_gas_price_statistics
data['mode'] = statistics.mode(gprices)
File "/mnt/data1/dev/projects/NervProj/.pyenvs/bsc_env/lib/python3.10/statistics.py", line 583, in mode
raise StatisticsError('no mode for empty data') from None
statistics.StatisticsError: no mode for empty data
* **Error 2**: Traceback (most recent call last):
File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 55, in
comp.run()
File "/mnt/data1/dev/projects/NervProj/nvp/nvp_component.py", line 69, in run
res = self.process_command(cmd)
File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 38, in process_command
self.collect_gas_price_statistics()
File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 261, in collect_gas_price_statistics
data = self.get_gas_price_statistics()
File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 255, in get_gas_price_statistics
data['std_dev'] = statistics.stdev(gprices)
File "/mnt/data1/dev/projects/NervProj/.pyenvs/bsc_env/lib/python3.10/statistics.py", line 828, in stdev
var = variance(data, xbar)
File "/mnt/data1/dev/projects/NervProj/.pyenvs/bsc_env/lib/python3.10/statistics.py", line 767, in variance
raise StatisticsError('variance requires at least two data points')
statistics.StatisticsError: variance requires at least two data points
* So typically, I think these may happen if we have respectively 0 or 1 transaction only in a given block which is of course totally legit, so let's fix that.
* **OK** here is the updated code which should take care of those problems: def get_gas_price_statistics(self, bnum=None):
""""Method used to retrieve the gas price statistics from a given block"""
data = self.get_gas_prices(bnum)
gprices = data['prices']
num = len(gprices)
data['num_transactions'] = num
data['mode'] = statistics.mode(gprices) if num > 0 else math.nan
data['mean'] = statistics.mean(gprices) if num > 0 else math.nan
data['std_dev'] = statistics.stdev(gprices) if num > 1 else math.nan
return data
def collect_gas_price_statistics(self):
"""Collect the latest gas price statistics"""
data = self.get_gas_price_statistics()
del data['prices']
# Note: we only store the block price data if there are at least 2 transactions:
if data['num_transactions'] < 2:
logger.info("Ignoring block %d: not enough transactions.", data['block_number'])
return
logger.info("Block %d: num_tx=%d, mode=%.3f, mean=%.3f, dev=%.3f",
data['block_number'], data['num_transactions'], data['mode'],
data['mean'], data['std_dev'])
self.get_db().insert_gas_price_stats(data)
* **Side note**: I also get some **warning** message on the ethereum blockchain, because checking every minute, there is sometimes no new block on the chain, so I get this kind of message (but this is not critical and already handled properly internally, so I'm leaving this as is): [nvh.crypto.blockchain.chain_db] WARNING: Gas price entry already provided for block 14834104
===== Using a dedicated rocketchat channel for problem reports =====
* While I'm at it, I think I should also try to use a dedicated channel to report those exceptions/errors on rochetchat: otherwise this is quickly filling my "general admin" channel which I would prefer to keep clean instead. So let's change that.
* Created the **problem-reports** channel: **OK**
* Updated the runner code to notify in that channel: try:
self.execute(cmd, cwd=cwd, env=env)
except subprocess.SubprocessError:
notify = self.config.get("notify_script_errors", True)
if notify:
# And exception occured in the sub process, so we should send a notification:
msg = ":warning: **WARNING:** an exception occured in the following command:\n"
msg += f"{cmd}\n"
msg += f"cwd={cwd}\n\n"
msg += "=> Check the logs for details."
rchat = self.get_component("rchat")
rchat.send_message(msg, channel="problem-reports")
* And also extended the rochetchat command line to support testing the channel: if __name__ == "__main__":
# Create the context:
context = NVPContext()
# Add our component:
comp = context.register_component("rchat", RocketChat(context))
context.define_subparsers("main", {
'send': None,
})
psr = context.get_parser('main.send')
psr.add_argument("message", type=str,
help="Simple message that we should send")
psr.add_argument("-c", "--channel", dest="channel", type=str,
help="Channel where to send the message.")
comp.run()
* And this simple command below works so we are good for now: $ nvp rchat "hello manu" -c problem-reports
===== Setting up backups for the new SQL databases =====
* In the implementation above I introduced 2 new databases on my postgresql server: **bsc_chain** for the BSC and **eth_chain** for Ethereum (to store tha gas prices only for the moment, but this will be extended soon)
* => So let's not forget to setup the corresponding backups for these!
* Which is super easy to add in my custom NervProj config file: // Backup of individual SQL databases:
"postgresql_crypto_databases": {
"type": "postgresql",
"container": "postgresql_server",
"user": "crypto_user",
"databases": [
"crypto_coins",
"crypto_prices_v2",
"bsc_chain",
"eth_chain"
],
"backup_dirs": [
"${slot1}/sql/crypto",
"${slot2}/sql/crypto",
"${slot3}/sql/crypto"
]
},/*//*/
===== Adding Ethereum dedicated panel in CryptoView =====
* And finally, let's also add a dedicated tab in **CryptoView** to report Ethereum specific informations (only the gas price for now, but I expect to be able to extend that soon ;-))
* => For this I simply convert my previous **BSCPanel** into a more generic **EVMChainPanel** taking the chain component name as input: """EVMChainPanel widget module"""
import logging
from PyQt5 import QtCore
from PyQt5.QtCore import Qt
import PyQt5.QtWidgets as qwd
import numpy as np
from nvp.nvp_context import NVPContext
import nvh.crypto.utils as utl
from nvh.gui.app_base import AppBase
from nvh.gui.plot_canvas import PlotCanvas
from nvh.crypto.blockchain.evm_blockchain import EVMBlockchain
logger = logging.getLogger(__name__)
class EVMChainPanel(qwd.QWidget):
"""EVMChainPanel class"""
def __init__(self, comp_name):
"""Constructor of the EVMChainPanel class
using the comp_name to retrieve the desired chain component"""
super().__init__()
# Create a CoinGecko() instance:
ctx = NVPContext.get()
self.app: AppBase = ctx.get_component("app")
self.chain: EVMBlockchain = ctx.get_component(comp_name)
# chart display offset:
self.offset = 0.04
self.build_panel()
setattr(self, "sizeHint", lambda: QtCore.QSize(360, 300))
self.timer = QtCore.QTimer()
self.timer.setInterval(1000*60)
self.timer.timeout.connect(self.update_gas_price_chart)
self.timer.start()
self.update_gas_price_chart()
def build_panel(self):
"""Build the panel for this widget"""
lyt = qwd.QVBoxLayout()
lyt.setContentsMargins(5, 5, 5, 5)
lyt.setSpacing(5)
self.chart = PlotCanvas(self, width=2, height=2, dpi=100)
self.toolbar = self.chart.create_toolbar(self)
self.ax2 = self.chart.axes.twinx()
lyt.addWidget(self.toolbar)
lyt.addWidget(self.chart)
self.setLayout(lyt)
def update_gas_price_chart(self):
"""Update the gas price chart"""
logger.info("Updating gas price chart.")
# Collect the available entries:
# Get at max 7 days of data:
period = 20
data = self.chain.get_db().get_gas_price_stats(max_num=7*24*60+period)
self.chain.check(data.shape[1] == 6, "Invalid number of gas price columns: %d", data.shape[1])
# we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values:
now_ts = self.chain.get_timestamp()
times = now_ts - data[:, 1]
# convert times to hours:
times /= 3600
mean = utl.compute_ema(data[:, 4], period)
# logger.info("Mean values: %s", mean)
mode = data[:, 3]
# txn = data[:, 2]
txn = utl.compute_ema(data[:, 2], period)
# dev = utl.compute_ema(data[:, 5], period)
# mean_up = mean + dev
self.chart.clear()
sname = self.chain.get_short_name().upper()
self.chart.set_title(f"{sname} Gas Price")
self.chart.axes.plot(times, mode, color='orange', linewidth=1, label="Mode price")
self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean price")
# self.chart.axes.plot(times, mean_up, linewidth=1, label="Mean + 1sig")
self.ax2.cla()
self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx")
# self.ax2.plot(times, txn2, color='black', linewidth=1, label="Num. tx EMA")
# self.chart.axes.plot(times, mean_up, color='red', linewidth=1, label="Mean + 1sig")
self.chart.axes.set_xlabel("Time offset (hours)")
self.chart.axes.set_ylabel("Gas price (Gwei)")
self.ax2.set_ylabel("Number of tx")
# cf. https://www.adamsmith.haus/python/answers/how-to-add-secondary-axis-to-a-legend-in-python
lines_1, labels_1 = self.chart.axes.get_legend_handles_labels()
lines_2, labels_2 = self.ax2.get_legend_handles_labels()
lines = lines_1 + lines_2
labels = labels_1 + labels_2
self.chart.axes.legend(lines, labels, loc=0)
xmin = np.amin(times)
xmax = np.amax(times)
offset = self.offset * (xmax - xmin)
self.chart.axes.set_xlim([xmax+offset, xmin-offset])
self.chart.draw()
* Then I create a couple of those panels on my main window: dock3 = DockWidget("BSC")
panel = EVMChainPanel("bsc_chain")
self.widgets["bsc_panel"] = panel
dock3.setWidget(panel)
self.widgets["bsc_panel_dock"] = dock3
dock4 = DockWidget("Ethereum")
panel = EVMChainPanel("eth_chain")
self.widgets["eth_panel"] = panel
dock4.setWidget(panel)
self.widgets["eth_panel_dock"] = dock4
dock1.setAllowedAreas(Qt.AllDockWidgetAreas)
dock2.setAllowedAreas(Qt.AllDockWidgetAreas)
dock3.setAllowedAreas(Qt.AllDockWidgetAreas)
dock4.setAllowedAreas(Qt.AllDockWidgetAreas)
win.addDockWidget(Qt.RightDockWidgetArea, dock1)
win.addDockWidget(Qt.RightDockWidgetArea, dock2)
win.addDockWidget(Qt.RightDockWidgetArea, dock3)
win.addDockWidget(Qt.RightDockWidgetArea, dock4)
win.tabifyDockWidget(dock1, dock2)
win.tabifyDockWidget(dock1, dock3)
win.tabifyDockWidget(dock1, dock4)
* And this does the trick 👍!:
{{ blog:2022:0523:eth_gas_price_v1.png }}
* Except that... it's a pretty dirty graph here lol => I think this is mainly because the **mode price** value is changing significantly on Ethereum (as opposed to on the BSC): So I should maybe applied some EMA filtering to that one also: def update_gas_price_chart(self):
"""Update the gas price chart"""
logger.info("Updating gas price chart.")
# Collect the available entries:
# Get at max 7 days of data:
period = 20
data = self.chain.get_db().get_gas_price_stats(max_num=7*24*60+period)
self.chain.check(data.shape[1] == 6, "Invalid number of gas price columns: %d", data.shape[1])
# we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values:
now_ts = self.chain.get_timestamp()
times = now_ts - data[:, 1]
# convert times to hours:
times /= 3600
mean = utl.compute_ema(data[:, 4], period)
# logger.info("Mean values: %s", mean)
mode = utl.compute_ema(data[:, 3], period)
# txn = data[:, 2]
txn = utl.compute_ema(data[:, 2], period)
# dev = utl.compute_ema(data[:, 5], period)
# mean_up = mean + dev
self.chart.clear()
sname = self.chain.get_short_name().upper()
self.chart.set_title(f"{sname} Gas Price")
self.chart.axes.plot(times, mode, color='orange', linewidth=2, label="Mode price")
self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean price")
# self.chart.axes.plot(times, mean_up, linewidth=1, label="Mean + 1sig")
self.ax2.cla()
self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx")
* **Note**: I also bumped the mode price linewitdh to 2 pixels to try to make it more visible:
{{ blog:2022:0523:eth_gas_price_v2.png }}
* That's a bit better already, but I feel I should maybe not put the plots of the transactions just mixed with the prices: this is getting confusing, so let's try to separate them: self.chart.axes.legend(lines, labels, loc=2)
xmin = np.amin(times)
xmax = np.amax(times)
offset = self.offset * (xmax - xmin)
self.chart.axes.set_xlim([xmax+offset, xmin-offset])
# update the vertical ranges:
txn_min = np.nanmin(txn)
txn_max = np.nanmax(txn)
txn_range = txn_max - txn_min
price_min = min(np.nanmin(mean), np.nanmin(mode))
price_max = max(np.nanmax(mean), np.nanmax(mode))
price_range = price_max - price_min
self.chart.axes.set_ylim([price_min-self.offset*price_range,
price_max+price_range*(1.0+3.0*self.offset)])
self.ax2.set_ylim([txn_min-txn_range*(1.0+3.0*self.offset),
txn_max+txn_range*self.offset])
self.chart.draw()
* And with that change the display feels much better already (from my perspective at least):
{{ blog:2022:0523:eth_gas_price_v3.png }}
* And here the updated display for the BSC gas prices/transactions:
{{ blog:2022:0523:bsc_gas_price_v3.png }}
===== Conclusion =====
* Okay, so, with the experiments conducted here we see that it is fairly easy to manually collect gas prices for a given EVM blockchain, and from that, to generate your own "gas price oracle".
* => Eventually I could extend that to more EVM blockchains that I'm interested in, but for the moment, let's say this will be good enough and stop this long article here ;-)! See ya!