====== Crypto: Computing EVM blockchain gas price statistics ====== {{tag>dev python crypto nervhome finance bsc blockchain}} Continuing from my prevvous article on the [[blog:2022:0522_crypto_linear_finance_claims|Linear Finance reward claims]], I think I need to push it a bit further on the gas price I use for my transactions: using an hardcoded value of 7Gwei is certainly not ideal, instead I should build a dedicated system to continuously update the statistics on the gas price, so let's implement that system here. ====== ====== ===== Initial experiments ===== * Actually I already have the required function to compute the gas price from statistics. But I feel I should also prepare a jupyter env to be able to test those jsonrpc functions manually, so I'm setting this up first: "bscjup": { "custom_python_env": "bsc_env", "cmd": "${PYTHON} -m jupyter lab", "cwd": "${PROJECT_ROOT_DIR}", "python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"] }, * I then tested a minimal script in jupyter: from nvp.nvp_context import NVPContext if NVPContext.instance is None: NVPContext() ctx = NVPContext.get() chain = ctx.get_component('bsc_chain') bck = chain.get_block(full_tx=True) * And indeed, in the resulting block above ('bck') we can retrieve the list of transactions from bck['transactions'], and in the the gas price specified for that transaction * So I just added the following methods in the EVMBlockchain class: def get_gas_prices(self, bnum=None): """Collect the gas prices in a given block""" # Retrieve the block details with all the transactions: bck = self.get_block(block_num=bnum, full_tx=True) # logger.info("block content: %s", self.pretty_print(bck)) # logger.info("block content: %s", bck) # We retrieve the list of transactions from the block: transactions = bck['transactions'] tnum = len(transactions) bnum = bck['number'] logger.info("Block %d contains %d transactions", bnum, tnum) # We collect all the gas prices from the transactions: gprices = [] for i in range(tnum): # We should convert the hex string to a number: txn = transactions[i] price = txn['gasPrice']/1e9 if price > 0.0: gprices.append(price) return { 'prices': gprices, 'block_number': bnum, 'timestamp': bck['timestamp']} # Function to collect the mode, mean, stddev statistics from a given list of gas prices # from a given block number. If not specified then return the last block details. def get_gas_price_statistics(self, bnum=None): """"Method used to retrieve the gas price statistics from a given block""" data = self.get_gas_prices(bnum) gprices = data['prices'] data['num_prices'] = len(gprices) data['mode'] = statistics.mode(gprices) data['mean'] = statistics.mean(gprices) data['dev'] = statistics.stdev(gprices) return data * Now we have the question of the storage of that data: basically 2 options here: either in a dedicated remote SQL database or some local storage DB or file: * => The remote storage option is probably the best option: even if I then need some fast crypto system on an extern computer, "updating the gas price" is not something that would have to be done super quickly, and I could still update the storage mechanism at that time anyway. * So let's prepare a database for that: SQL_CREATE_TOKENS_TABLE = """ CREATE TABLE IF NOT EXISTS tokens ( id SERIAL PRIMARY KEY, address char(42) UNIQUE, name varchar(64), symbol varchar(32) UNIQUE, decimals SMALLINT, min_slippage INT, swap_fees INT, classification SMALLINT ); """ SQL_INSERT_TOKEN = ''' INSERT INTO tokens(address,name,symbol,decimals,min_slippage,swap_fees,classification) VALUES(%s,%s,%s,%s,%s,%s,%s) ''' SQL_UPDATE_TOKEN = ''' UPDATE tokens SET (name,symbol,decimals,min_slippage,swap_fees,classification) = (%s,%s,%s,%s,%s,%s) WHERE address=%s; ''' SQL_CREATE_GASPRICE_TABLE = """ CREATE TABLE IF NOT EXISTS gasprice ( block_num integer PRIMARY KEY, timestamp integer, num_transactions integer, mode float, mean float, std_dev float ); """ SQL_INSERT_GASPRICE = ''' INSERT INTO gasprice(block_num, timestamp, num_transactions, mode, mean, std_dev) VALUES(%s,%s,%s,%s,%s,%s)''' class ChainDB(NVPObject): """ChainDB component class""" def __init__(self, db_name, user, password): """ChainDB constructor""" self.sql_db = PostgreSQLDB(db_name=db_name, user=user, password=password) self.sql_db.execute(SQL_CREATE_TOKENS_TABLE, commit=True) self.sql_db.execute(SQL_CREATE_GASPRICE_TABLE, commit=True) def execute(self, *args, **kaargs): """Forward execute code to the SQL DB""" return self.sql_db.execute(*args, **kaargs) def insert_gas_price_stats(self, data, check=True): """Insert a new gasprice row in the table""" # We should insert (block_num,mode,mean,std_dev) logger.info("Inserting gasprice data: %s", data) keys = ['block_number', 'timestamp', 'num_transactions', 'mode', 'mean', 'std_dev'] row = [data[key] for key in keys] if check: # Check if the row already exists: bnum = data['block_number'] cur = self.execute("select exists(select 1 from gasprice where block_num=%s)", (bnum,)) if cur.fetchone()[0]: logger.warning('Gas price entry already provided for block %d', bnum) return self.execute(SQL_INSERT_GASPRICE, row, commit=True) * And I now have a dedicate method in the EVMBlockchain to store new rows in the gasprice table: def collect_gas_price_statistics(self): """Collect the latest gas price statistics""" data = self.get_gas_price_statistics() del data['prices'] self.get_db().insert_gas_price_stats(data) * Tested that in jupyter and it works as expected: chain.collect_gas_price_statistics() * Command above will output: 2022/05/23 08:27:21 [nvh.crypto.blockchain.evm_blockchain] INFO: Block 18045926 contains 150 transactions 2022/05/23 08:27:21 [nvh.crypto.blockchain.chain_db] INFO: Inserting gasprice data: {'block_number': 18045926, 'timestamp': 1653290838, 'num_transactions': 149, 'mode': 5.0, 'mean': 5.84451191375839, 'std_dev': 2.419240396353444} * => And I can see that a new row is added to the table, all **OK**. * Now I just need to add new data every minute, so let's add a command line entry to handle that: class BinanceSmartChain(EVMBlockchain): """BinanceSmartChain component class""" def __init__(self, ctx): """blockchain constructor""" cfg = ctx.get_config()['blockchains']['bsc'] EVMBlockchain.__init__(self, ctx, cfg) def get_contract_abi_url(self, address): """Retrieve the ABI URL for a given contract""" api_key = self.config.get("bscscan_api_key", None) url = f"https://api.bscscan.com/api?module=contract&action=getabi&address={address}" if api_key is not None: url = f"{url}&apikey={api_key}" return url, 1.0/5 if api_key is not None else 5.0 def process_command(self, cmd): """Check if this component can process the given command""" if cmd == 'collect-gas-price': self.collect_gas_price_statistics() return True return False if __name__ == "__main__": # Create the context: context = NVPContext() # Add our component: comp = context.register_component("bsc_chain", BinanceSmartChain(context)) context.define_subparsers("main", { 'collect-gas-price': None, }) comp.run() * And we add the corresponding script: "bsc": { "custom_python_env": "bsc_env", "cmd": "${PYTHON} ${PROJECT_ROOT_DIR}/nvh/crypto/bsc/binance_smart_chain.py", "python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"] }, * And finally the entry in our **every_1min_operations.sh** file: log_dir="/mnt/data1/admin/logs" lfile="${log_dir}/bsc_collect_gas_price.log" nvp bsc collect-gas-price 2>&1 | tee -a $lfile ===== Retrieving the latest statistic values ===== * I then added the following function in the **ChainDB** class to get back the gasprice data: def get_gas_price_stats(self, max_num=None, min_ts=None): """Function returning the num_transactions, mode, mean, std_dev columns from the gas price table""" if max_num is not None: sql = f"SELECT * FROM gasprice ORDER BY block_num DESC LIMIT {max_num}" sql = f"SELECT * FROM ({sql}) as sub ORDER BY sub.block_num ASC;" elif min_ts is not None: if min_ts < 0: # Assume relative to current timestamp and given in seconds: min_ts = self.get_timestamp() + min_ts # logger.info("using min_ts: %d", min_ts) sql = f"SELECT * FROM gasprice WHERE timestamp > {min_ts} ORDER BY timestamp ASC;" else: sql = "SELECT * FROM gasprice;" cur = self.execute(sql) results = cur.fetchall() # logger.debug("Retrieved %d gasprice rows", len(results)) return np.array(results) * And with that I should be able to compute a more appropriate gas price before I perform a transaction: def get_current_gas_price_emas(self, period=10): """Compute the EMAs for the gas price data""" # Retrieve the gas price statistics with a given smoothing EMA period: # Note: we should retrieve at least twice as much data as the period length: data = self.get_db().get_gas_price_stats(min_ts=-period*2*60) # Now compute the ema for each col: # Note: we only keep the EMAs for the last 4 columns res = [ta.trend.EMAIndicator(pd.Series(data[:, i+2]), window=period).ema_indicator().iloc[-1] for i in range(4)] return dict(zip(("num_transactions", "mode", "mean", "std_dev"), res)) def compute_gas_price_estimate(self, mode="default"): """Compute an advanced gas price estimate.""" # Retrieve the weights from the selected mode: weights = self.config["gas_price_weights"][mode] # Compute an estimated gas price, using weights for the mode,mean, mean+std_dev price values: # the weights should also contain the estimation period: self.check(len(weights) == 4, "Invalid number of weights") emas = self.get_current_gas_price_emas(period=weights[0]) weights = weights[1:] totw = sum(weights) self.check(totw > 0.0, "Invalid total weights.") price = (emas['mode'] * weights[0] + emas['mean'] * weights[1] + (emas['mean']+emas['std_dev']) * weights[2])/totw price = math.floor(price*1000)/1000 # Clamp if bigger than max gas price: maxp = self.max_gas_price if price > maxp: logger.warning("Clamping estimated gas price of %f to max gas price value of %f", price, maxp) price = maxp else: logger.info("Computed estimated gas price of %f gwei for mode %s", price, mode) return price I just discovered the **ta** package above: it's using pandas, but add support for many technical indicators without any fancy dependency build requirement like **TA-lib** so I will go with that one for now! * And if I call the ''compute_gas_price_estimate()'' method above in jupyter I get the following kind of result (as expected): 2022/05/23 10:26:27 [nvh.crypto.blockchain.evm_blockchain] INFO: Computed estimated gas price of 5.275000 gwei for mode default ===== Displaying the gas price chart in cryptoview ===== * One last bonus step I would like to take now would be to display the current gas price as a chart in my cryptoview app: could be interesting to have a look at that chart sometimes 😉! * So here is my first "BSCPanel" implementation: """BSCPanel widget module""" import logging from PyQt5 import QtCore from PyQt5.QtCore import Qt import PyQt5.QtWidgets as qwd import numpy as np from nvp.nvp_context import NVPContext from nvh.gui.app_base import AppBase from nvh.gui.plot_canvas import PlotCanvas from nvh.crypto.bsc.binance_smart_chain import BinanceSmartChain logger = logging.getLogger(__name__) class BSCPanel(qwd.QWidget): """BSCPanel class""" def __init__(self): """Constructor of the BSCPanel class""" super().__init__() # Create a CoinGecko() instance: ctx = NVPContext.get() self.app: AppBase = ctx.get_component("app") self.chain: BinanceSmartChain = ctx.get_component("bsc_chain") # chart display offset: self.offset = 0.04 self.build_panel() setattr(self, "sizeHint", lambda: QtCore.QSize(360, 300)) self.timer = QtCore.QTimer() self.timer.setInterval(1000*60) self.timer.timeout.connect(self.update_gas_price_chart) self.timer.start() self.update_gas_price_chart() def build_panel(self): """Build the panel for this widget""" lyt = qwd.QVBoxLayout() lyt.setContentsMargins(5, 5, 5, 5) lyt.setSpacing(5) self.chart = PlotCanvas(self, width=2, height=2, dpi=100) self.toolbar = self.chart.create_toolbar(self) self.ax2 = self.chart.axes.twinx() lyt.addWidget(self.toolbar) lyt.addWidget(self.chart) self.setLayout(lyt) def update_gas_price_chart(self): """Update the gas price chart""" logger.info("Updating gas price chart.") # Collect the available entries: # Get at max 7 days of data: data = self.chain.get_db().get_gas_price_stats(min_ts=-7*24*60*60) self.chain.check(data.shape[1] == 6, "Invalid number of gas price columns: %d", data.shape[1]) # we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values: now_ts = self.chain.get_timestamp() times = now_ts - data[:, 1] # convert times to hours: times /= 3600 mean = data[:, 4] mode = data[:, 3] txn = data[:, 2] # mean_up = mean + data[:, 5] self.chart.clear() self.chart.set_title("BSC Gas Price") self.chart.axes.plot(times, mode, color='orange', linewidth=1, label="Mode") self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean") self.ax2.cla() self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx") # self.chart.axes.plot(times, mean_up, color='red', linewidth=1, label="Mean + 1sig") self.chart.axes.set_xlabel("Time offset (hours)") self.chart.axes.set_ylabel("Gas price (Gwei)") self.ax2.set_ylabel("Number of tx") # cf. https://www.adamsmith.haus/python/answers/how-to-add-secondary-axis-to-a-legend-in-python lines_1, labels_1 = self.chart.axes.get_legend_handles_labels() lines_2, labels_2 = self.ax2.get_legend_handles_labels() lines = lines_1 + lines_2 labels = labels_1 + labels_2 self.chart.axes.legend(lines, labels, loc=0) xmin = np.amin(times) xmax = np.amax(times) offset = self.offset * (xmax - xmin) self.chart.axes.set_xlim([xmax+offset, xmin-offset]) self.chart.draw() * And here is the kind of display this will produce: {{ blog:2022:0523:bsc_gas_price_v1.png }} * => yeah... It's a bit messy when we disply raw data directly: let's add a EMA filtering on those values instead 😉 * So I added a **utility function** ''compute_ema()'' in my **nvh.crypto.utils** module: def compute_ema(arr, period): """Compute the EMA of a numpy array""" return ta.trend.EMAIndicator(pd.Series(arr), window=period).ema_indicator().values * And updated the gas price plot drawing to this: def update_gas_price_chart(self): """Update the gas price chart""" logger.info("Updating gas price chart.") # Collect the available entries: # Get at max 7 days of data: period = 20 data = self.chain.get_db().get_gas_price_stats(max_num=7*24*60+period) self.chain.check(data.shape[1] == 6, "Invalid number of gas price columns: %d", data.shape[1]) # we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values: now_ts = self.chain.get_timestamp() times = now_ts - data[:, 1] # convert times to hours: times /= 3600 mean = utl.compute_ema(data[:, 4], period) # logger.info("Mean values: %s", mean) mode = data[:, 3] # txn = data[:, 2] txn = utl.compute_ema(data[:, 2], period) # dev = utl.compute_ema(data[:, 5], period) # mean_up = mean + dev self.chart.clear() self.chart.set_title("BSC Gas Price") self.chart.axes.plot(times, mode, color='orange', linewidth=1, label="Mode") self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean") # self.chart.axes.plot(times, mean_up, linewidth=1, label="Mean + 1sig") self.ax2.cla() self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx") # self.ax2.plot(times, txn2, color='black', linewidth=1, label="Num. tx EMA") # self.chart.axes.plot(times, mean_up, color='red', linewidth=1, label="Mean + 1sig") self.chart.axes.set_xlabel("Time offset (hours)") self.chart.axes.set_ylabel("Gas price (Gwei)") self.ax2.set_ylabel("Number of tx") # cf. https://www.adamsmith.haus/python/answers/how-to-add-secondary-axis-to-a-legend-in-python lines_1, labels_1 = self.chart.axes.get_legend_handles_labels() lines_2, labels_2 = self.ax2.get_legend_handles_labels() lines = lines_1 + lines_2 labels = labels_1 + labels_2 self.chart.axes.legend(lines, labels, loc=0) xmin = np.amin(times) xmax = np.amax(times) offset = self.offset * (xmax - xmin) self.chart.axes.set_xlim([xmax+offset, xmin-offset]) self.chart.draw() * Which gives me the result: {{ blog:2022:0523:bsc_gas_price_v2.png }} * => There isn't much data to display here for the moment, so this still feels a bit noisy, but I think it should get better when I have few days of history data to display instead, so let's just keep it this way fro the moment ✌. ===== Testing with other EVM blockchains ===== * Actually, in all the steps above we only used some basic JSONRPC block/transaction retrieval: so this should in theory work the same for other EVM compatible blockchains ? * => Let's try to implement this for Ethereum: * Here is the simple **EthereumChain** class I create as a starting point: """ETH blockchain.""" import logging from nvp.nvp_context import NVPContext from nvh.crypto.blockchain.evm_blockchain import EVMBlockchain logger = logging.getLogger(__name__) def create_component(ctx: NVPContext): """Create an instance of the component""" return EthereumChain(ctx) class EthereumChain(EVMBlockchain): """EthereumChain component class""" def __init__(self, ctx): """blockchain constructor""" cfg = ctx.get_config()['blockchains']['eth'] EVMBlockchain.__init__(self, ctx, cfg) def get_contract_abi_url(self, address): """Retrieve the ABI URL for a given contract""" api_key = self.config.get("etherscan_api_key", None) url = f"https://api.etherscan.io/api?module=contract&action=getabi&address={address}" if api_key is not None: url = f"{url}&apikey={api_key}" return url, 1.0/5 if api_key is not None else 5.0 def process_command(self, cmd): """Check if this component can process the given command""" if cmd == 'collect-gas-price': self.collect_gas_price_statistics() return True return False if __name__ == "__main__": # Create the context: context = NVPContext() # Add our component: comp = context.register_component("eth_chain", EthereumChain(context)) context.define_subparsers("main", { 'collect-gas-price': None, }) comp.run() * Then we add a simple script to use it: "eth": { "custom_python_env": "bsc_env", "cmd": "${PYTHON} ${PROJECT_ROOT_DIR}/nvh/crypto/eth/ethereum_chain.py", "python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"] }, * And this works pretty damn well already: $ nvp eth collect-gas-price 2022/05/23 22:29:06 [nvh.crypto.blockchain.evm_blockchain] INFO: Block 14831913: num_tx=21, mode=13.580, mean=16.079, dev=4.162 * => So also adding that in my 1min operation script: lfile="${log_dir}/eth_collect_gas_price.log" nvp eth collect-gas-price 2>&1 | tee -a $lfile ===== Investigating gas price collect error on Ethereum ===== * Remember the system I built in my previous article to [[https://wiki.nervtech.org/doku.php?id=blog:2022:0522_crypto_linear_finance_claims#before_we_start|receive automatic notifications]] in case there is an exception in one of the scripts I run with cron ? Well, I got a few of those notifications over night about the ethereum gas price retrieval 😅: {{ blog:2022:0523:eth_errors.png }} * Let's see what this is about... Ohhh okay, that's interesting: I found at least the following type of error in the logs: * **Error 1**: Traceback (most recent call last): File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 55, in comp.run() File "/mnt/data1/dev/projects/NervProj/nvp/nvp_component.py", line 69, in run res = self.process_command(cmd) File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 38, in process_command self.collect_gas_price_statistics() File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 261, in collect_gas_price_statistics data = self.get_gas_price_statistics() File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 253, in get_gas_price_statistics data['mode'] = statistics.mode(gprices) File "/mnt/data1/dev/projects/NervProj/.pyenvs/bsc_env/lib/python3.10/statistics.py", line 583, in mode raise StatisticsError('no mode for empty data') from None statistics.StatisticsError: no mode for empty data * **Error 2**: Traceback (most recent call last): File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 55, in comp.run() File "/mnt/data1/dev/projects/NervProj/nvp/nvp_component.py", line 69, in run res = self.process_command(cmd) File "/mnt/data1/dev/projects/NervHome/nvh/crypto/eth/ethereum_chain.py", line 38, in process_command self.collect_gas_price_statistics() File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 261, in collect_gas_price_statistics data = self.get_gas_price_statistics() File "/mnt/data1/dev/projects/NervHome/nvh/crypto/blockchain/evm_blockchain.py", line 255, in get_gas_price_statistics data['std_dev'] = statistics.stdev(gprices) File "/mnt/data1/dev/projects/NervProj/.pyenvs/bsc_env/lib/python3.10/statistics.py", line 828, in stdev var = variance(data, xbar) File "/mnt/data1/dev/projects/NervProj/.pyenvs/bsc_env/lib/python3.10/statistics.py", line 767, in variance raise StatisticsError('variance requires at least two data points') statistics.StatisticsError: variance requires at least two data points * So typically, I think these may happen if we have respectively 0 or 1 transaction only in a given block which is of course totally legit, so let's fix that. * **OK** here is the updated code which should take care of those problems: def get_gas_price_statistics(self, bnum=None): """"Method used to retrieve the gas price statistics from a given block""" data = self.get_gas_prices(bnum) gprices = data['prices'] num = len(gprices) data['num_transactions'] = num data['mode'] = statistics.mode(gprices) if num > 0 else math.nan data['mean'] = statistics.mean(gprices) if num > 0 else math.nan data['std_dev'] = statistics.stdev(gprices) if num > 1 else math.nan return data def collect_gas_price_statistics(self): """Collect the latest gas price statistics""" data = self.get_gas_price_statistics() del data['prices'] # Note: we only store the block price data if there are at least 2 transactions: if data['num_transactions'] < 2: logger.info("Ignoring block %d: not enough transactions.", data['block_number']) return logger.info("Block %d: num_tx=%d, mode=%.3f, mean=%.3f, dev=%.3f", data['block_number'], data['num_transactions'], data['mode'], data['mean'], data['std_dev']) self.get_db().insert_gas_price_stats(data) * **Side note**: I also get some **warning** message on the ethereum blockchain, because checking every minute, there is sometimes no new block on the chain, so I get this kind of message (but this is not critical and already handled properly internally, so I'm leaving this as is): [nvh.crypto.blockchain.chain_db] WARNING: Gas price entry already provided for block 14834104 ===== Using a dedicated rocketchat channel for problem reports ===== * While I'm at it, I think I should also try to use a dedicated channel to report those exceptions/errors on rochetchat: otherwise this is quickly filling my "general admin" channel which I would prefer to keep clean instead. So let's change that. * Created the **problem-reports** channel: **OK** * Updated the runner code to notify in that channel: try: self.execute(cmd, cwd=cwd, env=env) except subprocess.SubprocessError: notify = self.config.get("notify_script_errors", True) if notify: # And exception occured in the sub process, so we should send a notification: msg = ":warning: **WARNING:** an exception occured in the following command:\n" msg += f"{cmd}\n" msg += f"cwd={cwd}\n\n" msg += "=> Check the logs for details." rchat = self.get_component("rchat") rchat.send_message(msg, channel="problem-reports") * And also extended the rochetchat command line to support testing the channel: if __name__ == "__main__": # Create the context: context = NVPContext() # Add our component: comp = context.register_component("rchat", RocketChat(context)) context.define_subparsers("main", { 'send': None, }) psr = context.get_parser('main.send') psr.add_argument("message", type=str, help="Simple message that we should send") psr.add_argument("-c", "--channel", dest="channel", type=str, help="Channel where to send the message.") comp.run() * And this simple command below works so we are good for now: $ nvp rchat "hello manu" -c problem-reports ===== Setting up backups for the new SQL databases ===== * In the implementation above I introduced 2 new databases on my postgresql server: **bsc_chain** for the BSC and **eth_chain** for Ethereum (to store tha gas prices only for the moment, but this will be extended soon) * => So let's not forget to setup the corresponding backups for these! * Which is super easy to add in my custom NervProj config file: // Backup of individual SQL databases: "postgresql_crypto_databases": { "type": "postgresql", "container": "postgresql_server", "user": "crypto_user", "databases": [ "crypto_coins", "crypto_prices_v2", "bsc_chain", "eth_chain" ], "backup_dirs": [ "${slot1}/sql/crypto", "${slot2}/sql/crypto", "${slot3}/sql/crypto" ] },/*//*/ ===== Adding Ethereum dedicated panel in CryptoView ===== * And finally, let's also add a dedicated tab in **CryptoView** to report Ethereum specific informations (only the gas price for now, but I expect to be able to extend that soon ;-)) * => For this I simply convert my previous **BSCPanel** into a more generic **EVMChainPanel** taking the chain component name as input: """EVMChainPanel widget module""" import logging from PyQt5 import QtCore from PyQt5.QtCore import Qt import PyQt5.QtWidgets as qwd import numpy as np from nvp.nvp_context import NVPContext import nvh.crypto.utils as utl from nvh.gui.app_base import AppBase from nvh.gui.plot_canvas import PlotCanvas from nvh.crypto.blockchain.evm_blockchain import EVMBlockchain logger = logging.getLogger(__name__) class EVMChainPanel(qwd.QWidget): """EVMChainPanel class""" def __init__(self, comp_name): """Constructor of the EVMChainPanel class using the comp_name to retrieve the desired chain component""" super().__init__() # Create a CoinGecko() instance: ctx = NVPContext.get() self.app: AppBase = ctx.get_component("app") self.chain: EVMBlockchain = ctx.get_component(comp_name) # chart display offset: self.offset = 0.04 self.build_panel() setattr(self, "sizeHint", lambda: QtCore.QSize(360, 300)) self.timer = QtCore.QTimer() self.timer.setInterval(1000*60) self.timer.timeout.connect(self.update_gas_price_chart) self.timer.start() self.update_gas_price_chart() def build_panel(self): """Build the panel for this widget""" lyt = qwd.QVBoxLayout() lyt.setContentsMargins(5, 5, 5, 5) lyt.setSpacing(5) self.chart = PlotCanvas(self, width=2, height=2, dpi=100) self.toolbar = self.chart.create_toolbar(self) self.ax2 = self.chart.axes.twinx() lyt.addWidget(self.toolbar) lyt.addWidget(self.chart) self.setLayout(lyt) def update_gas_price_chart(self): """Update the gas price chart""" logger.info("Updating gas price chart.") # Collect the available entries: # Get at max 7 days of data: period = 20 data = self.chain.get_db().get_gas_price_stats(max_num=7*24*60+period) self.chain.check(data.shape[1] == 6, "Invalid number of gas price columns: %d", data.shape[1]) # we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values: now_ts = self.chain.get_timestamp() times = now_ts - data[:, 1] # convert times to hours: times /= 3600 mean = utl.compute_ema(data[:, 4], period) # logger.info("Mean values: %s", mean) mode = data[:, 3] # txn = data[:, 2] txn = utl.compute_ema(data[:, 2], period) # dev = utl.compute_ema(data[:, 5], period) # mean_up = mean + dev self.chart.clear() sname = self.chain.get_short_name().upper() self.chart.set_title(f"{sname} Gas Price") self.chart.axes.plot(times, mode, color='orange', linewidth=1, label="Mode price") self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean price") # self.chart.axes.plot(times, mean_up, linewidth=1, label="Mean + 1sig") self.ax2.cla() self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx") # self.ax2.plot(times, txn2, color='black', linewidth=1, label="Num. tx EMA") # self.chart.axes.plot(times, mean_up, color='red', linewidth=1, label="Mean + 1sig") self.chart.axes.set_xlabel("Time offset (hours)") self.chart.axes.set_ylabel("Gas price (Gwei)") self.ax2.set_ylabel("Number of tx") # cf. https://www.adamsmith.haus/python/answers/how-to-add-secondary-axis-to-a-legend-in-python lines_1, labels_1 = self.chart.axes.get_legend_handles_labels() lines_2, labels_2 = self.ax2.get_legend_handles_labels() lines = lines_1 + lines_2 labels = labels_1 + labels_2 self.chart.axes.legend(lines, labels, loc=0) xmin = np.amin(times) xmax = np.amax(times) offset = self.offset * (xmax - xmin) self.chart.axes.set_xlim([xmax+offset, xmin-offset]) self.chart.draw() * Then I create a couple of those panels on my main window: dock3 = DockWidget("BSC") panel = EVMChainPanel("bsc_chain") self.widgets["bsc_panel"] = panel dock3.setWidget(panel) self.widgets["bsc_panel_dock"] = dock3 dock4 = DockWidget("Ethereum") panel = EVMChainPanel("eth_chain") self.widgets["eth_panel"] = panel dock4.setWidget(panel) self.widgets["eth_panel_dock"] = dock4 dock1.setAllowedAreas(Qt.AllDockWidgetAreas) dock2.setAllowedAreas(Qt.AllDockWidgetAreas) dock3.setAllowedAreas(Qt.AllDockWidgetAreas) dock4.setAllowedAreas(Qt.AllDockWidgetAreas) win.addDockWidget(Qt.RightDockWidgetArea, dock1) win.addDockWidget(Qt.RightDockWidgetArea, dock2) win.addDockWidget(Qt.RightDockWidgetArea, dock3) win.addDockWidget(Qt.RightDockWidgetArea, dock4) win.tabifyDockWidget(dock1, dock2) win.tabifyDockWidget(dock1, dock3) win.tabifyDockWidget(dock1, dock4) * And this does the trick 👍!: {{ blog:2022:0523:eth_gas_price_v1.png }} * Except that... it's a pretty dirty graph here lol => I think this is mainly because the **mode price** value is changing significantly on Ethereum (as opposed to on the BSC): So I should maybe applied some EMA filtering to that one also: def update_gas_price_chart(self): """Update the gas price chart""" logger.info("Updating gas price chart.") # Collect the available entries: # Get at max 7 days of data: period = 20 data = self.chain.get_db().get_gas_price_stats(max_num=7*24*60+period) self.chain.check(data.shape[1] == 6, "Invalid number of gas price columns: %d", data.shape[1]) # we got 'blcok_number', 'timestamp', 'num_txs', 'mode', 'mean', 'std_dev' values: now_ts = self.chain.get_timestamp() times = now_ts - data[:, 1] # convert times to hours: times /= 3600 mean = utl.compute_ema(data[:, 4], period) # logger.info("Mean values: %s", mean) mode = utl.compute_ema(data[:, 3], period) # txn = data[:, 2] txn = utl.compute_ema(data[:, 2], period) # dev = utl.compute_ema(data[:, 5], period) # mean_up = mean + dev self.chart.clear() sname = self.chain.get_short_name().upper() self.chart.set_title(f"{sname} Gas Price") self.chart.axes.plot(times, mode, color='orange', linewidth=2, label="Mode price") self.chart.axes.plot(times, mean, color='blue', linewidth=1, label="Mean price") # self.chart.axes.plot(times, mean_up, linewidth=1, label="Mean + 1sig") self.ax2.cla() self.ax2.plot(times, txn, color='red', linewidth=1, label="Num. tx") * **Note**: I also bumped the mode price linewitdh to 2 pixels to try to make it more visible: {{ blog:2022:0523:eth_gas_price_v2.png }} * That's a bit better already, but I feel I should maybe not put the plots of the transactions just mixed with the prices: this is getting confusing, so let's try to separate them: self.chart.axes.legend(lines, labels, loc=2) xmin = np.amin(times) xmax = np.amax(times) offset = self.offset * (xmax - xmin) self.chart.axes.set_xlim([xmax+offset, xmin-offset]) # update the vertical ranges: txn_min = np.nanmin(txn) txn_max = np.nanmax(txn) txn_range = txn_max - txn_min price_min = min(np.nanmin(mean), np.nanmin(mode)) price_max = max(np.nanmax(mean), np.nanmax(mode)) price_range = price_max - price_min self.chart.axes.set_ylim([price_min-self.offset*price_range, price_max+price_range*(1.0+3.0*self.offset)]) self.ax2.set_ylim([txn_min-txn_range*(1.0+3.0*self.offset), txn_max+txn_range*self.offset]) self.chart.draw() * And with that change the display feels much better already (from my perspective at least): {{ blog:2022:0523:eth_gas_price_v3.png }} * And here the updated display for the BSC gas prices/transactions: {{ blog:2022:0523:bsc_gas_price_v3.png }} ===== Conclusion ===== * Okay, so, with the experiments conducted here we see that it is fairly easy to manually collect gas prices for a given EVM blockchain, and from that, to generate your own "gas price oracle". * => Eventually I could extend that to more EVM blockchains that I'm interested in, but for the moment, let's say this will be good enough and stop this long article here ;-)! See ya!