====== NervProj: Restoring my crypto coins monitoring project ======

{{tag>dev python crypto nervhome finance}}

Okay so, I have a coin monitoring system I built some time ago: it's in python and it will retrive price data for many coins on a regular basis, and store that data into a postgresql database. It was producing some errors or conflicts or corruptions sometimes but on the whole it was working reasonably well. Until I decided to get ride of my fully corrupted RAID array, and thus mess up completely my main server structure lol.

Yet that tool was actually very handy (I then created a GUI on top of it to get some visuals on the coin price actions), so I think it's now high time I try and put this back on rails!

====== ======

===== Our current status =====

  * First I need to clarify my current status on this point, because, as usual, I have zero idea were I'm at in this mess 😅.
  * Okay, so, before, I had this cron script running every minute: <sxh bash; highlight: []>* * * * * su kenshin -c "bash /home/kenshin/scripts/cron/every_1min_operations.sh"</sxh>
  * Let's check the content of that... So, basically that script is just executing another script to collect highres data on coingecko: <sxh bash; highlight: []>defi_gecko --updateHighResDatasets >>$lfile 2>&1</sxh>
  * Let's find the defi_gecko command. [//searching...//] Here it is: <sxh bash; highlight: []>defi_gecko()
{
  cd `defi_dir`
  nv_py_setup_paths
  nv_call_python gecko.py "$@"
  cd - > /dev/null
}</sxh>

  * Now to the **gecko.py** file: okay, this mostly just a launcher for **Coingecko** class which will do all the heavy lifting. So the Coingecko class is the final class I should re-integrate in NervProj.

===== Thinking about NervProj current design =====

  * And in fact, this is getting me thinking a little: I could keep adding "components" into a single "NVPContext" in a given python environment, 
  * But this will progressively make the application larger and larger, with more dependency packages in a single env, etc.
  * => So what I want instead is to be able to "split" this (future giant) collection of components into simpler apps in dedicated python envs.
  * In this case for instance, I need this script to work: <sxh javascript; highlight: []>  "custom_python_envs": {
    "defi_env": {
      "packages": ["requests"]
    }
  },
  "scripts": {
    // Update the coingecko prices
    "defi_gecko_update": {
      "custom_python_env": "defi_env",
      "cmd": "${PYTHON} nvh/defi/coingecko.py --updateHighResDatasets",
      "cwd": "${PROJECT_ROOT_DIR}",
      "python_path": ["${PROJECT_ROOT_DIR}"]
    }
  }</sxh>

  * But the launcher I'm going to build in the Coingecko file should still use an **NVPContext**: except that this will be in a different python env than the main NVPContext running the initial "nvp run defi_gecko_update" command, and it should only load the components we really need in there.

  * Let's get coding and see where we go with that... 😎

===== Auto python env creation =====

  * I thus started a minimal coingecko script: <sxh python; highlight: []>"""Module for Coingecko class definition"""
import logging

logger = logging.getLogger(__name__)

if __name__ == "__main__":
    logger.info("This is the main script for coingecko")
</sxh>

  * But then I get an error when trying to run that command: <sxh bash; highlight: []>$ nvp run defi_gecko_update
Traceback (most recent call last):
  File "D:\Projects\NervProj\cli.py", line 5, in <module>
    ctx.run()
  File "D:\Projects\NervProj\nvp\nvp_context.py", line 315, in run
    if comp.process_command(cmd):
  File "D:\Projects\NervProj\nvp\components\runner.py", line 39, in process_command
    self.run_script(sname, proj)
  File "D:\Projects\NervProj\nvp\components\runner.py", line 120, in run_script
    self.execute(cmd, cwd=cwd, env=env)
  File "D:\Projects\NervProj\nvp\nvp_object.py", line 383, in execute
    subprocess.check_call(cmd, stdout=stdout, stderr=stderr, cwd=cwd, env=env)
  File "D:\Projects\NervProj\tools\windows\python-3.10.1\lib\subprocess.py", line 364, in check_call
    retcode = call(*popenargs, **kwargs)
  File "D:\Projects\NervProj\tools\windows\python-3.10.1\lib\subprocess.py", line 345, in call
    with Popen(*popenargs, **kwargs) as p:
  File "D:\Projects\NervProj\tools\windows\python-3.10.1\lib\subprocess.py", line 966, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "D:\Projects\NervProj\tools\windows\python-3.10.1\lib\subprocess.py", line 1435, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] Le fichier spécifié est introuvable
</sxh>

  * => Actually that make sense since I have not created the **defi_env** environment yet! 😅 => Let's just try to automate that in our runner component (ie. create the pyenv if it doesn't exist yet)

  * **OK**! better now: script execution is OK, but I don't get anything displayed, because I did not setup logging correctly yet, let's continue.


===== Initial version of coingecko script =====

  * So I made some progress on my coingecko class which currently looks like this: <sxh python; highlight: []>"""Module for Coingecko class definition"""

import logging
from nvp.nvp_context import NVPContext
from nvp.nvp_component import NVPComponent

logger = logging.getLogger(__name__)


class Coingecko(NVPComponent):
    """Coingecko component class"""

    def __init__(self, ctx: NVPContext):
        """Script runner constructor"""
        NVPComponent.__init__(self, ctx)

        # override the config entry with our dedicated config read
        # from the current base dir:
        self.config = self.config["coingecko"]

    def process_command(self, cmd):
        """Check if this component can process the given command"""

        if cmd == 'update-highres':
            self.update_highres_datasets()
            return True

        return False

    # def get_price_db(self, user=None, password=None):
    #     if self.priceDb is None:
    #         db_name = self.config['price_db_name']

    #         # logDEBUG2("Loading PriceDB from %s" % dbFile)
    #         if user is None:
    #             user = self.config['price_db_user']
    #         if password is None:
    #             password = self.config['price_db_password']

    #         sqldb = PostgreSQLDB(dbName=db_name, user=user, password=password)
    #         self.priceDb = PriceDB(sqldb)

    #     # Now return that object:
    #     return self.priceDb

    def get_monitored_coins(self):
        """Retrieve the full list of monitored coins"""
        return self.config["monitored_coins"]

    def update_highres_datasets(self, cids=None, period=60):
        """High frequency dataset update of the selected coins"""
        if cids is None:
            cids = self.get_monitored_coins()

        logger.info("Should update the monitored coins: %s", cids)

        # start_time = self.get_time()

        # pdb = self.get_price_db()

        # last_datas = {}

        # # check for too long gaps in the datasets:
        # now_stamp = self.get_timestamp()


if __name__ == "__main__":
    # Create the context:
    context = NVPContext()

    # Add our component:
    comp = context.register_component("coingecko", Coingecko(context))

    context.define_subparsers("main", {'update-highres': None})

    # psr = context.get_parser('main')
    # psr.add_argument("--updateHighResDatasets", dest="update_highres", action="store_true",
    #                  help="Update the highres data from coingecko")

    comp.run()
</sxh>

  * The next step is to add support for the postgresql_db class, but this requires dependency on the **psycopg2**
  * Yet, of course, I dont want to keep adding more dependencies in my main NVP python environment only for those subprojects, so I definitely need a dedicated python env for this tool
  * But I also want to get the intellisense support from Visual Studio Code so I need to setup a **local** python env for the **NervHome** project: <sxh bash; highlight: []>$ nvp -p nvh admin init -p
  $ nvp home nvh
  # => Update the tools/requirements.txt file here.
  $ ./cli.sh --install-py-reqs</sxh>

  * **OK**, so after some good refactoring of the previous version of my coingecko class, I now finally have a new component for it integrated in my NervProj framework [//but in a dedicated python env as described above!//] yeepeee 🥳!

  * And retrieving the "highres" prices will work just fine (after a good debugging session of course): <sxh bash; highlight: []>$ nvp run defi_gecko_update
2022/05/07 17:52:29 [nvh.core.postgresql_db] INFO: Opening connection to crypto_prices_v2
2022/05/07 17:52:33 [__main__] INFO: Writing 8 new price entries
2022/05/07 17:52:33 [__main__] INFO: Done updating highres data for bitcoin
2022/05/07 17:52:37 [__main__] INFO: Writing 8 new price entries
2022/05/07 17:52:37 [__main__] INFO: Done updating highres data for ethereum
2022/05/07 17:52:41 [__main__] INFO: Writing 8 new price entries
2022/05/07 17:52:41 [__main__] INFO: Done updating highres data for litecoin
2022/05/07 17:52:45 [__main__] INFO: Writing 8 new price entries
2022/05/07 17:52:45 [__main__] INFO: Done updating highres data for binancecoin
2022/05/07 17:52:48 [__main__] INFO: Writing 9 new price entries
2022/05/07 17:52:48 [__main__] INFO: Done updating highres data for elrond-erd-2
2022/05/07 17:52:50 [__main__] INFO: Last written timestamp for bitcoin: 1651941900
2022/05/07 17:52:50 [__main__] INFO: Writing 1 new price observations for bitcoin
2022/05/07 17:52:50 [__main__] INFO: Last written timestamp for ethereum: 1651941900
2022/05/07 17:52:50 [__main__] INFO: Writing 1 new price observations for ethereum
2022/05/07 17:52:50 [__main__] INFO: Last written timestamp for litecoin: 1651941900
2022/05/07 17:52:50 [__main__] INFO: Writing 1 new price observations for litecoin
2022/05/07 17:52:50 [__main__] INFO: Last written timestamp for binancecoin: 1651941900
2022/05/07 17:52:50 [__main__] INFO: Writing 1 new price observations for binancecoin
2022/05/07 17:52:50 [__main__] INFO: Last written timestamp for elrond-erd-2: 1651942200
2022/05/07 17:52:50 [__main__] INFO: Updated all high res datasets in 20.780 seconds</sxh>

  * Yet I find those final "Writing 1 new price observations for" lines unexpected here... maybe I should try to look a bit more carefully into this 🤔, you know, just in case...
  * OOhhh, OK, now I kind of remember how this works: the first part with the "Writing 9 new price entries" comesfrom the retrieval of "market range" data in case our last entry is too far away in the past. And then the single added observation is from the just "retrieved price" which may or may not lead to a new entry. I think it sort of make sense in the end 😁.

  * => So all I have left to do now is to activate running that script every 5mins (yeah: initially I was considering running that every 1min, but for now I don't really think I need to be that precise. And later if I really need it, its only a default parameter value to change in the coingecko module function: <sxh python; highlight: []>def update_highres_datasets(self, cids=None, period=300):</sxh>

  * Created a some script **every_5mins_operations.sh** for this: <sxh bash; highlight: []>source /mnt/data1/dev/projects/NervProj/cli.sh

log_dir="/mnt/data1/admin/logs"

lfile="${log_dir}/coingecko_highres_updates.log"
nvp run defi_gecko_update 2>&1 | tee -a $lfile
</sxh>

  * And now deploying the support on my server machine: <sxh bash; highlight: []>$ nvp git pull
$ nvp -p nvh git pull
$ nvp pyenv setup defi_env
$ nvp run defi_gecko_update</sxh>

  * Finally installing the cron script with the entry: <sxh bash; highlight: []>*/5 * * * * su kenshin -c "bash /home/kenshin/cron/every_5mins_operations.sh"</sxh>
  * => Now just checking our new log file **coingecko_highres_updates.log** => All good in there 👍!


===== Adding support to store history datasets =====

  * Next logical step now is to also restore the support for the history data retrieval part: basically same thing as the highres data but only once per day... or is it per hour ? 🤔 I don't quite remember... [//arrff, well, actually it's a bit of both: day period for too old data, and otherwise hourly data for recent enough timestamps (ie. less than 3 years ago)//]

  * And here there is a new small issue which I need to address: the storage of the coins from coingecko, and the retrieval of their start timestamp: <sxh python; highlight: []>        # By default our start_stamp should be the date of release of the coin:
        start_stamp = self.getCoin(cid).getStartTimestamp()</sxh>
  
  * => So I basically also had to restore my previous **CoinDB** and **Coin** classes.

  * And now let's create a new script to call the 'update-history' command: <sxh javascript; highlight: []>    "gecko_update_history": {
      "custom_python_env": "defi_env",
      "cmd": "${PYTHON} nvh/defi/coingecko.py update-history",
      "cwd": "${PROJECT_ROOT_DIR}",
      "python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"]
    }</sxh>

  * And time to check the results: <sxh bash; highlight: []>$ nvp run gecko_update_history
2022/05/08 15:27:47 [__main__] INFO: Default start timestamp for bitcoin is: 1367107200
2022/05/08 15:27:47 [__main__] INFO: Creating history price table for bitcoin
2022/05/08 15:27:51 [__main__] INFO: Writing 89 new price entries
2022/05/08 15:27:56 [__main__] INFO: Writing 89 new price entries
2022/05/08 15:28:01 [__main__] INFO: Writing 89 new price entries
2022/05/08 15:28:06 [__main__] INFO: Writing 89 new price entries
2022/05/08 15:28:12 [__main__] INFO: Writing 89 new price entries
2022/05/08 15:28:16 [__main__] INFO: Writing 89 new price entries
2022/05/08 15:28:21 [__main__] INFO: Writing 89 new price entries
2022/05/08 15:28:26 [__main__] INFO: Writing 89 new price entries
</sxh>

  * **Not bad!** 🤪😎👍!

  * Next, as before, I should call this additional function once per day, let's add that in **daily_operations.sh**: <sxh bash; highlight: []>lfile="${log_dir}/coingecko_history_updates.log"
nvp run gecko_update_history 2>&1 | tee -a $lfile
</sxh>

===== Arbitrary arguments for nvp scripts =====

  * Continuing on this project, I need to handle my list of "monitored coins" properly: this is currently stored in a PostgreSQL table, and I'm only using a small "fixed list" read from config file in this refreshed version of my coingecko module.
  * I would like to restart that table and add the coins I want incrementally to keep things under control.
  * For this I need to support some additional command line arguments to:
    * list the current content of the table 
    * Remove elements from that table (drop the full table ?)
    * Add elements one by one to that list.
    * Remove a single element from that list.
  * => but obviously I'm not going to create a dedicated script for each of those commands. Instead I should find a way to pass arbitrary arguments to my scripts... let's see how this can be done...

  * Okay, so that was easy in the end, now just parsing the command line arguments with support for unknown args in the NVPContext: <sxh python; highlight: []>    def parse_args(self):
        """Parse the command line arguments"""
        # self.settings = vars(self.parsers['main'].parse_args())
        # cf. https://docs.python.org/3.4/library/argparse.html#partial-parsing
        self.settings, self.additional_args = self.parsers['main'].parse_known_args()
        self.settings = vars(self.settings)</sxh>
  * And then appending those args to the script command line itself in our runner component: <sxh python; highlight: []>        # Check if we have additional args to pass to the command:
        args = self.ctx.get_additional_args()
        if len(args) > 0:
            cmd += args

        # Execute that command:
        logger.info("Executing script command: %s (cwd=%s)", cmd, cwd)
        self.execute(cmd, cwd=cwd, env=env)</sxh>

  * I then added a **generic** script defined this way: <sxh javascript; highlight: []>    "coingecko": {
      "custom_python_env": "defi_env",
      "cmd": "${PYTHON} nvh/defi/coingecko.py",
      "cwd": "${PROJECT_ROOT_DIR}",
      "python_path": ["${PROJECT_ROOT_DIR}", "${NVP_ROOT_DIR}"]
    }</sxh>

  * And this seems to be working just fine: <sxh bash; highlight: []>$ nvp run coingecko update-history
2022/05/08 19:07:07 [nvp.components.runner] INFO: Executing script command: ['D:\\Projects\\NervProj\\.pyenvs\\defi_env\\python.exe',
'nvh/defi/coingecko.py', 'update-history'] (cwd=D:\Projects\NervHome)
2022/05/08 19:07:09 [__main__] INFO: Default start timestamp for bitcoin is: 1367107200
2022/05/08 19:07:13 [__main__] INFO: Writing 4 new price entries
2022/05/08 19:07:13 [__main__] INFO: Done updating history data for bitcoin
2022/05/08 19:07:13 [__main__] INFO: Default start timestamp for ethereum is: 1438905600
2022/05/08 19:07:17 [__main__] INFO: Writing 4 new price entries
2022/05/08 19:07:17 [__main__] INFO: Done updating history data for ethereum
2022/05/08 19:07:17 [__main__] INFO: Default start timestamp for litecoin is: 1367107200
2022/05/08 19:07:21 [__main__] INFO: Writing 4 new price entries
2022/05/08 19:07:21 [__main__] INFO: Done updating history data for litecoin
2022/05/08 19:07:21 [__main__] INFO: Default start timestamp for binancecoin is: 1505520000
2022/05/08 19:07:26 [__main__] INFO: Writing 4 new price entries
2022/05/08 19:07:26 [__main__] INFO: Done updating history data for binancecoin
2022/05/08 19:07:26 [__main__] INFO: Default start timestamp for elrond-erd-2 is: 1599091200
2022/05/08 19:07:30 [__main__] INFO: Writing 4 new price entries
2022/05/08 19:07:30 [__main__] INFO: Done updating history data for elrond-erd-2
</sxh>

  * Now updating my previous cron script to replace the **gecko_update_highres** and **gecko_update_history** by their respective command line (no need to defined those specific scripts anymore): **OK**

<note>One small issue I see with this "additional args" mechanism is that for instance this line ''nvp run coingecko --help'' will should the help for the run command: the --help argument is not "unknown" to it's collected as usual... 😐 Not quite sure how to deal with this, but well, I can live with this for now anyway.</note>

  * **Update**: well actually there is a simple workaround for that: I just added an optional command line arg ''--show-help'' to the runner parser, and when specified, this will add the ''--help'' to our script command line (tested and working 👍!): <sxh python; highlight: []>        if len(args) > 0:
            cmd += args

        if self.get_param("show_script_help", False):
            cmd += ["--help"]
</sxh>

===== Support to list/add/remove monitored coins =====

  * Now that I can pass arbitrary args to my scripts, I continued with adding control for the "monitored coins": <sxh python; highlight: []>    context.define_subparsers("main", {
        'update-highres': None,
        'update-history': None,
        'monitored': {
            'list': None,
            'add': None,
            'remove': None,
            'drop': None
        }
    })</sxh>
  * Implementing the corresponding action was easy: <sxh python; highlight: []>        if cmd == 'monitored':
            cmd1 = self.ctx.get_command(1)

            if cmd1 == 'list':
                # We should list all the monitored coins:
                mcoins = self.get_coin_db().get_all_monitored_coins()
                logger.info("List of monitored coins: %s", self.pretty_print(mcoins))
                return True
            if cmd1 == 'remove':
                # We should list all the monitored coins:
                mcoin = self.get_param("mcoin_name")
                self.get_coin_db().delete_monitored_coin(mcoin)
                logger.info("Removed monitored coin %s", mcoin)
                return True
            if cmd1 == 'add':
                # We should list all the monitored coins:
                mcoin = self.get_param("mcoin_name")
                # Convert to list:
                mcoin = mcoin.split(",")
                self.get_coin_db().insert_monitored_coins(mcoin)
                logger.info("Added monitored coins: %s", mcoin)
                return True
            if cmd1 == 'drop':
                logger.info("TODO: Should drop monitored coins table here.")
                return True</sxh>
  * And list/remove/add were tested successfully: <sxh bash; highlight: []>$ nvp run coingecko monitored list
2022/05/08 20:11:39 [__main__] INFO: List of monitored coins: [ 'bitcoin',
  'ethereum',
  'litecoin',
  'binancecoin',
  'elrond-erd-2',
  # ( large list here )
  'meld']</sxh>
  * <sxh bash; highlight: []>$ nvp run coingecko monitored remove meld
2022/05/08 20:12:27 [__main__] INFO: Removed monitored coin meld</sxh>
  * <sxh bash; highlight: []>$ nvp run coingecko monitored add meld
2022/05/08 20:12:52 [__main__] INFO: Added monitored coins: ['meld']</sxh>

<note>I can actually add multiple monitored coins at once separating them with a comma on the command line</note>

===== Restoring full history of coin prices =====

  * Now time to use that full list of monitored coins to get the corresponding history price [//this will take some time...//]:
  * First ensuring that that read the list of monitored coins from the database now: <sxh python; highlight: []>    def get_monitored_coins(self):
        """Retrieve the full list of monitored coins"""
        # return self.config["monitored_coins"]
        return self.get_coin_db().get_all_monitored_coins()
</sxh>
  * And then executing the command to get the data: <sxh bash; highlight: []>$ nvp run coingecko update-history</sxh>
  * All good: history data restored!
  * And I actually did the same for the highres data overnight, and now this is running smoothly also, as, in the ''coingecko_highres_updates.log'', I get: <code>2022/05/09 07:55:04 [__main__] INFO: Updated 69 highres datasets in 1.929 seconds
2022/05/09 08:00:05 [__main__] INFO: Updated 69 highres datasets in 2.541 seconds
2022/05/09 08:05:04 [__main__] INFO: Updated 69 highres datasets in 1.922 seconds
2022/05/09 08:10:04 [__main__] INFO: Updated 69 highres datasets in 2.024 seconds
2022/05/09 08:15:04 [__main__] INFO: Updated 69 highres datasets in 1.939 seconds</code>


===== Backing up those databases correctly =====

  * The next [//critical//] part to handle, is the backup of those new databases
  * First, let's check if I have an overall backup in place for my postgresql server: **Yes**, I have a backup element for that server in my backup config: <sxh javascript; highlight: []>      // Backup of postgresql server data:
      "postgresql_server": {
        "source_dir": "/mnt/data1/containers/postgresql_server",
        "backup_dir": "${slot1}/containers",
        "repository_url": "ssh://git@gitlab.nervtech.org:22002/backups/postgresql_server.git",
        "clone_dirs": [
          "${slot1}/git/archives_1",
          "${slot2}/git/archives_1",
          "${slot3}/git/archives_1"
        ]
      },</sxh>
  * So in theory, I don't really need to keep backups of the individual databases... but still, when it comes to backups, I now think that **the more you have, the better**, so let's have a look at how to backup individual databases anyway.

<note>"Why am I getting so obcessed with backups ?" you may ask... well, as I said at the beginning of this post I got a recurring corruption issue on one disc in a RAID 6 array of 7 discs where I was basically storing all my data for, well, everything. So Everything I was getting a new problem, broken docker container, corrupted database, lost files, etc... a nightmare. And I don't want this to happen to me anymore, no way lol. </note>

  * Basically I need a command like this one I believe: <sxh bash; highlight: []>docker exec -t postgresql_server pg_dump crypto_coins -c -U crypto_user > my_dump.sql</sxh>
  * That command seems to work just fine, so now I need to integrate this as part of my "backup" component processing... let's get to work.

  * Starting with a simple backup target definition as follow: <sxh javascript; highlight: []>      // Backup of individual SQL databases:
      "postgresql_crypto_databases": {
        "type": "postgresql",
        "container": "postgresql_server",
        "user": "crypto_user",
        "databases": ["crypto_coins", "crypto_prices_v2"],
        "backup_dirs": [
          "${slot1}/sql/crypto",
          "${slot2}/sql/crypto",
          "${slot3}/sql/crypto"
        ]
      },</sxh> /*//*/
  
  * And preparing the corresponding handling method in the **BackupManager** component: <sxh python; highlight: []>    def backup_postgresql_databases(self, tgt_name, desc):
        """Handle a postgresql database backup target"""

        logger.info("Should backup postgresql databases with desc: %s", desc)</sxh>
  
  * let's check if this works: <sxh bash; highlight: []>$ nvp backup run postgresql_crypto_databases
2022/05/09 08:21:58 [components.backup_manager] INFO: Should backup postgresql databases with desc: {'type': 'postgresql', 'container': '
postgresql_server', 'user': 'crypto_user', 'databases': ['crypto_coins', 'crypto_prices_v2'], 'backup_dirs': ['${slot1}/sql/crypto', '${s
lot2}/sql/crypto', '${slot3}/sql/crypto']}
2022/05/09 08:21:58 [components.backup_manager] INFO: Backup target postgresql_crypto_databases processed in 0.00 seconds</sxh>

  * => **OK**, now let's add some content in there.

  * Alright, it's working just fine now 👍! Here are the main functions I added in my Backup manager component to handle this process: <sxh python; highlight: []>    def backup_postgresql_databases(self, tgt_name, desc):
        """Handle a postgresql database backup target"""

        # prepare the work folder:
        backup_dirs = self.collect_valid_slot_paths(desc["backup_dirs"])
        num_bdirs = len(backup_dirs)
        if num_bdirs == 0:
            logger.warning("No valid backup dir provided for %s", tgt_name)
            return

        logger.info("postgresql db backup dirs: %s", backup_dirs)

        # Prepare the work folder:
        work_dir = backup_dirs[0]

        # Ensure our work folder is created:
        self.make_folder(work_dir)

        # Iterate on each database:
        dbs = desc["databases"]
        user = desc["user"]
        container = desc["container"]
        now = self.get_now()
        now_str = now.strftime("%Y%m%d_%H%M%S")

        tools = self.get_component('tools')

        btype = self.get_rolling_backup_type(now.date())
        suffix = f"{btype}_{now_str}.sql"

        for db_name in dbs:
            # Prepare the command line:
            cmd = ["docker", "exec", "-t", container, "pg_dump", db_name, "-c", "-U", user]
            filename = f"{db_name}_{suffix}"
            outfile = self.get_path(work_dir, filename)

            logger.info("Dumping database %s...", db_name)

            with open(outfile, "w", encoding="utf-8") as file:
                self.execute(cmd, outfile=file)

            # Next we should compress that file:
            logger.info("Generating archive for %s...", db_name)

            pkg_name = f"{filename}.tar.xz"
            tools.create_package(outfile, work_dir, pkg_name)

            # remove the source sql file:
            self.remove_file(outfile)
            pkg_file = self.get_path(work_dir, pkg_name)

            dest_folder = self.get_path(work_dir, db_name)
            self.make_folder(dest_folder)
            dest_file = self.get_path(dest_folder, pkg_name)
            self.rename_file(pkg_file, dest_file)

            self.remove_old_backup_files(dest_folder)

            # Finally we also copy that file into the additional backup slots:
            for i in range(1, num_bdirs):
                bck_dir = backup_dirs[i]
                dest_folder = self.get_path(bck_dir, db_name)
                self.make_folder(dest_folder)

                logger.debug("Copying %s into %s", pkg_name, dest_folder)
                self.copy_file(dest_file, self.get_path(dest_folder, pkg_name))

                # Remove the old backups there too:
                self.remove_old_backup_files(dest_folder)

    def get_rolling_backup_type(self, date):
        """Retrieve the type of rolling backup based on the given date"""
        if date.day == 1:
            # first day of the month:
            return "mbak"

        if date.weekday() == 0:
            # This is a monday:
            return "wbak"

        # ordinary day backup:
        return "dbak"

    def remove_old_backup_files(self, folder):
        """Remove the too old backup files in a given folder"""

        files = self.get_all_files(folder, recursive=True)

        now_stamp = self.get_timestamp()

        # Iterate on each file:
        for fname in files:

            if "_mbak_" in fname:
                # keep for 366 days!
                offset = 366*24*3600
            elif "_wbak_" in fname:
                # keep for 28 days:
                offset = 28*24*3600
            elif "_dbak_" in fname:
                # Keep the files for 7 days
                offset = 7*24*3600
            else:
                logger.warning("Ignoring non-rolling backup file %s", fname)
                continue

            min_stamp = now_stamp - offset

            # Check the file timestamp:
            file_path = self.get_path(folder, fname)
            mtime = self.get_file_mtime(file_path)
            if mtime < min_stamp:
                logger.info("Removing old backup file %s", file_path)
                self.remove_file(file_path)
            else:
                ndays = (mtime - min_stamp)/(24*3600)
                logger.debug("Keeping %s for %.3f more days", file_path, ndays)</sxh>

  * A bit long and complex sure, but its doing its job:
    * Dumping the databases to an SQL file,
    * Generating a tar.xz of that sql dump,
    * Placing that into the correct backup folder,
    * Duplicating the backup file in all the provided backup slots,
    * Removing the old backup files from all those slots.
  
  * I also considered generating PAR2 recovery files for those SQL archives, with something like that: <sxh python; highlight: []># Create The par files:
redun = 10
pkg_file = self.get_path(work_dir, pkg_name)
fsize = self.get_file_size(pkg_file)

# We allocate 1 block per 1kB with min=10, max=3000
nblocks = max(min(fsize/1024, 3000), 10)
logger.info("Generating PAR2 files with %d source blocks...", nblocks)
tools.create_par2_archives(pkg_file, redun, nblocks)</sxh>

  * But in the end I think this is a bit of an overkill, so I removed that layer ;-).

===== Conclusion & next steps =====

  * This post is getting a bit long now, so I will stop it here for this time.
  * I now have the history and highres data stored on my postgresql server again (and continuously updated), so naturally the next step will be to restore the GUI application itself, so I could finally get back to crying in front of so much red graphs I will get out of this 🤣 [yeahh... we are the 10th May 2022, crypto currencies are getting completely destroyed right now... UST depeg, BTC -20%, etc... you know: the usual fun part lol]


  * => So, see you next time ✌!