NervProj: support for custom python env in scripts

In addition to the support for “dedicated” python environment for a given sub project I described in this previous article I'm now realizing there is another feature that could be of great help to me: the possibility to generate any kind of python environment in a given folder using a list of required packages.

My idea on this point is that I could then easily setup a “testing environment” with all the packages I want in python + the jupyter notebook package, and then running a simple script, I could instantly start coding test snippets in a jupyter book using that custom environment without interfering with any other project, which I think would be super cool! So let's get started on this 😎.

• The python custom environments should be described with an arbitrary name, and a list of required packages,
• We should support reading those configs either from the main Nervproj config.json file or from any sub project loaded at start, so any sub project could effectively define one or more envs to test different things before making the actual changes in a project “production” environment.

• And here we go: I just added a NervProj component called PyEnvManager which can use use to handle the setup of those custom environments:

  """Collection of admin utility functions"""
  import logging
  
  from nvp.nvp_component import NVPComponent
  from nvp.nvp_context import NVPContext
  
  logger = logging.getLogger(__name__)
  
  
  def register_component(ctx: NVPContext):
      """Register this component in the given context"""
      comp = PyEnvManager(ctx)
      ctx.register_component('pyenvs', comp)
  
  
  class PyEnvManager(NVPComponent):
      """PyEnvManager component used to run scripts commands on the sub projects"""
  
      def __init__(self, ctx: NVPContext):
          """Script runner constructor"""
          NVPComponent.__init__(self, ctx)
  
          self.scripts = ctx.get_config().get("scripts", {})
  
          # Also extend the parser:
          ctx.define_subparsers("main", {'setup-pyenv': None})
          psr = ctx.get_parser('main.setup-pyenv')
          psr.add_argument("env_name", type=str,
                           help="Name of the python environment to setup/deploy")
          psr.add_argument("--dir", dest='env_dir', type=str,
                           help="Location where to install the environment")
          psr.add_argument("--renew", dest='renew_env', action='store_true',
                           help="Rebuild the environment completely")
          psr.add_argument("--update-pip", dest='update_pip', action='store_true',
                           help="Update pip module")
  
      def process_command(self, cmd):
          """Check if this component can process the given command"""
  
          if cmd == 'setup-pyenv':
  
              env_name = self.get_param('env_name')
              self.setup_py_env(env_name)
              return True
  
          return False
  
      def setup_py_env(self, env_name):
          """Setup a given python environment"""
  
          # If there is a current project we first search in that one:
          proj = self.ctx.get_current_project()
          desc = None
          if proj is not None:
              desc = proj.get_custom_python_env(env_name)
  
          if desc is None:
              # Then search in all projects:
              projs = self.ctx.get_projects()
              for proj in projs:
                  desc = proj.get_custom_python_env(env_name)
                  if desc is not None:
                      break
  
          if desc is None:
              all_envs = self.config.get("custom_python_envs")
              desc = all_envs.get(env_name, None)
  
          assert desc is not None, f"Cannot find python environment with name {env_name}"
  
          env_dir = self.get_param("env_dir")
  
          default_env_dir = self.get_path(self.ctx.get_root_dir(), ".pyenvs")
  
          if env_dir is None:
              # try to use the install dir from the desc if any or use the default install dir:
              env_dir = desc.get("install_dir", default_env_dir)
  
          # create the env folder if it doesn't exist yet:
          dest_folder = self.get_path(env_dir, env_name)
  
          tools = self.get_component("tools")
          new_env = False
  
          if self.dir_exists(dest_folder) and self.get_param('renew_env'):
              logger.info("Removing previous python environment at %s", dest_folder)
              self.remove_folder(dest_folder)
  
          if not self.dir_exists(dest_folder):
              # Should extract the python package first:
              logger.info("Extracting python package to %s", dest_folder)
              pdesc = tools.get_tool_desc("python")
              ext = "7z" if self.is_windows else "tar.xz"
              filename = f"python-{pdesc['version']}-{self.platform}.{ext}"
              pkg_file = self.get_path(self.ctx.get_root_dir(), "tools", "packages", filename)
  
              tools.extract_package(pkg_file, env_dir, target_dir=dest_folder, extracted_dir=f"python-{pdesc['version']}")
              new_env = True
  
          if new_env or self.get_param("update_pip"):
              # trigger the update of pip:
              py_path = self.get_path(dest_folder, pdesc['sub_path'])
              logger.info("Updating pip...")
              self.execute([py_path, "-m", "pip", "install", "--upgrade", "pip"])
  
          # Next we should prepare the requirements file:
          req_file = self.get_path(dest_folder, "requirements.txt")
          content = "\n".join(desc["packages"])
          self.write_text_file(content, req_file)
  
          logger.info("Installing python requirements...")
          self.execute([py_path, "-m", "pip", "install", "-r", req_file])
  

• With that I can for instance deploy an environment with a command such as:

  nvp setup-pyenv resp_t7

• The custom python environments can be defined in the main Nervproj config file, but I don't have any definition there yet
• Or in a given sub project config, for instance as follow:

    "custom_python_envs": {
      "resp_t7": {
        "packages": [
          "PyQt5",
          "vispy",
          "numpy",
          "Pillow",
          "jstyleson",
          "matplotlib"
        ]
      }
    },

• Next, as I said, the idea is to use such an environment on a script for instance:

    "scripts": {
      "t7_jupyter": {
        "custom_python_env": "resp_t7",
        "cmd": "${PYTHON} -m jupyter lab",
        "cwd": "${PROJECT_ROOT_DIR}",
        "python_path": ["${PROJECT_ROOT_DIR}"]
      }
    }

• ⇒ In the script above, the ${PYTHON} replacement should point to the python path in the specified custom env. Let's update our Runner component accordingly…

• And here is the updated run_script method taking into account the potential custom python env for a given script:

      def run_script(self, script_name: str, proj: NVPProject | None):
          """Run a given script on a given project"""
  
          # Get the script from the config:
          desc = None
          if proj is not None:
              desc = proj.get_script(script_name)
  
          if desc is None:
              desc = self.scripts.get(script_name, None)
  
          if desc is None:
              logger.warning("No script named %s found", script_name)
              return
  
          cmd = self.fill_placeholders(desc['cmd'], proj)
  
          # check if we should use python in this command:
          tools = self.get_component('tools')
          env_name = desc.get("custom_python_env", None)
  
          if env_name is not None:
              # Get the environment dir:
              pyenv = self.get_component("pyenvs")
  
              env_dir = pyenv.get_py_env_dir(env_name)
              pdesc = tools.get_tool_desc("python")
              py_path = self.get_path(env_dir, env_name, pdesc['sub_path'])
          else:
              # use the default python path:
              py_path = tools.get_tool_path('python')
  
          cmd = cmd.replace("${PYTHON}", py_path)
          cmd = cmd.split(" ")
          cmd = [el for el in cmd if el != ""]
  
          cwd = self.fill_placeholders(desc.get('cwd', None), proj)
  
          env = None
          if "python_path" in desc:
              elems = desc["python_path"]
              elems = [self.fill_placeholders(el, proj).replace("\\", "/") for el in elems]
              sep = ";" if self.is_windows else ":"
              pypath = sep.join(elems)
              logger.info("Using python path: %s", pypath)
              env = os.environ.copy()
              env['PYTHONPATH'] = pypath
  
          # Execute that command:
          logger.debug("Executing script command: %s (cwd=%s)", cmd, cwd)
          self.execute(cmd, cwd=cwd, env=env)

• OK then I try to run my script:

  $ nvp run t7_jupyter

• And I get the expected error (because I intentionally did not install jupyter yet):

  D:\Projects\NervProj\.pyenvs\resp_t7\python.exe: No module named jupyter
  Traceback (most recent call last):
    File "D:\Projects\NervProj\cli.py", line 5, in <module>
      ctx.run()
    File "D:\Projects\NervProj\nvp\nvp_context.py", line 315, in run
      if comp.process_command(cmd):
    File "D:\Projects\NervProj\nvp\components\runner.py", line 39, in process_command
      self.run_script(sname, proj)
    File "D:\Projects\NervProj\nvp\components\runner.py", line 113, in run_script
      self.execute(cmd, cwd=cwd, env=env)
    File "D:\Projects\NervProj\nvp\nvp_object.py", line 383, in execute
      subprocess.check_call(cmd, stdout=stdout, stderr=stderr, cwd=cwd, env=env)
    File "D:\Projects\NervProj\tools\windows\python-3.10.1\lib\subprocess.py", line 369, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['D:\\Projects\\NervProj\\.pyenvs\\resp_t7\\python.exe', '-m', 'jupyter', 'lab']' returned non-zero exit statu
  s 1.

• ⇒ Good thing is we see have that we are using the correct location for python.exe, yeahh ✌!

• Now let's add jupyterlab to the list of requirements:

    "custom_python_envs": {
      "resp_t7": {
        "packages": [
          "PyQt5",
          "vispy",
          "numpy",
          "Pillow",
          "jstyleson",
          "matplotlib",
          "jupyterlab"
        ]
      }
    },

• And we require an update of the python env (to install the additional requirement) simply calling again:

  nvp setup-pyenv resp_t7

• And then I can run my command again to start jupyter:

  $ nvp run t7_jupyter
  [I 2022-05-03 16:15:08.043 ServerApp] jupyterlab | extension was successfully linked.
  [I 2022-05-03 16:15:08.055 ServerApp] nbclassic | extension was successfully linked.
  [I 2022-05-03 16:15:08.062 ServerApp] Writing Jupyter server cookie secret to C:\Users\kenshin\AppData\Roaming\jupyter\runtime\jupyter_cookie_secret
  [I 2022-05-03 16:15:08.643 ServerApp] notebook_shim | extension was successfully linked.
  [I 2022-05-03 16:15:08.791 ServerApp] notebook_shim | extension was successfully loaded.
  [I 2022-05-03 16:15:08.793 LabApp] JupyterLab extension loaded from D:\Projects\NervProj\.pyenvs\resp_t7\lib\site-packages\jupyterlab
  [I 2022-05-03 16:15:08.793 LabApp] JupyterLab application directory is D:\Projects\NervProj\.pyenvs\resp_t7\share\jupyter\lab
  [I 2022-05-03 16:15:08.797 ServerApp] jupyterlab | extension was successfully loaded.
  [I 2022-05-03 16:15:08.807 ServerApp] nbclassic | extension was successfully loaded.
  [I 2022-05-03 16:15:08.808 ServerApp] Serving notebooks from local directory: D:\Projects\resp_t7
  [I 2022-05-03 16:15:08.808 ServerApp] Jupyter Server 1.17.0 is running at:
  [I 2022-05-03 16:15:08.809 ServerApp] http://localhost:8888/lab?token=a6abfc20422a96fe4aaba853b40983f846249d2ceb46dcf4
  [I 2022-05-03 16:15:08.809 ServerApp]  or http://127.0.0.1:8888/lab?token=a6abfc20422a96fe4aaba853b40983f846249d2ceb46dcf4
  [I 2022-05-03 16:15:08.809 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
  [C 2022-05-03 16:15:08.922 ServerApp]
  

• Oh my god 😳! That is working sooooo well I can hardly believe it lol:

• All right! So with that I can now start playing around with some jupyter books, with all the packages I need and zero risk to mess anything in another project or a production env, perfect for me 👍!