NervLand: Setting up proper C++ development environment
Continuing on my chaotic journey to build the next master piece 3D game everyone will want to play, I'm finally back our good old C++ 🤣 But this time I'm going to do this correctly! [Are you Manu ? Really ? 🙃] ⇒ I need a proper C++ development environment, not just Visual Studio Code without any kind of intellisense/linter/formatter etc. I need something to boost my productivity…
- I'm starting with a brand new Visual Studio 2022 installation… except that (of course) it's already complaining that it is not fully compatible with my Windows 10 “Arium” version lol, but I'm continuing anyway, and we will see where this leads us. OK VisualStudio 2022 installed.
- Next step is to prepare the dedicated gitlab project: this time I'm not going to “just put everything” into my main “NervSeed” project: that one is getting far too messy and going in all directions 😅 it's still soemthing definitely needed since that the fastest way “to learn”, but here it's rather about building something extra clean. ⇒ OK, created our new NervLand project on gitlab.
- Checking out the initial repository:
$ git clone ssh://git@gitlab.nervtech.org:22002/nerv/nervland.git NervLand
- Next I need to setup the environment scripts to access that project, I will use the project dedicated prefix nvl. Also preparing a script to run the “CLI”:
nvl_cli() { "`nvl_get_project_dir`/scripts/cli.sh" "$@" }
- Note: Just found this cheatsheet on C++ standards: https://github.com/AnthonyCalandra/modern-cpp-features
- Also needed to config git in cygwin after my latest upgrade:
$ git config --global core.editor nano
- First thing I need to build before starting the project itself are some minimal dependencies: Boost and SDL.
- But here I want to use Python to control the project build environment instead of raw bash/batch scripts, so I need to embed that too. OK
- ⇒ So I have an initial scripts/cli.bat batch file that will take care of setting up the python tool package on windows if missing:
@echo off SETLOCAL ENABLEDELAYEDEXPANSION @REM Retrieve the current folder: cd /D %~dp0.. FOR /F %%i IN (".") DO set NVL_ROOT_DIR=%%~fi set NVL_DIR=%NVL_ROOT_DIR% echo Using NervLand root folder: %NVL_DIR% @REM Extract the python env if needed: set TOOLS_DIR=%NVL_DIR%\tools\windows\ set UNZIP=%TOOLS_DIR%\7zip\7za.exe set PYTHON=%TOOLS_DIR%\python\python.exe @REM Check if python is extracted already: if not exist "%PYTHON%" ( echo Extracting python tool... %UNZIP% x -o"%TOOLS_DIR%" "%TOOLS_DIR%\python-win.7z" > nul @REM Upgrade pip: %PYTHON% -m pip install --upgrade pip )
- ⇒ Now time to have a look at the boost dependency…
- Disabling filemode in project .git/config as this is confusing VisualStudio Code.
- Built minimal cli.sh script to call the cli.bat on windows:
#!/usr/bin/env bash # cf. https://stackoverflow.com/questions/59895/how-can-i-get-the-source-directory-of-a-bash-script-from-within-the-script-itsel SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) # On windows we should simply rely on the cli.bat script below: SCRIPT_DIR="`cygpath -w $SCRIPT_DIR`" cmd /C "$SCRIPT_DIR\\cli.bat" "$@"
- Next I should also ensure I have an approprite python setup in visual studio code: so preparing the settings accordingly:
{ "python.linting.pylintEnabled": true, "python.linting.enabled": true, "python.linting.pylintPath": "${workspaceFolder}\\tools\\windows\\python\\Scripts\\pylint.exe", "python.linting.pylintArgs": [ "--max-line-length=120", ], "python.defaultInterpreterPath": "${workspaceFolder}\\tools\\windows\\python\\python.exe", "python.formatting.autopep8Path": "${workspaceFolder}\\tools\\windows\\python\\Scripts\\autopep8.exe", "python.formatting.provider": "autopep8", "python.formatting.autopep8Args": [ "--max-line-length=120", "--experimental" ], "editor.formatOnSave": true, // "python.envFile": "${workspaceFolder}/.vs_env" }
- Now working my way in my cli.py script to support building boost:
- The cli script create an instance of NVLBuilder
- In there, we call check_dependencies() to check all dependencies
- This may lead to deploy_dependency() if deployment is needed
- And this in turn can lead to extract_dependency_package if we already have a pre-built dep package
- Or it will call the proper _build_xxx() method if building from sources is required.
- Now we need to implement that _build_boost() method…
- OK: I now have a working mechanism to build boos from sources, and generated a boost-1.78.0-msvc64.7z package from the resulting build if that pacakge is missing.
- My cli.py file is getting a bit large already, but let's first consider adding an additional dependency before splitting it into smaller elements.
- Note: to build the dependencies I simply need to execute:
nvl_cli --check-deps -v
- Next logical step would be compilation of the SDL2 library…
- Additional note: I'm cheating a little bit here: for the boost and SDL2 library, I'm currently mostly following the builder implementation I already implemented some time ago in python in my NervSeed project (cf. python/modules/nv/core/builders.py)
- ⇒ OK: SDL2 version 2.0.20 now compiled (⇒ generated SDL2-2.0.20-msvc64.7z package): this was pretty quick in fact.
- Next stop should be LuaJIT: I will want support for scripting at some point and LuaJIt seems the best option to me (even if it's not really evolving anymore…)
- All good! LuaJIT now compiled directly from gti repository (so I'm also adding Ggit as part of the tool packages)
- So what should I do next ? I'm thinking that now I should really consider preparing the support on a linux system too. So let's prepare an ubuntu docker image to handle that: this way I can always rebuild that machine from scratches as needed and I could even maybe use the same image to build a CI environment in gitlab when we get to this point later ?
- First thing to do is to checkout the project on a linux base system (so neptune first here):
git clone ssh://git@gitlab.nervtech.org:22002/nerv/nervland.git NervLand
- Next I updated my cli.sh script to support setting python as needed on linux:
#!/usr/bin/env bash # cf. https://stackoverflow.com/questions/59895/how-can-i-get-the-source-directory-of-a-bash-script-from-within-the-script-itsel SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) run_cli_windows() { # On windows we should simply rely on the cli.bat script below: SCRIPT_DIR="`cygpath -w $SCRIPT_DIR`" cmd /C "$SCRIPT_DIR\\cli.bat" "$@" } run_cli_linux() { # On linux we should call the python cli directly: # Get the project root folder: local root_dir=`readlink -f $SCRIPT_DIR/../` echo "NervLand root dir is: $root_dir" # Check if we already have python: local tools_dir=$root_dir/tools/linux local tmp_dir=$root_dir/temp if [[ ! -d $tmp_dir ]]; then echo "Creating temp folder..." mkdir $tmp_dir fi local unzip_dir=$tools_dir/7zip local unzip_path=$unzip_dir/7zzs if [[ ! -d $unzip_dir ]]; then echo "Extracting 7zip tool..." pushd $tools_dir > /dev/null tar xvJf 7zip.tar.xz popd > /dev/null fi local python_dir=$tools_dir/python local python_path=$python_dir/bin/python3 if [[ ! -d $python_dir ]]; then # Check if we already have the python.7z if [[ -e "${python_dir}.7z" ]]; then echo "Extracting $python_dir.7z..." $unzip_path x -o"$tools_dir" "${python_dir}.7z" > /dev/null else local pyversion="3.10.2" local pyfolder="Python-$pyversion" local tarfile="$pyfolder.tar.xz" local url="https://www.python.org/ftp/python/$pyversion/$tarfile" pushd $tmp_dir > /dev/null # Remove any previous build folder: if [[ -d $pyfolder ]]; then echo "Removing previous $pyfolder..." rm -Rf $pyfolder fi if [[ -e $tarfile ]]; then echo "Removing previous $tarfile..." rm -Rf $tarfile fi echo "Downloading python sources from $url" wget -O $tarfile $url tar xvJf $tarfile # Enter into the python source folder: pushd $pyfolder > /dev/null # should ensure that the dependency packages are installed (?) # sudo apt-get install libbz2-dev liblzma-dev echo "Configuring python..." ./configure --enable-optimizations --prefix=$python_dir.tmp CFLAGS=-fPIC CXXFLAGS=-fPIC # --enable-loadable-sqlite-extensions --with-system-expat --with-system-ffi CPPFLAGS=-I/usr/local/include LDFLAGS=-L/usr/local/lib echo "Building python..." # Note: Building with optimizations is very slow: make echo "Installing python..." make install popd > /dev/null popd > /dev/null # Now we rename the destination folder: mv $python_dir.tmp $python_dir # And we create the 7z package: echo "Generating python tool package..." $unzip_path a -t7z $python_dir.7z $python_dir -m0=lzma2 -mx=9 -aoa -mfb=64 -md=32m -ms=on -r # removing python build folder: echo "Removing python build folder..." rm -Rf temp/$pyfolder* echo "Done generating python package." fi # Once we have deployed the base python tool package we start with upgrading pip: echo "Upgrading pip..." $python_path -m pip install --upgrade pip # Finally we install the python requirements: echo "Installing python requirements..." $python_path -m pip install -r $root_dir/tools/requirements.txt fi # Execute the command in python: $python_path $root_dir/scripts/cli.py "$@" } # Check if we are on a windows or a linux system: pname=`uname -s` case $pname in CYGWIN*) run_cli_windows "$@" ;; *) run_cli_linux "$@" ;; esac
- Now it's time to select a compiler on linux 🤔… Basically gcc or clang, right ? I'm not quite sure if there is a “best option” here but anyway, clang seems a bit either to build, so i'm thinking I should start with a try on that one first, let's go!
- Clang getting started page: https://clang.llvm.org/get_started.html
- ⇒ I will need cmake as a tool here, so let's get that on linux. ⇒ using version 3.22.3 OK
- Instruction on how to build GCC 11: https://iamsorush.com/posts/build-gcc11/
- And I could eventually setup a build python function correctly for boost:
def _build_boost_linux64(self, build_dir, prefix, desc): """Build method for boost with linux64 compiler.""" comp = self.get_compiler_config() # cf. https://gist.github.com/Shauren/5c28f646bf7a28b470a8 # Note: the bootstrap.sh script above is crap, so instead we build b2 manually ourself here: bs_cmd = ["./tools/build/src/engine/build.sh", "clang", f"--cxx={comp['path']}", f"--cxxflags={comp['cxxflags']}" ] logger.info("Building B2 command: %s", bs_cmd) self.execute(bs_cmd, cwd=build_dir) bjam_file=self.get_path(build_dir, "bjam") self.copy_file(self.get_path(build_dir, "tools/build/src/engine/b2"), bjam_file) self.add_execute_permission(bjam_file) with open(os.path.join(build_dir, "user-config.jam"), "w") as file: # Note: Should not add the -std=c++11 flag below as this will lead to an error with C files: file.write(f"using clang : : {comp['path']} : <compileflags>\"{comp['cxxflags']} -fPIC\" <linkflags>\"{comp['linkflags']}\" ;\n") # Note: below we need to run bjam with links to the clang libraries: build_env = os.environ.copy() build_env['LD_LIBRARY_PATH'] = comp['library_path'] bjam_cmd = ['./bjam', "--user-config=user-config.jam", "--buildid=clang", "-j", "8", "toolset=clang", "--prefix="+prefix, "--without-mpi", "-sNO_BZIP2=1", "architecture=x86", "variant=release", "link=static", "threading=multi", "target-os=linux", "address-model=64", "install"] logger.info("Executing bjam command: %s", bjam_cmd) self.execute(bjam_cmd, cwd=build_dir, env=build_env) - Note on the code above: using “bootstrap.sh” directly doesn't seem to work in my configuration (I'm currently simply using the prebuild clang 13.0.1 package downloaded from that link: https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.1/clang+llvm-13.0.1-x86_64-linux-gnu-ubuntu-18.04.tar.xz). So instead I had to call the b2 buid/sh script manually myself, not quite sure if this is the best option, but at least it is working this way.
- Note: some instructions on how to configure bjam for boost: https://www.boost.org/doc/libs/1_78_0/tools/build/doc/html/index.html
- And now I'm trying to get the SDL2 library to compile with clang, but this is less straightforward thann what I expected 🤔… I keep getting an error related to the call to 'atexit' in src/core/linux/SDL_evdev_kbd.c.
- ⇒ Arrff, okay, my mistake was to try to assing some C++ compiler/linker flags (ie stdlib=c++) to the C compiler, build script now updated to handle that properly in the construction of the build environment:
def _build_sdl2_linux64(self, build_dir, prefix, _desc): """Build method for sdl2 with linux64 compiler.""" build_env = self.setup_compiler_env() logger.info("Using CXXFLAGS: %s", build_env['CXXFLAGS']) cmd = [self.get_cmake_path(), "-DCMAKE_BUILD_TYPE=Release", "-DSDL_STATIC=ON", "-DSDL_STATIC_PIC=ON", f"-DCMAKE_INSTALL_PREFIX={prefix}", ".."] logger.info("Executing SDL2 build command: %s", cmd) self.execute(cmd, cwd=build_dir+"/src", env=build_env) # Build command: self.execute(["make"], cwd=build_dir+"/src", env=build_env) # Install command: self.execute(["make", "install"], cwd=build_dir+"/src", env=build_env)
- Now let's continue with the LuaJIT build script! Hmmm… No explicit instructions to build with clang on linux, so let's see if we can figure something out anyway… Well, that was not too hard in the end 😆:
def _build_luajit_linux64(self, build_dir, prefix, desc): """Build method for luajit with linux64 compiler.""" # End finally make install: build_env = self.setup_compiler_env() self.execute(["make", "install", f"PREFIX={prefix}", "HOST_CC=clang"], cwd=build_dir, env=build_env) # We should rename the include sub folder: "luajit-2.1" -> "luajit" dst_name=self.get_path(prefix, "include", "luajit") self.rename_folder(f"{dst_name}-{desc['version']}", dst_name)
- In the process I also updated the --check-deps command line entry on the nvl_cli to support listing the dependencies that should be checked/built (default to “all”) and added a --rebuild entry to force rebuilding from sources instead of using the prebuild 7z package. So we can now do something like that:
$ nvl_cli --check-deps luajit --rebuild -v 2>&1 | tee build_luajit.log
- Okay okay, all good so far. Now I was thinking I should continue with setting up the gitlab CI system but there is something more critical that just came into my mind: eventually I want to share a lot of this “dependency” + “project management” with my other main professional project (simcore), and I don't want to duplicate all of this… So I'm now thinking I should try to extract this “layer” into a dedicated project that I will store on github to all sharing between public/private projects 👍!
- ⇒ I think this deserve a dedicated page actually, so let's start a new one now 🤠.