NervLand Task list

DevLog 11 and previous

✅ Add support for lua-cjson library

✅ Add support for luasockets library
✅ Add support for lua MD5 library
✅ Add support for lua xml library
✅ Add support for openssl library
✅ Add support for libcurl library
- Comparing SSL engines: https://curl.se/docs/ssl-compared.html
✅ Add support for luasec library

✅ Add support to execute OpenSSL tests:
- Create the tests/luasec/server.lua and tests/luasec/client.lua files
- Generating the required certificates in assets/test_data/certs with the commands:
```
nvp gen-cert rootA rootA.cnf
nvp gen-cert clientA clientA.cnf -r rootA
nvp gen-cert serverA serverA.cnf -r rootA
```

✅ Add support for lua base64 library

✅ Add support for busted (for unit tests) in lua:
- Documentation at https://lunarmodules.github.io/busted/
- Add support for lua-term: https://github.com/roche-emmanuel/lua-term OK
- Add support for dkjson: OK
- Add support for lua cliargs: OK
- Add support for say, luaassert, luasystem, mediator_lua: OK

✅ Add support for lua-requests library

✅ Generate bindings for libcurl

⚠️ Add support for cpr library (?)
- cf. https://github.com/roche-emmanuel/cpr
- But no: will not add this now: we can just as well use libcurl directly, and this will be far less trouble for bindings generation.

✅ Fix issue with destruction of QMetaObject::Connection:
- Receiving this message on garbage collection: “Fatal exception: No public destructor available for class QMetaObject::Connection”
- OK ⇒ ignoring DDL import in QT bindings.

✅ Introduce initial support for GET request using libcurl in lua:
- Considering how to dispose of the curl_easy object: cannot use luajit directly for this.
- But we have the gc metamethod for the userdata: let's try to attach a lua callback there.
- OK We can now attach custom gc callback to a luna userdata as follow:
```
    self.easy = curl.easy_init()

    __luna.set_gc_callback(self.easy, function(obj)
        logDEBUG("Releasing curl easy object.")
        curl.easy_cleanup(obj)
        logDEBUG("Done releasing curl object.")
    end)
```
- Considering building the Request object in C++ directly instead ?
  - That's interesting, but the idea of using Lua is really to be able to prototype such constructions more quickly. If we build the class in C++ directly, then every change we make on this class will require re-generating the bindings and rebuilding the shared modules.
  - ⇒ So it's a better idea to keep C++ implementation only for the most critical performance elements or for finalized constructions only. And here we should stick with lua implementation instead.
- So we need fixing for the easy_setopt function…
- Arrfff… reconsidering the C++ implementation again: otherwise the support for setopt is pretty complex to build, so we might jsut as well build the C++ version already.
- Next, I got a PEER_FAILED_VERIFICATION error, thus checked with verification disable:
```
self.req:set_opt(curl.option.SSL_VERIFYPEER, false)
```
- ⇒ This works, so I need to do something about the SSL certificates…
- Reading this page: https://curl.se/docs/sslcerts.html
- Downloading a certificate bundle from: https://curl.se/docs/caextract.html
- Now assigning the CA bundle as follow (and the get request will now work):
```
self.req:set_opt(curl.option.CAINFO, self:getPath(nvland:getAppDir(), "assets", "certs", "cacert-2023-01-10.pem"))
```

✅ Adding more bindings for QT6: QFrame, QLayoutItem, QLayout, QBoxLayout, QGridLayout, QDockWidget

✅ Build cryptoview on top of qtapp
⚠️ Consider building https://luarocks.org/modules/brimworks/lua-ev ⇒ discarded, built libuv+luv instead

✅ Add support to run another lua state in an auxiliary thread:
- Implement mechanism to start/stop lua threads from LuaManager: OK
- Implement support for message passing between threads in luaQT: OK

✅ How to pass arbitrary data between lua states in a thread-safe way ?
- When using QT we could setup an Event management system: objects in different threads would connect to given event given by names
- And each thread could emit the corresponding event.
- This requires a centralized EventManager component.
- Otherwise with lzmq we could serialize/deserialize objects and send/receive them on a socket.
- In any case, when not using QT we would still have an event loop somehow, so we could still post “new events” on the main thread indicating that some data is ready for usage.
- It might also be worth it to have a generic data container class… let's see how we could build that. ⇒ OK Added implementation for DataMap
- Should also add support RefObject or other datamaps from inside a datamap: OK

✅ Add support for base classes in ClassTemplate instantiation.

✅ Advanced class template instantiation in NervLuna:
- Forcing parsing of otherwise unreachable template classes in base resolution
- Advanced resolution of template class parameters for bases

DevLog 12

✅ Fix the issue with: Ignoring TypeAliasTemplateDecl cursor for base template class: nv::Vector<T>
✅ Fix invalid unnamed union binding issue

✅ Fix anonymous class binding issue:

Added the correct class name on its corresponding type when calling setName:

function Class:setName(name)
    if name ~= self._name then
        -- Check if we had a type for the previous name:
        local luna = import "bind.LunaManager"
        local t = luna.tm:getType(self._name)
        -- self._fullNames = {}
        self._name = name

        if t then
            t:addName(name)
            -- logDEBUG("==> Added name ", name, " to type ", t:getNames())
        end
    end
end

But then still getting the errors in case the “anonymous name” is shorter than the actual class name.

The name of the corresponding type is retrieved on this line:

self:writeLine("%s res = (%s)%s%s;", rtype:getName(), rtype:getName(), funcName, callstr)

So needed to to ensure that luna_anonymous is never considered as a valid name:

    if defName ~= baseName and not defName:startsWith("luna_anonymous") and defName:len() <= baseName:len() then
        -- logDEBUG("Replacing baseName: ", baseName, " with default name: ", defName)
        baseName = defName
    end

✅ Add support for preferred type names:

Updated process to add a name to a type to check for name preference:

function Class:_addName(name)
    if luna:isPreferredName(name) or #self._names == 0 or name:len() < self._names[1] then
        -- add this as the first name for this type:
        table.insert(self._names, 1, name)
    else
        -- Add this name at the end of the list:
        table.insert(self._names, name)
    end
    self._nameMap[name] = true
end

And now defining a default list in the Configurator config:

cfg.preferredNames = {
    "nv::U8",
    "nv::I8",
    "nv::U16",
    "nv::I16",
    "nv::U32",
    "nv::I32",
    "nv::U64",
    "nv::I64",
}

⚙️ Build the QEventManager system for lua (Stage 01)

Preparing initial classes for QEventManager and QEventHandler
Introduced support for nv::RefValue<T> in the process
Provided list of base types in nv_base_types.inl to simplify procedural types definitions

Updating class fullname when detecting template arguments using the best arg names:

    logDEBUG("ClassWriter detected templated class: ", tplcl, " with args: ", args)

    -- Find what are the best argument names:
    local bestArgs = {}
    for _, arg in ipairs(args) do
        -- We really expect to have a type for this arg or it could be a number, or a bool ?
        local t = luna.tm:getType(arg)
        if t then
            local tname = t:getName()
            if tname ~= arg then
                nvDEBUG("=> Renaming template argument from '%s' to '%s'", arg, tname)
                arg = tname
            end
        end

        table.insert(bestArgs, arg)
    end

Adding dedicated ClangParser:getOrCreateClassTemplate() method

Fix issue with invalid QFlags::enum_type type, ignoring invalid template dependent type names:

        -- Check if the reported name is some kind of template dependent name,
        local parts = putils.split(fname, "::")
        table.remove(parts)
        local pname = table.concat(parts, "::")
        local tplcl = luna:getClassTemplateByName(pname)
        if tplcl then
            logWARN("Ignoring invalid template dependant typedef name: ", fname)
        else
            -- CHECK(fname ~= "QFlags::enum_type", "Detected invalid type name.")
            rtype:addName(fname)
        end

Restored support for F32List in vulkan bindings: OK

✅ Adding support for generic referenced vectors with the nv::RefVector<T> template class.
✅ Fixing invalid String storage for entrypoints in VulkanGraphicsPipelineCreateInfo: still using a vector of RefString instead of our new RefVector of strings:
```
nv::Vector<nv::RefString> shaderEntrypoints;
```
✅ Read WASM tutorial: https://marcoselvatici.github.io/WASM_tutorial/
⇒ Should consider preparing an Emscripten dev env one day.

DevLog 13

✅ Build the QEventManager system for lua (Stage 02)

Added mechanism to push the arguments on the lua state with generic RefValPusher:

template <typename T, bool copy = false> struct RefValPusher {
    static void push(lua_State* L, RefValue<T>* val) {
        luna::LunaPusher<T>::pushValue(L, val->get());
    }
};

template <typename T> struct RefValPusher<T, true> {
    static void push(lua_State* L, RefValue<T>* val) {

        T* obj = new T(val->get());
        // Note: not using LunaPusher since we would not perform gc with it.
        // luna::LunaPusher<T*>::pushValue(L, obj);
        luna::Luna<T>::push(L, obj, true);
    }
};

template <> struct RefValPusher<nv::String, true> {
    static void push(lua_State* L, RefValue<nv::String>* val) {

        nv::String& str = val->get();
        lua_pushlstring(L, str.data(), str.size());
    }
};

✅ Added support for RefValue on VoidPtr and preparing implementation of QEventManager::trigger(…)

✅ Updated implementation of QEventManager::call_function() (now working with simple types)
✅ Updated implementation of QEventManager::trigger() (now working with simple types)

✅ Prepare skeleton classes for Coingecko and PriceManager components in lua
✅ Use global 'nvl' table to store utilities from 'base/utils.lua'
✅ Encapsulate pl.path functions inside nvl table

✅ Adding support to load config in nvl.Context and preparing nvl.Component support

DevLog 14

✅ Adding support for get_cwd and get_home_dir in nvCore
✅ Replacing usage of boost::any, boost::filesystem by equivalent std versions
✅ Replacing usage of boost::replace_all by internal version.
✅ Implement support to build yaml-cpp library
✅ Updated nv::Any to also store the type of the value contained in the std::any (as a StringID)
✅ Renaming DataMap to AnyMap
✅ Removing Any from base nv types list (preparing to create dedicated AnyList class)
✅ Renaming all XXXList and RefXXXList typedefs to XXXVector and XXXRefVector respectively.
✅ Adding initial support for AnyList class

✅ Add initial support to read AnyMap/AnyList from yaml file

✅ Implement support to cast AnyList into a given type Vector of Bool, I64, F64, or Strings.
✅ Add bindings for AnyMapVector and AnyListVector
✅ Add support for ref vectors of AnyList and AnyMap in YAML parsing.
✅ Load NervApp config from yaml file (and store it in an AnyMap)

✅ Add automatic call to AppComponent init/uninit functions.
✅ Automatic creation of AppComponent instance with default implementation support.
✅ Replace lua RequestManager with C++ version

✅ Implement support for get request handling in RequestManager (equivalent of make_get_request in python)

DevLog 15

✅ Move YAML parsing support in AnyMap
✅ Add support to parse JSON to AnyMap/AnyList with RapidJSON.
✅ Add support to push/set resolved AnyList in AnyMap and AnyList
✅ Add unit test for JSON parsing
✅ Provide initial support for emscripten compiler
- Project page: https://emscripten.org/
- Downloading SDK:
```
git clone https://github.com/emscripten-core/emsdk.git
```
- Added basic emsdk-git tool package OK
- Prepare emsdk_manager: OK
- Prepare emcc compiler: OK

DevLog 16

✅ Successfully compiled zlib with this build process on windows:

if self.compiler.is_emcc():
    self.patch_file(
        self.get_path(build_dir, "CMakeLists.txt"),
        "set_target_properties(zlib zlibstatic PROPERTIES OUTPUT_NAME z)",
        "set_target_properties(zlibstatic PROPERTIES OUTPUT_NAME z)\nset_target_properties(zlib PROPERTIES OUTPUT_NAME zdyn)",
    )
    self.run_emcmake(build_dir, prefix, ".")
    self.run_ninja(build_dir)

Then on linux with:

if self.compiler.is_emcc():
    self.run_emcmake(build_dir, prefix, ".", generator="Unix Makefiles")
    self.run_make(build_dir)

Now also adding support to use gnumake on windows:

    def __init__(self, bman: BuildManager, desc=None):
        """Initialize this builder"""
        self.ctx = bman.ctx
        self.man = bman
        self.compiler = bman.compiler
        self.env = None
        self.tools = self.ctx.get_component("tools")
        desc = desc or {}
        deftools = ["ninja", "make"] if self.is_windows else ["ninja"]
        self.tool_envs = desc.get("tool_envs", deftools)

⇒So we can now build on windows with no patch in zlib, using make instead of ninja:

if self.compiler.is_emcc():
    self.run_emcmake(build_dir, prefix, ".", generator="MinGW Makefiles")
    self.exec_emmake(build_dir)
    self.exec_emmake(build_dir, ["make", "install"])

Next we try to automatically call emconfigure/emmake/emcmake when running the corresponding build command (run_configure/run_make/run_cmake), for instance:

def run_make(self, build_dir, **kwargs):
    """Execute the standard make build/install commands"""
    cmd = ["make"]
    if self.compiler.is_emcc():
        ext = ".bat" if self.is_windows else ""
        folder = self.compiler.get_cxx_dir()
        emmake_path = self.get_path(folder, f"emmake{ext}")
        cmd = [emmake_path] + cmd

    self.check_execute(cmd, cwd=build_dir, env=self.env, **kwargs)
    self.check_execute(cmd + ["install"], cwd=build_dir, env=self.env, **kwargs)

So we could then reduce the build process to:

self.run_cmake(build_dir, prefix, ".", generator="MinGW Makefiles")
self.run_make(build_dir)

Build other “small” libraries with emcc:

Building yaml-cpp: OK (nothing to change 👍!)

Building openssl: OK but needed to patch the Makefile:

        if self.compiler.is_emcc():
            flags = ["no-asm", "no-shared", "no-hw", "no-engine"]
            self.run_configure(build_dir, prefix, flags=flags, configure_name="config")

            # Fixing this line in makefile:
            # CROSS_COMPILE=/mnt/data1/dev/projects/NervProj/tools/linux/emsdk-git/upstream/emscripten/em
            em_path = self.get_path(self.compiler.get_cxx_dir(), "em")
            self.patch_file(self.get_path(build_dir, "Makefile"), f"CROSS_COMPILE={em_path}", "CROSS_COMPILE=")
            self.exec_make(build_dir)
            self.exec_make(build_dir, ["install_sw"])

Building openjpeg: OK Just a minor change to disable building the shared library on both windows and linux:

flags = []
if self.compiler.is_emcc():
    flags = ["-DBUILD_SHARED_LIBS=OFF"]

self.run_cmake(build_dir, prefix, ".", flags=flags)
self.run_ninja(build_dir)

Building libpng: OK with PNG_SHARED=OFF

✅ Building QT6.4 with emscripten:

List of modules that can be built (+instructions): https://doc.qt.io/qt-6/wasm.html

Updated QT builder on windows: OK:

    def build_with_emcc_win(self, build_dir, prefix, _desc):
        """Build with emcc on windows"""
        # Building with emcc:

        # get the host path for QT:
        host_path = prefix.replace("windows_emcc", "windows_clang")
        logger.info("QT host path is: %s", host_path)
        self.check(self.dir_exists(host_path), "QT host path must exists.")

        # self.env["QT_HOST_PATH"]=host_path
        args = f"-qt-host-path {host_path} -platform wasm-emscripten -opensource"
        args += " -confirm-license -no-warnings-are-errors -feature-thread -static"
        args += " -skip qtwebengine -skip qtquick3d -skip qtquick3dphysics"

        em_dir = self.compiler.get_cxx_dir()
        cmake_args = f"-DCMAKE_TOOLCHAIN_FILE={em_dir}/cmake/Modules/Platform/Emscripten.cmake -DCMAKE_SUPPRESS_DEVELOPER_WARNINGS=1"
        self.generate_qt_config(build_dir, prefix, args, cmake_args)

        cmd = [
            self.tools.get_cmake_path(),
            "-DOPTFILE=config.opt",
            "-DTOP_LEVEL=TRUE",
            "-P",
            f"{build_dir}/qtbase/cmake/QtProcessConfigureArgs.cmake",
            "-Wno-dev",
        ]

        # let's run the configure.bat file:
        perl_dir = self.tools.get_tool_root_dir("perl")
        gperf_dir = self.tools.get_tool_dir("gperf")
        bison_dir = self.tools.get_tool_dir("bison")
        flex_dir = self.tools.get_tool_dir("flex")

        # py_dir = self.get_parent_folder(self.tools.get_tool_path("python"))

        dirs = [
            self.get_path(build_dir, "qtbase", "bin"),
            # py_dir,
            # nodejs_dir,
            gperf_dir,
            bison_dir,
            flex_dir,
            self.get_path(perl_dir, "perl", "site", "bin"),
            self.get_path(perl_dir, "perl", "bin"),
            self.get_path(perl_dir, "c", "bin"),
        ]
        logger.info("Adding additional paths: %s", dirs)

        self.env = self.append_env_list(dirs, self.env)

        logger.info("Post config command: %s", cmd)
        self.check_execute(cmd, cwd=build_dir, env=self.env)

        # Building the library now:
        logger.info("Building QT6 libraries...")
        # cmd = [self.tools.get_cmake_path(), "--build", ".", "-t", "qtbase", "-t", "qtdeclarative"]
        cmd = [self.tools.get_cmake_path(), "--build", ".", "--parallel"]
        logger.info("cmake command: %s", cmd)
        self.check_execute(cmd, cwd=build_dir, env=self.env)

        logger.info("Installing QT6 libraries...")
        cmd = [self.tools.get_cmake_path(), "--install", "."]
        self.check_execute(cmd, cwd=build_dir, env=self.env)

        logger.info("Done building QT6 with emcc.")
        return

✅ Remove dependency on boost in nervland
- ⇒ just realized that I'm already using c++20 in fact 😅 (So I should have support for “named arguments”)

⚠️ Try compiling nervland with emscripten (stage 01)
- Adding helper function set_compiler_flags in cmake to get the current compiler easily.
- ⇒ We need to setup a dummy boost package for emcc (otherwise the cmake manager will try to compile it from source): OK
- But we need to compile the fmt library anyway: OK (no change need in builder 😉)
- Next we will also need the package for LuaJIT 😵 ⇒ let's see if we can build that somehow 🤔 (without the JIT part of course…)
- For LuaJIT I tried this from linux, but it will not work:
```
        if self.compiler.is_emcc():
            # Compile for emscripten:
            # Should run the command:
            # make HOST_CC="emcc" BUILDMODE=static
            self.execute(
                ["make", "install", f"PREFIX={prefix}", "HOST_CC=emcc", "BUILDMODE=static"],
                cwd=build_dir, env=self.env)
```
- ⇒ So let's just provide a dummy package for luaJIT instead, and we will disable “everything lua related” when building for emscripten 😱!
- Next we need to compile SDL2:
  - Got a compilation error with version 2.0.20 -> but noticed we are already at version 2.26.5 😐 So trying to upgrade now. OK
- Reviewing/restoring support for msvc compilation in the process.
- Restoring compilation on linux: ⇒ This was pretty tricky (stdc++/clang header paths/mix with gcc/vector of bool/etc)
  - ⇒ Need to disable bindings for BoolVector copy constructor: OK

✅ Preparing dummy LLVM package for emcc compiler: OK
✅ Preparing dummy Vulkan package for emcc compiler: OK
✅ Preparing dummy glslang package for emcc compiler: OK
✅ Copying OpenSSL 3.0.8 for emcc from linux to windows: OK
✅ Building libcurl-7.88.1 for emcc compiler (on windows): OK
- ⇒ some minor updates in the libcurl builder.
✅ Preparing dummy libuv package for emcc compiler:

✅ Trying to compile nvCore for emscripten:

Need to compile as static lib:

if(NV_STATIC_BUILD OR IS_EMSCRIPTEN)
  add_subdirectory(static)
else()
  add_subdirectory(shared)
endif()

Need to fix `Spinlock::lock()` method:

    void lock() noexcept {
        for (;;) {
            // Optimistically assume the lock is free on the first try
            if (!lock_.exchange(true, std::memory_order_acquire)) {
                return;
            }
#ifndef __EMSCRIPTEN__
            // Wait for lock to be released without generating cache misses
            while (lock_.load(std::memory_order_relaxed)) {
                // Issue X86 PAUSE or ARM YIELD instruction to reduce contention
                // between hyper-threads
#if defined(_MSC_VER)
                _mm_pause();
#else
                __builtin_ia32_pause();
#endif
            }
#endif
        }
    }

Disabled check for 64bit target in core_common.cpp:

#ifndef __EMSCRIPTEN__
static_assert(sizeof(size_t) == 8, "Invalid size_t type");
#endif

Next we need to remove all references to lua in nvCore: OK (was mainly in NervApp.cpp)
⇒ Could successfully build nvCore_static.a for emscripten

Question: Is it possible to run a WASM file on an android phone?
- ⇒ seems to be “somewhat possible” with WasmEdge, but this seems pretty complex to do 😵.

✅ Build NervSeed app for emscripten: OK
- Need to disable the copy of the auxiliary binary files ⇒ added support for the `emscripten_dep_modules` entry.

✅ Generating html/wasm/js outputs for NervSeed:

if(IS_EMSCRIPTEN)
  install(
    FILES "$<TARGET_FILE_DIR:${TARGET_NAME}>/${TARGET_NAME}.js"
          "$<TARGET_FILE_DIR:${TARGET_NAME}>/${TARGET_NAME}.wasm"
          "$<TARGET_FILE_DIR:${TARGET_NAME}>/${TARGET_NAME}.html"
    DESTINATION ${TARGET_DIR})
else()
  install(
    TARGETS ${TARGET_NAME}
    RUNTIME DESTINATION ${TARGET_DIR}
    LIBRARY DESTINATION ${TARGET_DIR})
endif()

✅ Adding support to store config.yml file in compiled app:

if(IS_EMSCRIPTEN)
  set(assetFiles "config.yml")

  # Generate the command line that must be passed to emcc linker to produce the
  # given asset list.
  set(assertSourceDir ${CMAKE_INSTALL_PREFIX})
  # set(assetBundleCmdLine "--use_preload_cache --no-heap-copy")
  set(assetBundleCmdLine "--no-heap-copy")

  foreach(assetFile ${assetFiles})
    message(
      STATUS "Asset path '${assertSourceDir}/${assetFile}@/nvapp/${assetFile}'")
    set(assetBundleCmdLine
        "${assetBundleCmdLine} --preload-file \"${assertSourceDir}/${assetFile}@/nvapp/${assetFile}\" "
    )
  endforeach()

  # Use a response file to store all the --preload-file directives so that
  # windows max cmdline length limit won't be hit (there can be a lot of them!)
  file(WRITE "${CMAKE_BINARY_DIR}/${TARGET_NAME}_emcc_preload_file.rsp"
       "${assetBundleCmdLine}")
  set(linkFlags
      "${linkFlags} \"@${CMAKE_BINARY_DIR}/${TARGET_NAME}_emcc_preload_file.rsp\""
  )

  set(linkFlags "${linkFlags} -s WASM=1 -O3 -s ALLOW_MEMORY_GROWTH=1")

  # cf. https://github.com/emscripten-core/emscripten/issues/5414  -s
  # SAFE_HEAP=1 --bind  -o ../index.js -s LEGACY_GL_EMULATION=0  -s
  # GL_UNSAFE_OPTS=0 --pre-js pre-module.js --post-js post-module.js -s
  # ASSERTIONS=1 -s GL_ASSERTIONS=1 -s INVOKE_RUN=0 -std=c++11 -s USE_WEBGL2=1
  # -s FULL_ES3=1 -s USE_GLFW=3 -s OFFSCREENCANVAS_SUPPORT=1 --preload-file
  # shaders --preload-file extern --use-preload-plugins"

  set_target_properties(${TARGET_NAME} PROPERTIES LINK_FLAGS "${linkFlags}")
endif()

DevLog 17

✅ Build minimal sample with Qt and emscripten

Preparing minimal test_qt_wasm executable

linking to all required QT libraries:

set(QT_LIBS
    qwasm
    Qt6Gui
    Qt6Widgets
    Qt6OpenGLWidgets
    Qt6OpenGL
    Qt6Core
    Qt6Multimedia
    Qt6WebSockets
    Qt6BundledHarfbuzz
    Qt6BundledFreetype
    Qt6BundledLibpng
    Qt6BundledLibjpeg
    Qt6BundledPcre2
    Qt6BundledZLIB)

Enabling pthread (required by Qt) and disabling ALLOW_MEMORY_GROWTH for now (slow code otherwise ?)

Linking to missing Qt objects:

# also link the missing QT object files:
set(QT_OBJECTS
    "${QT6_DIR}/lib/objects-Release/Widgets_resources_1/.rcc/qrc_qstyle.cpp.o"
    "${QT6_DIR}/lib/objects-Release/Widgets_resources_2/.rcc/qrc_qstyle1.cpp.o"
    "${QT6_DIR}/lib/objects-Release/Widgets_resources_3/.rcc/qrc_qmessagebox.cpp.o"
)

⇒ Next we could try to run the test app from http://nervtech.org/test/test_qt_window.html
But on both Firefox and chrome this will produce the error:
```
Uncaught ReferenceError: SharedArrayBuffer is not defined
```
From initial investigations it seems this error is due to missing specific cross-origin headers: cf. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer

⇒ updating our nginx config to add those headers (in containers/nginx_server/config/sites/main_site):

  # Add COOP and COEP headers
  add_header Cross-Origin-Opener-Policy same-origin;
  add_header Cross-Origin-Embedder-Policy require-corp;

And restarting the server:

$ docker-compose down
docker-compose up -d

Argghh 😖, this not working as this time we get another error first:

Cross-Origin-Opener-Policy header has been ignored, because the URL's origin was untrustworthy. It was defined either in the final response or a redirect. Please deliver the response using the HTTPS protocol. You can also use the 'localhost' origin instead. See https://www.w3.org/TR/powerful-features/#potentially-trustworthy-origin and https://html.spec.whatwg.org/#the-cross-origin-opener-policy-header.
caught ReferenceError: SharedArrayBuffer is not defined

So it seems this is dues to not using HTTPS here… let's see.
⇒ Enabling SSL to access https://nervtech.org to, and now testing again: the SharedArrayBuffer exception is fixed 👍! … but I get other errors trying to load the file “test_qt_window.worker.js”
And indeed, that file is generated by emscripten but I'm not installing it yet, fixing that: OK
Now we can go a bit further, and it seems we are starting to init Qt, but there is still an error to be fixed:

Time to check some samples maybe ? cf. https://github.com/msorvig/qt-webassembly-examples (unfortunately this doesn't seem to help much for such low level construction issues…)
Found some infos on the qtloader.js on this page: https://bugreports.qt.io/browse/QTBUG-64061
So reading the wasm_shell.html/qtloader.js files content, it seems that those are used as a “wrapper” layer around our application, and will load the application instead of the default .html file generated by emscripten, so let's try to use them…
change the @APP_NAME@ value in wasm_shell.html, and then trying to load https://nervtech.org/test/wasm_shell.html, But we get yet another error:

⇒ Stupid me 😆 I was using the app name “test_qt_wasm” instead of “test_qt_window” (which is what I'm generating…)
So when using the correct app name, we can got a bit further, but of course, we then fall on more errors 😢:

⇒ Found a few references to createQtAppInstance in a few cmake files from QT folder: so checking what these are supposed to do. Found this cmake section:

set_target_properties(Qt6::Platform PROPERTIES
  INTERFACE_COMPILE_FEATURES "cxx_std_17"
  INTERFACE_COMPILE_OPTIONS "SHELL:-pthread"
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/mkspecs/wasm-emscripten;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "embind;Threads::Threads"
  INTERFACE_LINK_OPTIONS "SHELL:-s ERROR_ON_UNDEFINED_SYMBOLS=1;SHELL:-s MAX_WEBGL_VERSION=2;SHELL:-s FETCH=1;SHELL:-s WASM_BIGINT=1;SHELL:-s MODULARIZE=1;SHELL:-s EXPORT_NAME=createQtAppInstance;SHELL:-s DISABLE_EXCEPTION_CATCHING=1;SHELL:-pthread;\$<\$<CONFIG:Debug>:;SHELL:-s DEMANGLE_SUPPORT=1;SHELL:-s GL_DEBUG=1;--profiling-funcs>;SHELL:-sASYNCIFY_IMPORTS=qt_asyncify_suspend_js,qt_asyncify_resume_js"
  _qt_package_version "6.4.2"
)

Also found the file QT6-6.4.2\lib\cmake\Qt6\\QtWasmHelpers.cmake where many wasm related config elements are defined.

So adding the relevant emscripten config arguments:

set(linkFlags
    "${linkFlags} -s FETCH=1 -s WASM_BIGINT=1 -s MODULARIZE=1 -s EXPORT_NAME=createQtAppInstance"
)

And now also keeping the .js extension for the emscripten executable suffix (required with “EXPORT_NAME”)
But… yet another error now 🤣:

Just adding the additional exports:

set(linkFlags
    "${linkFlags} -s FETCH=1 -s WASM_BIGINT=1 -s MODULARIZE=1 -s EXPORT_NAME=createQtAppInstance -s EXPORTED_RUNTIME_METHODS=UTF16ToString,stringToUTF16"
)

Next error is on the memory usage, so we certainly need to use a large enough INITIAL_MEMORY value (done with -s INITIAL_MEMORY=1GB)

Finally I also had to specify -s NO_RUNTIME_EXIT=1, but then… Victory!! My empty QT window is finally displayed properly in the browser 🥳! (that's just an empty grey screen but that's really all I was expecting here 😄)

✅ Adding a menu bar and a status bar to our empty window

This is working fine when building the desktop app, but in webassembly, I get an invalid font display like this:

I think I've read somewhere that on webassembly only a few fonts are available, let's see…

⇒ To fix the display I had to include another missing input QT object:

set(QT_OBJECTS
    "${QT6_DIR}/lib/objects-Release/Widgets_resources_1/.rcc/qrc_qstyle.cpp.o"
    "${QT6_DIR}/lib/objects-Release/Widgets_resources_2/.rcc/qrc_qstyle1.cpp.o"
    "${QT6_DIR}/lib/objects-Release/Widgets_resources_3/.rcc/qrc_qmessagebox.cpp.o"
    "${QT6_DIR}/plugins/platforms/objects-Release/QWasmIntegrationPlugin_init/QWasmIntegrationPlugin_init.cpp.o"
    "${QT6_DIR}/lib/objects-Release/QWasmIntegrationPlugin_resources_1/.rcc/qrc_wasmfonts.cpp.o"
)

And now I get the menu elements to display correctly:

✅ Preparing to extend CryptoView app

Preparing a static nvQT library to store common GUI elements: OK
Building dedicated CryptoView app for webassembly and dektop: OK
Installing cryptoview app in dedicated folder dist/cryptoview

Update the cmake manager to support installing dependencies in multiple folders:

            # Iterate on each file to check if it's already installed or not:
            for src_file, dst_locs in file_map.items():
                # dst_locs could be a simple string or a list, we convert this to a list anyway:
                if isinstance(dst_locs, str):
                    dst_locs = [dst_locs]

                # We should iterate on each target location:
                for dst_loc in dst_locs:
                    # if the dst_loc ends with a "/" character, it means we want to use the source file name:
                    if dst_loc[-1] == "/":
                        fname = self.get_filename(src_file)
                        dst_file = self.get_path(dst_loc[:-1], fname)
                    else:
                        dst_file = dst_loc

                    src_path = self.get_path(root_dir, src_file)
                    dst_path = self.get_path(install_dir, dst_file)
                    copy_needed = False

                    if self.file_exists(dst_path):
                        # Check if the hash will match:
                        hash1 = self.compute_file_hash(src_path)
                        hash2 = self.compute_file_hash(dst_path)
                        if hash1 != hash2:
                            logger.info("Updating dep module %s...", dst_file)
                            self.remove_file(dst_path)
                            copy_needed = True
                    else:
                        # The destination file doesn't exist yet, we simply install it:
                        logger.info("Installing dep module %s...", dst_file)
                        copy_needed = True

                    if copy_needed:
                        # Check that the source file exists:
                        self.check(self.file_exists(src_path), "Invalid source file: %s", src_path)
                        folder = self.get_parent_folder(dst_path)
                        self.make_folder(folder)
                        self.copy_file(src_path, dst_path)

Build nvCore_lite module without lua dependency:

✅ Investigate and fix shared-memory issue

When linking against nvCore_lite we get this error from wasm-ld:

wasm-ld: error: --shared-memory is disallowed by memory.cpp.o because it was not compiled with 'atomics' or 'bulk-memory' features.

⇒ I could eventually figure out that the error comes from yaml-cpp which is not compiled with -pthread, now recompiling.

Updated global cmake builder command:

        if self.compiler.is_emcc():
            ext = ".bat" if self.is_windows else ""
            folder = self.compiler.get_cxx_dir()
            emcmake_path = self.get_path(folder, f"emcmake{ext}")
            cmd = [emcmake_path] + cmd
            # add -pthread for CXX compilation:
            cmd += ['-DCMAKE_CXX_FLAGS="-pthread"']

Then rebuild Yaml-cpp library:

nvp build libs yamlcpp -c emcc --rebuild

⇒ OK can then link properly with nvCore_lite

Add support to load icon from png:

auto create_icon(const char* name) -> QIcon* {
    // Get the application root path:
    const auto& root_path = NervApp::instance().get_root_path();
    auto fname = format_string("%s.png", name);
    return new QIcon(get_path(root_path, "assets", "icons", fname).c_str());
}

✅ Investigate how to serve webapps from simple python server

Preparing initial python HTTP server in component nvp/https_server:

    def serve_directory(self, root_dir, port):
        """Serve a given directory"""
        logger.info("Serving directory %s...", root_dir)
        os.chdir(root_dir)  # change the current working directory to the folder to serve

        class MyRequestHandler(http.server.SimpleHTTPRequestHandler):
            """Simple request handler"""

            def __init__(self, *args, **kwargs):
                """Constructor"""
                super().__init__(*args, directory=root_dir, **kwargs)

        httpd = http.server.HTTPServer(("localhost", port), MyRequestHandler)
        httpd.serve_forever()

        logger.info("Done serving directory.")

⇒ Need to enable SSL to be able to use SharedArrayBuffer

Preparing script to generate SSL certificate (for testing):

  gen_ssl_cert:
    notify: false
    cmd: $[TOOL_PATH:openssl] req -new -x509 -days 3650 -nodes -out server.pem -keyout server.key
    linux_env_vars:
      LD_LIBRARY_PATH: $[TOOL_DIR:openssl]/lib64

Also generating SSL certificate when serving files with https:

    def gen_ssl_cert(self):
        "Generate the ssl certificate if needed"

        cert_dir = self.get_path(self.ctx.get_root_dir(), "data", "certs")
        self.make_folder(cert_dir)

        pem_file = self.get_path(cert_dir, "localhost.pem")

        if not self.file_exists(pem_file):
            logger.info("Generating self-signed SSL certificate for localhost...")
            tools = self.get_component("tools")
            openssl_path = tools.get_tool_path("openssl")

            cmd = (
                "req -new -x509 -days 3650 -nodes -out localhost.pem -keyout localhost.key -subj /CN=localhost".split()
            )
            cmd = [openssl_path] + cmd

            self.execute(cmd, cwd=cert_dir)

        self.check(self.file_exists(pem_file), "Invalid %s file", pem_file)

        key_file = self.get_path(cert_dir, "localhost.key")
        return pem_file, key_file

And now using the SSL files:

        # Add the ssl layer:
        pem_file, key_file = self.gen_ssl_cert()
        cert = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
        cert.load_cert_chain(pem_file, key_file)

        # wrap the server socket with SSL and start the server
        httpd.socket = cert.wrap_socket(httpd.socket, server_side=True)
        logger.info("Serving at https://localhost:%d/%s", port, index_file)

⇒ But still no SharedArrayBuffer: we need to give trust to that certificate, so:

On windows: dding the Self-Signed Certificate to Trusted Root Certification Authorities

Spent a lot of time to build a certifcate authority “NervTechCA” and then a website certificate from it (“nervtech.local”), here is the updated process for that:

    def generate_certificate(self, cname, common_name, root_cert=None):
        """Generate an SSL certificate"""
        tools = self.get_component("tools")
        openssl = tools.get_tool_path("openssl")

        if common_name is None:
            common_name = cname

        # Write the config file:
        tpl_file = "ssl_root.cnf" if root_cert is None else "ssl_client.cnf"
        tpl_file = self.get_path(self.ctx.get_root_dir(), "assets", "templates", tpl_file)
        content = self.read_text_file(tpl_file)

        content = self.fill_placeholders(content, {"${COMMON_NAME}": common_name})
        if self.file_exists("config.cnf"):
            self.remove_file("config.cnf")

        self.write_text_file(content, "config.cnf")

        if root_cert is None:
            # Generate a root certificate:
            cmd1 = f"req -newkey rsa:2048 -sha256 -keyout {cname}_key.crt -out {cname}_req.crt -nodes -config ./config.cnf -batch"
            cmd2 = f"x509 -req -in {cname}_req.crt -sha256 -extfile ./config.cnf -extensions v3_ca -signkey {cname}_key.crt -out {cname}.crt -days 3650"
            cmd3 = f"x509 -subject -issuer -noout -in {cname}.crt"

        else:
            # Generate a non-root certificate:
            cmd1 = f"req -newkey rsa:2048 -sha256 -keyout {cname}_key.crt -out {cname}_req.crt -nodes -config ./config.cnf -batch"
            cmd2 = f"x509 -req -in {cname}_req.crt -sha256 -extfile ./config.cnf -extensions usr_cert -CA {root_cert}.crt -CAkey {root_cert}_key.crt -CAcreateserial -out {cname}_cert.crt -days 3650 -passin pass:"
            cmd3 = f"x509 -subject -issuer -noout -in {cname}.crt"

        cwd = self.get_cwd()
        logger.info("CWD: %s", cwd)
        # self.execute([openssl] + cmd0.split(), cwd=cwd)
        self.execute([openssl] + cmd1.split(), cwd=cwd)
        self.execute([openssl] + cmd2.split(), cwd=cwd)

        if root_cert is not None:
            # Combine the certificates:
            content1 = self.read_text_file(f"{cname}_cert.crt")
            content2 = self.read_text_file(f"{root_cert}.crt")
            self.write_text_file(content1 + content2, f"{cname}.crt")

        self.execute([openssl] + cmd3.split(), cwd=cwd)

Then calling:

nvp gen-cert NervTechCA
nvp gen-cert nervtech.local -r NervTechCA --name nervtech.local

Added NervtechCA as trusted root certificate ⇒ now local SSL website is valid, but still no SharedArrayBuffer

Also had to add the “Subject Alternative Names” (SANs) in the config for our client cert generation:

[ usr_cert ]

subjectAltName = @alt_names

[alt_names]

DNS.1 = ${COMMON_NAME}
DNS.2 = localhost
IP.1 = 127.0.0.1

⇒ Turns out I'm not sending the headers correctly! Trying to fix that, OK: updated request handler to:

        class MyRequestHandler(http.server.SimpleHTTPRequestHandler):
            """Simple request handler"""

            def __init__(self, *args, **kwargs):
                """Constructor"""
                super().__init__(*args, directory=root_dir, **kwargs)

            def end_headers(self):
                self.send_my_headers()

                http.server.SimpleHTTPRequestHandler.end_headers(self)

            def send_my_headers(self):
                """Senf my headers"""
                self.send_header("Cross-Origin-Opener-Policy", "same-origin")
                self.send_header("Cross-Origin-Embedder-Policy", "require-corp")

⇒ And now it's finally working 🥳!

Now in fact wondering if I really need SSL and/or 443 port here ? Testing…
- ⇒ ✔️ Port 443 not needed
- But SSL + valid secured context is needed

Next trying to open browser page automatically:

Cannot use webbrowser package if we want to support closing the HTTPS server automatically

So for now using user specified locatin for firefox to start the browser and control the lifetime of the process:

        firefox_path = os.getenv("FIREFOX_PATH")
        if firefox_path is not None:
            cmd = [firefox_path, url]

            def run_server():
                """Run the server"""
                logger.info("Running server...")
                try:
                    httpd.serve_forever()
                except KeyboardInterrupt:
                    logger.info("HTTPS server interrupted.")
                logger.info("Done with HTTPS server thread.")

            thread = threading.Thread(target=run_server)
            thread.start()

            self.execute(cmd, shell=True)

            logger.info("Stopping HTTPS server...")
            httpd.shutdown()

            thread.join()

        else:
            # Just run the server with no thread:
            httpd.serve_forever()

        logger.info("Done serving directory.")

✅ Adding support to load png icons in emscripten

Preparing cmake macros to setup emscripten targets: OK

Adding test icon in menu item:

    auto* menu = mbar->addMenu("&File");
    auto* act =
        new QAction(create_icon("folder-open-document"), "Open file...", this);

Added icon in asset list:

if(IS_EMSCRIPTEN)
  set(assetFiles "config.yml" "assets/icons/folder-open-document.png")
  setup_emscripten_qt_target(${TARGET_NAME} assetFiles ${TARGET_DIR})

And surprisingly, this works just fine:

✅ Adding display of console panel

Extending MainWindow to display a dockable widget for the ConsolePanel:

void MainWindow::build_dock_widgets() {
    auto* consoleDock = new DockWidget("Console");
    _consolePanel = new ConsolePanel();
    consoleDock->setWidget(_consolePanel);
    consoleDock->setAllowedAreas(Qt::AllDockWidgetAreas);

#ifdef __EMSCRIPTEN__
    // Disabling floating:
    consoleDock->setFeatures(QDockWidget::DockWidgetMovable |
                             QDockWidget::DockWidgetClosable);
#endif

    addDockWidget(Qt::BottomDockWidgetArea, consoleDock);
    setDockOptions(QMainWindow::GroupedDragging |
                   QMainWindow::AllowNestedDocks |
                   QMainWindow::AllowTabbedDocks);
    setTabPosition(Qt::AllDockWidgetAreas, QTabWidget::North);
}

Connecting ConsolePanel with dedicated LogManager sink:

class ConsoleLogSink : public nv::LogSink {
  public:
    explicit ConsoleLogSink(ConsolePanel* console) : _console(console){};

    void output(int level, const char* prefix, const char* msg,
                size_t size) override;

  private:
    ConsolePanel* _console{nullptr};
};

void ConsoleLogSink::output(int /*level*/, const char* prefix, const char* msg,
                            size_t size) {
    String str(msg, size);
    str = prefix + str;
    _console->write_text(str.c_str());
}

Also testing a simple menu item event handling:

    auto* menu = mbar->addMenu("&File");
    auto* act =
        new QAction(create_icon("folder-open-document"), "Open file...", this);
    QObject::connect(act, &QAction::triggered, [](bool /*checked*/) {
        logINFO("Handling open file event.");
    });

And now we have a dockable console panel displaying properly both on desktop and webassembly version of the app:

✅ Send request on app start to get bitcoin price

Setting up a simple request on MainWindow construction:

    // Get a price for bitcoin:
    String endpoint =
        "https://api.binance.com/api/v3/ticker/price?symbol=BTCUSDT";

    auto* rman = RequestManager::instance();
    auto resp = rman->get(endpoint.c_str());
    if (resp->is_ok()) {
        logDEBUG("Got response: {}", resp->text());
    } else {
        logWARN("Invalid response from binance.");
    }

⇒ This works fine in the windows app, but with emscripten we get another shared-memory error apparently from libcurl:

wasm-ld: error: --shared-memory is disallowed by easy.c.o because it was not compiled with 'atomics' or 'bulk-memory' features.

So rebuilding libcurl for emcc: OK
Arrgggg: it turns out that this is still not working: seems that we canno use libcurl in webassembly because we don't have sockets but instead we should use the fetch API: https://emscripten.org/docs/api_reference/fetch.html
⇒ We need to update the Request/RequestManager to deal with the emscripten fetch API
Added support for emscripten fetch API in request management: OK

Added support for async get (for emscripten implementation at list, for now just making a sync call on desktop), then calling:

    // Get a price for bitcoin:
    String endpoint =
        "https://api.binance.com/api/v3/ticker/price?symbol=BTCUSDT";

    auto* rman = RequestManager::instance();
    // perform an async get:
    rman->get_async(endpoint.c_str(), [](const RefPtr<RequestResponse>& resp) {
        if (resp->is_ok()) {
            logDEBUG("Got response: {}", resp->text());
        } else {
            logWARN("Invalid response from binance.");
        }
    });

And this is now working on both desktop and webassembly:

DevLog 18

✅ Build dawn library

Setting up support for depot_tools as a “git tool package” with the config entry:

  - name: depot_tools
    version: git
    sub_tools:
      gclient: gclient.bat
      fetch: fetch.bat
    sub_path: gclient.bat
    urls: http://files.nervtech.org/nvp_packages/tools/depot_tools-git-20230508-windows.7z
    # git: https://chromium.googlesource.com/chromium/tools/depot_tools.git

And added a dedicated python function to collect a tool from git and create a package for it automatically:

    def build_git_tool_package(self, full_name, desc):
        """Retrieve a tool package from git"""
        # Prepare installation dir:
        dest_dir = self.get_path(self.tools_dir, full_name)

        # Get the git component:
        git = self.get_component("git")

        self.check(not self.dir_exists(dest_dir), "Directory %s already exists", dest_dir)

        # if not self.dir_exists(dest_dir):
        # Request the cloning:
        git_url = desc["git"]
        git.clone_repository(git_url, dest_dir)

        # Prepare the current date:
        date_str = datetime.now().strftime("%Y%m%d")

        # Build the package for this tool ?
        ext = ".7z" if self.is_windows else ".tar.xz"
        pkgname = f"{full_name}-{date_str}-{self.platform}{ext}"

        self.create_package(dest_dir, self.tools_dir, pkgname)

Next, preparing a builder for dawn:

    def build_on_windows(self, build_dir, prefix, _desc):
        """Build method for dawn on windows"""

        # Bootstrap the gclient configuration
        # cp scripts/standalone.gclient .gclient
        self.copy_file(self.get_path(build_dir, "scripts", "standalone.gclient"), self.get_path(build_dir, ".gclient"))

        gclient_path = self.tools.get_tool_path("gclient")

        # Should also add the depot_tools folder in the PATH:
        depot_dir = self.tools.get_tool_dir("gclient")
        logger.info("Depot_tools dir: %s", depot_dir)
        git_dir = self.tools.get_tool_path("git")

        # Also add path to powershell:
        paths = [depot_dir, git_dir, "C:\\Windows\\System32\\WindowsPowerShell\\v1.0"]
        self.prepend_env_list(paths, self.env)

        # Fetch external dependencies and toolchains with gclient
        # gclient sync
        cmd = [gclient_path, "sync"]
        self.execute(cmd, cwd=build_dir, env=self.env)

        # Run cmake:
        flags = ["-S", ".", "-B", "release_build"]
        self.run_cmake(build_dir, prefix, flags=flags)
        sub_dir = self.get_path(build_dir, "release_build")
        self.run_ninja(sub_dir)

But this was failing or rather freezing on the message:

Updating depot_tools...
________ running 'python3 build/vs_toolchain.py update --force' in 'D:\Projects\NervProj\build\libraries\dawn-git-20230508\.'

So I tried to run the same steps on a batch prompt and got additional outputs there:

D:\Projects\NervProj\build\libraries>cd dawn-git-20230508

D:\Projects\NervProj\build\libraries\dawn-git-20230508>set PATH=D:\Projects\NervProj\tools\windows\depot_tools-git;%PATH%

D:\Projects\NervProj\build\libraries\dawn-git-20230508>gclient.bat sync
Updating depot_tools...
Syncing projects: 100% (33/33), done.

________ running 'python3 build/vs_toolchain.py update --force' in 'D:\Projects\NervProj\build\libraries\dawn-git-20230508\.'
ServiceException: 401 Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist).



No downloadable toolchain found. In order to use your locally installed version of Visual Studio to build Chrome please set DEPOT_TOOLS_WIN_TOOLCHAIN=0.
For details search for DEPOT_TOOLS_WIN_TOOLCHAIN in the instructions at https://chromium.googlesource.com/chromium/src/+/HEAD/docs/windows_build_instructions.md

So let's add this env var setting and try again the sync, and this time this seems to work as expected:

D:\Projects\NervProj\build\libraries\dawn-git-20230508>gclient.bat sync
Updating depot_tools...
Syncing projects: 100% (33/33), done.

________ running 'python3 tools/clang/scripts/update.py' in 'D:\Projects\NervProj\build\libraries\dawn-git-20230508\.'
Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Win/clang-llvmorg-17-init-2082-g6d4a674a-1.tar.xz .......... Done.
Hook 'python3 tools/clang/scripts/update.py' took 43.63 secs
Running hooks:  40% ( 8/20) rc_win
________ running 'download_from_google_storage --no_resume --no_auth --bucket chromium-browser-clang/rc -s build/toolchain/win/rc/win/rc.exe.sha1' in 'D:\Projects\NervProj\build\libraries\dawn-git-20230508\.'
0> Downloading build/toolchain/win/rc/win/rc.exe@7d3a485bb5bae0cf3c6b8af95d21f36aa7d02832...
Downloading 1 files took 2.396495 second(s)
Running hooks:  50% (10/20) clang_format_win
________ running 'download_from_google_storage --no_resume --no_auth --bucket chromium-clang-format -s buildtools/win/clang-format.exe.sha1' in 'D:\Projects\NervProj\build\libraries\dawn-git-20230508\.'
0> Downloading buildtools/win/clang-format.exe@d88796ff22d70c7d5ede62636daa220ccd238f06...
Downloading 1 files took 5.122753 second(s)
Running hooks: 100% (20/20), done.

D:\Projects\NervProj\build\libraries\dawn-git-20230508>

So let's continue with our builder adding this custom env var too. ⇒ OK got dawn built with msvc.

Next: manually handling dawn install process with this code:

        # logger.info("Installing dawn libraries...")
        def install_files(src_folder, exp, dst_folder, hint, **kwargs):
            # Get all the dawn libs:
            base_dir = kwargs.get("src_dir", sub_dir)
            flatten = kwargs.get("flatten", True)
            excluded = kwargs.get("excluded", [])
            src_dir = self.get_path(base_dir, src_folder)
            all_files = self.get_all_files(src_dir, exp=exp, recursive=True)

            dst_dir = self.get_path(prefix, dst_folder)
            self.make_folder(dst_dir)

            # copy the dawn libraries:
            for elem in all_files:
                ignored = False
                for pat in excluded:
                    if re.search(pat, elem) is not None:
                        ignored = True
                        break

                if ignored:
                    logger.info("Ignoring element %s", elem)
                    continue

                logger.info("Installing %s %s", hint, elem)
                src = self.get_path(src_dir, elem)
                dst_file = self.get_filename(src) if flatten else elem
                dst = self.get_path(dst_dir, dst_file)
                pdir = self.get_parent_folder(dst)
                self.make_folder(pdir)

                self.check(not self.file_exists(dst), "File %s already exists.", dst)
                self.copy_file(src, dst)

        install_files("src/dawn", r"\.lib$", "lib", "library")
        install_files("src/tint", r"\.lib$", "lib", "library")
        install_files("gen/include/dawn", r"\.h$", "include/dawn", "header")
        install_files("include", r"\.h$", "include", "header", src_dir=build_dir, flatten=False)
        install_files(".", r"\.exe$", "bin", "app", excluded=["CMake", "unittests"])

And also built dawn on linux without any serious trouble (just needed to install libxinerama-dev in my case)

✅ Rebuild a sample from dawn in NervLand

Sounds like we need the GLFW library support here: OK
⇒ in fact, now discarding the build of dawn samples and focusing on the webgpu native samples instead.
So we need to have cglm library OK
We also need the ktx library: OK on windows but failing on linux 😢 (we will get back to this one later)
Note: Also need to provide all the absl libs with dawn.
[crunching… crunching… crunching…]
⇒ And finally got a simple triangle example working (adapted from Dawn samples)
Refactored dedicated wgpu_app files (.h/.cpp): OK
Adding WGPU clear screen sample

✅ Start rebuilding WGPU native samples

Implement the WGPU native minimal sample: OK

DevLog 19

⚙️ Adding more WGPU native samples

Implement the WGPU native square sample: OK

Implement the WGPU native triangle sample:

Moved Camera/CameraController in nvCore

Added support to copy Mat4 with transposition:

    inline void set(const Mat4& rhs, bool transpose = false) {
        if (transpose) {
            // copy rhs with transposition at the same time:

            for (int i = 0; i < 4; ++i) {
                for (int j = 0; j < 4; ++j) {
                    _mat[i][j] = rhs._mat[j][i];
                }
            }
        } else {
            set(rhs.ptr());
        }
    }

DevLog 20

⚙️ Adding more WGPU native samples

Implement triangle example in SDL2 to get the TargetCameraController on rails.
- Refactor wgpu AppContext to use WGPUEngine
- Removed swapchain from AppContext class
- Removed device from AppContext
- Adding initial implementation for RenderPipelineBuilder
- Using RenderPipelineBuilder in minimal sample, reducing pipeline construction to a single line:
```
pipeline = eng->build_render_pipeline(vs, fs);
```
- Adding support for vertex buffer layouts in Render pipeline builder
- Embedding pipeline layout contruction ini pipeline builder
- Adding support to build bind groups with BindGroupBuilder
- Creating textures in WGPUEngine
- Added initial “TriangleSDL” sample: OK
- Adding support for FPS display in SDL wgpu app
Implement the WGPU native two_cubes sample:
- Preparing support to render 2 cubes with correct uniform buffer offset handling (ie. min alignment = 256 bytes).
- Support to display 3 cubes in two_cubes sample
- Added support for RenderBundleBuilder

DevLog 21

⚙️ Adding more WGPU native samples

Implement the WGPU native bind_groups sample:
- Adding support to load texture from KTX file in TextureLoader: OK
- Add initial support to load glTF models from file: OK (support for nodes/meshes/primitives/scenes)

Implement the WGPU native dynamic_uniform_buffer sample: OK
- Implementing support for RenderNode encapsulation

DevLog 22

✅ Adding more WGPU native samples

Implement the WGPU native textured_quad sample: OK

Implement the WGPU native textured_cube sample: OK
- Add support to load png images: OK
- Refactoring glTF model to support VBO/IBO usage: OK
- Adding support for resource paths and get_texture() in wgpuengine.
- Adding support to create gltf drawable by name:
```
rnode->add_drawable("tests/cube", 0);
```
- Removing the need to provide binding builder for render node construction
- Adding support to create buffers from structs:
```
    logDEBUG("Creating uniform buffers...");
    cam_ubo = eng->create_buffer(udata, BufferUsage::Uniform);
```

Implement the WGPU native cubemap sample: OK
- Added support for WGPUCamera
- Added support for cubemap texture generation and usage.

Implement the WGPU native skybox sample:
- Implement skybox shader in wgsl: OK
- Add support to draw without vertex/index buffers: OK
- Add support to load cubemap from ktx: OK

Implement the WGPU native texture_3d sample:
- Handle generation of perlin noise on cpu: OK
  - Was trying to use FastNoise2 but this will not compile with emscripten,
  - So instead, will use FastNoiseLite which also comes with a GLSL implementation!
- Assigned 3D texture to a cube model
- Add support to provide a time uniform value somehow in the shaders: introducing UBOProvider class
- Add support to define/bind multiple UBOProviders with a single function call:
```
        // Render a cube:
        WGPURenderNode rnode(std_vfmt_ipnu, "tests/textured_cube_3d");
        rnode
            .define_grp(defUBOs("camera", "frame_context"), def3dSampler(false),
                        defFrag3dTex(false))
            .add_bind_grp(0, bindUBOs("camera", "frame_context"),
                          BindNearestSampler, tex)
            .add_drawable("tests/cube", 0);
        rpass.add_bundle(rnode);
```
  /

Implement the WGPU native equirectangular sample: OK

Implement the WGPU basis universal texture loading example: OK
- Build the basis universal tool binary: OK with msvc, NOK with Clang on windows.

Implement the WGPU native reversed_z example:
- Added support for instanced drawDesc: OK
- Added support to render 2 viewports: OK
- Added support for reversed_z in WGPUCamera: OK

Implement shader file preprocessing engine:
- Add support for #include statements: OK
- Add support for #ifdef/#endif sections: OK
- Add support for #else statement: OK
- Refactored reversed_z example accordingly: OK

Implement the WGPU native Wireframe and Thick-Line Rendering example:
- Add support to create storage buffers: OK
- Add support to render wireframe in linelist mode: OK

Add support to render in thick wireframe mode: OK

Implement the WGPU native Offscreen rendering example:
- Introducing support for RenderNode::bind() and RenderNode::draw() methods
- Adding support for rendeer_node() in RenderPass.

Implement the WGPU native Stencil buffer example:
- Refactored a bit support for Depth state settings.

TODO

Need to implement nv::Request::post for emscripten
Build support for a free flying camera controller:
To read: https://toji.github.io/webgpu-gltf-case-study/
Official WebGPU documentation: https://www.w3.org/TR/webgpu/
Official WGSL documentation: https://www.w3.org/TR/WGSL/

WGPU native: should implement the “coordinate_system” sample
To read on Webgpu: https://eliemichel.github.io/LearnWebGPU

Consider implementing base components directly in C++ (to get support for WASM out of the box) ?
Provide all required support function to initialize coingecko component properly
Implement support to get a price from coingecko component
Implement support to display the current price of a few coins, read from config file:

Read the zguide at https://zguide.zeromq.org/

Build lua sqlite bindings: http://lua.sqlite.org/index.cgi/indexty
Consider providing support in NervLuna for class inheritance/virtual/abstract methods ?
- Check https://github.com/lqt5/lqt/tree/qt5 maybe ?

Handle bindings correctly for classes introducing additional overloads for a function from a parent class

Improvements on the NervLuna binding system:
Move ClangParser:resolveType() method in type manager or other class ?

Table of Contents

NervLand Task list

DevLog 11 and previous

DevLog 12

DevLog 13

DevLog 14

DevLog 15

DevLog 16

DevLog 17

✅ Build minimal sample with Qt and emscripten

✅ Adding a menu bar and a status bar to our empty window

✅ Preparing to extend CryptoView app

✅ Investigate and fix shared-memory issue

✅ Investigate how to serve webapps from simple python server

✅ Adding support to load png icons in emscripten

✅ Adding display of console panel

✅ Send request on app start to get bitcoin price

DevLog 18

✅ Build dawn library

✅ Rebuild a sample from dawn in NervLand

✅ Start rebuilding WGPU native samples

DevLog 19

⚙️ Adding more WGPU native samples

DevLog 20

⚙️ Adding more WGPU native samples

DevLog 21

⚙️ Adding more WGPU native samples

DevLog 22

✅ Adding more WGPU native samples

TODO