HomeCtrl: Considering compression with Brotli

Allright guys, as mentioned in my previous article, I should now find how to deal with brotli decompression for my wasm file when downloading that from the server. Let's rock it baby 😆!

Youtube video for this article available at:

My wasm file is currently 11MB large, not really unmanageable, but still, I've seen it was recommended from everywhere to compress those files with brotli, so, where do we start ?

First of course I tried a simple command to compress this WASM file:

  brotli:
    notify: false
    cmd: ${NVP_ROOT_DIR}/libraries/windows_msvc/brotli-git-1.0.9/bin/brotli.exe

Then calling this script as:

D:\Projects\NervLand\dist\webapps\homeview>nvp brotli -k -Z homeview.wasm

This will produce the homectrl.wasm.br file, of size of about 3.59MB which is not bad compared to 11.92MB (that's just about 30% of the full size).

Now problem with the command above is that it depends on the brotli executable I built myself, so that's platform dependent. I think I could rather use the brotli package in python directly: let's try to write a very simple command for that.

Adding the brenc script:

  brenc:
    notify: false
    custom_python_env: brotli_env
    cmd: ${PYTHON} ${PROJECT_ROOT_DIR}/nvp/admin/brotli_handler.py encode
    python_path: ["${NVP_ROOT_DIR}"]

And preparing a skeleton component for brotli:

class BrotliHandler(NVPComponent):
    """BrotliHandler component class"""

    def __init__(self, ctx: NVPContext):
        """Component constructor"""
        NVPComponent.__init__(self, ctx)

    def process_cmd_path(self, cmd):
        """Re-implementation of process_cmd_path"""

        if cmd == "encode":
            file = self.get_param("input_file")
            logger.info("Shoulc compress file %s", file)
            return True

        return False


if __name__ == "__main__":
    # Create the context:
    context = NVPContext()

    # Add our component:
    comp = context.register_component("BrotliHandler", BrotliHandler(context))

    psr = context.build_parser("encode")
    psr.add_str("input_file")("File to compress")

    comp.run()

Now let's put some meat in this component. Here are the compress() and decompress() implemnetations:

    def compress_file(self, input_file, output_file=None):
        """Compress a file"""
        if output_file is None:
            output_file = input_file + ".br"

        params = {
            # 'mode': brotli.MODE_TEXT  # Set to brotli.MODE_TEXT for text-based files
            "mode": brotli.MODE_GENERIC,
            "quality": 11,
            "lgwin": 22,
            "lgblock": 0,
        }

        start_time = time.time()
        content = self.read_binary_file(input_file)
        logger.info("Compressing %s...", input_file)
        compressed = brotli.compress(content, **params)

        # write the compressed data:
        self.write_binary_file(compressed, output_file)

        elapsed = time.time() - start_time
        logger.info("Compressed %s in %.2fsecs", input_file, elapsed)

        return True

    def decompress_file(self, input_file, output_file=None):
        """Compress a file"""
        if output_file is None:
            output_file = self.set_path_extension(input_file, "")

        start_time = time.time()
        content = self.read_binary_file(input_file)
        logger.info("Decompressing %s...", input_file)
        decompressed = brotli.decompress(content)

        # write the compressed data:
        self.write_binary_file(decompressed, output_file)

        elapsed = time.time() - start_time
        logger.info("Decompressed %s in %.2fsecs", input_file, elapsed)

        return True

I just tested that on my homeview.wasm file and it works just fine:

D:\Projects\NervLand\dist\webapps\homeview>nvp brenc homeview.wasm
2023/07/13 15:52:54 [nvp.nvp_compiler] INFO: MSVC root dir is: D:\Softs\VisualStudio\VS2022
2023/07/13 15:52:54 [nvp.nvp_compiler] INFO: Found msvc-14.34.31933
2023/07/13 15:52:54 [nvp.core.build_manager] INFO: Selecting compiler msvc-14.34.31933 (in D:\Softs\VisualStudio\VS2022)
2023/07/13 15:52:54 [__main__] INFO: Compressing homeview.wasm...
2023/07/13 15:53:18 [__main__] INFO: Compressed homeview.wasm in 23.55secs

D:\Projects\NervLand\dist\webapps\homeview>nvp brdec homeview.wasm.br -o home.wasm
2023/07/13 15:54:54 [nvp.nvp_compiler] INFO: MSVC root dir is: D:\Softs\VisualStudio\VS2022
2023/07/13 15:54:54 [nvp.nvp_compiler] INFO: Found msvc-14.34.31933
2023/07/13 15:54:54 [nvp.core.build_manager] INFO: Selecting compiler msvc-14.34.31933 (in D:\Softs\VisualStudio\VS2022)
2023/07/13 15:54:54 [__main__] INFO: Decompressing homeview.wasm.br...
2023/07/13 15:54:54 [__main__] INFO: Decompressed homeview.wasm.br in 0.05secs

Next I need to somehow call this compressor when I'm about to serve my app in firefox. How should I proceed 🤔? I guess I should check if the .wasm file is more recent than the .wasm.br file ? This should be handled directly in my https_server.py component.

I'm now adding the following code at the beginning of the HttpsServer.serve_directory function:

        # Check if we have .wasm files in this folder:
        wasm_files = self.get_all_files(root_dir, exp=r"\.wasm$", recursive=False)
        for wfile in wasm_files:
            in_file = self.get_path(root_dir, wfile)
            logger.info("Found WASM file %s", in_file)

            # check if we have a corresponding wasm.br file:
            brfile = in_file + ".br"
            if not self.file_exists(brfile):
                logger.info("Generating %s...", brfile)
                brotli = self.get_component("brotli")
                brotli.compress_file(in_file, brfile)

OK, this is working fine (provided I remove the existing .wasm.br file manually here):

PS D:\Projects\NervHome> nvp nvl_serve_homeview
2023/07/13 16:21:26 [nvp.nvp_compiler] INFO: MSVC root dir is: D:\Softs\VisualStudio\VS2022
2023/07/13 16:21:26 [nvp.nvp_compiler] INFO: Found msvc-14.34.31933
2023/07/13 16:21:26 [nvp.core.build_manager] INFO: Selecting compiler msvc-14.34.31933 (in D:\Softs\VisualStudio\VS2022)
2023/07/13 16:21:26 [__main__] INFO: Serving directory D:\Projects\NervLand/dist/webapps/homeview...
2023/07/13 16:21:26 [__main__] INFO: Found WASM file D:\Projects\NervLand/dist/webapps/homeview\homeview.wasm
2023/07/13 16:21:26 [__main__] INFO: Generating D:\Projects\NervLand/dist/webapps/homeview\homeview.wasm.br...
2023/07/13 16:21:26 [nvp.admin.brotli_handler] INFO: Compressing D:\Projects\NervLand/dist/webapps/homeview\homeview.wasm...
2023/07/13 16:21:50 [nvp.admin.brotli_handler] INFO: Compressed D:\Projects\NervLand/dist/webapps/homeview\homeview.wasm in 23.70secs
2023/07/13 16:21:50 [__main__] INFO: Serving at https://nervtech.local:444/homeview.html

For the second part now we need to check the modification time of both files if the .br file already exists, so here is the final code:

        # Check if we have .wasm files in this folder:
        wasm_files = self.get_all_files(root_dir, exp=r"\.wasm$", recursive=False)
        for wfile in wasm_files:
            in_file = self.get_path(root_dir, wfile)
            logger.info("Found WASM file %s", in_file)

            # check if we have a corresponding wasm.br file:
            brfile = in_file + ".br"
            if not self.file_exists(brfile):
                logger.info("Generating %s...", brfile)
                brotli = self.get_component("brotli")
                brotli.compress_file(in_file, brfile)
            elif self.get_file_mtime(wfile) > self.get_file_mtime(brfile):
                # The brfile already exists but the wasm file is more recent
                logger.info("Updating %s...", brfile)
                self.remove_file(brfile)
                brotli = self.get_component("brotli")
                brotli.compress_file(in_file, brfile)
            else:
                logger.info("%s is OK", brfile)

Now another fun part: I will need brotli in the browser to do the decoding, but I can't seem to be able to find just a simple brotli.min.js file anywhere online, come on guys… 🤣 I keep searching… Feewww, finally found this location which I think could be what I'm after.

OF course, then I had to fight a little more to figure out how to earth one can start using the decode.js file provided in that folder without first setting up a full nodejs environment to build/bundle that as a working module etc etc. But eventually I found an example of how to do it, which turns to be super easy:

    <script type="module">
      import {BrotliDecode} from './decode.min.js';
      let ukkonooa = [
        0x1b, 0x76, 0x00, 0x00, 0x14, 0x4a, 0xac, 0x9b, 0x7a, 0xbd, 0xe1, 0x97,
        0x9d, 0x7f, 0x8e, 0xc2, 0x82, 0x36, 0x0e, 0x9c, 0xe0, 0x90, 0x03, 0xf7,
        0x8b, 0x9e, 0x38, 0xe6, 0xb6, 0x00, 0xab, 0xc3, 0xca, 0xa0, 0xc2, 0xda,
        0x66, 0x36, 0xdc, 0xcd, 0x80, 0x8d, 0x2e, 0x21, 0xd7, 0x6e, 0xe3, 0xea,
        0x4c, 0xb8, 0xf0, 0xd2, 0xb8, 0xc7, 0xc2, 0x70, 0x4d, 0x3a, 0xf0, 0x69,
        0x7e, 0xa1, 0xb8, 0x45, 0x73, 0xab, 0xc4, 0x57, 0x1e
      ];
      let lyrics = BrotliDecode(new Uint8Array(ukkonooa));
      console.log("The lyrics are:");
      console.log(String.fromCharCode.apply(null, new Uint16Array(lyrics)));
      console.log("Done printing lyrics.");
      window.BrotliDecode = BrotliDecode
    </script>

⇒ if we use the type “module” for a script, then we can use the import {xxx} syntax in there! (I didn't know that lol) (I found this example on this page by the way).

Running the app with this code in place will produce the following outputs in the javascript console, indicating that the decompression is indeed working as expected:

The lyrics are:
ukko nooa, ukko nooa oli kunnon mies, kun han meni saunaan, pisti laukun naulaan, ukko nooa, ukko nooa oli kunnon mies.
Done printing lyrics.

I have zero what the lyrics language is here, but since it's not totally random/garbage, it's certainly a real language

With this Brotli decompressor read, I moved to the next step: trying to actually put that to use in the qtloade.js script to download/decompress the homeview.wasm.br file.

To achieve this, the easiest option from my perspective was to replace the fetchCompileWasm() function which is called only to download compile the wasm file, with a customized fetchCompileCompressedWasm() function:

        // Fetch and compile wasm module
        var wasmModule = undefined;
        // var wasmModulePromise = fetchCompileWasm(applicationName + ".wasm").then(function (module) {
        var wasmModulePromise = fetchCompileCompressedWasm(applicationName + ".wasm").then(function (module) {
            wasmModule = module;
        });

And the new function is as follow:

    function fetchCompileCompressedWasm(filePath) {
        console.log("Fetching resource: "+filePath)
        // return fetchResource(filePath)
        return fetchResource(filePath+".br")
        .then(function(response) {
              // Content is Brotli-encoded, decode it
              return response.arrayBuffer().then(function(buffer) {
                console.log("Trying to decode buffer of size "+ buffer.byteLength)
                var decodedBuffer = BrotliDecode(new Uint8Array(buffer));
                console.log("Decoded buffer of size "+ decodedBuffer.length)
                return new Response(decodedBuffer.buffer, { status: response.status, statusText: response.statusText });
              });
            })
        .then(function(response) {
            // if (typeof WebAssembly.compileStreaming !== "undefined") {
            //     self.loaderSubState = "Downloading/Compiling";
            //     setStatus("Loading");
            //     return WebAssembly.compileStreaming(response).catch(function(error) {
            //         // compileStreaming may/will fail if the server does not set the correct
            //         // mime type (application/wasm) for the wasm file. Fall back to fetch,
            //         // then compile in this case.
            //         return fetchThenCompileWasm(response);
            //     });
            // } else {
                // Fall back to fetch, then compile if compileStreaming is not supported
                return fetchThenCompileWasm(response);
            // }
        });
    }

And this works 🥳!

Allright, next step now: could we reprot some process percentage while downloading the compressed wasm file ? Let's see… Asking our dear friend ChatGPT about this, and he said we need something like this (which sounds quite appropriate to me):

return fetch(fullPath)
  .then(function (response) {
    if (!response.ok) {
      let err = response.status + " " + response.statusText + " " + response.url;
      handleError(err);
      return Promise.reject(err);
    } else {
      // Start reading the response as a stream
      const reader = response.body.getReader();
      const contentLength = +response.headers.get('Content-Length');
      let receivedBytes = 0;

      // Define a function to handle each chunk of data
      function handleData({ done, value }) {
        if (done) {
          // All data has been received
          return response;
        }

        // Process the chunk of data here
        receivedBytes += value.length;
        const progress = (receivedBytes / contentLength) * 100;
        reportProgress(progress);

        // Read the next chunk of data
        return reader.read().then(handleData);
      }

      // Start reading the data
      return reader.read().then(handleData);
    }
  });

hmmm… 🤔, actually, I'm not sure I can retrieve the downloaded data with the code above, so an alternative here is to keep all the downloaded “chunks”, and then create a blob out of them, so with something like that:

    const contentLength = +response.headers.get('Content-Length');
    const reader = response.body.getReader();
    let receivedBytes = 0;
    let chunks = [];

    function readChunk() {
      return reader.read().then(function ({ done, value }) {
        if (done) {
          // All data has been downloaded
          return chunks;
        }

        chunks.push(value);
        receivedBytes += value.length;
        const progress = (receivedBytes / contentLength) * 100;
        progressCallback(progress);

        return readChunk(); // Read the next chunk
      });
    }

    return readChunk();

And then:

  .then(function (chunks) {
    // All data has been downloaded
    const blob = new Blob(chunks);
    // Process the downloaded blob as needed
    console.log("Download completed");
  })

And from that blob we could retrieve an arrayBuffer, right ? Yes indeed, this is working just fine once more 🥳! I can see the progress reported for a faction of a secs since i'm on local network and it's support fast anyway, but later when I deploy the app on my public website, this will give a better user experience for sure.

Okay, time to make a leap of faith now: let's try to deploy the app directly on nervtech.org 😳🤣

Adding the required location in the nervtech.org site config:

  location /homeview {
    root /mnt/data1/web/homeview;
    try_files $uri $uri/ /homeview.html;
  }

And then restarting the nginx server as usual and crossing fingers 🤞…

But of course… not working 🤣: it seems I have a problem with the MIME type of my served files 🤔:

Checking the nginx config is OK:

$ dk_enter nginx_server
root@de35904c66d4:/# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Okay, what else could be wrong here? […burning some neurons on this…] hmmm, allright, not really sure what this was, but I finally got it working with this location definition:

  location /homeview {
      alias /mnt/data1/web/homeview/;
      index homeview.html;
      try_files $uri $uri/ /homeview.html;
  }

With that change, it works very well on desktop 👍!

But then, trying on my mobile phone I get an exception when trying to allocate the memory for the app 😞. Now, if I remember correctly, when building QT for webassembly apps I have requested 1GB of mem at start to be able to use the multi threading feature from emscripten: maybe I could reduce this a bit ? Let's try with something like 256MB maybe.

Here is the updated macro in cmake, preparing support to customize the initial memory value:

macro(get_emscripten_qt_linkflags link_fvar)
  # Check if initial memory is provided:
  if(NOT NV_WASM_INIT_MEM)
    set(NV_WASM_INIT_MEM "256MB")
  endif()

  # -s ALLOW_MEMORY_GROWTH=1 -s USE_ZLIB=1 -s USE_HARFBUZZ=1 -s USE_FREETYPE=1
  # -s USE_LIBPNG=1 -s DISABLE_EXCEPTION_CATCHING=1

  set(${link_fvar}
      "${${link_fvar}} -s WASM=1 -O2 --bind -pthread -sPTHREAD_POOL_SIZE=4 -s ASSERTIONS "
  )
  set(${link_fvar}
      "${${link_fvar}} -s FETCH=1 -s WASM_BIGINT=1 -s MODULARIZE=1 -s EXPORT_NAME=createQtAppInstance"
  )
  set(${link_fvar}
      "${${link_fvar}} -s EXPORTED_RUNTIME_METHODS=UTF16ToString,stringToUTF16 -s INITIAL_MEMORY=${NV_WASM_INIT_MEM}"
  )
  set(${link_fvar}
      "${${link_fvar}} -s FULL_ES2=1 -s USE_WEBGL2=1 -s NO_EXIT_RUNTIME=1 -s ERROR_ON_UNDEFINED_SYMBOLS=1"
  )

  set(${link_fvar} "${${link_fvar}} -lidbfs.js -s FORCE_FILESYSTEM=1")
  # Might need to include "--use-preload-plugins" here. cf.
  # https://emscripten.org/docs/tools_reference/emcc.html?highlight=preload
endmacro()

And building with this change, the app is finally working on my android phone too in chrome! Yeeepeee 🥳! Credentials are remembered and all, so all good so far!

Next I think I should spend some time refactoring a bit the design of this app to make it more convinient to use from a phone screen, but that will be for another session. meanwhile, happy coding and see ya next time !