public:projects:nervland:0002_landjs_terrainnode_review

TerrainNode simple scene review

  • So last time I was working on this project, I think I was working around this “TerrainNode” scene implementation, which is available in the file src/scenes/test7_terrainNode.ts
  • This is the scene currently loaded in the main app when using `landjs_serve` since we have in src/app/app.component.ts:
      ngOnInit(): void {
        // retrieve the WebGL2 context:
        log.DEBUG("Retriving webGL2 context...");
        let ctx = this.view.nativeElement.getContext('webgl2');
        RenderContext.setCurrent(ctx);
        log.DEBUG("Assigned Render context!");
    
        // Create a resource loader here:
        let loader = new ResourceLoader(this.http);
    
        // let scene = new Test2Scene();
        // let scene = new Test2Scene(this.view.nativeElement);
        // let scene = new Test3Scene(this.view.nativeElement);
        // let scene = new Test4Scene(this.view.nativeElement);
        // let scene = new Test5Scene(this.view.nativeElement, loader);
        // let scene = new Test6Scene(this.view.nativeElement, loader);
        let scene = new Test7Scene(this.view.nativeElement, loader);
      }
  • We have a loader passed as argument here: this reminds me of how it works (roughtly): this loader is responsible for loading the “actual resources” in the context of the angular app (I think 😋…)
  • Most (all) of the important code is in the src/nvland folder. And in there we have multiple sub folders for the main sub-systems like “core”, “ecs”, “math”, “render”, etc.
  • Which reminds me that lately I spend a lot of time on the Entity Component System (ECS), but that's not the last thing I was working on:
  • According to the git log messages, last thing I touched was the implementation of the ElevationProducer
  • And just before that, some implementation/tests around webworkers… I'm wondering what was the conclusion on that point ? 😏
  • Checking the file src/nvland/workers/Mat4Mult.worker.ts:
    // <reference lib="webworker" />
    // import * as log from '../core/Log';
    import { sprintf } from "sprintf-js";
    import { Mat4 } from "../math/Mat4";
    // import { BaseObject } from "../core/BaseObject";
    
    // class MyClass extends BaseObject
    // {
    //     public constructor() {
    //         super();
    //     }
    // };
    
    // function multMat(n: number, a: Mat4, b: Mat4): Mat4
    // {
    //   for(let i=0;i<n;i++) {
    //     Mat4.multiply(a,b,a);
    //   }
    
    //   return a;
    // }
    
    // We send a message back to the main thread
    addEventListener("message", (event) => {
    
        // self.console.log("Received data: "+JSON.stringify(event))
        // let a = new Mat4();
        // a.rotateX(45.0);
        // // Get the limit from the event data
        // let a = new Mat4();
        // a.fromArray(event.data.mat1);
        // let b = new Mat4();
        // b.fromArray(event.data.mat2);
        // let obj = new MyClass();
        // log.DEBUG("Hello!");
        // let msg = sprintf("Hello %s", obj.name);
        let msg = sprintf("Hello %s", "manu");
        // let n = event.data.count;
    
        // let res = multMat(n,a,b);
    
        // Send the primes back to the main thread
        // postMessage({ result: res });
        // postMessage({ result: a.getElements() });
        postMessage({ result: [42] });
        // postMessage({ result: [msg.length] });
    });
    
  • ⇒ This reminds me that I could not successfully “import” some of my other modules and use them in a worker inside a unit test, but I think this was working when executed inside the main app directly 🤔 (or am I dreaming ? ⇒ I should redo that test to confirm it or find where this is done.)
  • Arrrggg…. trying to restore some of the Mat4 computation in the worker and then call that in the main app on init:
        protected async init() {
    
            // Test the webworker:
            let promise = new Promise(function (resolve: ((val: number[]) => void), reject) {
    
                console.log("Creating Mat4Mult...")
                // const worker = new Worker(new URL('./Mat4Mult.worker', import.meta.url));
                const worker = new Worker(new URL('src/nvland/workers/Mat4Mult.worker', import.meta.url));
                // const worker = new Worker('Mat4Mult.worker.ts', {type: "module"})
                worker.onmessage = (event) => {
                    resolve(event.data.result);
                };
    
                let a = Mat4.makeRotateX(43.0);
                let b = Mat4.makeRotateY(12.0);
                console.log("Mat1: " + a.toString());
                console.log("Mat2: " + b.toString());
    
                // worker.onmessage = ({ data }) => {
                //   console.log(`page got message: ${JSON.stringify(data)}`);
                // };
    
                // worker.postMessage({ mat1: a.getElements(), mat2: b.getElements(), count: 100 });
    
                // worker.postMessage('hello');
            });
    
            let res = await promise;
            this.DEBUG("Received result data: " + JSON.stringify(res));
      }

… But right now it seems I only get a black screen out of this 😖 not good… Arrff, no: stupid me, it's working, but I have to uncomment the worker.postMessage() statement of course lol.

  • But now in the unit tests, I get no test executed at all:
    $ landjs_test
    
    > nerv-land-js@0.0.0 test
    > ng test --no-watch --code-coverage
    
    \ Generating browser application bundles (phase: building)...04 02 2022 13:38:22.538:INFO [karma-server]: Karma v6.3.9 server started at http://loca
    lhost:9876/
    04 02 2022 13:38:22.540:INFO [launcher]: Launching browsers ChromeHeadless with concurrency unlimited
    04 02 2022 13:38:22.549:INFO [launcher]: Starting browser ChromeHeadless
    √ Browser application bundle generation complete.
    04 02 2022 13:38:34.874:INFO [Chrome Headless 97.0.4692.99 (Windows 10)]: Connected on socket fO2t8ywxepFNO-UmAAAB with id 68198556
    Chrome Headless 97.0.4692.99 (Windows 10): Executed 0 of 0 SUCCESS (0.005 secs / 0 secs)
    TOTAL: 0 SUCCESS
    
    =============================== Coverage summary ===============================
    Statements   : Unknown% ( 0/0 )
    Branches     : Unknown% ( 0/0 )
    Functions    : Unknown% ( 0/0 )
    Lines        : Unknown% ( 0/0 )
    ================================================================================
    

… which is surprising, because I actually renamed the file Mat4Mult.spec.ts to Mat4Mult.bad.ts before running those tests! 😳 So, what now ?

  • Trying to rename the “Mat4Mult.bad.ts” file to “Mat4Mult.bad.old” ⇒ nope still not working! 😨 So, it this due to my “Mat4Mult.worker.ts” file then ? -> Commenting some content there… [nope], then commenting even more code there… Now it's OK, running 28 tests without error.
  • ⇒ So it is something on how I process the files named “.worker.ts” ? Hmmm 🤔 I kind of remember I was doing something special about that, let's see if I can find it back.
  • Okay, so, in my angular.json file I have that section for the tests:
            "test": {
              "builder": "@angular-builders/custom-webpack:karma",
              "options": {
                "main": "src/test.ts",
                "polyfills": "src/polyfills.ts",
                "tsConfig": "tsconfig.spec.json",
                "karmaConfig": "karma.conf.js",
                "inlineStyleLanguage": "scss",
                "assets": [
                  "src/favicon.ico",
                  "src/assets"
                ],
                "styles": [
                  "src/styles.scss"
                ],
                "customWebpackConfig": {
                  "path": "./webpack.partial.js"
                },
                "scripts": [],
                "webWorkerTsConfig": "tsconfig.worker.json"
              }
            }
  • So we have the webWorkerTsConfig pointing to a dedicated tsconfig file:
    {
      "extends": "./tsconfig.json",
      "compilerOptions": {
        "outDir": "./out-tsc/worker",
        "lib": [
          "es2018",
          "webworker"
        ],
        "types": []
      },
      "include": [
        "src/**/*.worker.ts"
      ]
    }
    

… and in there our reference to the .worker.ts files. ⇒ looking for more infos on that “webWorkerTsConfig” config entry.

  • Okay, so, time for a recap: I could loose a lot of time trying to figure out how to run those unit test on webworkers, so let's cut it short for now:
    • I should ensure that the unit tests do not try to load my complex webworkers, so should use a dedicated extensions for those that can be tested:
    • ⇒ My testing environment will now use the tsconfig.lightworker.ts config and only find the .lightworker.ts scripts.
    • Updated the reference in angular.json as needed: “webWorkerTsConfig”: “tsconfig.lightworker.json”
    • And no… still getting an error with that:
      ./src/nvland/workers/Mat4Mult.worker.ts - Error: Module build failed (from ./node_modules/@ngtools/webpack/src/ivy/index.js):
      Error: D:\Projects\NervLandJS\src\nvland\workers\Mat4Mult.worker.ts is missing from the TypeScript compilation. Please make sure it is in your tscon
      fig via the 'files' or 'include' property.
          at D:\Projects\NervLandJS\node_modules\@ngtools\webpack\src\ivy\loader.js:59:26
    • I gonna cry… 😭😭
  • Okay, so, now, considering if there could be a different way to “load” a webworker script: maybe using soem feature from webpack itself ? Investigating…
  • So now trying with the worker-loader package:
    • First installing the package with npm:
      landjs_npm install --save worker-loader
    • Then no special rule to add in our webpack partial, but we need the typings/worker-loader.d.ts file:
      declare module "worker-loader!*" {
          class WebpackWorker extends Worker {
              constructor();
          }
          export default WebpackWorker;
      }
    • Now we update our worker typescript file to use the framework:
      const ctx: Worker = self as any;
      
      onmessage = async (event) => {
          setTimeout(() => {
              ctx.postMessage(`[WORKER_TS] Waited ${event.data}ms`);
          }, event.data)
      
      }
  • Next we setup loading that worker in the main app:
    import MatWorker from "worker-loader?inline=no-fallback!src/nvland/workers/Mat4Mult.worker.ts";
        
    // Later in the code: 
    const worker = new MatWorker()
    
    worker.onmessage = (event:any) => {
        resolve(event.data.result);
    };
    
    let a = Mat4.makeRotateX(43.0);
    let b = Mat4.makeRotateY(12.0);
    console.log("Mat1: " + a.toString());
    console.log("Mat2: " + b.toString());
    
    worker.postMessage({ mat1: a.getElements(), mat2: b.getElements(), count: 100 });
    

/

  • But of course, this is not working for use and will produce a typescript error when compile the app:
    Error: src/scenes/test7_terrainNode.ts:21:23 - error TS2691: An import path cannot end with a '.ts' extension. Consider importing 'worker-loader?inl
    ine=no-fallback!src/nvland/workers/Mat4Mult.worker.js' instead.
    
    21 import MatWorker from "worker-loader?inline=no-fallback!src/nvland/workers/Mat4Mult.worker.ts";
                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    ** Angular Live Development Server is listening on localhost:4200, open your browser on http://localhost:4200/ **
    
    × Failed to compile.
  • I'm so desperate now lol… I even try loading my worker script as a JS file, and still no luck: this is such a mess…
  • Now trying to follow the instructions from https://v4.webpack.js.org/loaders/worker-loader/ more carefully. Nope. Pointless.
  • OKAY, so now discarding this whole worker-loader usage experiment: I'm not getting anywhere with that unfortunately.
  • hmmm 🤔… Actually, wait a minute… I'm now just realizing that in my previous experiment I actually had 2 locations where I was trying to load my “Mat4Mult.worker.ts” file, but using on one side the worker-loader, and on the other location the webpack 5 mechanism 😐! So basically: I should try that again now before moving to the next option 🤣
  • So installing worker-loader again:
    landjs_npm install --save worker-loader
  • Adding typings/worker-loader.d.ts file:
    declare module "worker-loader!*" {
        class WebpackWorker extends Worker {
            constructor();
        }
        export default WebpackWorker;
    }
  • And updating webpack.partial.json:
    module.exports = {
        module: {
            rules: [
                {
                    enforce: 'pre',
                    test: /\.worker\.ts$/,
                    loader: "worker-loader",
                },
                { 
                    test: /\.glsl$/, 
                    exclude: /node_modules/,
                    loader: 'raw-loader' 
                }
            ]
        }
    };
  • But no: compilation is still failing without any clear error message:
    √ Compiled successfully.
    √ Browser application bundle generation complete.
    
    Initial Chunk Files | Names   |      Size
    main.js             | main    | 306.21 kB
    runtime.js          | runtime |   6.75 kB
    
    3 unchanged chunks
    
    Build at: 2022-02-04T21:27:27.836Z - Hash: 6e2c2425dc9e35b3e572 - Time: 968ms
    
    × Failed to compile.
  • So let's leave that path behind us, and try comlink as suggested above:
  • Installing that package:
    $ landjs_npm install comlink
  • Nayy… ⇒ This might be worth using eventually, but it will not lead us anywhere here: the problem is deeper than that: when we simply have to load one worker file in the testing environment, and it has some non trivial imports…
  • Finally! Found a working/acceptable solution!
    • We are now defining our worker creation directly in the main.ts file:
      // And building the app we should support loading webworkers:
      (window as any).createWorker = function() : any {
        return new Worker(new URL('nvland/workers/Mat4Mult.worker', import.meta.url))
      }
    • And that file and all its dependencies don't get loaded in the testing environment! So we finally don't have to load a .worker.ts file during testing :-) !
    • Feeew… that was an hard one.

⇒ So, the solution we have here is not the cleanest, but I feel happy I have something working at least! So let's call it a day now. Tomorrow I should try to get back to the Elevation Producer

  • public/projects/nervland/0002_landjs_terrainnode_review.txt
  • Last modified: 2022/02/05 22:51
  • by 127.0.0.1