TerrainNode simple scene review
- So last time I was working on this project, I think I was working around this “TerrainNode” scene implementation, which is available in the file src/scenes/test7_terrainNode.ts
- This is the scene currently loaded in the main app when using `landjs_serve` since we have in src/app/app.component.ts:
ngOnInit(): void { // retrieve the WebGL2 context: log.DEBUG("Retriving webGL2 context..."); let ctx = this.view.nativeElement.getContext('webgl2'); RenderContext.setCurrent(ctx); log.DEBUG("Assigned Render context!"); // Create a resource loader here: let loader = new ResourceLoader(this.http); // let scene = new Test2Scene(); // let scene = new Test2Scene(this.view.nativeElement); // let scene = new Test3Scene(this.view.nativeElement); // let scene = new Test4Scene(this.view.nativeElement); // let scene = new Test5Scene(this.view.nativeElement, loader); // let scene = new Test6Scene(this.view.nativeElement, loader); let scene = new Test7Scene(this.view.nativeElement, loader); }
- We have a loader passed as argument here: this reminds me of how it works (roughtly): this loader is responsible for loading the “actual resources” in the context of the angular app (I think 😋…)
- Most (all) of the important code is in the src/nvland folder. And in there we have multiple sub folders for the main sub-systems like “core”, “ecs”, “math”, “render”, etc.
- Which reminds me that lately I spend a lot of time on the Entity Component System (ECS), but that's not the last thing I was working on:
- According to the git log messages, last thing I touched was the implementation of the ElevationProducer
- And just before that, some implementation/tests around webworkers… I'm wondering what was the conclusion on that point ? 😏
- Checking the file src/nvland/workers/Mat4Mult.worker.ts:
// <reference lib="webworker" /> // import * as log from '../core/Log'; import { sprintf } from "sprintf-js"; import { Mat4 } from "../math/Mat4"; // import { BaseObject } from "../core/BaseObject"; // class MyClass extends BaseObject // { // public constructor() { // super(); // } // }; // function multMat(n: number, a: Mat4, b: Mat4): Mat4 // { // for(let i=0;i<n;i++) { // Mat4.multiply(a,b,a); // } // return a; // } // We send a message back to the main thread addEventListener("message", (event) => { // self.console.log("Received data: "+JSON.stringify(event)) // let a = new Mat4(); // a.rotateX(45.0); // // Get the limit from the event data // let a = new Mat4(); // a.fromArray(event.data.mat1); // let b = new Mat4(); // b.fromArray(event.data.mat2); // let obj = new MyClass(); // log.DEBUG("Hello!"); // let msg = sprintf("Hello %s", obj.name); let msg = sprintf("Hello %s", "manu"); // let n = event.data.count; // let res = multMat(n,a,b); // Send the primes back to the main thread // postMessage({ result: res }); // postMessage({ result: a.getElements() }); postMessage({ result: [42] }); // postMessage({ result: [msg.length] }); });
- ⇒ This reminds me that I could not successfully “import” some of my other modules and use them in a worker inside a unit test, but I think this was working when executed inside the main app directly 🤔 (or am I dreaming ? ⇒ I should redo that test to confirm it or find where this is done.)
- Arrrggg…. trying to restore some of the Mat4 computation in the worker and then call that in the main app on init:
protected async init() { // Test the webworker: let promise = new Promise(function (resolve: ((val: number[]) => void), reject) { console.log("Creating Mat4Mult...") // const worker = new Worker(new URL('./Mat4Mult.worker', import.meta.url)); const worker = new Worker(new URL('src/nvland/workers/Mat4Mult.worker', import.meta.url)); // const worker = new Worker('Mat4Mult.worker.ts', {type: "module"}) worker.onmessage = (event) => { resolve(event.data.result); }; let a = Mat4.makeRotateX(43.0); let b = Mat4.makeRotateY(12.0); console.log("Mat1: " + a.toString()); console.log("Mat2: " + b.toString()); // worker.onmessage = ({ data }) => { // console.log(`page got message: ${JSON.stringify(data)}`); // }; // worker.postMessage({ mat1: a.getElements(), mat2: b.getElements(), count: 100 }); // worker.postMessage('hello'); }); let res = await promise; this.DEBUG("Received result data: " + JSON.stringify(res)); }
… But right now it seems I only get a black screen out of this 😖 not good… Arrff, no: stupid me, it's working, but I have to uncomment the worker.postMessage() statement of course lol.
- But now in the unit tests, I get no test executed at all:
$ landjs_test > nerv-land-js@0.0.0 test > ng test --no-watch --code-coverage \ Generating browser application bundles (phase: building)...04 02 2022 13:38:22.538:INFO [karma-server]: Karma v6.3.9 server started at http://loca lhost:9876/ 04 02 2022 13:38:22.540:INFO [launcher]: Launching browsers ChromeHeadless with concurrency unlimited 04 02 2022 13:38:22.549:INFO [launcher]: Starting browser ChromeHeadless √ Browser application bundle generation complete. 04 02 2022 13:38:34.874:INFO [Chrome Headless 97.0.4692.99 (Windows 10)]: Connected on socket fO2t8ywxepFNO-UmAAAB with id 68198556 Chrome Headless 97.0.4692.99 (Windows 10): Executed 0 of 0 SUCCESS (0.005 secs / 0 secs) TOTAL: 0 SUCCESS =============================== Coverage summary =============================== Statements : Unknown% ( 0/0 ) Branches : Unknown% ( 0/0 ) Functions : Unknown% ( 0/0 ) Lines : Unknown% ( 0/0 ) ================================================================================
… which is surprising, because I actually renamed the file Mat4Mult.spec.ts to Mat4Mult.bad.ts before running those tests! 😳 So, what now ?
- Trying to rename the “Mat4Mult.bad.ts” file to “Mat4Mult.bad.old” ⇒ nope still not working! 😨 So, it this due to my “Mat4Mult.worker.ts” file then ? -> Commenting some content there… [nope], then commenting even more code there… Now it's OK, running 28 tests without error.
- ⇒ So it is something on how I process the files named “.worker.ts” ? Hmmm 🤔 I kind of remember I was doing something special about that, let's see if I can find it back.
- Okay, so, in my angular.json file I have that section for the tests:
"test": { "builder": "@angular-builders/custom-webpack:karma", "options": { "main": "src/test.ts", "polyfills": "src/polyfills.ts", "tsConfig": "tsconfig.spec.json", "karmaConfig": "karma.conf.js", "inlineStyleLanguage": "scss", "assets": [ "src/favicon.ico", "src/assets" ], "styles": [ "src/styles.scss" ], "customWebpackConfig": { "path": "./webpack.partial.js" }, "scripts": [], "webWorkerTsConfig": "tsconfig.worker.json" } }
- So we have the webWorkerTsConfig pointing to a dedicated tsconfig file:
{ "extends": "./tsconfig.json", "compilerOptions": { "outDir": "./out-tsc/worker", "lib": [ "es2018", "webworker" ], "types": [] }, "include": [ "src/**/*.worker.ts" ] }
… and in there our reference to the .worker.ts files. ⇒ looking for more infos on that “webWorkerTsConfig” config entry.
- Found this page: https://github.com/angular/angular-cli/issues/19372
- ⇒ Maybe I need
"exclude": ["* */*.worker.ts"]
in the other tsconfig files ? ⇒ nope, this doesn't seem to help 😭
- Okay, so, time for a recap: I could loose a lot of time trying to figure out how to run those unit test on webworkers, so let's cut it short for now:
- I should ensure that the unit tests do not try to load my complex webworkers, so should use a dedicated extensions for those that can be tested:
- ⇒ My testing environment will now use the tsconfig.lightworker.ts config and only find the .lightworker.ts scripts.
- Updated the reference in angular.json as needed:
“webWorkerTsConfig”: “tsconfig.lightworker.json”
- And no… still getting an error with that:
./src/nvland/workers/Mat4Mult.worker.ts - Error: Module build failed (from ./node_modules/@ngtools/webpack/src/ivy/index.js): Error: D:\Projects\NervLandJS\src\nvland\workers\Mat4Mult.worker.ts is missing from the TypeScript compilation. Please make sure it is in your tscon fig via the 'files' or 'include' property. at D:\Projects\NervLandJS\node_modules\@ngtools\webpack\src\ivy\loader.js:59:26
- I gonna cry… 😭😭
- Okay, so, now, considering if there could be a different way to “load” a webworker script: maybe using soem feature from webpack itself ? Investigating…
- So now trying with the worker-loader package:
- First installing the package with npm:
landjs_npm install --save worker-loader
- Then no special rule to add in our webpack partial, but we need the typings/worker-loader.d.ts file:
declare module "worker-loader!*" { class WebpackWorker extends Worker { constructor(); } export default WebpackWorker; }
- Now we update our worker typescript file to use the framework:
const ctx: Worker = self as any; onmessage = async (event) => { setTimeout(() => { ctx.postMessage(`[WORKER_TS] Waited ${event.data}ms`); }, event.data) }
- Next we setup loading that worker in the main app:
import MatWorker from "worker-loader?inline=no-fallback!src/nvland/workers/Mat4Mult.worker.ts"; // Later in the code: const worker = new MatWorker() worker.onmessage = (event:any) => { resolve(event.data.result); }; let a = Mat4.makeRotateX(43.0); let b = Mat4.makeRotateY(12.0); console.log("Mat1: " + a.toString()); console.log("Mat2: " + b.toString()); worker.postMessage({ mat1: a.getElements(), mat2: b.getElements(), count: 100 });
/
- But of course, this is not working for use and will produce a typescript error when compile the app:
Error: src/scenes/test7_terrainNode.ts:21:23 - error TS2691: An import path cannot end with a '.ts' extension. Consider importing 'worker-loader?inl ine=no-fallback!src/nvland/workers/Mat4Mult.worker.js' instead. 21 import MatWorker from "worker-loader?inline=no-fallback!src/nvland/workers/Mat4Mult.worker.ts"; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ** Angular Live Development Server is listening on localhost:4200, open your browser on http://localhost:4200/ ** × Failed to compile.
- I'm so desperate now lol… I even try loading my worker script as a JS file, and still no luck: this is such a mess…
- Now trying to follow the instructions from https://v4.webpack.js.org/loaders/worker-loader/ more carefully. Nope. Pointless.
- ⇒ OKAY, so now discarding this whole worker-loader usage experiment: I'm not getting anywhere with that unfortunately.
- So, trying with comlink now: https://blog.lacolaco.net/2018/12/enjoyable-webworkers-in-angular/ 🥴 ?
- hmmm 🤔… Actually, wait a minute… I'm now just realizing that in my previous experiment I actually had 2 locations where I was trying to load my “Mat4Mult.worker.ts” file, but using on one side the worker-loader, and on the other location the webpack 5 mechanism 😐! So basically: I should try that again now before moving to the next option 🤣
- So installing worker-loader again:
landjs_npm install --save worker-loader
- Adding typings/worker-loader.d.ts file:
declare module "worker-loader!*" { class WebpackWorker extends Worker { constructor(); } export default WebpackWorker; }
- And updating webpack.partial.json:
module.exports = { module: { rules: [ { enforce: 'pre', test: /\.worker\.ts$/, loader: "worker-loader", }, { test: /\.glsl$/, exclude: /node_modules/, loader: 'raw-loader' } ] } };
- But no: compilation is still failing without any clear error message:
√ Compiled successfully. √ Browser application bundle generation complete. Initial Chunk Files | Names | Size main.js | main | 306.21 kB runtime.js | runtime | 6.75 kB 3 unchanged chunks Build at: 2022-02-04T21:27:27.836Z - Hash: 6e2c2425dc9e35b3e572 - Time: 968ms × Failed to compile.
- So let's leave that path behind us, and try comlink as suggested above:
- Installing that package:
$ landjs_npm install comlink
- Nayy… ⇒ This might be worth using eventually, but it will not lead us anywhere here: the problem is deeper than that: when we simply have to load one worker file in the testing environment, and it has some non trivial imports…
- Finally! Found a working/acceptable solution!
- We are now defining our worker creation directly in the main.ts file:
// And building the app we should support loading webworkers: (window as any).createWorker = function() : any { return new Worker(new URL('nvland/workers/Mat4Mult.worker', import.meta.url)) }
- And that file and all its dependencies don't get loaded in the testing environment! So we finally don't have to load a .worker.ts file during testing !
- Feeew… that was an hard one.
⇒ So, the solution we have here is not the cleanest, but I feel happy I have something working at least! So let's call it a day now. Tomorrow I should try to get back to the Elevation Producer