blog:2021:0406_nervluna_practical_usage_case_part_1

NervLuna: A first practical usage case - Part 1

Hi readers! So here we are back on our NervLuna project for lua bindings generation… Or at least, that's what I “kind of” remember it was about ? I mean, all this stuff seems so far away already lol. But anyway, lately I've been working on lua bindings generation for one of the other professional projects I'm working on (ie. the SimCore project).

I had to get something very quickly on rails, so I immediately turned to the Sol3 library of course. And oh boy… (don't get me wrong, I really like the sol3 library, the syntax is cool, you have all the needed features and all in there, but…) this experiment definetely reminded me how complex it was to use that library properly and why I decided to start working on the nervluna project in the first place!

So here I am again: back on NervLuna, and I want to try to use it to generate the bindings I need in SimCore while those bindings are still not too big. So let's get started!

Hmmm, let's be honest, I haven't been working on this project for ages… and given my really poor memory I hardly remember anything about it ah ah ah LOL. The first question is thus: “Where should I start from ?” ⇒ Let me re-read the first article on this project to see if this helps refreshing my mind a bit.

Okay, so first thing I'm starting to remember here is that I was using lua to run the binding generation, given some configuration file, soo let's see if I can find again an example of how this was run in my sources… [🔎searching…] OK! So now I have something else: just found in my “dist” folder that I have a collection of lua files, and among those files I have definition of “lua apps” entry points! And a very tiny tiny voice in my head is telling me that the lua app I've need using to generate bindings is this NervBind.lua app.

So now I just need to find where I'm using this “NervBind” script filename… [🔎searching again with visual studio code search feature… but finding nothing this time] Hmmm, this would have been too easy of course. So it seems I need to put some more thinking here.

Oh! One more step forward now: I was checking the binary folders in my “dist” folder and realized there was no executable file (and not even a luajit.exe), so I tried to rebuild all my code from sources, with one of my commands that I still remember [surprising isn't it ?]:

nv_build msvc64

This is producing a great lot of errors lol. Still, eventually it will install a few files:

Install the project...
-- Install configuration: "Release"
-- Up-to-date: D:/Projects/NervSeed/dist/bin/msvc64/./cef_process.exe
-- Up-to-date: D:/Projects/NervSeed/dist/bin/msvc64/./cef_process.pdb
-- Up-to-date: D:/Projects/NervSeed/dist/bin/msvc64/./cef_process.map
-- Installing: D:/Projects/NervSeed/dist/bin/msvc64/./nvCore.dll
-- Installing: D:/Projects/NervSeed/dist/bin/msvc64/./nvCore.pdb
-- Installing: D:/Projects/NervSeed/dist/bin/msvc64/./nvCore.map
-- Installing: D:/Projects/NervSeed/dist/bin/msvc64/./nvCEF.dll
-- Installing: D:/Projects/NervSeed/dist/bin/msvc64/./nvCEF.pdb
-- Installing: D:/Projects/NervSeed/dist/bin/msvc64/./nvCEF.map
-- Installing: D:/Projects/NervSeed/dist/bin/msvc64/./nvLand.dll
-- Installing: D:/Projects/NervSeed/dist/bin/msvc64/./nvLand.pdb
-- Installing: D:/Projects/NervSeed/dist/bin/msvc64/./nvLand.map
-- Installing: D:/Projects/NervSeed/dist/bin/msvc64/./nvView.dll
-- Installing: D:/Projects/NervSeed/dist/bin/msvc64/./nvView.pdb
-- Installing: D:/Projects/NervSeed/dist/bin/msvc64/./nvView.map
CMake Error at sources/nvVulkan/src/cmake_install.cmake:41 (file):
  file INSTALL cannot find
  "D:/Projects/NervSeed/build/msvc64/sources/nvVulkan/src/nvVulkan.dll": No
  error.
Call Stack (most recent call first):
  sources/nvVulkan/cmake_install.cmake:42 (include)
  sources/cmake_install.cmake:49 (include)
  cmake_install.cmake:42 (include)

… But still no executable generated here! Yet this made me realize that the executable I'm after is my NervSeed app! And that's most probably the one I'm using to load/execute lua apps too. So my first step here should be to rebuild NervSeed correctly: let's try that.

Getting the NervSeed sources to compile was actually pretty easy: I just had to disable all the non compiling modules and stuff:

  • nvVulkan: will not compile because I didn't even install the vulkan runtime yet on my computer, so this is just normal I guess.
  • nvOCCT: I had some serious problems/errors while quickly trying to build the openCASCADE dependency, so I simply bypassed that for now, and of course this module will not compile anymore for now.
  • lua_bindings: I had a lot of errors in the compilation here too so just starting with this disabled first.

So basically I have the CmakeList content for my sources folder:

ADD_SUBDIRECTORY(soil2)
ADD_SUBDIRECTORY(libcef_dll)
ADD_SUBDIRECTORY(cef_process)
ADD_SUBDIRECTORY(nvCore)
ADD_SUBDIRECTORY(nvCEF)
ADD_SUBDIRECTORY(nvLand)
# ADD_SUBDIRECTORY(nvPy)
ADD_SUBDIRECTORY(nvView)
# ADD_SUBDIRECTORY(nvVulkan)
ADD_SUBDIRECTORY(nvDX12)
ADD_SUBDIRECTORY(nvGL)
# ADD_SUBDIRECTORY(nvOCCT)
ADD_SUBDIRECTORY(nvLLVM)
# ADD_SUBDIRECTORY(NervSeed0)
ADD_SUBDIRECTORY(NervSeed)
 
ADD_SUBDIRECTORY(llvm_syms)
# ADD_SUBDIRECTORY(lua_bindings)
ADD_SUBDIRECTORY(lua_lfs)
 
# if(${FLAVOR} STREQUAL "WIN64")
#     ADD_SUBDIRECTORY(nvUnity)
#     ADD_SUBDIRECTORY(nvUnityProvider)
#     ADD_SUBDIRECTORY(scUnityGateway)
# ENDIF()

Then from the tests/ folder I also had to disable my test_nvVulkan and test_vulkan_win tests obviously.

⇒ And then I got NervSeed rebuilt as desired! Yeepeee!

Checking the code of the NervSeed app, the “main()” function is actually very minimal and it will delegate everything directly to a Lua environment from what I see [Yes I know… it's strange to speak this way of your own code lol], disregarding the error/exception handling parts we only have this:

		LogManager::instance().setNotifyLevel(LogManager::DEBUG0);

		{
			// First we retrieve the NervApp instance:
			auto& app = NervApp::instance();

			// First we retrieve the LuaManager:
			auto& lm = LuaManager::instance();

			// We assign the command line arguments to the NervApp:
			{
				auto& state = lm.getMainState();
				auto t = state["arg"].get_or_create<sol::table>();

				for(U32 i=0; i<argc; ++i) {
					t.set(i, argv[i]);
				}
			}

			// std::cout << "Running app with main thread "<<std::this_thread::get_id()<<std::endl;
			lm.executeFile("run.lua");
		}

        // Finally we destroy the app:
		// std::cout<<"Destroying NervApp"<<std::endl;
        NervApp::destroy();

		logDEBUG2("Destroying memory manager.");
		MemoryManager::destroy();

		std::cout<<"Exiting."<<std::endl;
		return 0;

So here we only collect all the provided arguments into the lua “arg” table, and then we call the “run.lua” script. Cool, this one, I already know where to find: in dist/assets/lua :-). Let's have a look:

logDEBUG3("Lua: this is the main run script.")
-- logDEBUG("Lua: the provided arguments are: '",arg,"'")
-- logDEBUG("Num arg: ", #arg)
-- for i=0,#arg do
--     logDEBUG("arg ",i,": ", arg[i])
-- end

-- The app name should always be the first argument we provide:
if #arg < 1 then
    logERROR("No command provided.")
    return
end


-- Otherwise we first check if the first argument correspond to the valid app name:
local cmd = arg[1]
table.remove(arg,1)

-- Prepare a command name mapping
local commandMap = {
    vkapp = "app.VulkanApp",
    sdlapp = "app.SDLApp",
    bind = "app.NervBind",
    bind1 = {"app.NervBind", "-c", "config/bind_test.lua"},
    bind2 = {"app.NervBind", "-c", "config/bind_vulkan.lua"},
    bind3 = {"app.NervBind", "-c", "config/bind_core.lua"},
    -- bind3d = {"app.NervBind", "-c", "config/bind_core.lua", "--debug"},
    parse1 = {"app.NervBind", "-p", "sources/lua_bindings/test1/interface/bind_test1.h"},
    parse1b = {"app.NervBind", "-p", "sources/lua_bindings/test1/interface/bind_test2.h"},
    parse1c = {"app.NervBind", "-p", "sources/lua_bindings/test1/interface/bind_test3.h"},
    parse1d = {"app.NervBind", "-p", "sources/lua_bindings/test1/interface/bind_test4.h"},
    parse1e = {"app.NervBind", "-p", "sources/lua_bindings/test1/interface/bind_test5.h"},
    parse2 = {"app.NervBind", "-p", "sources/lua_bindings/wip_Vulkan/interface/vk_layer.h"},
    tests = "app.Telescope",
    bind0 = "app.NervBind_v0",
    tsgen = "app.TSDclGen",
    test1 = "test.Test1"
}

-- Check if we have a mapping for that command name:
local rep = commandMap[cmd]

if type(rep) == "string" then
    cmd = rep
elseif type(rep) == "table" then
    cmd = rep[1]
    local newargs = {}
    for i=2,#rep do
        table.insert(newargs, rep[i])
    end
    -- Add the remaining command line args:
    for i=1,#arg do
        table.insert(newargs, arg[i])
    end

    -- Replace the command line args:
    arg = newargs
end

if hasScript(cmd) then
    -- We run that script providing it all the input args except thi script name itself:
    local app = import(cmd)
    app(arg)
    return
end

logERROR("Unknown script: ", cmd)

And now, this is all starting to fall back in place: I now remember this command mapping I was using to avoid having to type long command lines all the time! So here for instance, I should be able to call `NervSeed bind1` and this should automatically call the “NervBind” app giving it the config file “config/bind_test.lua”

On top of that, now I'm also remembering I had a dedicated shell script nv_seed to call NervSeed from any location with additional setups, and here it is:

nv_seed()
{
  local flavor="linux64"
  local suffix=""
  local pname=`nv_os`
  case "${pname}" in
  cygwin|wsl)
    flavor="msvc64"
    suffix=".exe"
    ;;
  *)
    export LD_LIBRARY_PATH="`nv_get_project_dir`/tools/linux/${__nv_tool_gcc}/lib64"
    ;;
  esac

  export VK_INSTANCE_LAYERS="VK_LAYER_LUNARG_standard_validation"

  local tapp="`nv_get_dist_dir`/bin/$flavor/NervSeed$suffix"
  $tapp "$@"

  # unset LD_LIBRARY_PATH
}

⇒ So now trying the command: nv_seed bind1 [crossing fingers…] Arff, maybe this was a little bit too pretencious afterall:

Remember that I disabled the building of all the lua modules above, but the luaCode.dll is one of then, and this is definitely required in NervSeed, so we need to build at least that one ⇒ fixing this an rebuilding: OK

And now that I'm thinking about it, I'm pretty sure I will need at least also the Clang lua module: since we will use clang to parse the input source files… but let's predend I didn't mention that yet.

Trying the command nv_seed bind1 again… and sure enough we have an error on luaClang:

So I activated the build of the luaClang module and after that, it worked! Great!:

$ nv_seed bind1
[Debug]               Loading dynamic library: luaClang.dll
[Debug]               Parsed args: {
  config = "config/bind_test.lua"
}
[Debug]               Loading config file: config/bind_test.lua
[Debug]               Using include path: D:\Projects\NervSeed\sources\lua_bindings\test1\interface
[Debug]               Found 26 existing files.
[Debug]               Using input file: D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h
[Debug]               Using input file: D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test2.h
[Debug]               Using input file: D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test3.h
[Debug]               Using input file: D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test4.h
[Debug]               Using input file: D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test5.h
[Debug]               Writing file D:\Projects\NervSeed\sources\lua_bindings\test1\luna_bindings.h
[Debug]               No change in file D:\Projects\NervSeed\sources\lua_bindings\test1\luna_bindings.h
[Debug]               Command line args: {
  "-x",
  "c++",
  "-std=c++17",
  "-ID:\\Projects\\NervSeed\\sources\\lua_bindings\\test1\\include",
  "-ID:\\Projects\\NervSeed\\sources\\lua_bindings\\test1\\interface"
}
[Debug]               Parsing file: D:\Projects\NervSeed\sources\lua_bindings\test1\luna_bindings.h
[Info]     Num diagnostics for file D:\Projects\NervSeed\sources\lua_bindings\test1\luna_bindings.h: 3
[Info]      - diag 1: D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:185:31: warning: direct base 'nvt::OtherClass' is inaccessible due to ambiguity:
    class nvt::ClassC -> class nvt::ClassB -> class nvt::OtherClass
    class nvt::ClassC -> class nvt::OtherClass [-Winaccessible-base]
[Info]      - diag 2: D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:232:15: warning: anonymous non-C-compatible type given name for linkage purposes by t
ypedef declaration; add a tag name here [-Wnon-c-typedef-for-linkage]
[Info]      - diag 3: D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:291:15: warning: anonymous non-C-compatible type given name for linkage purposes by t
ypedef declaration; add a tag name here [-Wnon-c-typedef-for-linkage]
[Debug]               Processing: 'my_enum' kind: 'EnumDecl
[Debug]               Creating enum ::my_enum
[Debug]               Adding enum constant: my_enum::ENUM_VALUE1
[Debug]               Adding enum constant: my_enum::ENUM_VALUE2
[Debug]               Adding enum constant: my_enum::ENUM_VALUE3

(... much more outputs here ... )

[Debug]               Writing file D:\Projects\NervSeed\sources\lua_bindings\test1\src/luna/bind_std_string.cpp
[Debug]               No change in file D:\Projects\NervSeed\sources\lua_bindings\test1\src\luna\bind_std_string.cpp
[Debug]               Writing class binding for std::basic_ostream<char,std::char_traits<char>>
[Debug]               Preprocessing function list...
[Debug]               Keeping 0 on 0
[Debug]               Writing function type checks...
[Debug]               Writing function bindings...
[Debug]               Done writing function bindings.
[Debug]               Preprocessing field list...
[Debug]               Keeping 0/0 fields.
[Debug]               Writing fields type checks...
[Debug]               Writing fields bindings...
[Debug]               Done writing fields bindings.
[Debug]               Preprocessing field list...
[Debug]               Keeping 0/0 fields.
[Debug]               Writing fields type checks...
[Debug]               Writing fields bindings...
[Debug]               Done writing fields bindings.
[Debug]               Writing file D:\Projects\NervSeed\sources\lua_bindings\test1\src/luna/bind_std_basic_ostream_char_std_char_traits_char.cpp
[Debug]               Updating content of D:\Projects\NervSeed\sources\lua_bindings\test1\src\luna\bind_std_basic_ostream_char_std_char_traits_char.cpp
[Debug]               Writing Package bindings...
[Debug]               Writing file D:\Projects\NervSeed\sources\lua_bindings\test1\src/luna/register_package.cpp
[Debug]               Updating content of D:\Projects\NervSeed\sources\lua_bindings\test1\src\luna\register_package.cpp
[Debug]               Done writing bindings.
[Debug]               Searching for obsolete files...
[Debug]               Removing obsolete/unused file D:\Projects\NervSeed\sources\lua_bindings\test1\src\luna\bind_std_ostream.cpp
[Debug]               Done searching for obsolete files.
[Debug]               Uninitializing Clang parser.
[Debug]               Disposing cidx
[Debug]               Done uninitializing Clang parser.
Deleted LogManager object.
Looking for memory leaks...
No memory leak detected.
Exiting.

The previous call of 'nv_seed bind1' seems to have been mostly working fine, but checking the debug outputs more carefully, I eventually noticed this exception somewhere in the middle of the processing:

[Warning]       No class template registered for 'MyTemplateTest<1>' creating default place holder class in namespace 'nvt'
[Debug]               Creating class nvt::MyTemplateTest<1>
[Debug]               Creating concrete class nvt::MyTemplateTest<1> for initial type name: nvt::MyTemplateTest<true>
[Error]         Error in cursorVisitor: [string "bind.ClangParser"]:366: attempt to index local 't' (a nil value)
stack traceback:
        [string "bind.ClangParser"]:366: in function 'parseTemplateType'
        [string "bind.ClangParser"]:1121: in function 'parseClass'
        [string "bind.ClangParser"]:1381: in function <[string "bind.ClangParser"]:1352>
        [C]: in function 'visitChildren'
        [string "bind.ClangParser"]:1352: in function 'parseNamespace'
        [string "bind.ClangParser"]:1379: in function <[string "bind.ClangParser"]:1352>
        [C]: in function 'visitChildren'
        [string "bind.ClangParser"]:1352: in function 'parseNamespace'
        [string "bind.ClangParser"]:1418: in function 'parseFile'
        [string "app.NervBind"]:137: in function 'parseInputs'
        [string "app.NervBind"]:154: in function 'run'
        [string "app.AppBase"]:23: in function '__init'
        [string "app.NervBind"]:12: in function '__init'
        [string "setup.lua"]:155: in function 'app'
        [string "run.lua"]:63: in main chunk
[Debug]               clang done parsing file: D:\Projects\NervSeed\sources\lua_bindings\test1\luna_bindings.h
[Debug]               Resolving target types...

So this error occurs in parseTemplateType(), but from what I see that would be because the type parameter “t” that we receive in that function call is already nil: and this should come from the parseClass calls just before, so let's check that one.

arrf, and indeed in the parseClass() function we have this:

logDEBUG("Creating template specialization: ", tname)
        t = self:parseTemplateType(nil, cxtype, tname)
        CHECK(t, "Cannot parse template type: ",tname)

⇒ So indeed the first parameter (ie. “t”) is nil in that case :-S. And right now I'm starting to have an hard time understanding what this all means exactly lol.

Anyway, just noticed that at other locations I will create “Type” object when required, so let's do that here too:

-- @luna2: If no type is provided already then we create one here with tname:
        if t == nil then
          t = Type(tname)
        end

⇒ And with that change the binding code generation seems to work fine without errors! OK

Before trying to build our “newly” generated bindings for this test1 project, I thought I should at least have a quick look at the generated code… just in case… And of course, here is what I found in the first file I opened:

// Typecheck for (anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5) (1) with signature: void ()
static bool _check__anonymous_union_at_D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5_sig1(lua_State* L) {
	int luatop = lua_gettop(L);
	if( luatop!=0 ) return false;

	return true;
}


// Function bindings

// Bind for (anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5) constructor (1) with signature: void ()
static nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5)* _bind__anonymous_union_at_D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5_sig1(lua_State* L) {

	auto res = luna_callDefaultConstructor<nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5)>(L);
	if(res == nullptr) luaL_error(L, "Cannot default construct instance of class nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5).");
	return res;
}

… 🤢… Yeahh… I always feel the same with those kind of errors: “How on earth could this happen, in something that was working OK so far and that I didn't touch for months ?!”. But eventually you'll realize it is just always like this, “bugs” just appear out of nowhere! 🥳

Anyway, let's now try to investigate where this could come from. Point is, I'm pretty sure I had something in place already to handle (ie. rename) some “anonymous things”, but maybe this is currently not applied on unions ?

So the problem seems to come from this code section (inside my StrutB structure definition):

    union {
        int intValue;
        bool boolValue;
    } my_anon_union;

And now I'm thinking this union finding part could be something new that was not happening at all before. Time to gather some debug inputs from clang on our little bind_test1.h file with:

nv_seed parse1 | tee parse1.log

So this gives us the following outputs:

[Debug] 	      level 3: '' of kind 'UnionDecl', type: 'nvt::StructB::(anonymous union at sources/lua_bindings/test1/interface/bind_test1.h:217:5)'
WARN: clang_getTypeFullSpelling: no decl found for int
[Debug] 	      level 4: 'intValue' of kind 'FieldDecl', type: 'int'
WARN: clang_getTypeFullSpelling: no decl found for bool
[Debug] 	      level 4: 'boolValue' of kind 'FieldDecl', type: 'bool'
[Debug] 	      level 3: 'my_anon_union' of kind 'FieldDecl', type: 'nvt::StructB::(anonymous union at sources/lua_bindings/test1/interface/bind_test1.h:217:5)'
[Debug] 	      level 4: '' of kind 'UnionDecl', type: 'nvt::StructB::(anonymous union at sources/lua_bindings/test1/interface/bind_test1.h:217:5)'
WARN: clang_getTypeFullSpelling: no decl found for int
[Debug] 	      level 5: 'intValue' of kind 'FieldDecl', type: 'int'
WARN: clang_getTypeFullSpelling: no decl found for bool
[Debug] 	      level 5: 'boolValue' of kind 'FieldDecl', type: 'bool'

Right now in ClangParser, when I find a “UnionDecl”, I simply write a new class for it apparently:

elseif ckind == clang.CursorKind.ClassDecl or ckind == clang.CursorKind.StructDecl or ckind == clang.CursorKind.UnionDecl then
      obj = self:parseClass(cur, scope, cname)

This can make sense to some extents because we have “fields” defined in this union, but then if the union is anonymous (and that's most cases I believe ?) it doesn't make sense to write class bindings for it. So I need to deal with that first point.

⇒ So as a first step I decided I should introduce a new entity type for “UNION”, and create a “Union” class as well as an “UnionHolder” as below:

local Class = createClass{name="Union", bases={"reflection.Scope", "reflection.FieldHolder"}}

local luna = import "bind.LunaManager"

function Class:__init(name, parent)
  Class.__initBases(self, name, parent)
  self:setEntityType(self.EntityType.UNION)
end

function Class:isAnonymous()
  return self._name:startsWith(luna.anonymous_union_prefix)
end

function Class:isBindable()
  return not self:isIgnored()
end

return Class
local Class = createClass{name="UnionHolder", bases="base.Object"}
local Vis = import "reflection.Visibility"

function Class:__init(name, parent)
  Class.super.__init(self)
  self._children.unions = {}
end

function Class:getOrCreateUnion(name)
  CHECK(name~="", "Invalid union name.")
  for _,obj in ipairs(self._children.unions) do
    if obj:getName() == name then
      return obj
    end
  end

  -- We should create this child namespace:
  logDEBUG("Creating enum ", self:getFullName(), "::", name)
  local Union = import "reflection.Union"
  
  local obj = Union(name, self)
  table.insert(self._children.unions, obj)
  return obj
end

-- Retrieve all the public unions
function Class:getPublicUnions(tt)
  tt = tt or {}
  for _,el in ipairs(self._children.unions) do
    if el:getVisibility() == Vis.PUBLIC then
      table.insert(tt, el)
    end
  end

  return tt
end

return Class
The Union class is also a “FieldHolder” so that I can then add fields like “int intValue” or “bool boolValue” inside the union definition.

Then I simply added the UnionHolder base for my “Class” and “Namesapce” classes. And I updated the handling of Unions declarations to be similar to what I was doing for enums:

elseif ckind == clang.CursorKind.UnionDecl then
      -- @luna3: We can also have anonymous unions in fact:
      if cname=="" then
        cname = luna:getNextName(luna.anonymous_union_prefix)
        logDEBUG("Registering anonymous union '", cname, "' with type ", typename)
      end
      
      -- We add that enum to the current scope:
      obj = scope:getOrCreateUnion(cname)
      self:parseUnion(cur, obj)

And I built the required dedicated function to parse unions of course:

function Class:parseUnion(cursor, union)
  if union:isParsed() then
    logDEBUG2("Union ", union:getFullName()," is already parsed.")
    return;
  end
  
  if not cursor:isDefinition() then
    -- Nothing more to do here, we won't get any definition from this cursor.
    return
  end

  -- Keep going, parsing the class:
  union:setParsed(true)

  -- We retrieve all the values inside that enum:
  cursor:visitChildren(function(cur, parent)
    local ckind = cur:getKind()
    CHECK(ckind == clang.CursorKind.FieldDecl, "Unexpect cursor type in union: ", cursor:getKindSpelling())
    
    local cname = cur:getSpelling()
    
    obj = union:getOrCreateField(cname)
    self:parseField(cur, obj)

    return clang.VisitResult.Continue
  end)
end

… So now let's see how lucky/unlucky I can be with those changes… (⇒ building again with “nv_seed bind1 | tee bind1.log”) And okay, now the processing gives us:

[Debug] 	      Registering anonymous union 'luna_anonymous_union_2' with type nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5)
[Debug] 	      Creating enum nvt::StructB::luna_anonymous_union_2
[Debug] 	      Creating field nvt::StructB::luna_anonymous_union_2::intValue
WARN: clang_getTypeFullSpelling: no decl found for int
[Debug] 	      Creating field nvt::StructB::luna_anonymous_union_2::boolValue
WARN: clang_getTypeFullSpelling: no decl found for bool
[Debug] 	      Creating field nvt::StructB::my_anon_union
[Debug] 	      Resolving type nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5)
[Debug] 	      Type kind: Elaborated
[Debug] 	      Adding type with class target name: nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5), (canonical type: nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5))
[Debug] 	      TypeManager: Registering type {
  "nvt::StructB::(anonymous union at D:\\Projects\\NervSeed\\sources\\lua_bindings\\test1\\interface\\bind_test1.h:217:5)"
}

But I still have (some of) those crazy bindings generated ggrrrrrr!! (yet the “typecheck/function bindings” previously reported are now gone):

nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5)* LunaTraits< nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5) >::construct(lua_State* L) {
	luaL_error(L, "No public constructor available for class nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5)");
	return nullptr;
}

void LunaTraits< nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5) >::destruct(nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5)* obj) {
	THROW_MSG("No public destructor available for class nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5)");
}

The actual problem comes from the type resolution at the end of the binding generation process:

[Debug] 	      Resolving target types...
[Debug] 	      Resolving target for type: nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5)
[Debug] 	      Couldn't resolve type target, using target name instead: nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5)
[Debug] 	      Creating class nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5)

So this is now given us anything meaningfull (but that's expected), and then we still write a class binding for that type:

[Debug] 	      Writing class binding for nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5)

That part we should really avoid somehow: if we have some kind of “anonymous type” then we should not try to write a class binding for it at all.

⇒ So now I have updated my “final type resolution” process to catch those kind of invalid anonymous names and avoid creating default class binding for them:

if tgtType then
      local tname = type:getName()
      logDEBUG("Resolving target for type: ", tname)
      count = count + 1
      local cl = luna:getClass(tgtType)
      if not cl then
        tgtName = tgtType:getFullName()
        logDEBUG("Couldn't resolve type target, using target name instead: ", tgtName)
        

        -- Once we have a type name then we split it in parts:
        local parts = utils.splitEntityName(tgtName)

        local ns = luna:getRootNamespace()
        local nparts = #parts
        for i=1,nparts-1 do
          if ns:hasClass(parts[i]) then
            ns = ns:getOrCreateClass(parts[i])
          else
            ns = ns:getOrCreateNamespace(parts[i])
          end
        end

        -- Then we create our target class:
        local clname = parts[nparts]

        -- @luna3: Check if that class name only contains valid characters, otherwise we should ignore it:
        if self:isValidTypeName(clname) then
          cl = ns:getOrCreateClass(clname)
        else
          logDEBUG("Ignoring unresolvable/invalid type with resulting name: ", clname)
        end
      end
      
      type:setTarget(cl)

    end

With the helper function:

function Class:isValidTypeName(tname)
  -- Chekc if the provided typename is valid:
  if tname:find("anonymous enum at") then
    logDEBUG("Detected anonymous enum type name in ", tname)
    return false
  elseif tname:find("anonymous union at") then
    logDEBUG("Detected anonymous union type name in ", tname)
    return false
  end

  return true
end

And finally those invalid binding names are gone! Ouuff! :

[Debug] 	      Resolving target types...
[Debug] 	      Resolving target for type: nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5)
[Debug] 	      Couldn't resolve type target, using target name instead: nvt::StructB::(anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5)
[Debug] 	      Detected anonymous union type name in (anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5)
[Debug] 	      Ignoring unresolvable/invalid type with resulting name: (anonymous union at D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:217:5)
[Debug] 	      Done resolving 1 target types.
Note that here we only really handled “avoiding” writing garbage when unions are found, but in the end, we still have no support at all to generate usable bindings for union variables. This is some I will have to deal with eventually.

And now of course you guess it, since the generated binding files were looking OK I moved to the compilation stage of the test1 module. And I got errors… 😭 The first one being:

[ 90%] Building CXX object sources/lua_bindings/test1/src/CMakeFiles/luaTest1.dir/luna/bind_nvt_MyTemplateTest_1.cpp.obj
bind_nvt_MyStringTest.cpp
[ 90%] Building CXX object sources/lua_bindings/test1/src/CMakeFiles/luaTest1.dir/luna/bind_nvt_MyVectorNameTest.cpp.obj
bind_nvt_MyTemplateTest_1.cpp
D:\Projects\NervSeed\sources\lua_bindings\test1\src\luna\bind_nvt_MyTemplateTest_1.cpp(49): error C2065: '_bind_MyTemplateTest' : identificateur non déclaré
bind_nvt_MyVectorNameTest.cpp
jom: D:\Projects\NervSeed\build\msvc64\sources\lua_bindings\test1\src\CMakeFiles\luaTest1.dir\build.make [sources\lua_bindings\test1\src\CMakeFiles\luaTest1.dir\luna\bind_nvt_My
TemplateTest_1.cpp.obj] Error 2

Having a look at that binding file we have indeed:

// Overall bind for MyTemplateTest<1>
static nvt::MyTemplateTest<1>* _bind_MyTemplateTest_1(lua_State* L) {
	if(_check_MyTemplateTest_1_sig1(L)) return _bind_MyTemplateTest_1_sig1(L);

	logERROR("Current lua stack: "<<luna_dumpStack(L));
	luaL_error(L, "Binding error for function MyTemplateTest<1>, cannot match any of the 1 signature(s):\n  sig1: void ()");
	return nullptr;
}

nvt::MyTemplateTest<1>* LunaTraits< nvt::MyTemplateTest<1> >::construct(lua_State* L) {
	return _bind_MyTemplateTest<1>(L);
}

⇒ So it's cool: this one sounds simple to fix: we are already generating a construction function name correctly (on line 2 above), but then when we call it on line 11 it seems I'm not sanitizing the function name correctly. Let's fix that: 🆗!

Next compilation errors on the list are:

bind_nvt_TplBool.cpp
D:\Projects\NervSeed\sources\lua_bindings\test1\src\luna\bind_nvt_TplBool.cpp(23): error C2039: 'TplBool' : n'est pas membre de 'nvt'
D:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test5.h(9): note: voir la déclaration de 'nvt'
D:\Projects\NervSeed\sources\lua_bindings\test1\src\luna\bind_nvt_TplBool.cpp(23): error C2143: erreur de syntaxe : absence de ';' avant '*'
D:\Projects\NervSeed\sources\lua_bindings\test1\src\luna\bind_nvt_TplBool.cpp(23): error C4430: spécificateur de type manquant - int est pris en compte par défaut. Remarque : C+
+ ne prend pas en charge int par défaut
D:\Projects\NervSeed\sources\lua_bindings\test1\src\luna\bind_nvt_TplBool.cpp(23): error C2923: 'nv::LunaTraits' : 'nv::TplBool' n'est pas un argument de type modèle valide pour
 le paramètre 'T'
D:\Projects\NervSeed\sources\lua_bindings\test1\src\luna\bind_nvt_TplBool.cpp(23): note: voir la déclaration de 'nv::TplBool'
D:\Projects\NervSeed\sources\lua_bindings\test1\src\luna\bind_nvt_TplBool.cpp(23): error C2955: 'nv::LunaTraits' : l'utilisation de classe modèle requiert une liste d'arguments
modèle
D:\Projects\NervSeed\sources\nvCore\include\lua/luna.h(157): note: voir la déclaration de 'nv::LunaTraits'
D:\Projects\NervSeed\sources\lua_bindings\test1\src\luna\bind_nvt_TplBool.cpp(23): error C2761: 'T *nv::LunaTraits<T>::construct(lua_State *)' : la redéclaration du membre n'est
 pas autorisée

Now, looking at the binding file, the content looks OK. Ahh! but now I get it: the TplBool (and TplInt) types are defined in bind_test2.h:

typedef MyTemplateClass<int> TplInt;
typedef MyTemplateClass<bool> TplBool;

But the “bind_test2.h” was not included anywhere in the actuall build process since I was only using this global header:

#ifndef BIND_COMMON_
#define BIND_COMMON_

#if defined(_MSC_VER) || defined(__CYGWIN__) || defined(__MINGW32__) || defined( __BCPLUSPLUS__)  || defined( __MWERKS__)
#  if defined( BIND_LIBRARY_STATIC )
#    define BIND_EXPORT
#  elif defined( BIND_LIBRARY )
#    define BIND_EXPORT   __declspec(dllexport)
#  else
#    define BIND_EXPORT   __declspec(dllimport)
#  endif
#else
#  define BIND_EXPORT
#endif

//#pragma warning( disable: 4251 )

#include <core_lua.h>
#include <lua/luna.h>

#include <bind_context.h>

#include <bind_test1.h>
#include <bind_test3.h>
#include <bind_test4.h>
#include <bind_test5.h>

#endif

So just adding “#include <bind_test2.h>” there, and then we were good 🤟!

So, after the compilation stage, more errors at link stage! mouaahh ahhh ahh… This is insane… That's what I have now:

[100%] Linking CXX shared library luaTest1.dll
LINK: command "D:\Softs\VisualStudio2017CE\VC\Tools\MSVC\14.16.27023\bin\Hostx64\x64\link.exe /nologo @CMakeFiles\luaTest1.dir\objects1.rsp /out:luaTest1.dll /implib:luaTest1.li
b /pdb:D:\Projects\NervSeed\build\msvc64\sources\lua_bindings\test1\src\luaTest1.pdb /dll /version:0.0 /DEBUG /OPT:REF,NOICF /MAP /MAPINFO:EXPORTS -LIBPATH:D:\Projects\NervSeed\
scripts\..\deps\msvc64\boost_1_68_0\lib -LIBPATH:D:\Projects\NervSeed\scripts\..\deps\msvc64\LuaJIT-2.0.5\lib ..\..\..\nvCore\src\nvCore.lib ..\..\..\..\..\..\scripts\..\deps\ms
vc64\LuaJIT-2.0.5\lib\lua51.lib opengl32.lib winmm.lib Imm32.lib version.lib ..\..\..\..\..\..\scripts\..\deps\msvc64\LuaJIT-2.0.5\lib\lua51.lib ..\..\..\soil2\soil2.lib kernel3
2.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:luaTest1.dll.manifest" failed (exit code 11
69) with the following output:
bind_std_Simple_types_double.cpp.obj : error LNK2005: "public: static struct std::_Simple_types<double> * __cdecl nv::LunaTraits<struct std::_Simple_types<double> >::construct(s
truct lua_State *)" (?construct@?$LunaTraits@U?$_Simple_types@N@std@@@nv@@SAPEAU?$_Simple_types@N@std@@PEAUlua_State@@@Z) déjà défini(e) dans bind_class_1.cpp.obj
bind_std_Simple_types_double.cpp.obj : error LNK2005: "public: static void __cdecl nv::LunaTraits<struct std::_Simple_types<double> >::destruct(struct std::_Simple_types<double>
 *)" (?destruct@?$LunaTraits@U?$_Simple_types@N@std@@@nv@@SAXPEAU?$_Simple_types@N@std@@@Z) déjà défini(e) dans bind_class_1.cpp.obj
bind_std_allocator_double.cpp.obj : error LNK2005: "public: static class std::allocator<double> * __cdecl nv::LunaTraits<class std::allocator<double> >::construct(struct lua_Sta
te *)" (?construct@?$LunaTraits@V?$allocator@N@std@@@nv@@SAPEAV?$allocator@N@std@@PEAUlua_State@@@Z) déjà défini(e) dans bind_std_Rebind_alloc_t_std_allocator_double_double.cpp.
obj
bind_std_allocator_double.cpp.obj : error LNK2005: "public: static void __cdecl nv::LunaTraits<class std::allocator<double> >::destruct(class std::allocator<double> *)" (?destru
ct@?$LunaTraits@V?$allocator@N@std@@@nv@@SAXPEAV?$allocator@N@std@@@Z) déjà défini(e) dans bind_std_Rebind_alloc_t_std_allocator_double_double.cpp.obj
luaTest1.dll : fatal error LNK1169: un ou plusieurs symboles définis à différentes reprises ont été rencontrés
jom: D:\Projects\NervSeed\build\msvc64\sources\lua_bindings\test1\src\CMakeFiles\luaTest1.dir\build.make [sources\lua_bindings\test1\src\luaTest1.dll] Error 2
jom: D:\Projects\NervSeed\build\msvc64\CMakeFiles\Makefile2 [sources\lua_bindings\test1\src\CMakeFiles\luaTest1.dir\all] Error 1
jom: D:\Projects\NervSeed\build\msvc64\Makefile [all] Error 1

And well… hmmm… this is pretty crazy stuff at this level now: in my “bind_class_1.cpp” file I'm actually generating bindings for “std::conditional_t<1,std::_Simple_types<double>,std::_Vec_iter_types<double,size_t,int64_t,double*,const double*,double&,const double&>>”

But in fact, this type ends up being the same thing as just “std::_Simple_types<double>” 🥴 Oh my…

It would be very nice if we could figure this out automatically with clang, but I'm not sure this is something that can be done easily (or at all in fact ?), and I really don't need to push it that far for my initial needs here, so I'm jsut giving it a quick try here and otherwise, will keep that for later.

Hmmm… interesting… here is what I have in my bind1.log file:

[Debug] 	      Creating class std::conditional_t<1,std::_Simple_types<double>,std::_Vec_iter_types<double,size_t,int64_t,double*,const double*,double&,const double&>>
[Debug] 	      Creating concrete class std::conditional_t<1,std::_Simple_types<double>,std::_Vec_iter_types<double,size_t,int64_t,double*,const double*,double&,const double&>> for initial type name: std::_Simple_types<double>
[Debug] 	      Adding name std::conditional_t<1,std::_Simple_types<double>,std::_Vec_iter_types<double,size_t,int64_t,double*,const double*,double&,const double&>> to type {
  "std::_Simple_types<double>"
}
[Debug] 	      Type already registered for 'std::_Simple_types<double>'

⇒ So it seems I can actually already figure out that those are the same type somehow [and that's absolutely amazing if you ask me🤩!] So from there, I should really be able to avoid this error then ?!

Okay, so I think I see it now: it all comes down to those lines:

[Debug] 	      Creating concrete class std::_Simple_types<double> for initial type name: std::_Simple_types<double>
[Debug] 	      Creating concrete class std::conditional_t<1,std::_Simple_types<double>,std::_Vec_iter_types<double,size_t,int64_t,double*,const double*,double&,const double&>> for initial type name: std::_Simple_types<double>

So this means I'm creating 2 classes for a single type here: instead of always creating “concrete classes” in that case I should rather check if the type already has a target class assigned to it first. OK! Now I get those debug outputs:

[Warning] 	No class template registered for 'conditional_t<1,std::_Simple_types<double>,std::_Vec_iter_types<double,size_t,int64_t,double*,const double*,double&,const double&>>' creating default place holder class in namespace 'std'
[Debug] 	      Detected a type already registered for type name: std::_Simple_types<double> adding alternative name: std::conditional_t<1,std::_Simple_types<double>,std::_Vec_iter_types<double,size_t,int64_t,double*,const double*,double&,const double&>>
[Debug] 	      Adding name std::conditional_t<1,std::_Simple_types<double>,std::_Vec_iter_types<double,size_t,int64_t,double*,const double*,double&,const double&>> to type {
  "std::_Simple_types<double>"
}
[Debug] 	      Type already registered for 'std::_Simple_types<double>'

And I have no more “bind_class_1.cpp” file at all ;-) Great!

It turns out that the “bind_common.h” file where I placed my “bind_test2.h” header is in fact autogenerated in the process! so my entry disappeared again, because the system considered there was no class defined in that file that was binded, to it was not necessary to include it… God damn it! Give me a break… 😫

Continuing on this path I had to also provide the binding generator with the location of the MSVC include folder to support really finding all the required include files when we automatically build concrete classes below:

-- We may proceed to create the class:
        cl = tplNS:getOrCreateClass(clname)
        CHECK(cl~=nil, "Invalid class generated from template.")
        
        logDEBUG("Creating concrete class ", fname, " for initial type name: ", tname)
        
        -- @luna3: We should also set the header file for the class here using the cursor from the cxtype:
        cl:setHeaderFile(self:getHeaderFile(cxtype:getTypeDeclaration()))

        -- We assign this class as target:
        t:setTarget(cl)

But then, this means that clang could find even more stuff to play with, and so I get errors on the type “WChar” which I didn't consider at all so far… So, I just added an entry to support that type:

elseif tkind == tk.WChar then
    logDEBUG("Found wchar type: ", tname)
    local t2 = typeManager:getType("wchar_t")
    if t2 then
      logDEBUG("Using already registered type for 'wchat_t'")
      t = t2
      t:addName(tname)
    end

[Oh man… I have a feeling this will mean even more trouble… :-S ]

Same thing then required for the NullPtr type:

elseif tkind == tk.NullPtr then
    logDEBUG("Found nullptr type: ", tname)
    local t2 = typeManager:getType("nullptr_t")
    if t2 then
      logDEBUG("Using already registered type for 'nullptr_t'")
      t = t2
      t:addName(tname)
    end

And the “Auto” type… :-S

And then I got “unions” defined in unions… :-S, so for course, also not really handled so far:

[FATAL]         [FATAL] Unexpect cursor type in union: UnionDecl
stack traceback:
        [string "setup.lua"]:215: in function 'THROW'
        [string "setup.lua"]:210: in function 'CHECK'
        [string "bind.ClangParser"]:200: in function <[string "bind.ClangParser"]:198>
        [C]: in function 'visitChildren'
        [string "bind.ClangParser"]:198: in function 'parseUnion'
        [string "bind.ClangParser"]:1307: in function <[string "bind.ClangParser"]:1265>
        [C]: in function 'visitChildren'

And then I got even more crashes along that path.. So not definitely don't want to go there for now!

⇒ Let's just add our missing header file manually lol. Or at least, ignore those include paths from clang side of things. ⇒ So I updated my LunaManager class to avoid parsing files further if they are part of the “ignored include paths” while still generating the header file correctly (hopefully):

function Class:isLocationParseable(file)
  if self:isIgnoredHeaderFile(file) then
    return false
  end

  if self:isHeaderFile(file) or self:isInputFile(file) then
    return true
  end

  return false
end

function Class:isIgnoredHeaderFile(file)
  file = path.normpath(file)
  for _,dir in ipairs(self._config.ignoredIncludePaths or {}) do 
    if file:startsWith(path.normpath(dir)) then
      return true
    end
  end

  return false
end

And I specified the MSVC include folder as ignored include folder in the binding config file:

local msvcIncludeDir = "D:/Softs/VisualStudio2019CE/VC/Tools/MSVC/14.27.29110/include"
cfg.ignoredIncludePaths = {msvcIncludeDir}

With those changes the autogenerated bind_common.h file looks like this:

#ifndef BIND_COMMON_
#define BIND_COMMON_

#if defined(_MSC_VER) || defined(__CYGWIN__) || defined(__MINGW32__) || defined( __BCPLUSPLUS__)  || defined( __MWERKS__)
#  if defined( BIND_LIBRARY_STATIC )
#    define BIND_EXPORT
#  elif defined( BIND_LIBRARY )
#    define BIND_EXPORT   __declspec(dllexport)
#  else
#    define BIND_EXPORT   __declspec(dllimport)
#  endif
#else
#  define BIND_EXPORT
#endif

//#pragma warning( disable: 4251 )

#include <core_lua.h>
#include <lua/luna.h>

#include <bind_context.h>

#include <bind_test1.h>
#include <bind_test2.h>
#include <bind_test3.h>
#include <bind_test4.h>
#include <bind_test5.h>
#include <xstring>
#include <xmemory>
#include <iosfwd>
#include <vector>
#include <initializer_list>
#include <map>


#endif

Let's try to build once more with those changes… And bingo! Finally, this is compiling fine! Feeeeeewwww!

So with this refreshing stage completed, now I can finally think about trying to build the bindings I'm after for SimCore lol. But given how long this article already is I think I should just keep that for a part 2 blog post, and we should call it a day here. See you next time ✌!

  • blog/2021/0406_nervluna_practical_usage_case_part_1.txt
  • Last modified: 2021/06/07 09:39
  • (external edit)