Generating bindings for the nvCore module

The nvCore module is the base module I built to encapsulate all the features that I consider as “base blocks” for any other sub-project. As a result, this is the first useful module I should try the NervLuna binding generation on. And this is was we will try to achieve in this article. Let's get started!

Currently I'm using a manually generated binding layer (using the sol3 library) for the nvCore. Clearly, we could consider we are good to go when the auto-generated bindings can provide all the features currently binded (and more).

So I should start with a simple brute force approach first, and then we should provide iterative small corrections until we reach the desired target.

⇒ Let's create the binding config file then:

-- Configuration file for the nvCore bindings generation.

local path = import "pl.path"
local utils = import "base.utils"
local Set = import "base.Set"

local coreIncludeDir = path.abspath(root_path.."../sources/nvCore/include")

local inputDir = path.abspath(root_path.."../sources/nvCore/include")
local outputDir = path.abspath(root_path.."../sources/lua_bindings/wip_core")

local cfg = {}

cfg.moduleName = "Core"
cfg.includePaths = {inputDir}
cfg.clangArgs = {
    "-Wno-pragma-once-outside-header",
    "-I"..coreIncludeDir
}

cfg.outputDir = outputDir

return cfg</sxhjs>

<note>For the moment I'm generating the bindings in a "wip_" (ie. "Work in progress") folder to avoid overriding the existing "core" bindings too soon.</note>

So now let's see what kind of result we get with that...

LOL, and of course, I get a lot of errors with this "brute force" approach. So now it's time to analyze what's going wrong and start fixing.

===== Ignoring some input files =====

First thing I noticed is that the binding generator will find all the files in the include folders I provided [that's expected of course], but in fact I really want to ignore some of those files: for instance all the **sol3** header files should not be used at all to generate my lua bindings:

<code>
[Debug] 	      Found input file: W:\Projects\NervSeed\sources\nvCore\include\sol\usertype_proxy.hpp
[Debug] 	      Found input file: W:\Projects\NervSeed\sources\nvCore\include\sol\usertype_storage.hpp
[Debug] 	      Found input file: W:\Projects\NervSeed\sources\nvCore\include\sol\usertype_traits.hpp
[Debug] 	      Found input file: W:\Projects\NervSeed\sources\nvCore\include\sol\variadic_args.hpp
[Debug] 	      Found input file: W:\Projects\NervSeed\sources\nvCore\include\sol\variadic_results.hpp
[Debug] 	      Found input file: W:\Projects\NervSeed\sources\nvCore\include\sol\wrapper.hpp
[Debug] 	      Found input file: W:\Projects\NervSeed\sources\nvCore\include\task\MultiThreadTaskScheduler.h
[Debug] 	      Found input file: W:\Projects\NervSeed\sources\nvCore\include\task\Task.h
[Debug] 	      Found input file: W:\Projects\NervSeed\sources\nvCore\include\task\TaskGraph.h
</code>

=> I introduced the following config function to handle this:

<sxhjs lua>
local ignoredInputs = {
    "/sol/",
    "/rapidjson/",
    "/debug/",
    "/glm/",
    "/lua/luna",
}
cfg.ignoreInputFile = function(filename)
    -- We should ignore that file if it contains one of the ignore patterns:
    local fname = filename:gsub("\\","/")
    for _,p in ipairs(ignoredInputs) do 
        if fname:find(p) then
            return true
        end
    end

    -- File is not ignored.
    return false
end</sxhjs>

Then I call this function (if any) for each input file that is found ignoring or keeping the file accordingly. And this seems to work as expected.

===== "Invalid enumeration name" issue =====

With the undesired input files excluded with the previous fix, the first really error I get is an "Invalid enumeration name" error.

As far as I understand the error I face comes from this kind of code:

<sxh cpp; highlight: [9]>
class NVCORE_EXPORT Vec2f
{
    public:

        /** Data type of vector components.*/
        typedef float value_type;

        /** Number of vector components. */
        enum { num_components = 2 };
        
        /** Vec member variable. */
        value_type _v[2];
        
}

Simply because I was always expecting enums to have a valid name, but that's silly of course, and so in this case, I should accept an annonymous enum, and simply not try to use it's name when binding it inside the class (but I think that part is already OK)

Let's test this in our simple test module, using:

class MyClass
{
public:
    enum SubType {
        SUBTYPE_ONE,
        SUBTYPE_TWO = 4,
        SUBTYPE_THREE,
    };

    typedef float value_type;

    enum { num_components = 2 };

    // More stuff here 
};

⇒ OK, after updating the generation code, I'm now producing the expected enum array content for that class (and compilation is going just fine):

luna_RegEnumType LunaTraits< nvt::MyClass >::enumValues[] = {
	{"SUBTYPE_ONE", nvt::MyClass::SUBTYPE_ONE},
	{"SUBTYPE_TWO", nvt::MyClass::SUBTYPE_TWO},
	{"SUBTYPE_THREE", nvt::MyClass::SUBTYPE_THREE},
	{"num_components", nvt::MyClass::num_components},
	{0,0}
};

The next error I receive then is:

[Debug] 	      Adding signature const char *() const to function getClassName
[FATAL] 	[FATAL] signature const char *() const already registered for function nv::RefObject::getClassName

In this module code, I add this function to some classes using the following macro:

#define NV_DECLARE_CLASS(cname) \
public:                         \
    virtual const char *getClassName() const { return #cname; }

So clearly, the fact that the function is virtual should be taken into account somehow, but still I don't understand how this could be a problem here.

⇒ OK, now I finally get it: this error happens because when we have multiple input files, each file is parsed as a separated translation unit. And as a result, we may find the definition of a given class or function (or enum, etc) multiple times. Ideally, we should concatenate all the input files into a single translation unit [⇒ more on this below]. But I think I should also be able to deal with the situation properly in case an element is found multiple times. So let's fix that here.

Solution: I just added support to mark a given “scope” (ie. applicable to classes and enums only) as already parsed. So I can avoid parsing them twice from 2 different translation units:

function Class:__init(name, parent)
  Class.super.__init(self, name, parent)
  self._children = {}
  self._isParsed = false
end

function Class:isParsed()
  return self._parsed
end

function Class:setParsed(val)
  self._parsed = val
end
</sxhjs>

<note>Another thing i just noticed is that parsing is going to be somewhat slow if we parse all the input files one by one => so I should really add support to "pack" those files into a single translation unit.</note>

<note>Another important consideration is that we get diagnostics separately for each translation units anyway: so it would definitely help to produce less of those TUs.</note>

===== Unsupported template type issue =====

Next error on my path is: <code>[Debug] 	      Creating field nv::LinearAllocator::_pools
[Debug] 	      Type kind: Unexposed
[FATAL] 	[FATAL] Unsupported type: Vector<nv::LinearAllocator::Pool>, kind: Unexposed</code>

So we see the type as "Unexposed" here, this seems strange to me, so let's add a test in our test module to check how this is parsed exactly:

<sxh cpp>
template <typename T>
class Vector
{
public:
    T* ptr;
    unsigned int size;

    Vector(T* input, unsigned int count) : ptr(input), size(count) {};
};

struct Point
{
    float x;
    float y;
};

struct PointList
{
public:
    Vector<Point> _ctrls;
    
protected:
    Vector<Point> _points;
};

Using that test code above I get exactly the same error message:

[Debug] 	      Processing: 'PointList' kind: 'StructDecl
[Debug] 	      Creating class nvt::PointList
[Debug] 	      Creating field nvt::PointList::_ctrls
[Debug] 	      Type kind: Unexposed
[FATAL] 	[FATAL] Unsupported type: Vector<nvt::Point>, kind: Unexposed

And well yes, I was right in my last note above I'm afraid I have to start dealing with class template… [Disclaimer: this is gonna be tricky lol]

So I'm starting with adding support to parse a new kind of element: a ClassTemplate that will actually inherit my class representing a Class: this is not absolutely correct because a ClassTemplate is not a kind of Class, but it's behavior is so similar that this seems to be a reasonnable choice [a better choice would be to setup a common parent with all the common features… but I'll get to that later.]

Next I have to retrieve the list of template parameters, so I'm extending the “parseClass” method with the following section:

    elseif ckind == clang.CursorKind.TemplateTypeParameter then
      -- We should add this as a template parameter:
      CHECK(scope:isClassTemplate(), "Cannot add template parameter to non class template element.")
      logDEBUG("Adding template parameter ", cname, " to class template ", scope:getFullName())
      scope:addTemplateParameter(cname)
</sxhjs>

So far, template parameters are only "simple names" like "T", and the idea is that a class template with such parameters could then be used to generate a real class, by replacing all the template parameters with actual types [this sounds complex to achieve, but it should be possible.]

Next problem I have to deal with after that is when I'm trying to parse the content of a template class. For instance with the test code mentioned above, I eventually enter an infinite type resolution loop with this:

<code>
[Debug] 	      Creating field nvt::Vector::ptr
[Debug] 	      Type kind: Pointer
[Debug] 	      Found pointer type: T *
[Debug] 	      Type kind: Unexposed
[Debug] 	      base type: 'type-parameter-0-0' for unexposed: T
[Debug] 	      Type kind: Unexposed
[Debug] 	      base type: 'type-parameter-0-0' for unexposed: type-parameter-0-0
[Debug] 	      Type kind: Unexposed
[Debug] 	      base type: 'type-parameter-0-0' for unexposed: type-parameter-0-0
[Debug] 	      Type kind: Unexposed
[Debug] 	      base type: 'type-parameter-0-0' for unexposed: type-parameter-0-0
[Debug] 	      Type kind: Unexposed
[Debug] 	      base type: 'type-parameter-0-0' for unexposed: type-parameter-0-0
[Debug] 	      Type kind: Unexposed
[Debug] 	      base type: 'type-parameter-0-0' for unexposed: type-parameter-0-0
[Debug] 	      Type kind: Unexposed
[Debug] 	      base type: 'type-parameter-0-0' for unexposed: type-parameter-0-0</code>

Let's see what we can do about it...

=> Okay, so we really have to somehow "catch" the "TypeRefs/TemplateRefs/NamespaceRefs" that might be associated with the current "Cursor" when building new Types: with these we can figure out that a type is based on template parameters and thus, is only valid in a given "templating context" (ie. basically a stack of Class templates) [Again, more on this below]

===== Default constructor deleted issue =====

I made some interesting progress on the class templating system, but now one additional thing that is not working as expected is the classes "default constructor" that I'm using for the classes where I don't have any constructor explicitly default. So for instance with the code:

<sxh cpp>
struct PointList
{
public:
    Vector<Point> _ctrls;
    
protected:
    Vector<Point> _points;
};

I'm making use of a default generated constructor (which I assumed would be available), so I produce the code:

static nvt::PointList* _bind_PointList_sig1(lua_State* L) {
	// When reaching this call, we assume that the type checking is already done.

	return new nvt::PointList();
}

Unfortunately, this will not compile and produce the error:

W:\Projects\NervSeed\sources\lua_bindings\test1\src\luna\bind_nvt_PointList.cpp(22): error C2280: 'nvt::PointList::PointList(void)': attempting to reference a deleted function
W:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h(259): note: compiler has generated 'nvt::PointList::PointList' here
W:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h(259): note: 'nvt::PointList::PointList(void)': function was implicitly deleted because a data member 'nvt::PointList::_points' has either no appropriate default constructor or overload resolution was ambiguous

… So, I seems I cannot just blindly assume that a default constructor will be available.

Yet the nice thing here with C++11 is that we have the helper std::is_default_constructible that we can use to know for sure if the default constructor is available or not. So we should use this when dealing with a “presumably” default generated constructor.

Here is the little additional “C++ gymnastic” I added to handle this consideration:

template <typename T, bool val>
struct Constructor
{
    static T* construct(lua_State* L) {
        // luaL_error(L, "Class is not default constructible.");
        return nullptr;
    }
};

template <typename T>
struct Constructor<T, true>
{
    static T* construct(lua_State* L) {
        // Use the default constructor:
        return new T();
    }
};

template <typename T>
T* luna_callDefaultConstructor(lua_State* L) {
    return Constructor<T, std::is_default_constructible<T>::value>::construct(L);
}

And now I simply generate the default constructors of each class as follow to let the compiler decide if we should be able to create an instance this way or not:

static nvt::PointList* _bind_PointList_sig1(lua_State* L) {
	// When reaching this call, we assume that the type checking is already done.

	auto res = luna_callDefaultConstructor<nvt::PointList>(L);
	if(res == nullptr) luaL_error(L, "Cannot default construct instance of class nvt::PointList.");
	return res;
}

And this is now compiling just fine cool.

For the moment I'm not completely handling the class templates in C++: I'm just providing some minimal support as follow:

• When we are parsing a translation unit an find the definition of a ClassTemplate then we create the corresponding class template object in lua, and we parse the “class” as usual (except that we might also have references on the template parameters in this case). Then we continue with the parsing and if a find a “type” corresponding to a class template instanciation then we request the class template to create an actual class representation in lua [For instance, a template class “Vector<T>” could be used to create the real type “Vector<float>”]

• But, when creating the real class, we should use the class template to populate the class with concrete methods, fields, enums etc. ⇒ yet, I'm not handling that part for the moment, the the reflection system only knows that there is a class “Vector<float>” that we should bind, but that class is just empty

• Still, this should be enough to fix the issue that was stopping us in the first binding generation attempt for nvCore. So it's time to give this another try now [crossing fingers…]

Okay, so, of course, the second attempt failed lol. The error message I receive now is:

[Debug] 	      Creating field nv::RefObject::_refCount
[Debug] 	      Type kind: Elaborated
[Debug] 	      Detected template record type: std::atomic<U64> with 1 template arguments
[FATAL] 	[FATAL] Non-templateRef cursor detected: NamespaceRef (at W:\Projects\NervSeed\sources\lua_bindings\Core\core_bindings.cpp:152)

Basically, what I think is happening here is:

• I'm parsing the field “nv::RefObject::_refCount” of type “std::atomic<U64>”, so I was expecting the first child cursor of the field cursor to be a TemplateRef cursor: This was working fine in my intial test above because the type I was using for my fields was “Vector<nvt::Point>” (ie. notice that I'm not specifying any namespace before Vector in that case). But here we have the “std” namespace specified as part of the type name… so I think that's why we have a “NamespaceRef” first. And I would expect the next child cursor to be the “TemplateRef” I'm expecting to receive in that case. Let's update our test code to double check this little theory:

struct PointList
{
public:
    nvt::Vector<Point> _ctrls;
    
protected:
    Vector<Point> _points;
};

And indeed, the complete AST I get with this small change contains:

[Debug] 	      level 3: '_ctrls' of kind 'FieldDecl', type: 'nvt::Vector<Point>'
[Debug] 	      level 4: 'nvt' of kind 'NamespaceRef', type: ''
[Debug] 	      level 4: 'Vector' of kind 'TemplateRef', type: ''
[Debug] 	      level 4: 'struct nvt::Point' of kind 'TypeRef', type: 'nvt::Point'

So I should really be more flexible, and parse all the child cursor until I find the “TemplateRef” I need . Let's do that.

Unfortunately, this is still not good enough because with the previous issue fixed, I immediately get the following next error:

[Debug] 	      Creating field nv::RefObject::_refCount
[Debug] 	      Type kind: Elaborated
[Debug] 	      Detected template record type: std::atomic<U64> with 1 template arguments
[Debug] 	      TemplateRef definition: atomic, kind: ClassTemplate
[Error] 	Cannot retrieve class template from cursor location: D:\Apps\VisualStudio2017_CE\VC\Tools\MSVC\14.16.27023\include\atomic:634:9:22078
[FATAL] 	[FATAL] Cannot retrieve class template for type std::atomic<U64>

And this, again, makes perfect sense, because, so far I was expecting that the class template would be registered (and parsed) before we ever try to build a type with it. Expect that, in practice, I'm only parsing the content I'm interested in, not all the classes, functions, templates etc that are included in the translation unit. So here, I did not register the “std::atomic” template class before trying to use it.

So how should I proceed here ? I see basically 2 paths I could take:

• Option A: If I reach a situation where I'm trying to use a class template that is not registered yet, I simply register a class of the appropriate templated type, with no function or field or anything in it. This could then just be used as a simple placeholder.

• Option B: I use the cursor corresponding to the ClassTemplate of interest to immediately parse the class template.

⇒ Option B is clearly more complex to achieve [in fact, I'm not even sure this will really work somehow], but that would make the bindings generation more complete too. So eventually I think it will be worth it to investigate that path further. But for now, let's rather keep things simple and try Option A instead, with a simple place holder class.

OK, this seems to work as expected now:

[Debug] 	      Creating field nv::RefObject::_refCount
[Debug] 	      Type kind: Elaborated
[Debug] 	      Detected template record type: std::atomic<U64> with 1 template arguments
[Debug] 	      Entering template namespace: std
[Debug] 	      Creating namespace ::std
[Debug] 	      TemplateRef definition: atomic, kind: ClassTemplate
[Error] 	Cannot retrieve class template from cursor location: D:\Apps\VisualStudio2017_CE\VC\Tools\MSVC\14.16.27023\include\atomic:634:9:22078
[Debug] 	      Resolved template argument 0: nv::U64
[Warning] 	No class template registered for 'atomic' creating default place holder class: 'atomic<nv::U64>' in namespace 'std'
[Debug] 	      Creating class std::atomic<nv::U64>

Continuing with our class template related issues, the next problem to deal with occurs when trying to parse the “RefPtr” template class which has a template parameter usage in some of its constructor:

template <typename T>
class RefPtr {
public:
	typedef T element_type;

	RefPtr() : _ptr(0) {}
	RefPtr(T* ptr) : _ptr(ptr) { if (_ptr) _ptr->ref(); }
	RefPtr(const RefPtr& rp) : _ptr(rp._ptr) { if (_ptr) _ptr->ref(); }

    // (more stuff here)
private:
	T* _ptr;
};

This will produce the following infinite loop + overflow:

[Debug] 	      Processing: 'RefPtr' kind: 'ClassTemplate
[Debug] 	      Creating class template nv::RefPtr
[Debug] 	      Parsing class template: RefPtr
[Debug] 	      Registering class template: nv::RefPtr
[Debug] 	      Class template cursor location: W:\Projects\NervSeed\sources\nvCore\include\base/RefPtr.h:14:7:176
[Debug] 	      Adding template parameter T to class template nv::RefPtr
[Debug] 	      parseClass: ignoring cursor 'element_type' of kind 'TypedefDecl'
[Debug] 	      Creating function nv::RefPtr::RefPtr<T>
[Debug] 	      Adding signature void () to function RefPtr<T>
[Debug] 	      Return type: void
[Debug] 	      cursor num arguments: 0
[Debug] 	      Adding signature void (T *) to function RefPtr<T>
[Debug] 	      Return type: void
[Debug] 	      cursor num arguments: 1
[Debug] 	      Arg 0 cursor: ptr, type: T *
[Debug] 	      Type kind: Pointer
[Debug] 	      Found pointer type: T *
[Debug] 	      Type kind: Unexposed
[Debug] 	      Using template contextual type name: T@nv::RefPtr
[Debug] 	      Adding signature void (const RefPtr<T> &) to function RefPtr<T>
[Debug] 	      Return type: void
[Debug] 	      cursor num arguments: 1
[Debug] 	      Arg 0 cursor: rp, type: const RefPtr<T> &
[Debug] 	      Type kind: LValueReference
[Debug] 	      Type kind: Unexposed
[Debug] 	      Found const qualified type: const RefPtr<T>
[Debug] 	      Found base type RefPtr<T> for const type const RefPtr<T>
[Debug] 	      Type kind: Unexposed
[Debug] 	      base type: 'RefPtr<T>' for unexposed: RefPtr<T>
[Debug] 	      Type kind: Unexposed
[Debug] 	      base type: 'RefPtr<T>' for unexposed: RefPtr<T>
[Debug] 	      Type kind: Unexposed
[Debug] 	      base type: 'RefPtr<T>' for unexposed: RefPtr<T>
[Debug] 	      Type kind: Unexposed
[Debug] 	      base type: 'RefPtr<T>' for unexposed: RefPtr<T>
[Debug] 	      Type kind: Unexposed

(And eventually)
[Error] 	Error in cursorVisitor: stack overflow

Full parsing of AST would produce in this case:

[Debug] 	      level 3: 'RefPtr<T>' of kind 'CXXConstructor', type: 'void ()'
[Debug] 	      level 3: 'RefPtr<T>' of kind 'CXXConstructor', type: 'void (T *)'
[Debug] 	      level 4: 'ptr' of kind 'ParmDecl', type: 'T *'
[Debug] 	      level 5: 'T' of kind 'TypeRef', type: 'T'
[Debug] 	      level 3: 'RefPtr<T>' of kind 'CXXConstructor', type: 'void (const RefPtr<T> &)'
[Debug] 	      level 4: 'rp' of kind 'ParmDecl', type: 'const RefPtr<T> &'
[Debug] 	      level 5: 'RefPtr<T>' of kind 'TypeRef', type: 'RefPtr<T>'
[Debug] 	      level 3: '' of kind 'CXXAccessSpecifier', type: ''
[Debug] 	      level 3: '_ptr' of kind 'FieldDecl', type: 'T *'
[Debug] 	      level 4: 'T' of kind 'TypeRef', type: 'T'

So, when parsing a signature, we could retrieve TypeRef cursor from inside the ParmDecl cursors.

⇒ I updated the Type resolution mechanism to be able to deal with this problem. And now I can get a bit further, with this kind of result:

[Debug] 	      Processing: 'RefPtr' kind: 'ClassTemplate
[Debug] 	      Creating class template nvt::RefPtr
[Debug] 	      Parsing class template: RefPtr
[Debug] 	      Registering class template: nvt::RefPtr
[Debug] 	      Class template cursor location: W:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:274:7:4956
[Debug] 	      Adding template parameter T to class template nvt::RefPtr
[Debug] 	      parseClass: ignoring cursor 'element_type' of kind 'TypedefDecl'
[Debug] 	      Creating function nvt::RefPtr::RefPtr<T>
[Debug] 	      Adding signature void () to function RefPtr<T>
[Debug] 	      Return type: void
[Debug] 	      Using already resolved type for 'void'
[Debug] 	      cursor num arguments: 0
[Debug] 	      Adding signature void (T *) to function RefPtr<T>
[Debug] 	      Return type: void
[Debug] 	      Using already resolved type for 'void'
[Debug] 	      cursor num arguments: 1
[Debug] 	      Arg 0 cursor: ptr, type: T *
[Debug] 	      Using already resolved type for 'T *'
[Debug] 	      Adding signature void (const RefPtr<T> &) to function RefPtr<T>
[Debug] 	      Return type: void
[Debug] 	      Using already resolved type for 'void'
[Debug] 	      cursor num arguments: 1
[Debug] 	      Arg 0 cursor: rp, type: const RefPtr<T> &
[Debug] 	      Type kind: LValueReference
[Debug] 	      Type kind: Unexposed
[Debug] 	      Found const qualified type: const RefPtr<T>
[Debug] 	      Found base type RefPtr<T> for const type const RefPtr<T>
[Debug] 	      Type kind: Unexposed
[Debug] 	      base type: 'RefPtr<T>' for unexposed: RefPtr<T>
[Debug] 	      Found templated type reference: RefPtr<T>
[Debug] 	      Using template contextual type name: RefPtr<${T}>@nvt::RefPtr
[Debug] 	      templated type referenced location: W:\Projects\NervSeed\sources\lua_bindings\test1\interface\bind_test1.h:274:7:4956
[Debug] 	      Base class for reference const RefPtr<T> & is: const RefPtr<T>
[Debug] 	      parseClass: ignoring cursor 'RefPtr<T>' of kind 'FunctionTemplate'
[Debug] 	      Creating function nvt::RefPtr::operator*
[Debug] 	      Adding signature T &() const to function operator*
[Debug] 	      Return type: T &
[Debug] 	      Type kind: LValueReference
[Debug] 	      Type kind: Unexposed
[Debug] 	      Using template contextual type name: ${T}@nvt::RefPtr for unexposed type T
[Debug] 	      Base class for reference T & is: T
[Debug] 	      cursor num arguments: 0
[Debug] 	      Creating function nvt::RefPtr::operator->
[Debug] 	      Adding signature T *() const to function operator->
[Debug] 	      Return type: T *
[Debug] 	      Using already resolved type for 'T *'
[Debug] 	      cursor num arguments: 0
[Debug] 	      Creating function nvt::RefPtr::get
[Debug] 	      Adding signature T *() const to function get
[Debug] 	      Return type: T *
[Debug] 	      Using already resolved type for 'T *'

This seems pretty good already, because when I detect a “TypeRef”, it might either be a “raw template parameter”, such as “T”, or some other more complex type that will be built on top of some of the template parameters available in the current templating context, such as “RefPtr<T>”, and in both cases, I can process that type name, to extract the actual template parameters that should be used when building a concrete type, and I update the “template type” name accordingly. So above we see for instance the registration of the types: “${T}@nvt::RefPtr" or "RefPtr<${T}>@nvt::RefPtr”. This means that in both cases we are detecting properly that the template parameter “T” will be used here to build the real type.

What is not so good here are the entries such as: “Using already resolved type for 'T *'”

⇒ This means that the reflection system will at some point just register a type named “T *”, and the next time we see that type name, the already registered type will be used directly. But obviously this type doesn't really mean anything outside of a templating context so it should really not be registered this way.

The problem here is that the types are built recursively: so to build the type for “T *”, we retrieve the type for “T” [that one will be registered correctly with a template contextual name], but then we simply copy the type “T” and a we add a pointer on it. Not taking into account here that the type “T *” should also be in the same templating context. Let's fix that.

⇒ OK, now it seems I'm able to handle the template type recursively as desired:

[Debug] 	      Adding signature T &() const to function operator*
[Debug] 	      Return type: T &
[Debug] 	      Type kind: LValueReference
[Debug] 	      Type kind: Unexposed
[Debug] 	      Using template contextual type name: ${T}@nvt::RefPtr for unexposed type T
[Debug] 	      Using already resolved type for '${T}@nvt::RefPtr'
[Debug] 	      Collecting template parameters for T &
[Debug] 	      Updated type name is: ${T} &
[Debug] 	      Base class for reference T & is: T, contextualTypeName: ${T}@nvt::RefPtr
[Debug] 	      cursor num arguments: 0
[Debug] 	      Creating function nvt::RefPtr::operator->
[Debug] 	      Adding signature T *() const to function operator->
[Debug] 	      Return type: T *
[Debug] 	      Type kind: Pointer
[Debug] 	      Found pointer type: T *
[Debug] 	      Type kind: Unexposed
[Debug] 	      Using template contextual type name: ${T}@nvt::RefPtr for unexposed type T
[Debug] 	      Using already resolved type for '${T}@nvt::RefPtr'
[Debug] 	      Collecting template parameters for T *
[Debug] 	      Updated type name is: ${T} *
[Debug] 	      Using already resolved type for '${T} *@nvt::RefPtr'

And still continuing on our binding generation for nvCore, it seems we are back to a missing class template registration issue with the following error message:

[Debug] 	      Adding signature std::ostream &(std::ostream &, const nv::Vec2f &) to function operator<<
[Debug] 	      Return type: std::ostream &
[Debug] 	      Type kind: LValueReference
[Debug] 	      Type kind: Elaborated
[Debug] 	      Detected template record type: std::ostream with 2 template arguments
[Debug] 	      Entering template namespace: std
[Debug] 	      Using already resolved type for 'char'
[Debug] 	      Resolved template argument 0: char
[Debug] 	      Type kind: Unexposed
[Debug] 	      base type: 'std::char_traits<char>' for unexposed: char_traits<char>
[Debug] 	      Type kind: Record
[Debug] 	      Detected template record type: std::char_traits<char> with 1 template arguments
[Debug] 	      Entering template namespace: std
[Debug] 	      Using already resolved type for 'char'
[Debug] 	      Resolved template argument 0: char
[FATAL] 	[FATAL] Cannot retrieve class template name for type std::char_traits<char>

For testing, I'm now adding the following function in our test module:

namespace std {
inline std::ostream& operator<< (std::ostream& os, const nvt::Point& pt)
{
  os << "nvt::Point(" << pt.x << ", " << pt.y << ")";
  return os;
}
}

Hmmm… But unfortunately, this doesn't help me much: I see no statement of “std::char_traits<char>” or anything alike. I suppose this is because I'm following some of the “TypeRef” cursors and thus get into different locations… So let's just manually build a class type name in this case!

Next error on the list is:

[Debug] 	      Creating function nv::STLAllocator::address
[Debug] 	      Adding signature nv::STLAllocator::pointer (nv::STLAllocator::reference) const to function address
[Debug] 	      Return type: nv::STLAllocator::pointer
[Debug] 	      Type kind: Typedef
[Debug] 	      Using base type: 'type-parameter-0-0 *' for typedef type nv::STLAllocator::pointer
[Debug] 	      Type kind: Pointer
[Debug] 	      Found pointer type: type-parameter-0-0 *
[Debug] 	      Type kind: Unexposed
[FATAL] 	[FATAL] Cannot find template parameter with name type-parameter-0-0 in current templating context

The function address() we are binding above uses the typedef pointer: so we should gracefully retrieve the template parameter here too. I thus added the following section in my TypeManage.collectTemplateParameters method:

      -- If somewhere we find the text "type-parameter-X-Y", then we consider this as a reference to our current template parameter too:
      local fullSymPat = "type%-parameter%-"..(cId-1).."%-"..(pId-1)
      local newtname = tname:gsub(fullSymPat,"%${"..tpname.."}")
      if newtname ~= tname then
        -- We used that template name so we should register it:
        refTps:insert(tp)
        tname = newtname
      end
</sxhjs>

<note>Note that I have no real idea yet what the "-0-0" suffix in the type parameter name means. So for now I'm simply assuming, this would be the template context index, followed by the template parameter index in that context. We'll see with time if this should be any different ;-)</note>

===== Incorrect parsing order for some files =====

And now finally something that is not directly related to class templates: I have a problem with the parsing of some files simply because they are not supposed to be parsed on their own. This is the case for my "base/containers.h" header file for instance. That file will include "core_common.h", and "core_common.h" will also include it (so we have a cyclic dependency). 

As a result we get the following diagnostic errors: <code>[Info] 	   Num diagnostics for file W:\Projects\NervSeed\sources\nvCore\include\base\containers.h: 4
[Info] 	    - diag 1: W:\Projects\NervSeed\sources\nvCore\include\core_common.h:70:15: error: unknown type name 'String'
[Info] 	    - diag 2: W:\Projects\NervSeed\sources\nvCore\include\core_common.h:72:55: error: unknown type name 'String'
[Info] 	    - diag 3: W:\Projects\NervSeed\sources\nvCore\include\core_utils.h:78:41: error: unknown type name 'String'
[Info] 	    - diag 4: W:\Projects\NervSeed\sources\nvCore\include\core_utils.h:178:16: warning: 'getenv' is deprecated: This function or variable may be unsafe. Consider using _dupenv_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. 
</code>

=> Basically, as already mentioned above, parsing all the include files one by one doesn't seem to be an appropriate solution: a lot of the parsing job is duplicated, and sometimes this can lead to diagnostic errors + parsing errors as just above.

Instead I think we should really setup a dedicated single header file that would be used to get all the elements we need to bind into a single translation unit, and thus reduce significantly the time needed to generate the bindings (and fix those errors at the same time). Let's go!

=> So I'm now providing a single header file that should contain all the elements I want to generate the bindings for, currently with the following content:

<sxh cpp>
#include <core_common.h>

Then I had to fix some additional issues on some function signatures being binded even if they were using “unbindable types”. And after that: Oh miracle!: the binding generation went fine! :

Of course, I haven't try the code compilation yet and this is definitely going to fail because I have some content such as:

// Bind for operator-= (1) with signature: nv::Vec2d &(const nv::Vec2d &)
static int _bind_operator-=_sig1(lua_State* L) {
	// When reaching this call, we assume that the type checking is already done.
	nv::Vec2d* self = Luna< nv::Vec2d >::get(L,1);
	ASSERT(self!=nullptr);

	nv::Vec2d* rhs = Luna< nv::Vec2d >::get(L,2,false);

	nv::Vec2d & res = self->operator-=(*rhs);
	Luna< nv::Vec2d >::push(L, &res, false);

	return 1;
}

And of course this is not valid syntax but I can now keep going and progressively fix those problems and it feels globably less complex to handle than the class templates stuff.

So, we can't really say we have a working nvCore binding module generated already. But we definitely made some significant progress in that direction with this article. Now, this is already an awfully long post so I should really stop here for this time. But, if we are lucky, we might be able to achieve our goal in the next development session on this topic ?! [hey… you have to stay positive, don't you ? ].