NervLuna: Adding more input folders for nvCore bindings
In this long and terrifying blog post, I was basically just trying to send all the input folders I have as headers in my nvCore module for binding generation with nervluna… trying to add the folders one by one and fixing the problems as their occur progressively. That was really tedious 
And desperating… And frustrating… But anyway: not much choice here: if I want to build something usable, then I should at least be able to generate the bindings for a single of my existing module, no ?! [and actually, that would only be the “beginning” lol]
In the end, I got all (well, “almost all”) headers parsed, and the binding generator was not crashing anymore. Yet, the resulting code is still not compiling because I'm facing some issues with the type names etc. But I really think I should keep that part for another article, so more on this in another episode 
So, not sure anyone would find it interesting to read the following, but at the very least, I can keep this as reference in case I need to remember why I did something somehow etc.
Missing header file during compilation
I've been trying to extend the bindings for my nvCore module, so, the first thing I did, was to add some additional files in my interface definition file:
#include <core_common.h> #include <base/SystemTime.h> #include <base/utils.h> #include <hash/Sha1.h>
But then, the code will not compile anymore, because the file “hash/Sha1.h” is not included anywhere in the bindings. Hmmm, what could I do about this ?
⇒ So I added a dedicated method to find what should be used as header file for a given cursor (either the current input file if it is a valid “include file”, and the actual location where this cursor is defined, assuming that one would be a valid include file in the worst case):
function Class:getHeaderFile(cur)
-- To retrieve an header file for a given cursor, we
-- first check if we have a valid input file:
local hfile = self:getCurrentInputFile()
if hfile then
return hfile
end
-- If we don't have a valid current header,
-- we retrieve the current location of that cursor,
-- and we check if it is a valid include file:
hfile = cursor:getLocation():getFullPath()
local idir, relfile = luna:asHeaderFile(hfile)
CHECK(idir~=nil, "File '", relfile, "' is not in the include paths ?")
return hfile
end</sxhjs>
This did the trick apparently, and the compilation went fine after this change.
===== Incorrect array type detection =====
Then I continue with the introduction of the Sha256 class in the interface: <sxh cpp; highlight: []>#include <core_common.h>
#include <base/SystemTime.h>
#include <base/utils.h>
#include <hash/Sha1.h>
#include <hash/Sha256.h>
And then of course I get a stack overflow somehow with the following outputs:
[Debug] Type kind: IncompleteArray [Debug] Found const qualified type: const nv::SHA256::uint32 [] [Debug] Found base type const nv::SHA256::uint32 [] for const type const nv::SHA256::uint32 [] [Debug] Resolving type const nv::SHA256::uint32 [] [Debug] Creating new type object for const nv::SHA256::uint32 [] [Debug] Type kind: IncompleteArray [Debug] Found const qualified type: const nv::SHA256::uint32 [] [Debug] Found base type const nv::SHA256::uint32 [] for const type const nv::SHA256::uint32 [] [Debug] Resolving type const nv::SHA256::uint32 [] [Debug] Creating new type object for const nv::SHA256::uint32 [] [Debug] Type kind: IncompleteArray
So when resolving the base type for our “const qualified” type, we fallback on the same type again: actually I think that could make sense here: it's simply that we should consider the “array” aspect before trying to resolve the type further… Let's see if we can do that…
Hhmmm, well, in fact we already have in the code:
elseif cxtype:isArray() then
-- (...stuff here...)
elseif cxtype:isConstQualifiedType() then
-- (...more stuff here...)
</sxhjs>
So, it seems our call to :isArray() is not giving us the expected result here ? How could that be ? Okay, so in our C++ definition for that function, we actually try to retrieve the size of the array: <sxh cpp; highlight: []> SOL_CUSTOM_FUNC(isArray) = [](class_t&obj) {
return clang_getArraySize(obj)!=-1;
};
But obviously, this will not really make sense for an “incomplete array” ⇒ I think I should instead try to verify if there is a valid “element type”. ⇒ OK this will fix the binding generation issue, but now I have a compilation issue:
W:\Projects\NervSeed\sources\lua_bindings\wip_core\src\luna\bind_nv_SHA256.cpp(90): error C2664: 'void nv::SHA256::update(const unsigned char *,unsigned int)': cannot convert argument 1 from 'const char *' to 'const unsigned char *'
The problematic function is:
// Bind for update (1) with signature: void (const unsigned char *, unsigned int)
static int _bind_update_sig1(lua_State* L) {
// When reaching this call, we assume that the type checking is already done.
nv::SHA256* self = Luna< nv::SHA256 >::get(L,1);
ASSERT(self!=nullptr);
const char* message = (const char*)lua_tostring(L,2);
uint32_t len = (uint32_t)lua_tointeger(L,3);
self->update(message, len);
return 0;
}
How could it be I'm not casting to const unsigned char* here ? Let's check that. OK: in my CharConverter class I was simply not taking into account that my target “const char*” type could also be unsigned . Should be easy to fix:
local atype = arg:getType()
CHECK(atype:isConst() and atype:isPointer() and atype:getRawName() == "char", "Expected const char* type instead of: ", atype:getName())
local typeName = atype:isUnsigned() and "const unsigned char*" or "const char*"
local argName = arg:getName()
local defVal = nil
if not isField then
defVal = arg:getDefault()
end</sxhjs>
**Fixed!** Compilation is OK again.
===== Binding generation for complete "base" folder =====
Then I thought I should try generating bindings for a complete folder, so I updated the config as follow: <code>cfg.moduleName = "Core"
cfg.libraryName = "luaCoreWIP"
cfg.includePaths = { inputDir, outputDir.."/include" }
cfg.inputPaths = { outputDir.."/include/core_interface.h",
inputDir.."/base" }</code>
But then we have 2 major problems with that:
* **A** the binding generation will take significantly more time, because I'm sending each file for parsing separetely, so Clang has to re-do the same preprocessing work over and over again.
* **B** More importantly, I then get an error when parsing a function: <code>[Debug] Signature is *not* defined: nv::readFile::int (const char *)
[Debug] Return type: int
[Debug] Resolving type int
[Debug] Using already resolved type for 'int', type name: int
[Debug] cursor num arguments: 1
[Debug] Detected DLL imported function signature: nv::readFile::int (const char *)
[Error] Unexpected current cursor AST:
[FATAL] [FATAL] Mismatch in num function args and num param decls: 1 != 0 (at W:\Projects\NervSeed\sources\lua_bindings\Core\core_bindings.cpp:152)</code>
But if we check the diagnostics for that file we see that there are some errors already that are related to the problem we have here: <code>[Debug] Parsing file: W:\Projects\NervSeed\sources\nvCore\include\base\containers.h
[Info] Num diagnostics for file W:\Projects\NervSeed\sources\nvCore\include\base\containers.h: 4
[Info] - diag 1: W:\Projects\NervSeed\sources\nvCore\include\core_common.h:70:15: error: unknown type name 'String'
[Info] - diag 2: W:\Projects\NervSeed\sources\nvCore\include\core_common.h:72:55: error: unknown type name 'String'
[Info] - diag 3: W:\Projects\NervSeed\sources\nvCore\include\core_utils.h:78:41: error: unknown type name 'String'
[Info] - diag 4: W:\Projects\NervSeed\sources\nvCore\include\core_utils.h:178:16: warning: 'getenv' is deprecated: This function or variable may be unsafe. Consider using _dupenv_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. [-Wdeprecated-declarations]</code>
=> So I must find a way to parse all the input files together to ensure that clang will always find all the references it needs.
**OK**: now I'm generating a dedicated header file called **luna_bindings.h** where I place all the input files that should be parsed. At the beginning of that file we also include the newly created **bind_context.h** file where we may specify per project specify header include in proper order.
Now I have a problem with the generating of bindings for some private classes apparently: <code>bind_boost_lockfree_queue_nv_FiberPool_Task_boost_lockfree_queue_nv_STLAllocator_void_nv_DefaultPoolAllocator.cpp
W:\Projects\NervSeed\sources\lua_bindings\wip_core\src\luna\bind_boost_lockfree_queue_nv_FiberPool_Task_boost_lockfree_queue_nv_STLAllocator_void_nv_DefaultPoolAllocator.cpp(23): error C2248: 'nv::FiberPool::Task': cannot access private struct declared in class 'nv::FiberPool'
W:\Projects\NervSeed\sources\nvCore\include\base/FiberPool.h(34): note: see declaration of 'nv::FiberPool::Task'
W:\Projects\NervSeed\sources\nvCore\include\base/FiberPool.h(20): note: see declaration of 'nv::FiberPool'
</code>
Analyzing the situation, indeed we have a private class "Task" in the FiberPool class, but then declaration where this problematic type comes from is **also** private: <sxh cpp; highlight: []>
boost::lockfree::queue<Task, boost::lockfree::allocator<STLAllocator<void, DefaultPoolAllocator>>> _tasks;
⇒ Couldn't we just ignore the types that are built in private sections of a class ? OK I also had to ignore the protected classes actually, but then those invalid bindings were removed as expected.
Next I had an error on the type “std::size_t” that was of kind “Elaborated”, and I had to add an additional section in that case to resolve the type correctly from it's typedef.
Now compilation is OK again, cool
But, looking more carefully at the binding generation log, it seems we still have a significant problem there with a stack overflow:
[Debug] Type kind: Unexposed [Debug] base type: 'vector<nv::PoolAllocator::Pool, STLAllocator<nv::PoolAllocator::Pool, type-parameter-0-1>>' for unexposed: std::vector<nv::PoolAllocator<BlockSize, MemAlloc>::Pool, nv::STLAllocator<nv::PoolAllocator<BlockSize, MemAlloc>::Pool, type-parameter-0-1>> [Debug] Resolving type std::vector<nv::PoolAllocator<BlockSize, MemAlloc>::Pool, nv::STLAllocator<nv::PoolAllocator<BlockSize, MemAlloc>::Pool, type-parameter-0-1>> [Debug] Creating new type object for std::vector<nv::PoolAllocator<BlockSize, MemAlloc>::Pool, nv::STLAllocator<nv::PoolAllocator<BlockSize, MemAlloc>::Pool, type-parameter-0-1>> [Debug] Type kind: Unexposed [Debug] base type: 'vector<nv::PoolAllocator::Pool, STLAllocator<nv::PoolAllocator::Pool, type-parameter-0-1>>' for unexposed: std::vector<nv::PoolAllocator<BlockSize, MemAlloc>::Pool, nv::STLAllocator<nv::PoolAllocator<BlockSize, MemAlloc>::Pool, type-parameter-0-1>> [Debug] Resolving type std::vector<nv::PoolAllocator<BlockSize, MemAlloc>::Pool, nv::STLAllocator<nv::PoolAllocator<BlockSize, MemAlloc>::Pool, type-parameter-0-1>> [Debug] Creating new type object for std::vector<nv::PoolAllocator<BlockSize, MemAlloc>::Pool, nv::STLAllocator<nv::PoolAllocator<BlockSize, MemAlloc>::Pool, type-parameter-0-1>> [Debug] Type kind: Unexposed [Debug] base type: 'vector<nv::PoolAllocator::Pool, STLAllocator<nv::PoolAllocator::Pool, type-parameter-0-1>>' for unexposed: std::vector<nv::PoolAllocator<BlockSize, MemAlloc>::Pool, nv::STLAllocator<nv::PoolAllocator<BlockSize, MemAlloc>::Pool, type-parameter-0-1>> [Debug] Resolving type std::vector<nv::PoolAllocator<BlockSize, MemAlloc>::Pool, nv::STLAllocator<nv::PoolAllocator<BlockSize, MemAlloc>::Pool, type-parameter-0-1>> [Error] Error in cursorVisitor: stack overflow
Let's investigate that one… OK, so it seems I should also expect to see templated type when parsing an Unexposed type. So I created a dedicated function parseTemplateType() that is used to handle all templated types parsing (applied for Unexposed, Record or Elaborated types so far) ⇒ Which that change the stack overflow issue is gone [but note that this is not changing the generated code anyway, since we are only parsing a template class at this level].
One thing I should keep in mind is that currently when parsing a template class I don't have any support for non-type template parameters (ie. such as int or bool values): this should really be added eventually. [“eventually”? Ohh boy… you have no idea yet how “soon” “eventually” will really be here lol]
Adding other folders for binding generation
- Adding the “math” folder didn't generate anything new since all classes were already bound anyway.
- Then I added the “log” folder, this will generate the bindings for the “FileLogger” and “StdLogger” properly, but then of course I get compilation errors:
W:\Projects\NervSeed\sources\lua_bindings\wip_core\src\luna\bind_nv_FileLogger.cpp(93): error C2143: syntax error: missing ';' before 'return' W:\Projects\NervSeed\sources\lua_bindings\wip_core\src\luna\bind_nv_StdLogger.cpp(40): error C2143: syntax error: missing ';' before 'return'
⇒ Fixed: this was only a minor typo in the generation code (one missing semi-colon )
- Next issue is when I had the “lua” folder: compilation itself is fine, but I get a linking error this time:
register_package.cpp.obj : error LNK2019: unresolved external symbol "public: static class std::allocator<class sol::state *> * __cdecl nv::LunaTraits<class std::allocator<class sol::state *> >::construct(struct lua_State *)" (?construct@?$LunaTraits@V?$allocator@PEAVstate@sol@@@std@@@nv@@SAPEAV?$allocator@PEAVstate@sol@@@std@@PEAUlua_State@@@Z) referenced in function "public: static int __cdecl nv::Luna<class std::allocator<class sol::state *> >::new_T(struct lua_State *)" (?new_T@?$Luna@V?$allocator@PEAVstate@sol@@@std@@@nv@@SAHPEAUlua_State@@@Z) register_package.cpp.obj : error LNK2019: unresolved external symbol "public: static void __cdecl nv::LunaTraits<class std::allocator<class sol::state *> >::destruct(class std::allocator<class sol::state *> *)" (?destruct@?$LunaTraits@V?$allocator@PEAVstate@sol@@@std@@@nv@@SAXPEAV?$allocator@PEAVstate@sol@@@std@@@Z) referenced in function "public: static int __cdecl nv::Luna<class std::allocator<class sol::state *> >::gc_T(struct lua_State *)" (?gc_T@?$Luna@V?$allocator@PEAVstate@sol@@@std@@@nv@@SAHPEAUlua_State@@@Z) register_package.cpp.obj : error LNK2001: unresolved external symbol "public: static char const * const nv::LunaTraits<class std::allocator<class sol::state *> >::className" (?className@?$LunaTraits@V?$allocator@PEAVstate@sol@@@std@@@nv@@2QBDB) register_package.cpp.obj : error LNK2001: unresolved external symbol "public: static char const * const nv::LunaTraits<class std::allocator<class sol::state *> >::fullName" (?fullName@?$LunaTraits@V?$allocator@PEAVstate@sol@@@std@@@nv@@2QBDB)
So, what's happening with the bindings for the class “std::allocator<class sol::state *>” ? Let's have a look. Ohh, lol, I see:
[Debug] Writing file W:\Projects\NervSeed\sources\lua_bindings\wip_Core\src/luna/bind_std_allocator_sol_state_*.cpp [Debug] Updating content of W:\Projects\NervSeed\sources\lua_bindings\wip_Core\src\luna\bind_std_allocator_sol_state_*.cpp [Error] Cannot open file W:\Projects\NervSeed\sources\lua_bindings\wip_Core\src\luna\bind_std_allocator_sol_state_*.cpp for writing. (file is locked or parent path doesnt exist ?)
⇒ So this is another case of incorrect file name issue. Just updated the sanitize function I'm using:
function Class:sanitizeName(name)
name = name:gsub("::","_")
name = name:gsub("[<>,%s]","_")
name = name:gsub("%*","_ptr")
name = name:gsub("__+","_")
name = name:gsub("_$", "")
return name
end</sxhjs>
And then we are back on rails :-)!
* I continue with the "noise" folder, again, binding generation is OK, but the compilation is failing: <code>W:\Projects\NervSeed\sources\nvCore\include\noise/ARM/cpu-features.h(31): fatal error C1083: Cannot open include file: 'sys/cdefs.h': No such file or directory
jom: W:\Projects\NervSeed\build\msvc64\sources\lua_bindings\wip_Core\src\CMakeFiles\luaCoreWIP.dir\build.make [sources\lua_bindings\wip_Core\src\CMakeFiles\luaCoreWIP.dir\bind_precomp.cpp.obj] Error 2</code>
Hmmm... I'm on windows, so, it's pretty obvious I should not be including an ARM folder there, no :-) ? Let's just ignore that file for now, because I'm not quite sure how I could bypass it automatically: <sxhjs lua>local ignoredInputs = {
"/sol/",
"/rapidjson/",
"/debug/",
"/glm/",
"noise/ARM/",
"/lua/luna",
}</sxhjs>
* Then I added the "osg" folder, and stack overflows are back with that one: <code>[Debug] Type kind: Unexposed
[Debug] base type: 'typename type-parameter-0-0::value_type' for unexposed: typename type-parameter-0-0::value_type
[Debug] Resolving type typename type-parameter-0-0::value_type
[Debug] Creating new type object for typename type-parameter-0-0::value_type
[Debug] Type kind: Unexposed
[Debug] base type: 'typename type-parameter-0-0::value_type' for unexposed: typename type-parameter-0-0::value_type
[Debug] Resolving type typename type-parameter-0-0::value_type
[Debug] Creating new type object for typename type-parameter-0-0::value_type
[Debug] Type kind: Unexposed
[Debug] base type: 'typename type-parameter-0-0::value_type' for unexposed: typename type-parameter-0-0::value_type
[Debug] Resolving type typename type-parameter-0-0::value_type
[Debug] Creating new type object for typename type-parameter-0-0::value_type
[Debug] Type kind: Unexposed
[Error] Error in cursorVisitor: stack overflow</code>
[//My my my... Is this going to stop one day ? :-)//]
=> it seems this comes from the **BoundingBoxImpl** class... More precisely, the source of the issue is this line: <sxh cpp; highlight: []>typedef typename VT::value_type value_type;
So, I'm now parsing those template types to with the following section:
if baseTypeName:startsWith("typename ") then
-- In that case we remove the typename prefix:
local actualTypename = baseTypeName:gsub("^typename%s+","")
-- Then we try to collect the template parameters in there:
local refTname, tparams = typeManager:collectTemplateParameters(actualTypename)
-- note that we might be referencing multiple template types here, so we
-- don't have the constraint #tparams==1.
local contextualTypeName = typeManager:getTemplateContextFullName(refTname)
logDEBUG("Using template contextual type name: ", contextualTypeName, " for unexposed type ", tname)
t:setName(refTname)
t:setTemplateParameters(tparams)
elseif baseTypeName:startsWith("type-parameter-") then</sxhjs>
So, no more stack overflow after that, but then I'm not instantiating the types as I would expect: <code> Using template contextual type name: ${VT}::value_type@osg::BoundingBoxImpl for unexposed type typename type-parameter-0-0::value_type
[Debug] Ignoring alias osg::BoundingBoxImpl<Vec3f>::value_type for template type ${VT}::value_type
[Debug] Arg 1 cursor: ymin, type: osg::BoundingBoxImpl::value_type
[Debug] Resolving type osg::BoundingBoxImpl<Vec3f>::value_type
[Debug] Using already resolved type for 'osg::BoundingBoxImpl<Vec3f>::value_type', type name: ${VT}::value_type
[Debug] Arg 2 cursor: zmin, type: osg::BoundingBoxImpl::value_type
[Debug] Resolving type osg::BoundingBoxImpl<Vec3f>::value_type
[Debug] Using already resolved type for 'osg::BoundingBoxImpl<Vec3f>::value_type', type name: ${VT}::value_type
[Debug] Arg 3 cursor: xmax, type: osg::BoundingBoxImpl::value_type
[Debug] Resolving type osg::BoundingBoxImpl<Vec3f>::value_type
[Debug] Using already resolved type for 'osg::BoundingBoxImpl<Vec3f>::value_type', type name: ${VT}::value_type
[Debug] Arg 4 cursor: ymax, type: osg::BoundingBoxImpl::value_type
[Debug] Resolving type osg::BoundingBoxImpl<Vec3f>::value_type
[Debug] Using already resolved type for 'osg::BoundingBoxImpl<Vec3f>::value_type', type name: ${VT}::value_type
[Debug] Arg 5 cursor: zmax, type: osg::BoundingBoxImpl::value_type
[Debug] Resolving type osg::BoundingBoxImpl<Vec3f>::value_type
[Debug] Using already resolved type for 'osg::BoundingBoxImpl<Vec3f>::value_type', type name: ${VT}::value_type
[Debug] Adding signature void (const osg::BoundingBoxImpl::vec_type &, const osg::BoundingBoxImpl::vec_type &) to function BoundingBoxImpl<VT></code>
[//Yes... that's a total mess lol...//]
So, I eventually traced this down to a problem with my function to get a fully qualified name for a templated type: <sxh cpp; highlight: []>CXString clang_getTypeFullSpelling(CXType CT) {
QualType T = GetQualType(CT);
if (T.isNull())
return cxstring::createEmpty();
CXTranslationUnit TU = GetTU(CT);
PrintingPolicy PP(cxtu::getASTUnit(TU)->getASTContext().getLangOpts());
PP.FullyQualifiedName = 1;
PP.SuppressScope = false;
PP.AnonymousTagLocations = false;
std::string fname = clang::TypeName::getFullyQualifiedName(T, cxtu::getASTUnit(TU)->getASTContext(),PP);
return cxstring::createDup(fname);
// SmallString<64> Str;
// llvm::raw_svector_ostream OS(Str);
// T.print(OS, PP);
// return cxstring::createDup(OS.str());
}
⇒ it seems that this function will return an “instanciated name” such as nvt::MyTemplateClass<bool>::value_t even in a case where we would really only expect nvt::MyTemplateClass<T>::value_t instead, as soon as we have a typedef such as:
typedef MyTemplateClass<bool> TplBool;
OK So I fixed that [i think] by using the canonical type name in the getTypeFullSpelling function:
CXString clang_getTypeFullSpelling(CXType CT) {
QualType T = GetQualType(CT);
if (T.isNull())
return cxstring::createEmpty();
CXTranslationUnit TU = GetTU(CT);
PrintingPolicy PP(cxtu::getASTUnit(TU)->getASTContext().getLangOpts());
PP.FullyQualifiedName = 1;
PP.SuppressScope = false;
PP.AnonymousTagLocations = false;
PP.PrintCanonicalTypes = true;
std::string fname = clang::TypeName::getFullyQualifiedName(T, cxtu::getASTUnit(TU)->getASTContext(),PP);
return cxstring::createDup(fname);
}
This seems to work, but then it will introduce another error when resolving our “MemAlloc” template parameter later:
[Debug] Resolved template argument 0: nv::PoolAllocator::Pool [Debug] Resolving type type-parameter-0-1 [Debug] Creating new type object for type-parameter-0-1 [Debug] Type kind: Unexposed [Debug] Begin of template contexts. [Debug] Context 1: nv::PoolAllocator [Debug] Param 1: MemAlloc [Debug] End of template contexts. [FATAL] [FATAL] Cannot find template parameter with name type-parameter-0-1 in current templating context: (at W:\Projects\NervSeed\sources\lua_bindings\Core\core_bindings.cpp:152) [FATAL]
Actually, this makes sense! And it is related to the fact that we are not handling non-type template parameters yet [I think ?] and there is one in use here, so the “offset” 0-1 is incorrect in this context.
⇒ I really need to also deal with the non-type template parameters. Let's get to it.
Adding support for non-type template parameters
- First thing I did on this point was to add the following in the parseClass() method:
elseif ckind == clang.CursorKind.NonTypeTemplateParameter then -- We should add this as a template parameter: CHECK(scope:isClassTemplate(), "Cannot add template parameter to non class template element.") logDEBUG("Adding template parameter ", cname, " to class template ", scope:getFullName()) scope:addTemplateParameter(cname)</sxhjs> And with that change the latest crash reported above is fixed already ?! [//That was almost too simple ?//] ... All good, but... the problem now, with this "canonical type usage" in getTypeFullSpelling is that all of our nice short aliases are gone :-S (so no more nv::String for instance, but instead just a class called "%%std::basic_string<char, std::char_traits<char>, nv::STLAllocator<char, nv::DefaultPoolAllocator>>%%"): That's too bad :-( I really want to keep my short class names! So let's try to think about this a bit harder. ===== Fixing incorrect full type spelling ===== This full type spelling issue really looks like a bug in the clang library to me... So I found maybe my best bet is to upgrade the llvm dependency ? [that might actually lead to even more trouble, but well not much choice left I guess] Meanwhile, I've also been trying to build a specific method to retrieve the "correct" name for a templated type, but that just doesn't seem to work as expected for the moment: <sxhjs lua>function Class:getTypeFullName(cxtype) -- If this type has template arguments, then we should retrieve its *correct* fullname: local fname = cxtype:getFullSpelling() local nargs = cxtype:getNumTemplateArguments() if nargs>0 then -- We only extract the class name from the name: local clname = utils.extractTemplateClassName(fname) logDEBUG("Extracted template class name: ", clname) tpArgs = {} for i=0,nargs-1 do local aname = self:getTypeFullName(cxtype:getTemplateArgumentType(i)) logDEBUG("Found template argument ",i," name: ", aname) table.insert(tpArgs, aname) end return clname.."<".. table.concat(tpArgs,",")..">" end -- Given a type, we may return its full spelling here: return fname end</sxhjs> Anyway, after rebuilding LLVM/Clang from the current git master version, I now get an error when linking my luaClang module: <code>LINK : fatal error LNK1181: cannot open input file 'LLVMXCoreDisassembler.lib'</code> Nothing too serious I'd say: I can probably just remove that library if it is really not available in my new build ? Hmmm well actually many other libs are missing because I didn't build the support for those archs. So let's regenerate the list of libraries directly: <code>llvm-config --libs</code> * Next error on the list is in my NervJIT.cpp file: <code>W:\Projects\NervSeed\sources\nvLLVM\src\NervJIT.cpp(1169): error C2039: 'addToSearchOrder': is not a member of 'llvm::orc::JITDylib' W:\Projects\NervSeed\scripts\..\deps\msvc64\llvm-20200724\include\llvm/ExecutionEngine/Orc/Core.h(779): note: see declaration of 'llvm::orc::JITDylib' W:\Projects\NervSeed\sources\nvLLVM\src\NervJIT.cpp(1170): error C2039: 'addToSearchOrder': is not a member of 'llvm::orc::JITDylib' W:\Projects\NervSeed\scripts\..\deps\msvc64\llvm-20200724\include\llvm/ExecutionEngine/Orc/Core.h(779): note: see declaration of 'llvm::orc::JITDylib'</code> From what I see in the lli/lli.cpp source file now, that **addToSearchOrder()** function should simply be renamed **addToLinkOrder()**: OK, this seems to do the trick. But then I have the same kind of link error as with the luaClang module: <code>LINK : fatal error LNK1181: cannot open input file 'LLVMX86Utils.lib'</code> => Also fixing the LLVM libs list for the nvLLVM module... OK. * But unfortunately, even with this update to the latest version of LLVM, I still get the **same invalid data reported** :-( So that probably means what is really wrong here is my function to retrieve a full type spelling. So I really need additional investigations in that direction. => So I've been fighting for a long time with this type:getFullSpelling() issue: I tried to implement some kind of fix in lua for it, but that really didn't work: far too complex to handle. Eventually I decided to just get back to the initial implementation for that function :-( [I can't spend so much time on it]. And for now, I'm trying to investigate other issues, since the problem I discovered above was apparently not reproduced in the actual nvCore bindings so far: so that's not really critical after all ? ===== Double class registration issue with default template parameters ===== * Next major issue I noticed, was with some kind of double class registration, as follow: <code> No class template registered for 'map<std::string, nv::RefPtr<nv::LuaScript>, std::less<std::string>, std::allocator<std::pair<const std::string, nv::RefPtr<nv::LuaScript>>>>' creating default place holder class in namespace 'std' No class template registered for 'map<std::string, nv::RefPtr<nv::LuaScript>>' creating default place holder class in namespace 'std' </code> => So in the end, I get 2 classes defined for "map<std::string, nv::RefPtr<nv::LuaScript>>", because one of them doesn't explicitly specify the default values. How on earth am I going to make the system realize that both classes are the same thing here ?! Oh my... Okay, so what is strange here is that we first have: <code>[Debug] Using underlying type: std::map<std::string, RefPtr<LuaScript>> for typedef ScriptMap [Debug] Resolving type std::map<std::string, nv::RefPtr<nv::LuaScript>></code> And shortly after that: <code>[Debug] Resolving type nv::LuaManager::ScriptMap [Debug] Creating new type object for nv::LuaManager::ScriptMap [Debug] Type kind: Typedef [Debug] Using base type: 'std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char>>, nv::RefPtr<nv::LuaScript>, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::allocator<std::pair<const std::basic_string<char, std::char_traits<char>, std::allocator<char>>, nv::RefPtr<nv::LuaScript>>>>' for typedef type nv::LuaManager::ScriptMap</code> => So I think the problem here is due to the fact that I'm trying to register a type directly for a typedef, but not defining the typedef name as a type itself (?). Let's see if I can do something about it...! **OK, fixed!** The key was to **not** try to use the canonical type when facing a typedef type and instead retrieve the "underlyting typedef type" from its declaration: <sxhjs lua> -- local baseType = cxtype:getCanonicalType() local decl = cxtype:getTypeDeclaration() -- Should be a typedefdecl: CHECK(decl:getKind() == clang.CursorKind.TypedefDecl, "Invalid typedef cursor kind: ", decl:getKindSpelling()) local baseType = decl:getTypedefDeclUnderlyingType() local baseTypeName = baseType:getSpelling() logDEBUG("Using base type: '", baseTypeName,"' for typedef type ",tname ) local resolved = self:resolveType(baseType) </sxhjs> * Now I can finally continue with adding more input folders: we add the "osg" folder back. And I get some diagnostic errors from clang: <code>[Info] - diag 2: W:\Projects\NervSeed\sources\nvCore\include\noise/FastNoiseSIMD_internal.h:30:2: error: Dont include this file without defining SIMD_LEVEL_H [Info] - diag 3: W:\Projects\NervSeed\sources\nvCore\include\osg/BoundingBox.h:231:25: error: use of undeclared identifier 'Vec3d' [Info] - diag 4: W:\Projects\NervSeed\sources\nvCore\include\osg/BoundingSphere.h:295:28: error: use of undeclared identifier 'Vec3d'</code> So we start with ignoring the **FastNoiseSIMD_internal.h** file: **OK**. Next we check the Vec3d usage in BoundingBox.h/BoundingSphere.h **OK**. Bindings generation goes well, but then we have some obvious problems because we have the bindings for a class called **osg::BoundingBoxImpl<VT>** => It seems this was because I wasn't checking if the template parameters were actual types or still templates before building a class from a class template. With that small update, the unexpected class "osg::BoundingBoxImpl<VT>" is now gone: <sxhjs lua> elseif not isTemplateType then -- Once we have the templace class definition cursor we can retrieve the actual class template: CHECK(cltpl, "Cannot retrieve class template for type ", tname) logDEBUG("Full template class name is: ", cltpl:getFullName()) -- We now have access to all the required elements to create a class from our template: cl = cltpl:buildClass(templateArgs) -- We assign this class as target: CHECK(cl~=nil, "Invalid class generated from template.") t:setTarget(cl) cl:setAlias(tname) end</sxhjs> ===== Default value variable declaration ref ===== * Next issue on the list is with this kind of code: <sxh cpp; highlight: []>namespace nvt { const double MY_VALUE = 3.0; class Test3 { public: Test3(double val = MY_VALUE) { _val = val; }; void setValue(double val2 = 4.0) { _val = val2; } protected: double _val; }; }
This will produce this kind of code:
// Bind for Test3 constructor (1) with signature: void (double)
static nvt::Test3* _bind_Test3_sig1(lua_State* L) {
// When reaching this call, we assume that the type checking is already done.
int luatop = lua_gettop(L);
double val = luatop>=1? (double)lua_tonumber(L,1) : (double)MY_VALUE;
return new nvt::Test3(val);
}
And this will not compile because the value “MY_VALUE” is defined in the “nvt” namespace which we are not specifying here… not good 
.
⇒ This means that when trying to retrieve the default value for an argument I should more carefully look for additional information on the value itself. Let's see how we can do that…
Okay, with some minimal refactoring in the resolveArgument function we can now retrieve the correct full name for those referenced declarations:
function Class:resolveArgument(cursor, aname)
-- And we resolve the type:
local atype = self:resolveType(cursor:getType())
-- #123: before searching for a tokenized default value, we should
-- first ensure we don't have a declaration ref inside this parameter:
local dclRef = self:getFirstChild(cursor, clang.CursorKind.DeclRefExpr, true)
local defValue = nil
if dclRef then
-- We get the referenced cursor:
local defCursor = dclRef:getCursorReferenced()
logDEBUG("Default value cursor is defined at: ", defCursor:getLocation():getFullPath())
-- The default value we should use is the cursor full name:
defValue = self:getCursorFullName(defCursor)
logDEBUG("Using default value: '", defValue,"'")
end
-- ( continuing here will manual parsing of the cursor tokens)
end</sxhjs>
But now this make me realize: why should I stop here ? why not trying to find the value "programmatically" for the other possible cursor types such as "IntegerLiteral" or "FloatingLiteral" ?
Eventually I found that we could normally extract the AST object from the CXCursor, so I'm adding a bunch of functions to quickly get my int/float/bool values: <sxh cpp; highlight: []> // cf. https://stackoverflow.com/questions/10692015/libclang-get-primitive-value
SOL_CUSTOM_FUNC(getIntegerLiteralValue) = [](class_t&obj) {
ASSERT(CXCursor_IntegerLiteral==obj.kind);
const clang::IntegerLiteral* lit = static_cast<const clang::IntegerLiteral*>(obj.data[1]);
return lit->getValue().getLimitedValue();
};
SOL_CUSTOM_FUNC(getFloatingLiteralValue) = [](class_t&obj) {
ASSERT(CXCursor_FloatingLiteral==obj.kind);
const clang::FloatingLiteral* lit = static_cast<const clang::FloatingLiteral*>(obj.data[1]);
return lit->getValue().convertToDouble();
};
SOL_CUSTOM_FUNC(getBoolLiteralValue) = [](class_t&obj) {
ASSERT(CXCursor_CXXBoolLiteralExpr==obj.kind);
const clang::CXXBoolLiteralExpr* lit = static_cast<const clang::CXXBoolLiteralExpr*>(obj.data[1]);
return lit->getValue();
};
SOL_CUSTOM_FUNC(getStringLiteralValue) = [](class_t&obj) {
ASSERT(CXCursor_StringLiteral==obj.kind);
const clang::StringLiteral* lit = static_cast<const clang::StringLiteral*>(obj.data[1]);
return std::string(lit->getString().data(), lit->getString().size());
};
And with these I can find my literals without the access to the raw tokens, that's cool:
local dclRef = self:getFirstChild(cursor, clang.CursorKind.DeclRefExpr, true)
if dclRef then
-- We get the referenced cursor:
local defCursor = dclRef:getCursorReferenced()
logDEBUG("Default value cursor is defined at: ", defCursor:getLocation():getFullPath())
-- The default value we should use is the cursor full name:
defValue = self:getCursorFullName(defCursor)
logDEBUG("Using default value: '", defValue,"'")
end
local lit = self:getFirstChild(cursor, clang.CursorKind.IntegerLiteral)
if lit then
defValue = "".. lit:getIntegerLiteralValue()
logDEBUG("Found integer literal value: ", defValue)
end
local lit = self:getFirstChild(cursor, clang.CursorKind.FloatingLiteral)
if lit then
defValue = "".. lit:getFloatingLiteralValue()
logDEBUG("Found float literal value: ", defValue)
end
local lit = self:getFirstChild(cursor, clang.CursorKind.CXXBoolLiteralExpr)
if lit then
defValue = lit:getBoolLiteralValue() and "true" or "false"
logDEBUG("Found bool literal value: ", defValue)
end
local lit = self:getFirstChild(cursor, clang.CursorKind.StringLiteral, true)
if lit then
defValue = lit:getStringLiteralValue()
logDEBUG("Found string literal value: ", defValue)
end</sxhjs>
* **Note**: The code above is quite far from optimal: we are visiting each parameter cursor recursively multiple times to find only one different kind of cursor each time. Eventually I should rather visit the cursor only once with a specific function that would search for any of the interesting nodes. Arrff.... Let's do that right now actually lol.
New version of my getFirstChild function: <sxhjs lua>function Class:getFirstChild(cursor, ckind, recursive)
local res = nil
cursor:visitChildren(function(cur, parent)
local curKind = cur:getKind()
-- if ckind is a table, then we should check any of the provided kinds:
if type(ckind) == "table" then
for _, ck in ipairs(ckind) do
if curKind == ck then
res = cur:clone()
break
end
end
elseif ckind==nil or ckind == curKind then
-- We need a copy for the cursors here:
res = cur:clone()
end
if not res and recursive then
res = self:getFirstChild(cur, ckind, recursive)
end
if res then
return clang.VisitResult.Break
end
return clang.VisitResult.Continue
end)
return res
end</sxhjs>
And with this, we can optimze the parameter resolution a bit (performing only a single traversal of the child elements): <sxhjs lua> -- #123: before searching for a tokenized default value, we should
-- first ensure we don't have a declaration ref inside this parameter:
local lit = self:getFirstChild(cursor, {
ck.IntegerLiteral, ck.FloatingLiteral, ck.CXXBoolLiteralExpr, ck.DeclRefExpr, ck.StringLiteral
}, true)
if lit then
local kind = lit:getKind()
if kind == ck.IntegerLiteral then
defValue = "".. lit:getIntegerLiteralValue()
logDEBUG("Found integer literal value: ", defValue)
elseif kind == ck.FloatingLiteral then
defValue = "".. lit:getFloatingLiteralValue()
logDEBUG("Found float literal value: ", defValue)
elseif kind == ck.CXXBoolLiteralExpr then
defValue = lit:getBoolLiteralValue() and "true" or "false"
logDEBUG("Found bool literal value: ", defValue)
elseif kind == ck.DeclRefExpr then
-- We get the referenced cursor:
local defCursor = lit:getCursorReferenced()
logDEBUG("Default value cursor is defined at: ", defCursor:getLocation():getFullPath())
-- The default value we should use is the cursor full name:
defValue = self:getCursorFullName(defCursor)
logDEBUG("Using default value: '", defValue,"'")
elseif kind == ck.StringLiteral then
defValue = '"'..lit:getStringLiteralValue()..'"'
logDEBUG("Found string literal value: ", defValue)
else
THROW("unexpected cursor kind ", lit:getKindSpelling())
end
end</sxhjs>
===== Default value from internal enum =====
* Next I considering the binding generation for the "Task" folder. And, unsurprisingly, I get yet another compilation issue: <code>W:\Projects\NervSeed\sources\lua_bindings\wip_core\src\luna\bind_nv_Task.cpp(470): error C2275: 'nv::Task::reason': illegal use of this type as an expression</code>
Problematic code is: <sxh cpp; highlight: [9]>static int _bind_setDone_sig1(lua_State* L) {
// When reaching this call, we assume that the type checking is already done.
int luatop = lua_gettop(L);
nv::Task* self = Luna< nv::Task >::get(L,1);
ASSERT(self!=nullptr);
bool done = (bool)lua_toboolean(L,2);
uint32_t t = (uint32_t)lua_tointeger(L,3);
nv::Task::reason r = luatop>=4? (nv::Task::reason)lua_tointeger(L,4) : (nv::Task::reason)nv::Task::reason;
self->setDone(done, t, r);
return 0;
}
Of course “nv::Task::reason” is not valid to use here. And I think I know where this comes from already: that would be because in mu getCursorFullName function I'm not considering the cases of enum/enum constants! Let's fix that.
So, simply adding the following case fixed the issue:
elseif pkind == ck.EnumConstantDecl then
-- If we are considering an enum constant value, then we should **bypass**
-- its parent EnumDecl.
local parent = cur:getSemanticParent():getSemanticParent()
if parent:getKind() == ck.TranslationUnit then
pfname = cur:getSpelling()
else
pfname = self:getCursorFullName(parent) .. "::" .. cur:getSpelling()
end
</sxhjs>
===== Undefined class template issue =====
* Still adding more input folders for the binding generation I then get a binding generation error when the system tries to parse some content from rapidjson:
<code>[Debug] Processing: 'STATIC_ASSERTION_FAILURE' kind: 'ClassTemplate
[Debug] Creating class template rapidjson::STATIC_ASSERTION_FAILURE
[Debug] Parsing class template: STATIC_ASSERTION_FAILURE
[Debug] Registering class template: rapidjson::STATIC_ASSERTION_FAILURE
[FATAL] [FATAL] Cannot find definition for template class rapidjson::STATIC_ASSERTION_FAILURE</code>
At first I thought: "What on Earth am i doinng wrong again ?!", but then I realized I was really not doing anything incorrectly, and the still the system was right: there is **no definition** for that class template lol, but actually, this is made **on purpose** to be able to trigger static assertion errors:
<sxh cpp; highlight: []>RAPIDJSON_NAMESPACE_BEGIN
template <bool x> struct STATIC_ASSERTION_FAILURE;
template <> struct STATIC_ASSERTION_FAILURE<true> { enum { value = 1 }; };
template<int x> struct StaticAssertTest {};
RAPIDJSON_NAMESPACE_END
So, this is a bit of a “meta-programming limit usage case”, but still, I should try to handle this more gracefully. Maybe I could fallback on the “canonical cursor” instead of the definition cursor if not found ? OK, this works
Non-type template parameter handling
- Unfortunately, shortly after the last error mentioned above we have another generation issue (seems that we are now parsing the content of rapidjson anyway so things are getting complex…):
[Debug] Creating new type object for rapidjson::StaticAssertTest<sizeof(::rapidjson::STATIC_ASSERTION_FAILURE<bool(sizeof(Ch) >= 2)>)> [Debug] Type kind: Elaborated [Debug] Detected templated type: rapidjson::StaticAssertTest<sizeof(::rapidjson::STATIC_ASSERTION_FAILURE<bool(sizeof(Ch) >= 2)>)> with 1 template arguments [Debug] TemplateRef full name: rapidjson::StaticAssertTest [Debug] Resolving type [Debug] Creating new type object for [Debug] Type kind: Invalid
So the thing here is that this StaticAssertTest class template will take a non-type template parameter, so it's really expected that we cannot “resolve” any type for so kind of integer value. I need to clarify this in the code now…
So I've been trying to update my template argument parsing as follow:
local ttype = cxtype:getTemplateArgumentType(i)
if ttype:isValid() then
self:pushContextCursor(ttype:getTypeDeclaration())
local atype = self:resolveType(ttype)
self:popContextCursor()
logDEBUG("Resolved template argument ",i,": ", atype:getName())
-- If the "type" we just retrieved is still a "template", then this
-- new type will also be a template itself.
isTemplateType = isTemplateType or atype:isTemplateType()
table.insert(templateArgs, atype)
else
-- This is a non-type template argument, so we should retrieve its "value"
local tCursor = cxtype:getTypeDeclaration()
local val = tCursor:getTemplateArgumentValue(i)
local tkind = tCursor:getTemplateArgumentKind(i)
logDEBUG("Non-type template argument ",i," value: ", val, ", tkind:", tkind)
logDEBUG("Declaration location: ", tCursor:getLocation():getFullPath())
-- We should simply insert a string as templateArg here:
table.insert(templateArgs, ""..val)
end</sxhjs>
... yet, this doesn't to work as expected: if I cannot retrieve a valid type for an argument, then the "kind" of template that I retrieve is always **"Invalid"**: So I suspect relying on the type declaration cursor is misleading here. Let's see what I could do instead.
Okay, so, since retrieving the "template parameter value" from the cursor seems to produce garbage, I decided I should try to implement this feature directly from the Type object. So I added the following function as extension in my custom libclang: <sxh cpp; highlight: []>long long clang_Type_getTemplateArgumentAsValue(CXType CT, unsigned index) {
QualType T = GetQualType(CT);
if (T.isNull())
return -1;
auto TA = GetTemplateArguments(T);
if (!TA)
return -1;
const TemplateArgument& arg = TA.getValue()[index];
switch(arg.getKind())
{
case TemplateArgument::Integral:
return arg.getAsIntegral().getExtValue();
default:
DEBUG_MSG("getTemplateArgumentAsValue: Cannot handle template argument of kind "<<(int)arg.getKind());
return -1;
}
}
Now, this is not quite producing the expected result in out case:
[Debug] TemplateRef full name: rapidjson::StaticAssertTest getTemplateArgumentAsValue: Cannot handle template argument of kind 7 [Debug] Non-type template argument 0 value: -1, tkind:9 [Debug] Declaration location: W:\Projects\NervSeed\sources\nvCore\include\rapidjson\rapidjson.h:416:24:15268 [Debug] Full template class name is: rapidjson::StaticAssertTest [Debug] Building class from template: StaticAssertTest<-1>
… but that could be expected here because we are handling some kind of complex expression for the template argument:
[Debug] Detected templated type: rapidjson::StaticAssertTest<sizeof(::rapidjson::STATIC_ASSERTION_FAILURE<bool(sizeof(Ch) >= 2)>)> with 1 template arguments
⇒ Yet the good news is that with this function at least clang will correct detect a parameter of kind 7 (ie. “Expression”) and not just kind 9 (“Invalid”)
According to the definition of the expression kind (cf. this page), maybe somehow we can resolve that expression to its canonical value ? How would I do that ?
⇒ Well, accoding to the Expr documentation, we have the method isIntegerConstantExpr() available on that type, so maybe we could use that ?
I thus extended my getValue function to also support expressions:
long long clang_Type_getTemplateArgumentAsValue(CXType CT, unsigned index) {
QualType T = GetQualType(CT);
if (T.isNull())
return -1;
auto TA = GetTemplateArguments(T);
if (!TA)
return -1;
CXTranslationUnit TU = GetTU(CT);
const auto& ctx = cxtu::getASTUnit(TU)->getASTContext();
const TemplateArgument& arg = TA.getValue()[index];
switch(arg.getKind())
{
case TemplateArgument::Integral:
return arg.getAsIntegral().getExtValue();
case TemplateArgument::Expression:
{
clang::Expr* expr = arg.getAsExpr();
clang::Expr::EvalResult res;
// if(expr->isCXX98IntegralConstantExpr(ctx))
if(expr->EvaluateAsInt(res, ctx))
{
return res.Val.getInt().getExtValue();
// This will crash:
// return expr->EvaluateKnownConstInt(ctx).getExtValue();
}
else {
DEBUG_MSG("getTemplateArgumentAsValue: Cannot extract integer from expression");
return -1;
}
}
default:
DEBUG_MSG("getTemplateArgumentAsValue: Cannot handle template argument of kind "<<(int)arg.getKind());
return -1;
}
}
But unfortunately, this was still not quite working… Yet, looking at the actual source code, this makes sense, because here we only have “Ch” as an unspecified template parameter. So no way we could say if the boolean condition “bool(sizeof(Ch) >= 2)” is true or false, so no way to select the correct assertion class, so no way to evaluate an integer here! 
So… what should I do then ? ⇒ In these cases, we cannot really get the value of the template parameter, so instead, maybe we should simply retrieve its string representation ?! So we could still handle the values in a generig way, and afterall, in the end all we need is a string representation to build our classes!
Good, so, I'm now returning string representing template parameter values. And in case I have an empty string for a given parameter I replace it with “???” and eventually, ignore the generation of the corresponding class:
[Debug] Detected templated type: rapidjson::StaticAssertTest<sizeof(::rapidjson::STATIC_ASSERTION_FAILURE<bool(sizeof(D) != 0)>)> with 1 template arguments [Debug] TemplateRef full name: rapidjson::StaticAssertTest getTemplateArgumentAsValue: Cannot extract integer from expression [Debug] Non-type template argument 0 value: '', tkind:9 [Debug] Full template class name is: rapidjson::StaticAssertTest [Debug] Cannot build templated class with invalid parameters: StaticAssertTest<???> [Warning] Could not build class for type rapidjson::StaticAssertTest<sizeof(::rapidjson::STATIC_ASSERTION_FAILURE<bool(sizeof(D) != 0)>)>
Handling type alias declarations
- Next problem on our path is this:
[Debug] Type kind: Typedef [FATAL] [FATAL] Invalid typedef cursor kind: TypeAliasDecl
The problem comes from this kind of line inside a class template definition in the system “xtree” file:
using _Val_types = typename _Alloc_types::_Val_types;
OK, so it seems using TypeAliasDecl or TypedefDecl would work:
CHECK(decl:getKind() == clang.CursorKind.TypedefDecl or decl:getKind() == clang.CursorKind.TypeAliasDecl,
"Invalid typedef cursor kind: ", decl:getKindSpelling(), ", location: ", decl:getLocation():getFullPath())
local baseType = decl:getTypedefDeclUnderlyingType()</sxhjs>
Yet, it seems I also have another stack overflow problem here... [investigating...] The stack overflow occured because the system didn't detect a template type in the following case: <code>[Debug] Type kind: Typedef
[Debug] Using base type: 'GenericMemberIterator<Const, Encoding, Allocator>' for typedef type rapidjson::GenericMemberIterator<false, UTF8<char>, MemoryPoolAllocator<CrtAllocator>>::Iterator
[Debug] Resolving type rapidjson::GenericMemberIterator<Const, Encoding, Allocator>
[Debug] Creating new type object for rapidjson::GenericMemberIterator<Const, Encoding, Allocator>
[Debug] Type kind: Unexposed
[Debug] base type: 'GenericMemberIterator<Const, Encoding, Allocator>' for unexposed: rapidjson::GenericMemberIterator<Const, Encoding, Allocator>
[FATAL] [FATAL] Failed to detect template type for rapidjson::GenericMemberIterator<Const, Encoding, Allocator></code>
I feel this mess could make sense: because here 'GenericMemberIterator<Const, Encoding, Allocator>' is a type that depend on the template parameters provided for the parent class. But honestly now, I have no idea how I could handle that gracefully...
=> Feeewww... I finally realized I had to push/pop the cursor context when resolving some typedef at that location. This code is getting really crazy lol.
And we are not done yet :-S Now I have another extremely weird error coming apparently from a crash in the C++ code: <code>[Debug] Processing 1 template arguments...
[Debug] Resolving type std::pair<const int, std::basic_string<char, std::char_traits<char>, std::allocator<char>>>
[Debug] Creating new type object for std::pair<const int, std::basic_string<char, std::char_traits<char>, std::allocator<char>>>
[Debug] Type kind: Unexposed
[Debug] base type: 'std::pair<const int, std::basic_string<char, std::char_traits<char>, std::allocator<char>>>' for unexposed: std::pair<const int, std::basic_string<char, std::char_traits<char>, std::allocator<char>>>
[Debug] Detected templated type: std::pair<const int, std::basic_string<char, std::char_traits<char>, std::allocator<char>>> with 2 template arguments
[Debug] Searching template ref in cursor context ''
[FATAL] </code>
=> OK, that's what you get when trying to do anything with a cursor of kind "NoDeclFound" :-) If I add a check to guard against this misusage, then no crash anymore.
===== Invalid bindings generated for "typename types" =====
* Now that I have the bingings generation passing with the additional rapidjson inputs, I can notice a few strange results... For instance I get bindings generated for "classes" such as this: <sxh cpp; highlight: []>typename std::_List_base_types<pair<const basic_string<char, char_traits<char>, allocator<char>>, any>, allocator<pair<const basic_string<char, char_traits<char>, allocator<char>>, any>>>::_Val_types* LunaTraits< typename std::_List_base_types<pair<const basic_string<char, char_traits<char>, allocator<char>>, any>, allocator<pair<const basic_string<char, char_traits<char>, allocator<char>>, any>>>::_Val_types >::construct(lua_State* L) {
luaL_error(L, "No public constructor available for class typename std::_List_base_types<pair<const basic_string<char, char_traits<char>, allocator<char>>, any>, allocator<pair<const basic_string<char, char_traits<char>, allocator<char>>, any>>>::_Val_types");
return nullptr;
}
I didn't even try to compile that: no way this will work So let's fix it already.
The filename for the file containing this binding is extremely long of course!!! And as one might guess here there are a lot of “std::string” alias that could be used in the example above, that's also another point I should check
Actually, even with those “typename” in their names some of those classes would compile, and thus could be considered legit. But really at this point, the filenames were becoming far too long. And I added a mechanism to use shorter names if we get above a given length threshold:
local filename = cl:getUdName()
-- if the filename is too long, then we should generate a short name for it:
if filename:len() >= maxFileNameLength then
local newName = luna:getNextFileName("class_")
logDEBUG("Replacing too long filename element '", filename, "' with '", newName,"'")
filename = newName
end
self:writeOutputFile(("src/luna/bind_%s.cpp"):format(filename))</sxhjs>
This helps a bit... but still, a lot of those classes were not compiling because we are outside of the "std" namespace, and still trying to use a raw class name such as "allocator<char>". And... I definitely don't want to spend my complete life trying to figure out how to inject the correct fullnames for such monstruous classes **nobody is ever going to want to bind or use anyway**!
So, no way around it: now it's really time to add support to discard some of this, and I thus implemented a mechanism to mark some classes as "ignored" in the post processing stage: <sxhjs lua>local ignoredEntities = {
"std::_",
"rapidjson::internal::",
"rapidjson::",
"std::initializer_list",
"^std::pair",
"::iterator$",
-- structs from pmath.h:
"^UIF32$",
"^ULLIF32$",
"^UIPTR$",
}
local ignoreEntity = function(ent)
local name = ent:getFullName()
for _,p in ipairs(ignoredEntities) do
if name:find(p) then
ent:setIgnored(true)
logDEBUG("=> Ignoring entity ", name)
return
end
end
end
cfg.processEntities = function(root)
-- We should register the "void" type:
local cl = root:getOrCreateClass("void")
cl:setDefined(false)
root:foreachClass(ignoreEntity)
end</sxhjs>
Feewww, cool! With these, I can finally throw away a lot of those undesired/polluting bindings! But I'm pretty sure this will break something in it's current state: because I'm requesting the system to ignore a given class, but then I might still have **functions** using that class somehow, and thus the required Luna references will be missing in that case... Or on this point later!
===== Generating shorter names for types =====
Now, with all this, I still feel we have far too long type names where really we could find more handy aliases for them... For instance in this case: <sxh cpp; highlight: []>static int _bind_begin_sig1(lua_State* L) {
nv::DataMap* self = Luna< nv::DataMap >::get(L,1);
ASSERT(self!=nullptr);
std::unordered_map<std::string, boost::any, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<const std::string, boost::any>>>::const_iterator res = self->begin();
std::unordered_map<std::_List_val<std::_List_base_types<std::pair<const std::string, boost::any>, std::allocator<std::pair<const std::string, boost::any>>>::_Val_types>>* res_ptr = new std::unordered_map<std::_List_val<std::_List_base_types<std::pair<const std::string, boost::any>, std::allocator<std::pair<const std::string, boost::any>>>::_Val_types>>(res);
Luna< std::unordered_map<std::_List_val<std::_List_base_types<std::pair<const std::string, boost::any>, std::allocator<std::pair<const std::string, boost::any>>>::_Val_types>> >::push(L, res_ptr, true);
return 1;
}
What I would rather want to see here is the use of the alias “nv::DataMap::MapType::const_iterator”. I think I need to reconsider the alias system for types: Instead of using only one “alias” for a given type, I should rather keep a list of all of them.
And also I could add a parenting system for types: if types are defined inside a class for instance, and we later give a short alias to that class, then the child types should also be able to use that shorter alias somehow.
Conclusion
Okay, so this article is already way too long, so I really need to stop here even if the end result is still not working as expected. Next time we we try to focus on the type aliasing system and improve on that to simplify the code and improve the robustness of the system further.
If you made it this far, congratulations man ! [and if not, well, then… boooOOoohh man! BooOOOOoooh! lol]