blog:2021:0706_nervluna_practical_usage_case_part_2 [NervTech's Wiki]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== NervLuna: A first practical usage case - Part 2 ======

{{tag>dev nervluna bindings}}

Hello everyone! So... is this going to be the day I can finally generate some bindings with NervLuna for SimCore as mentioned in [[blog:2021:0406_nervluna_practical_usage_case_part_1|my previous article about this]] ? And is this going to fail once more for so many reasons that we can't even enumerate them all here 🤣 ? Anyway, not much choice here, so let's get started and we'll see what we get on our way!

====== ======

===== Preparing a minimal binding configuration file =====

So to start with we only really need to build a very small binding config file for simcore, let's call it "bind_simcore.lua": <sxhjs lua>-- Configuration file for the SimCore bindings generation.

local path = import "pl.path"
local utils = import "base.utils"

local simcoreDir = "D:/Projects/SimCore"

local inputDir = path.abspath(simcoreDir .."/sources/lua_modules/luamx/interface")
local outputDir = path.abspath(simcoreDir.."/sources/lua_modules/luamx")

logDEBUG("Using include path: ", inputDir)

local cfg = {}

cfg.moduleName = "mx"
cfg.libraryName = "luamx"
cfg.clangArgs = {
    "-I"..inputDir
}

cfg.cmakeConfig = {
    includeDirs = {"../include", "../interface" },
    libDirs = {},
    libs = {}
}

-- local msvcIncludeDir = "D:/Softs/VisualStudio2017CE/VC/Tools/MSVC/14.16.27023/include"
local msvcIncludeDir = "D:/Softs/VisualStudio2019CE/VC/Tools/MSVC/14.27.29110/include"
cfg.ignoredIncludePaths = {msvcIncludeDir}

cfg.includePaths = {inputDir}
cfg.inputPaths = {inputDir}
cfg.outputDir = outputDir

return cfg</sxhjs>

I then started with a very minimal interface file with the content: <sxh cpp; highlight: []>#ifndef _BIND_LUAMX_H_
#define _BIND_LUAMX_H_


namespace mx {

class RefObject {
public:
    RefObject();
    ~RefObject();
};

};

#endif
</sxh>

And now let's run the bindings generation with a newly added command shortcut: **nv_seed bind4 | tee bind4.log**: **OK** The binding generation went fine.

===== Compiling luamx binding files =====

Next step will be to try to compile the luamx generate files as part of the whole simcore build process. But first thing that we can note here is that the generate CmakeList.txt file will need some customization: <code cmake>SET(TARGET_NAME "luamx")
SET(TARGET_DIR ".")

SET(SHARED_TGT ${TARGET_NAME})
FILE(GLOB_RECURSE SOURCE_FILES "*.cpp")


INCLUDE_DIRECTORIES(../include)
INCLUDE_DIRECTORIES(../interface)


ADD_MSVC_PRECOMPILED_HEADER("bind_precomp.h" "bind_precomp.cpp" SOURCE_FILES)

# Build the shared library:
ADD_LIBRARY (${SHARED_TGT} SHARED ${SOURCE_FILES})

TARGET_LINK_LIBRARIES(${SHARED_TGT} nvCore ${LUAJIT_LIBS} )

INSTALL(TARGETS ${SHARED_TGT}
RUNTIME DESTINATION ${TARGET_DIR}
LIBRARY DESTINATION ${TARGET_DIR})

# Install the pdb if applicable:
INSTALL_PDB()
</code>

  * The TARGET_DIR value should be changed to be instead: "${PROJECT_SOURCE_DIR}/lua/modules": **OK** made this configurable with a config entry called "targetDir"
  * The INSTALL_PDB() command should be removed here: **OK** made this configurable with a config entry called "installPdb"
  * And I'm wondering if the build could succeed with the link to the nvCore library (I really doubt it through...), so let's give this a try and see.

So I try a first integrated compilation of those bindings inside simcore (with **sc_build msvc64** command), and I get the error: <code>D:\Projects\SimCore\sources\lua_modules\luamx\include\bind_common.h(18): fatal error C1083: Impossible d'ouvrir le fichier include : 'core_lua.h' : No such file or directory
jom: D:\Projects\SimCore\build\msvc64\sources\lua_modules\luamx\src\CMakeFiles\luamx.dir\build.make [sources\lua_modules\luamx\src\CMakeFiles\luamx.dir\bind_precomp.cpp.obj] Error 2</code>

Okay, but quickly checking the core_lua.h content, it's not really obvious why we would need it: appart from the include of the LuaManager class it mainly contains SOL3 related macros, so maybe we could still do without this ?

Hmmm, continuing on this path, the next missing header is lua/luna.h, and well, that one sounds more critical: it contains declaration of all the "luna" functions/types we are actually using to build the bindings, so we really need that part.

But now I'm thinking: maybe I could build those luna related components as a dedicated shared or static library, I would make this a dependency of the nvCore library inside NervSeed but then could also easily include it in external projects like simcore here. => Let's try that option.


===== Turning nvLuna into standalone library =====

So I made quite a few changes on the luna bindings system to turn it into a dedicated standalone library:

  * I replaced the SID() (from nvCore lilbrary) macro with a dedicated LUNA_SID() equivalent
  * I started using a dedicated LunaID type instead of "StringID"
  * I replaced all the helper macros ASSERT/logDEBUG/THRROW_MSG/etc with custom versions LUNA_ASSERT/LUNA_DEBUG_MSG/LUNA_THROW_MSG
  * I changed the namespace from "nv" to "luna"
  * I added support for custom assignment of debug/error log handlers, as this can be handly to integrate into other external log systems: <sxh cpp; highlight: []>// In header file:
namespace luna
{
    typedef void (*luna_log_func)(const char *);
    void logDebugMsg(const char *msg);
    void logErrorMsg(const char *msg);
    NVLUNA_EXPORT void luna_setLogFunctions(luna_log_func debugFunc, luna_log_func errorFunc);
}

// Implementation file:
namespace luna
{

static luna_log_func debugLogFunc = nullptr;
static luna_log_func errorLogFunc = nullptr;

void luna_setLogFunctions(luna_log_func debugFunc, luna_log_func errorFunc)
{
	debugLogFunc = debugFunc;
	errorLogFunc = errorFunc;
}

void logDebugMsg(const char* msg)
{
  if(debugLogFunc) {
    debugLogFunc(msg);
  }
  else {
	  std::cout << msg << std::endl; \
  }
}

void logErrorMsg(const char* msg)
{
  if(errorLogFunc) {
    errorLogFunc(msg);
  }
  else {
	  std::cerr << msg << std::endl; \
  }
}

}
</sxh>
  * Then I actually started removing the "luna_" prefix on most of my luna functions since we are now working in the luna namespace directly.
  * And I replaced the module loading mechanism to be able to load it as a regular lua module (with a luaopen_xxx() registragion function), so my register_package.cpp files will now rather end with something like this: <sxh cpp; highlight: []>extern "C" {

	int luaopen_luaTest1 (lua_State *L) {
		return luna::load_luaTest1_bindings(L);
	}

}</sxh>

That last change above was pretty serious, and I think this will mean some additional trouble on the NervSeed side of things: because so far I had a dedicated mechanism in place to provide "entrypoints" to load those bindings to my nvCore "LuaManager". But well, this was really needed to avoid too deep links with the nvCore library, so I'll fix those other issues later as required ;-). Meanwhile, I could rebuild my luaTest1 module successfully (oh... and that's another thing actually, I cannot have a "module name" and a different "library name" to support the standard lua module loading process, so now I use a unified "module name" everywhere). So now it's time to try the simcore initial binding module build again [//crossing fingers...//🤞]

===== Cheery-picking elements to bind =====

Now that I can finally build a module for simcore bindings properly, I realized there is something critical that I needed: say I only really want to bind some parts of an existing library or only some functions. If I ask luna to parse those complete header files it's going to generate a lot more bindings that what I really want/need. So it's very handly to only specify what you want binded into one of those "interfaces" files as I've been doing in test phase so far, so for instance: <sxh cpp; highlight: []>#ifndef _BIND_LUAMX_H_
#define _BIND_LUAMX_H_


namespace mx {

class RefObject {
public:
    RefObject();
    ~RefObject();
};

};

#endif</sxh>

But then, that interface file is going to be included as header when actually compiling the bindings => yet, at that point, I don't want to be using that simple overview of the classes/functions I need, instead I'm going to include the "real header files" to access those! But if I keep the interface too then I will run into class/function duplications errors: not good.

The solution is simple though: I should encapsulate all my bindings interface definition stuff in a preprocessor enabled section that's only used when generating the bindings: so I only need to also pass that preprocessor macro to clang when parsing for bindings generation so here: <sxhjs lua>cfg.moduleName = "luamx"
cfg.clangArgs = {
    "-DLUNA_GEN_BINDINGS",
    "-I"..inputDir
}</sxhjs>

And then the updated interface file becomes: <sxh cpp; highlight: []>#ifndef _BIND_LUAMX_H_
#define _BIND_LUAMX_H_

#ifdef LUNA_GEN_BINDINGS

namespace sc {

class RefObject {
public:
    RefObject();
    ~RefObject();
};

};

#else
// provide the real RefObject class:
#include <base/RefObject.h>
using namespace sc;
#endif

#endif</sxh>

And with this the binding generation will work and during the compilation stage we use the proper header file. 

===== DLL importing classes and functions to bind =====

Yet I just noticed that I actually don't have any valid binding generated for the simple class constructors/destructors I specified above. Instead I get: <sxh cpp; highlight: []>sc::RefObject* LunaTraits< sc::RefObject >::construct(lua_State* L) {
	luaL_error(L, "No public constructor available for class sc::RefObject");
	return nullptr;
}

void LunaTraits< sc::RefObject >::destruct(sc::RefObject* obj) {
	LUNA_THROW_MSG("No public destructor available for class sc::RefObject");
}</sxh>

And checking the binding generation log I see that I got: <code>[Debug] 	      Writing class binding for mx::RefObject
[Debug] 	      Ignoring non DLL imported signature: mx::RefObject::RefObject [void ()]
[Debug] 	      Ignoring non DLL imported signature: mx::RefObject::~RefObject [void ()]
[Debug] 	      Preprocessing function list...
[Debug] 	      Processing function mx::RefObject::RefObject
[Debug] 	      Ignoring non DLL imported signature: mx::RefObject::RefObject [void ()]
[Debug] 	      Found non-convertible signature(s) in function mx::RefObject::RefObject
[Debug] 	      Processing function mx::RefObject::~RefObject
[Debug] 	      Ignoring non DLL imported signature: mx::RefObject::~RefObject [void ()]
[Debug] 	      Found non-convertible signature(s) in function mx::RefObject::~RefObject
[Debug] 	      Keeping 0 on 2</code>

So thinking about it I sort of remember there were special considerations for those importer / locally defined functions. And here I think the parser will notice that the functions at play are not "defined", but since the class itself is not "imported" then we would not be able to use those functions at all in the bindings module. And thus they are not binded in the end!

To fix this problem I think I need to specify clearly that I'm going to "import" that class I want to bind => I'm going to place this in a dedicated bind_export.h file, so I don't have to retype this logic multiple times. The content of this file is as follow (autogenerated during bindings generation): <sxh cpp; highlight: []>#ifndef BIND_EXPORT_
#define BIND_EXPORT_

#if defined(_MSC_VER) || defined(__CYGWIN__) || defined(__MINGW32__) || defined( __BCPLUSPLUS__)  || defined( __MWERKS__)
#  if defined( BIND_LIBRARY_STATIC )
#    define LUNA_EXPORT
#  else
#    define LUNA_EXPORT   __declspec(dllexport)
#  endif
#  if defined( BIND_LIBRARY_STATIC )
#    define LUNA_IMPORT
#  else
#    define LUNA_IMPORT   __declspec(dllimport)
#  endif
#else
#  define LUNA_EXPORT
#  define LUNA_IMPORT
#endif

//#pragma warning( disable: 4251 )
#endif

</sxh>

And I updated my interface definition for simcore to this: <sxh cpp; highlight: [10]>#ifndef _BIND_LUAMX_H_
#define _BIND_LUAMX_H_

#ifdef LUNA_GEN_BINDINGS

#include <bind_export.h>

namespace sc {

class LUNA_IMPORT RefObject {
public:
    RefObject();
    ~RefObject();
};

};

#else
// provide the real RefObject class:
#include <base/RefObject.h>
using namespace sc;
#endif

#endif
</sxh>

And with this change I finally get correct bindings for those simple constructor/destructor functions: <sxh cpp; highlight: []>sc::RefObject* LunaTraits< sc::RefObject >::construct(lua_State* L) {
	return _bind_RefObject(L);
}

void LunaTraits< sc::RefObject >::destruct(sc::RefObject* obj) {
	delete obj;
}</sxh>

Only problem left now is that this code is not "good enough to compile" in fact: from this definition clang will consider that RefObject is an ordinary class but in fact this is an abstract class so I won't be able to instantiate it like that 😄! Anyway, no big deal: let's just try a  more elaborated version.

<note important>All the considerations here for LUNA_IMPORT usage is only really needed when we try to build classes mockups like here for simple bindings generation: if instead we were requesting Clang to parse the real header directly, it would generate much more bindings by default sure, but at the same time it would also be able to use the real DLL import attributes on the input classes. So this is not a too big concern in the end.</note>

===== Smart pointers containers =====

Next stop on our journey is directly "extracted" from the destructor function we generated above: as we can see the destructor will right now just delete the created object. This is expected, but at this level we really need to push it a bit further and use smart pointers to deal with those memory considerations.

Hopefully this is already implemented in Luna, and we just need to make the compilation pass aware of the fact that all "RefObject" based classes should be stored inside a RefPtr<> container, with the following specialization (which I'm placing directly inside bind_context.h for clarity): <sxh cpp; highlight: []>#ifndef BIND_CONTEXT_
#define BIND_CONTEXT_

// Write here the headers that should be provided during parsing.
#include <base/RefObject.h>

namespace luna {

template <>
struct luna_provider<sc::RefObject>
{
    typedef sc::RefPtr<sc::RefObject> container_t;

    static sc::RefObject *get(const container_t &cont)
    {
        return cont.get();
    };

    static void set(container_t &cont, sc::RefObject *ptr)
    {
        cont = ptr;
    };

    static void release(container_t &cont)
    {
        cont = nullptr;
    };
};

};

#endif</sxh>

Okay, hmmm, but this will not change the result of the bindings generation in fact, and having a closer look at the luna code there is something that doesn't look right here to me: <sxh cpp; highlight: []>        static int gc_T(lua_State *L)
        {
            // typedef typename traits_t::parent_t ParentType;

            auto ud = luna_toUserData<T>(L, 1);
            LUNA_ASSERT(ud != nullptr); // Should be valid.

            if (ud->gc)
            {
                T *obj = (T *)provider_t::get(ud->pT);
                LUNA_ASSERT(obj != nullptr);

                traits_t::destruct(obj);
            }

            // Finally we release the pointer to the object:
            provider_t::release(ud->pT);

            return 0;
        }</sxh>

Obviously this **gc_T** function above is what is used to delete/destroy objects from lua side. And when we have ud->gc == true (ie. we consider lua is responsible for handling the memory for that object), then we will first call destruct() and then release() on that object: so even with out smart container declared above, we would still call that "delete obj" first: not quite what I really wanted here 😅. So let me fix that...

Okay! So I ended up updating my provider struct as follow: <sxh cpp; highlight: []>    template <typename T>
    struct luna_provider
    {
        typedef T *container_t;

        static inline T *get(const container_t &cont)
        {
            return cont;
        };

        static inline void set(container_t &cont, T *ptr)
        {
            cont = ptr;
        };

        static inline T* release(container_t &cont)
        {
            T* ptr = cont;
            cont = nullptr;
            return ptr;
        };

        static inline void destruct(container_t &cont)
        {
            LunaTraits<T>::destruct(cont);
        };
    };</sxh>

And then the gc_T handler will simply looks like this: <sxh cpp; highlight: []>        static int gc_T(lua_State *L)
        {
            // typedef typename traits_t::parent_t ParentType;

            auto ud = luna_toUserData<T>(L, 1);
            LUNA_ASSERT(ud != nullptr); // Should be valid.

            if (ud->gc)
            {
                // At this point we really expect the object to get destroyed if there is no reference to it any more
                provider_t::destruct(ud->pT);
            }
            else {
                // We just release the pointer:
                provider_t::release(ud->pT);
            }

            return 0;
        }</sxh>

=> So with those changes I should be able to:
  * Allocate simple objects and then delete them properly in lua using the default provider implementation + object destruct function
  * Handle "gracefully" the cases where the destructor is protected (Note that "gracefully" here means throwing an error such as ''LUNA_THROW_MSG("No public destructor available for class sc::RefObject")'' 🤣)
  * Handle the cases where I provided a smart provider implementation correctly.

===== Special handling of std::string type =====

I then tried to focus on bindings some simple functions before going deeper with the classes. So I used the following interface file content: <sxh cpp; highlight: []>#ifndef _BIND_LUAMX_H_
#define _BIND_LUAMX_H_

#ifdef LUNA_GEN_BINDINGS
#include <bind_export.h>
#include <string>

namespace sc {

// class LUNA_IMPORT RefObject {
// public:
//     RefObject();
//     ~RefObject();
// };
};

namespace mx {

LUNA_IMPORT void msleep(unsigned int ms);
LUNA_IMPORT void usleep(unsigned int ms);
LUNA_IMPORT double qRound(double val, int ii, int ff);
LUNA_IMPORT std::string toHex(const std::string& input);
LUNA_IMPORT std::string fromHex(const std::string& input);
LUNA_IMPORT std::string toBinary(const std::string& input, const char sep = ' ');
LUNA_IMPORT std::string fromBinary(const std::string& input);

};

#else

// provide the real RefObject class:
#include <base/RefObject.h>
#include <mx/utils.h>
using namespace sc;
using namespace mx;
#endif

// Custom functions here:
namespace mx {

inline uint32_t toUInt32(int32_t val) { return (uint32_t)val; }
inline int32_t toInt32(uint32_t val) { return (int32_t)val; }

};

#endif
</sxh>

This was working pretty fine, except for the handling of the std::string that should be returned from the functions such as ''std::string toHex(const std::string& input)'' above: binding system was initially creating a class named ''std::basic_string<char,std::char_traits<char>,std::allocator<char>>'' for the ''std::string'' type and then this was messing completely the automatic conversion to lua strings. So I had to fix a few glitches inside the ClangParser again to ensure that a concrete class mapped to a type would also use the shortest name available for the type, and now this is working just as expected 😁

===== Adding support for const char arguments =====

Oh one small issue with the functions binded just above though: the toBinary binding was not generated due to the not handled argument type "const char": <sxh cpp; highlight: [3-5]>namespace luna {

// NervLuna: Ignoring non lua convertible function 'mx::toBinary'
//  - std::string (const std::string &, const char)  => const char
// Function type checkers

// Typecheck for msleep (1) with signature: void (unsigned int)
static bool _check_mx_msleep_sig1(lua_State* L) {
	int luatop = lua_gettop(L);
	if( luatop!=1 ) return false;

	if( lua_isnumber(L,1)!=1 ) return false;
	return true;
}</sxh>

=> So it was time to extend a bit the CharConverter class! yet, in the end this was not a too big deal: right now I simply consider that I will receive a string from lua, and I ensure the length of that string is 1, and I retrieve the corresponding char. So this will generate this kind of code: <sxh cpp; highlight: [7-10]>// Bind for toBinary (1) with signature: std::string (const std::string &, const char)
static int _bind_mx_toBinary_sig1(lua_State* L) {
	int luatop = lua_gettop(L);
	size_t input_len = 0;
	const char* input_cstr = lua_tolstring(L,1,&input_len);
	std::string input(input_cstr, input_len);
	size_t sep_len = 0;
	const char* sep_cstr = luatop>=2 ? lua_tolstring(L,2,&sep_len) : nullptr;
	LUNA_ASSERT(luatop<2 || sep_len == 1);
	const char sep = sep_cstr ? sep_cstr[0] : ' ';

	std::string res = mx::toBinary(input, sep);
	lua_pushlstring(L, res.c_str(), res.size());

	return 1;
}</sxh>

===== Simplifying parse/compile contexts handling =====

In the end the "simple solution" I implemented above consisting in using a custom ''LUNA_GEN_BINDINGS'' macro to differentiate between the parse step and the compile step felt a bit too inconvinient for me... Visual Studio would display one part in "disabled code" mode, and it's still a lot of boiling code to add when generating bindings.

=> So I eventually updated the system to use 3 separated files now:

  * **interface/parse_context.h** : this one is only included in the parsing stage. So we can put here simplified mockups/interfaces definitions for existing real classes/functions that we don't want to fully parse.
  * **include/compile_context.h** : this one is only included in the actual compilation stage. So we put here addition content required for proper compilation but not for the bindings generation, like the real header files for the class/function "mockups" we defined in the "parsing stage"
  * **include/shared_context.h** : this one is included in both stages: that's where we will put content that should be binded, but also defined/implemented during compilation like new custom helper functions for instance.

And now the default logic I will take is that everything in the "interface" folder should be considered as only available during the parsing stage, so no more custom class definition there. On the whole, this was a bit tricky to setup, but now this seems to be working fine.

===== Conclusion =====

So, right now, I still need to write some more of the required bindings for simcore as started above. And this is taking some time unfortunately lol. And I really need a break. And I really need to get some other tasks moving forward too. So in the end, it seems we wont get our bindings fully ready within this article either 😅 but hey! We made a lot of progress in this direction and things are looking good so far, so, maybe next time then ? ;-)