NervLuna: Simplifying manual bindings

NervLuna: Simplifying manual bindings

In my last post I covered restoring the LLVM JIT compiler support in the NervLand project. That part seems to be working not too bad for now, but I realized that if I want to be able to dynamically inject new functions in the lua state then I need a more user friendly way to do it still using the Luna infrastructure, but with a mechanism a bit similar to what is done in the SOL library. So this will be the main topic that we will cover here. Let's get started [Spoiler alert: this is going to be a little tricky ]

Initial support for manual bindings generation

First, I now think it would be preferable to simply merge the Luna module directly into the Core module, so let's handle that task first. OK, that was a rather simple task.

Next, let's consider a simple case where we want to add a simple function binding in lua, something like the following:

int add(int a, int b)
{
   return a+b;
}

void register_function(LuaState& state)
{
  state.set_function("nvk.add_func", add);
}

And then, we want to be able to call that function in lua as follow:
```
local res = nvk.add_func(40, 2)
CHECK(res==42, "Unexpected result")
```

⇒ How could we do that ?

I have just created a luaSandBox module to store any precompiled extension I could think about, and that's where I should start experimenting with those manual bindings generation steps.

To be able to simply register lua bindings functions from different translation units in this sandbox module, I have now also implemented a concept of Luna binding functions registrar with the following macros:

#define LUNA_DECLARE_REGISTRAR()                                               \
    namespace luna {                                                           \
    auto get_bind_functions() -> LuaBindFunctionList&;                         \
    inline void register_bind_function(nv::LuaManager::BindingFunc func) {     \
        get_bind_functions().push_back(func);                                  \
    }                                                                          \
    struct BindingRegistration {                                               \
        BindingRegistration(nv::LuaManager::BindingFunc func) {                \
            register_bind_function(func);                                      \
        }                                                                      \
    };
}

#define LUNA_IMPLEMENT_REGISTRAR()                                             \
    namespace luna {                                                           \
    auto get_bind_functions() -> LuaBindFunctionList& {                        \
        static LuaBindFunctionList bindFuncs;                                  \
        return bindFuncs;                                                      \
    }                                                                          \
    }

#define LUNA_ADD_BINDING(func)                                                 \
    static const luna::BindingRegistration loader{func};

So now I can register some new bindings easily in a given cpp file:

#include <sandbox_precomp.h>

using namespace nv;

static auto add(int a, int b) -> int { return a + b; }

static void load_bindings(LuaState& lua) {
    logDEBUG("Should register my add function in the lua state here.");
}

LUNA_ADD_BINDING(load_bindings)

When trying to use this, I indeed get the expected outputs:

2022-11-26 13:13:24.719107 [DEBUG] Loading Sandbox bindings...
2022-11-26 13:13:24.719136 [DEBUG] Loading bindings for Sandbox with libname luaSandbox
2022-11-26 13:13:24.719160 [DEBUG] Loading dynamic library: luaSandbox.dll
2022-11-26 13:13:24.719182 [DEBUG] NervApp: load_library...
2022-11-26 13:13:24.719192 [DEBUG] lib file: modules/luaSandbox.dll
2022-11-26 13:13:24.746891 [DEBUG] Registering bindings for Sandbox
2022-11-26 13:13:24.746964 [DEBUG] Opened DynamicLibrary modules/luaSandbox.dll
2022-11-26 13:13:24.747013 [DEBUG] Should register my add function in the lua state here.

Now let's add some content in there…

Feeeww… that's getting a little bit tricky 😄: I had to add a pretty complex layers to be able to connect my C++ functions in Lua still using the Luna context. But this is starting to work now.

We have the LuaTable which is used as our main “container” in lua:

struct NVCORE_EXPORT LuaTable {
    using LuaCFunc = int (*)(lua_State*);

    LuaTable(const LuaTable&) = delete;
    auto operator=(const LuaTable&) -> LuaTable& = delete;
    LuaTable(LuaTable&&) noexcept;
    auto operator=(LuaTable&&) noexcept -> LuaTable&;

    lua_State* state{nullptr};
    int index{0};
    bool isRef{false};

    LuaTable() = delete;
    ~LuaTable();
    LuaTable(lua_State* L, int idx, bool makeRef);

    void push_self() const;
    [[nodiscard]] auto get_or_create_table(const nv::String& tname,
                                           nv::U64 offset = 0) const
        -> LuaTable;

    void set_function(const char* key, LuaCFunc func) const;
    void set_closure(const char* key, LuaCFunc func, nv::I32 nargs) const;

    auto operator[](const char* key) -> LuaTableSlot;
};

And from the table we can get access to LuaTableSlot objects:

struct NVCORE_EXPORT LuaTableSlot {
    LuaTableSlot(const LuaTableSlot&) = default;
    auto operator=(const LuaTableSlot&) -> LuaTableSlot& = default;
    LuaTableSlot(LuaTableSlot&&) = default;
    auto operator=(LuaTableSlot&&) -> LuaTableSlot& = default;

    LuaTableSlot(LuaTable* parent, const char* keyStr)
        : table(parent), key(keyStr){};

    ~LuaTableSlot() = default;

    template <typename T> auto operator=(T func) -> LuaTableSlot& {
        // Apply the connection:
        LuaConnector<T>::connect(*table, key, func);
        return *this;
    }

    LuaTable* table{nullptr};
    const char* key;
};

Then those slots can accept an arbitrary type in the assignment operator and for now the only LuaConnector we provide will be used to construct a closure around the provided function pointer:

template <typename T> struct LuaExecutor;

template <typename T> struct LuaConnector {
    static void connect(LuaTable& table, const char* key, T func) {
        lua_State* L = table.state;
        lua_pushlightuserdata(L, (void*)func);
        table.set_closure(key, LuaExecutor<T>::call, 1);
    }
};

Finally the LuaExecutor template is specialized depending on the function signature, for instance:

template <typename Ret, typename A0> struct LuaExecutor<Ret (*)(A0)> {
    using func_t = Ret (*)(A0);

    static auto call(lua_State* L) -> int {
        // Retrieve the function as an upvalue:
        auto func = (func_t)lua_touserdata(L, lua_upvalueindex(1));
        A0 arg0 = luna::LunaGetter<A0>::getValue(L, 1);
        Ret val = func(arg0);
        LunaPusher<Ret>::pushValue(L, val);
        return 1;
    }
};

Extending support for functions assignments

Next I wanted to extend the support for more function arguments in the LuaExecutor, but didn't want to write all the variable number of arguments so I've been fighting with variadic template arguments, but finally got this working:

template <unsigned int N, typename T, typename... R> struct type_at {
    using type = typename type_at<N - 1, R...>::type;
};

template <typename T, typename... R> struct type_at<0, T, R...> {
    using type = T;
};

// cf. http://loungecpp.wikidot.com/tips-and-tricks:indices
template <std::size_t... Is> struct indices {};

template <std::size_t N, std::size_t... Is>
struct build_indices : build_indices<N - 1, N - 1, Is...> {};

template <std::size_t... Is> struct build_indices<0, Is...> : indices<Is...> {};

template <typename... Args> struct LuaExecutor<void (*)(Args...)> {
    using func_t = void (*)(Args...);
    static constexpr size_t nArgs = sizeof...(Args);
    static constexpr build_indices<nArgs> indices{};

    template <unsigned int idx>
    static auto get_lua_value(lua_State* L) ->
        typename type_at<idx, Args...>::type {
        return luna::LunaGetter<typename type_at<idx, Args...>::type>::getValue(
            L, idx + 1);
    }

    template <std::size_t... Is>
    static void call_func(func_t func, lua_State* L,
                          const luna::indices<Is...>& /*ids*/) {
        func(get_lua_value<Is>(L)...);
    }

    static auto call(lua_State* L) -> int {
        // Retrieve the function as an upvalue:
        auto func = (func_t)lua_touserdata(L, lua_upvalueindex(1));
        // How to retrieve the arguments here ?
        call_func(func, L, indices);
        return 0;
    }
};

template <typename Ret, typename... Args> struct LuaExecutor<Ret (*)(Args...)> {
    using func_t = Ret (*)(Args...);
    static constexpr size_t nArgs = sizeof...(Args);
    // using indices = build_indices<nArgs>;
    static constexpr build_indices<nArgs> indices{};

    template <unsigned int idx>
    static auto get_lua_value(lua_State* L) ->
        typename type_at<idx, Args...>::type {
        return luna::LunaGetter<typename type_at<idx, Args...>::type>::getValue(
            L, idx + 1);
    }

    template <std::size_t... Is>
    static auto call_func(func_t func, lua_State* L,
                          const luna::indices<Is...>& /*ids*/) -> Ret {
        return func(get_lua_value<Is>(L)...);
    }

    static auto call(lua_State* L) -> int {
        // Retrieve the function as an upvalue:
        auto func = (func_t)lua_touserdata(L, lua_upvalueindex(1));
        Ret val = call_func(func, L, indices);
        LunaPusher<Ret>::pushValue(L, val);
        return 1;
    }
};

Unifying function return type

What I want to change next with the code above is the fact that I'm handling functions with a return type and functions returning void in fully separated templates while only the return value push on state mechanism should really differ, so let' optimize that part.

Here is the updated code: introducing an additional LuaCaller helper template to either call the provided functions directly if there is no need to retrieve any argument, or to forward the function to the LuaExecutor template if we need to retrieve some arguments to call it:

template <std::size_t... Is> struct indices {};

template <std::size_t N, std::size_t... Is>
struct build_indices : build_indices<N - 1, N - 1, Is...> {};

template <std::size_t... Is> struct build_indices<0, Is...> : indices<Is...> {};

template <typename Ret, typename... Args> struct LuaCaller<Ret (*)(Args...)> {
    using func_t = Ret (*)(Args...);

    static auto call(lua_State* L) -> int {
        return LuaExecutor<func_t>::call_ret_args(L);
    }
};

template <typename... Args> struct LuaCaller<void (*)(Args...)> {
    using func_t = void (*)(Args...);

    static auto call(lua_State* L) -> int {
        return LuaExecutor<func_t>::call_void_args(L);
    }
};

template <typename Ret> struct LuaCaller<Ret (*)()> {
    using func_t = Ret (*)();

    static auto call(lua_State* L) -> int {
        auto func = (func_t)lua_touserdata(L, lua_upvalueindex(1));
        Ret val = func();
        LunaPusher<Ret>::pushValue(L, val);
        return 1;
    }
};

template <> struct LuaCaller<void (*)()> {
    using func_t = void (*)();

    static auto call(lua_State* L) -> int {
        auto func = (func_t)lua_touserdata(L, lua_upvalueindex(1));
        func();
        return 0;
    }
};

template <typename Ret, typename... Args> struct LuaExecutor<Ret (*)(Args...)> {
    using func_t = Ret (*)(Args...);
    static constexpr size_t nArgs = sizeof...(Args);
    static constexpr build_indices<nArgs> indices{};

    template <unsigned int idx>
    static auto get_lua_value(lua_State* L) ->
        typename type_at<idx, Args...>::type {
        return luna::LunaGetter<typename type_at<idx, Args...>::type>::getValue(
            L, idx + 1);
    }

    template <std::size_t... Is>
    static auto call_func(func_t func, lua_State* L,
                          const luna::indices<Is...>& /*ids*/) -> Ret {
        return func(get_lua_value<Is>(L)...);
    }

    static auto call_ret_args(lua_State* L) -> int {
        // Retrieve the function as an upvalue:
        auto func = (func_t)lua_touserdata(L, lua_upvalueindex(1));
        Ret val = call_func(func, L, indices);
        LunaPusher<Ret>::pushValue(L, val);
        return 1;
    }

    static auto call_void_args(lua_State* L) -> int {
        // Retrieve the function as an upvalue:
        auto func = (func_t)lua_touserdata(L, lua_upvalueindex(1));
        call_func(func, L, indices);
        return 0;
    }
};

Adding support for lambdas

Unfortunately, one thing I just noticed is that I don't have support for lambdas yet with the signatures specializations I'm using currently, so the binding of the sub function below will fail:

#include <sandbox_precomp.h>

using namespace nv;

static auto meaning() -> int {
    logDEBUG("Returning meaning of life.");
    return 42;
};

static auto square2(int a) -> int { return a * a; }

static auto add(int a, int b) -> int { return a + b; }

static void load_bindings(LuaState& lua) {
    logDEBUG("Should register my add function in the lua state here.");
    auto space = lua.get_or_create_table("tests");

    space["meaning"] = meaning;
    space["square"] = square2;
    space["add"] = add;
    space["sub"] = [](int a, int b) { return a - b; };
}

LUNA_ADD_BINDING(load_bindings)

⇒ Let's clarify why and fix that.

Okay, so for non-capturing lambda, we can simply cast the lambda into the corresponding function pointer, and this will compile/work:

static void load_bindings(LuaState& lua) {
    logDEBUG("Should register my add function in the lua state here.");
    auto space = lua.get_or_create_table("tests");

    space["meaning"] = meaning;
    space["square"] = square2;
    space["add"] = add;
    auto func = [](int a, int b) { return a - b; };
    space["sub"] = (int (*)(int, int))func;
}

But let's try to handle that automatically instead. Okay: again, this was a little tricky to sort out, but I eventually got it working with the following specializations on the LuaConnector template:

template <typename T> struct PtrType;

template <typename Ret, typename C, typename... Args>
struct PtrType<Ret (C::*)(Args...) const> {
    using func_t = Ret (*)(Args...);
};

template <typename T> struct LuaConnector {
    using op_t = decltype(&T::operator());
    using func_t = typename PtrType<op_t>::func_t;

    static void connect(LuaTable& table, const char* key, T func) {
        lua_State* L = table.state;
        lua_pushlightuserdata(L, (void*)(func_t)func);
        table.set_closure(key, LuaCaller<func_t>::call, 1);
    }
};

template <typename Ret, typename... Args>
struct LuaConnector<Ret (*)(Args...)> {
    using func_t = Ret (*)(Args...);
    static void connect(LuaTable& table, const char* key, func_t func) {
        lua_State* L = table.state;
        lua_pushlightuserdata(L, (void*)(func_t*)func);
        table.set_closure(key, LuaCaller<func_t>::call, 1);
    }
};

In the template above, the defaut implementation will apply to the lambda objects, while the specilization will be used for static functions.

Providing the SimpleCmdBuffersProvider in the sandbox module

As a final test here let's move the SimpleCmdBuffersProvider into the sandbox:
Hmmm, I need to load the lua bindings traits to be able to read/write classes in Lua…
But currently the traits do not come with classes declarations ⇒ need to fix that as I don't want to include all the headers each time I use the tratits for a single class.

Actually, it's pretty tricky to handle declaring all the classes correctly at that level, since some classes are actual simple typedefs…

In the end, here is the part of the code where I will check if I can write a forward class declaration or if I need to include the corresponding header file:

        if not cl:isInnerClass() and not cl:isUnion() and not cname:find("<") and not nsIsClass and not isTemplate then
            self:writeSubLine("class ${1};", cname)
        else
            -- Get the header for this "class":
            local tgtcl = cl:isInnerClass() and cl:getParent() or cl
            local hfile = self:getHeaderForClass(tgtcl)
            if hfile ~= nil then
                logDEBUG("Traits: adding required header: ", hfile)
                required_headers:insert(hfile)
            else
                logDEBUG("Traits: No header file found for definition of: ", cfname)
            end
        end

Note: In the code above, I also had some trouble with the handling of “unions” in the bindings, but now this seems to work as expected.

But still some errors on the manual bindings generation process, since I have for instance:

D:\Projects\NervLand\sources\nvCore\src\lua\luna.h:257:56: error: no type named 'root_t' in 'luna::LunaTraits<const nv::ByteArray>'
    using provider_t = LunaProvider<typename traits_t::root_t>;
                                    ~~~~~~~~~~~~~~~~~~~^~~~~~
D:\Projects\NervLand\sources\nvCore\src\lua\luna.h:722:17: note: in instantiation of template class 'luna::Luna<const nv::ByteArray>' requested here
        return *Luna<T>::get(L, idx, false);
                ^
D:\Projects\NervLand\sources\nvCore\src\lua\LuaTable.h:158:72: note: in instantiation of member function 'luna::LunaGetter<const nv::ByteArray &>::getValue' requested here
        return luna::LunaGetter<typename type_at<idx, Args...>::type>::getValue()...

⇒ I need to handling the const qualifier here, which is simple enough with the following specilizations:

template <typename T> struct LunaGetter;

template <typename T> struct LunaGetter<T*> {
    static auto getValue(lua_State* L, int idx) -> T* {
        return Luna<T>::get(L, idx, true);
    }
};

template <typename T> struct LunaGetter<const T*> {
    static auto getValue(lua_State* L, int idx) -> T* {
        return Luna<T>::get(L, idx, true);
    }
};

template <typename T> struct LunaGetter<T&> {
    static auto getValue(lua_State* L, int idx) -> T& {
        return *Luna<T>::get(L, idx, false);
    }
};

template <typename T> struct LunaGetter<const T&> {
    static auto getValue(lua_State* L, int idx) -> T& {
        return *Luna<T>::get(L, idx, false);
    }
};

OK, but now I have another issue because I'm not providing the “behavior” for the Union bindings, let's fix that… Fixed!

Now that the compilation is finally working fine, it's time to try and use our provider from the sandbox. And cool! Using this code below works: cccll local prov = nvk.create_simple_cmd_buf_provider(self.renderer, renderpass, vbuf, pipelineLayout, cfg, pipelineCache,

self.cbufs, self.pushArr)

- local prov = nvk.SimpleCmdBuffersProvider(self.renderer, renderpass, vbuf, pipelineLayout, cfg, pipelineCache,
- self.cbufs, self.pushArr)

self.renderer:set_cmd_buffer_provider(prov)

At least until I clsoe the application 🙃, at which time I have 1 memory leak (I wonder what this could be ?)

2022-12-02 22:48:47.492428 [DEBUG] Closing DynamicLibrary modules/luaSandbox.dll
2022-12-02 22:48:47.492461 [DEBUG] Unregistering bindings for Sandbox
2022-12-02 22:48:47.493178 [DEBUG] Destroying LuaManager...
2022-12-02 22:48:47.493203 [DEBUG] Destroying memory manager.
Deleted LogManager object.
Looking for memory leaks...
[FATAL] Found 1 memory leaks!!!
[FATAL] Leaking object 0000019E7A93B450 with count: 4294967295
Exiting.

⇒ I think that could be because I'm not defining the LunaProvider for RefObjects in the sandbox module, so let's add that… OK: this was it, and now it's working just fine 👍!

I have now placed all my custom luna templates in in a single luna_shared_templates.h file to avoid repeating that code in each binding.

Now I'm back to the same rotating triangle as at the end of this previous post, except that the provider is manually bound from the Sandbox module install of directly from the nvVulkan module: amazing 😎!

Conclusion

Being able to manually add content into the lua state as described above is cool and will probably be useful at ome point, but still, thinking about the example provided here, I feel it would be more appropriate to just use the automated bindings generation exposing only a simple function to create our SimpleCmdBuffersProvider object.
Anyway, we'll see later which path we actually use 😆.

Table of Contents