Differences
This shows you the differences between two versions of the page.
Last revision | |||
— | blog:2017:1130_cef_direct_copy_to_d3d [2020/07/10 12:11] – external edit 127.0.0.1 | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ====== CEF Direct offscreen rendering to Direct3D Surfaces ====== | ||
+ | |||
+ | {{tag> | ||
+ | |||
+ | As I'm currently working on a project that make intensive use of CEF to render imagery overlays I finally reached the point where it could become very interesting to be able to render offscreen content with CEF, but still avoid copying the generated texture from GPU to CPU and then back to GPU to inject it in another 3D application. | ||
+ | |||
+ | Until very recently, I was just waiting for the [[https:// | ||
+ | |||
+ | ====== ====== | ||
+ | |||
+ | ===== The patch design ===== | ||
+ | |||
+ | The whole idea is based on the fact that CEF used the ANGLE library on Windows to convert the GLES commands into DirectX commands, and the ANGLE library supports [[https:// | ||
+ | |||
+ | From there, I had to spend a long time reading the sources and documentation on CEF to try to understand how it actually works and where exactly I could inject the changes I needed. The most complex aspect in this project is due to the fact that you basically have 2 separated " | ||
+ | * On one side you have "your client" | ||
+ | * On the other side, you have the CEF "GPU service" | ||
+ | |||
+ | Those two parts can communicate with **commands** sent from the client to the service via shared memory (with optional results received back), and some complex command buffer system. | ||
+ | |||
+ | So, the start point was in my software process, where I can create DirectX surfaces with **shared handle**, then with ANGLE, I can normally render on this shared handle from a GL ES context, but to do so, I needed to find a way to pass the shared handle value to the GPU service side, and thus, needed to create a new command in the CEF internals. | ||
+ | |||
+ | * I started from the current **CefRenderHandler** class which is used in the current offscreen rendering pipeline to retrieve the generated image from some CPU memory location (which its **OnPaint(...)** method). | ||
+ | * From there I figured out an instance of this class was used in the **CefRenderWidgetHostViewOSR** class, and more precisely in the helper class **CefCopyFrameGenerator** | ||
+ | |||
+ | * So I extended the CefRenderHandler with additional methods that can be overriden to specify that we want to use a shared handle: <sxh cpp> | ||
+ | // Check if shared handle should be used with this render handler. | ||
+ | /// | ||
+ | / | ||
+ | virtual bool UseSharedHandle() { return false; } | ||
+ | |||
+ | /// | ||
+ | // Return the shared handle for this renderhandler. If no shared handle | ||
+ | // is available then null is returned. | ||
+ | /// | ||
+ | / | ||
+ | virtual void* GetSharedHandle() { return nullptr; } | ||
+ | </ | ||
+ | |||
+ | < | ||
+ | |||
+ | * Then I ended up updating the function **CefCopyFrameGenerator:: | ||
+ | const gfx:: | ||
+ | std:: | ||
+ | DCHECK(result-> | ||
+ | base:: | ||
+ | base:: | ||
+ | weak_ptr_factory_.GetWeakPtr(), | ||
+ | |||
+ | const gfx:: | ||
+ | DEBUG_MSG2(" | ||
+ | | ||
+ | // Shared code: we need the gl_helper in both rendering paths: | ||
+ | content:: | ||
+ | content:: | ||
+ | viz:: | ||
+ | if (!gl_helper) { | ||
+ | DEBUG_MSG(" | ||
+ | return; | ||
+ | } | ||
+ | |||
+ | // Select the appropriate rendering path depending on the shared handle being provided or not: | ||
+ | CefRefPtr< | ||
+ | | ||
+ | if(handler.get() && handler-> | ||
+ | // Direct rendering on shared handle path: | ||
+ | |||
+ | // Prepare the texture mail box: | ||
+ | viz:: | ||
+ | std:: | ||
+ | result-> | ||
+ | DCHECK(texture_mailbox.IsTexture()); | ||
+ | if (!texture_mailbox.IsTexture()) | ||
+ | return; | ||
+ | |||
+ | ignore_result(scoped_callback_runner.Release()); | ||
+ | |||
+ | void* handle = handler-> | ||
+ | if(handle == nullptr) { | ||
+ | DEBUG_MSG(" | ||
+ | OnCopyFrameCaptureCompletion(false); | ||
+ | return; | ||
+ | } | ||
+ | |||
+ | // Here we call another function to copy the texture to the shared handle: | ||
+ | DEBUG_MSG2(" | ||
+ | |||
+ | gl_helper-> | ||
+ | OnCopyFrameCaptureCompletion(false); | ||
+ | } | ||
+ | else{ | ||
+ | // Regular on CPU rendering path: | ||
+ | |||
+ | // Allocate the Skia bitmap: | ||
+ | SkIRect bitmap_size; | ||
+ | if (bitmap_) | ||
+ | bitmap_-> | ||
+ | |||
+ | if (!bitmap_ || bitmap_size.width() != result_size.width() || | ||
+ | bitmap_size.height() != result_size.height()) { | ||
+ | // Create a new bitmap if the size has changed. | ||
+ | bitmap_.reset(new SkBitmap); | ||
+ | bitmap_-> | ||
+ | if (bitmap_-> | ||
+ | return; | ||
+ | } | ||
+ | | ||
+ | // Retrieve the pixel buffer: | ||
+ | uint8_t* pixels = static_cast< | ||
+ | |||
+ | // Prepare the texture mail box: | ||
+ | viz:: | ||
+ | std:: | ||
+ | result-> | ||
+ | DCHECK(texture_mailbox.IsTexture()); | ||
+ | if (!texture_mailbox.IsTexture()) | ||
+ | return; | ||
+ | |||
+ | ignore_result(scoped_callback_runner.Release()); | ||
+ | |||
+ | gl_helper-> | ||
+ | texture_mailbox.mailbox(), | ||
+ | gfx:: | ||
+ | base::Bind( | ||
+ | & | ||
+ | weak_ptr_factory_.GetWeakPtr(), | ||
+ | damage_rect, | ||
+ | viz:: | ||
+ | } | ||
+ | } | ||
+ | </ | ||
+ | |||
+ | * As shown on the code snippet above, when using the shared handle processing pipeline, we then rely on a new **command** created specifically to send the shared handle to the GPU service and to request a copy of a given GL texture_id onto that shared handle surface: **NervCopyMailboxToSharedHandle**. Again, building a new command in CEF/ | ||
+ | |||
+ | * For this new function, the main implementation is done in **GLES2DecoderImpl:: | ||
+ | |||
+ | * So, from there, the idea is to perform the following: | ||
+ | - Create a EGL context sharing content with the GLES2Decoder default context: so that we still have access to the texture data in our new context, but still, we avoid messing too much with the current state of rendering in the default context. (Actually, i'm not absolutely sure this is really needed: maybe we could use the default context directly to do the rendering ?) | ||
+ | |||
+ | - Setup a pbuffer with the shared handle (as described on the [[https:// | ||
+ | |||
+ | - Setup the resources required to render a screen aligned quad: we need **a program** that will simply copy the input texture on the render surface. A **vertex buffer** to draw 2 triangles, and an **index buffer**. | ||
+ | |||
+ | - Then each time our new command is executed, we carefully replace the current context with our own context, then we copy the provided texture onto our pbuffer, and then restore the default current context, just how it was: <sxh cpp> | ||
+ | EGLSurface drawSurface = eglGetCurrentSurface(EGL_DRAW); | ||
+ | EGLSurface readSurface = eglGetCurrentSurface(EGL_READ); | ||
+ | |||
+ | // Assign our surface as current draw/read target: | ||
+ | EGLSurface ourSurface = gSurfaces[handle]; | ||
+ | |||
+ | if(eglMakeCurrent(egl_display, | ||
+ | NV_LOG(" | ||
+ | // Try to restore the previous surfaces: | ||
+ | eglMakeCurrent(display, | ||
+ | return error:: | ||
+ | } | ||
+ | | ||
+ | // init the context if necessary: | ||
+ | if(program == 0) { | ||
+ | NV_LOG2(" | ||
+ | nvInitContext(); | ||
+ | } | ||
+ | |||
+ | // Convert the client texture_id to our service texture_id: | ||
+ | uint32_t service_tex_id=0; | ||
+ | if(GetServiceTextureId(texture_id, | ||
+ | NV_LOG2(" | ||
+ | nvDraw(service_tex_id); | ||
+ | } | ||
+ | else { | ||
+ | NV_LOG(" | ||
+ | } | ||
+ | |||
+ | // float v1 = 1.0f * (rand()/ | ||
+ | // float v2 = 1.0f * (rand()/ | ||
+ | // NV_LOG2(" | ||
+ | |||
+ | // First we try to just display some fixed color (red): | ||
+ | // glViewport(0, | ||
+ | // glClearColor(v1, | ||
+ | // glClear(GL_COLOR_BUFFER_BIT); | ||
+ | |||
+ | // Now that we are done, we restore the previous current context/ | ||
+ | eglMakeCurrent(display, | ||
+ | </ | ||
+ | |||
+ | * Initialization is done in the **nvInitContext()** function (only once). Then the custom **nvDraw()** function will use all the resources we allocated, to copy the input texture on the current render surface (ie. our pbuffer / ie. shared DirectX surface) as shown below: <sxh cpp>void nvDraw(uint32_t texture) | ||
+ | { | ||
+ | // NV_CHECK(eglReleaseTexImage(winDisplay, | ||
+ | // NV_CHECK(eglMakeCurrent(winDisplay, | ||
+ | |||
+ | // float v1 = 1.0f * (rand()/ | ||
+ | // float v2 = 1.0f * (rand()/ | ||
+ | // NV_LOG2(" | ||
+ | |||
+ | // // elapsed += 0.01; | ||
+ | // // glClearColor((sin(elapsed) + 1.0f) * 0.5f, 0.0f, 0.0f, 1.0f); | ||
+ | // glClearColor(v1, | ||
+ | // glClear(GL_COLOR_BUFFER_BIT); | ||
+ | |||
+ | // NV_CHECK(eglBindTexImage(winDisplay, | ||
+ | // NV_CHECK(eglMakeCurrent(winDisplay, | ||
+ | |||
+ | // Set the viewport | ||
+ | NV_CHECK_GL(" | ||
+ | glViewport(0, | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | float v1 = 1.0f * (rand()/ | ||
+ | float v2 = 1.0f * (rand()/ | ||
+ | |||
+ | // Ensure we have a proper color mask set: | ||
+ | glColorMask (GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE); | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | glClearColor (v1, v2, 0.0f, 1.0f); | ||
+ | // glClearColor(0.0f, | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | // Clear the color buffer | ||
+ | // glClearColor(1.0f, | ||
+ | glClear(GL_COLOR_BUFFER_BIT); | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | GLint curBuffer = 0; | ||
+ | glGetIntegerv(GL_ARRAY_BUFFER_BINDING, | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | // if(curBuffer == 0) { | ||
+ | // | ||
+ | // } | ||
+ | | ||
+ | GLint curIdxBuffer = 0; | ||
+ | glGetIntegerv(GL_ELEMENT_ARRAY_BUFFER_BINDING, | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | // if(curIdxBuffer == 0) { | ||
+ | // | ||
+ | // } | ||
+ | | ||
+ | // We bind the vertex buffer: | ||
+ | glBindBuffer(GL_ARRAY_BUFFER, | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | // We bind the index buffer: | ||
+ | glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | // Use the program object | ||
+ | glUseProgram(program); | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | // | ||
+ | glEnableVertexAttribArray(0); | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | glEnableVertexAttribArray(1); | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | // Load the vertex position | ||
+ | // glVertexAttribPointer(0, | ||
+ | glVertexAttribPointer(0, | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | // Load the texture coordinate | ||
+ | // glVertexAttribPointer(1, | ||
+ | glVertexAttribPointer(1, | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | // Bind the texture | ||
+ | glActiveTexture(GL_TEXTURE0); | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | // glBindTexture(GL_TEXTURE_2D, | ||
+ | glBindTexture(GL_TEXTURE_2D, | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | // Set the sampler texture unit to 0 | ||
+ | // logDEBUG(" | ||
+ | glUniform1i(texLoc, | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | glDrawElements(GL_TRIANGLES, | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | // Then we restore the bound buffer: | ||
+ | glBindBuffer(GL_ARRAY_BUFFER, | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | // Also unbind our texture here: | ||
+ | glBindTexture(GL_TEXTURE_2D, | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | glFlush(); | ||
+ | NV_CHECK_GL(" | ||
+ | |||
+ | // eglSwapBuffers(winDisplay, | ||
+ | // ValidateRect(hwnd, | ||
+ | }</ | ||
+ | |||
+ | * As a side note, alos note that when done with this copy operation, we delete the texture_id (or at least the client reference on the real texture) since this seems to be what was done anyway with the regular CPU rendering pipeline: <sxh cpp> | ||
+ | NV_LOG2(" | ||
+ | DeleteTexturesHelper(1, | ||
+ | NV_LOG2(" | ||
+ | |||
+ | < | ||
+ | |||
+ | < | ||
+ | |||
+ | ===== The build process ===== | ||
+ | |||
+ | During my initial tests, I was modifying the CEF/ | ||
+ | |||
+ | So I took a different path: instead I built a set of script functions and a separated collection of CEF files that I needed to modify: then I have script functions used to: | ||
+ | - checkout a given branch of CEF (branch 3163 by default): <sxh bash># Update of the CEF sources to a given branch | ||
+ | nv_cef_update() { | ||
+ | |||
+ | local branch=" | ||
+ | nv_cef_init_exports | ||
+ | |||
+ | local bdir=`cygpath -w " | ||
+ | # echo "using build dir: $bdir" | ||
+ | cd " | ||
+ | |||
+ | nv_cef_call_python ../ | ||
+ | |||
+ | cd - > /dev/null | ||
+ | }</ | ||
+ | |||
+ | - Override all the base files from CEF with the updated version I stored separetely and then generate the project files, this script will also take care of re-generating all the auto-generated files based on the modifications we just injected in the code: <sxh bash># create the cef projects: | ||
+ | nv_cef_create_project() { | ||
+ | nv_cef_init_exports | ||
+ | |||
+ | # Copy all the updated files: | ||
+ | cp -Rf " | ||
+ | |||
+ | # Once we are done copying the files ensure we call the translator tool: | ||
+ | cd " | ||
+ | nv_cef_call_python translator.py --root-dir .. | ||
+ | cd - > /dev/null | ||
+ | |||
+ | # Update the command buffer functions: | ||
+ | cd " | ||
+ | nv_cef_call_python " | ||
+ | cd - > /dev/null | ||
+ | |||
+ | # Create the project files: | ||
+ | cd " | ||
+ | nv_cef_call_python tools/ | ||
+ | cd - > /dev/null | ||
+ | }</ | ||
+ | |||
+ | - Then another script to perform the actual build: <sxh bash># build the cef library: | ||
+ | nv_cef_build() { | ||
+ | nv_cef_init_exports | ||
+ | | ||
+ | local btype=${1: | ||
+ | cd " | ||
+ | nv_cef_call_ninja -C " | ||
+ | |||
+ | echo "Done building CEF" | ||
+ | cd - > /dev/null | ||
+ | }</ | ||
+ | |||
+ | - And finally another script to generate the distrib folder: <sxh bash># Make CEF distrib: | ||
+ | nv_cef_make_distrib() { | ||
+ | # cf. https:// | ||
+ | local vsdir=`nv_get_visualstudio_dir` | ||
+ | vsdir=`nv_to_win_path $vsdir` | ||
+ | export CEF_VCVARS=" | ||
+ | |||
+ | cd " | ||
+ | |||
+ | nv_cef_call_python make_distrib.py --output-dir ../ | ||
+ | |||
+ | echo "Done packaging CEF" | ||
+ | cd - > /dev/null | ||
+ | }</ | ||
+ | |||
+ | < | ||
+ | |||
+ | Do be able to use those scripts, one should only need to provide 2 folder locations: | ||
+ | - The location where CEF should be built (ie. containing the official CEF sources) | ||
+ | - The location where the patched files are (which will be the folder containing the script file itself if you use my package below) | ||
+ | |||
+ | Those location can be configured at the beginning of the script file I'm providing below: <sxh bash># CEF Build dir: this is the root folder where CEF is built: | ||
+ | _cef_build_dir="/ | ||
+ | |||
+ | # CEF patch dir: this is the folder containing all the updated files required to build | ||
+ | # our patched version of CEF (with support for the Direct rendering to Direct3D) | ||
+ | _cef_patch_dir=" | ||
+ | </ | ||
+ | |||
+ | ===== Debug output systems ===== | ||
+ | |||
+ | Currently in use in this CEF patch, you will find 2 debug output logging mechanism I created specifically for my investigations (I don't know how to use the CEF logging system properly, and in fact I didn't even want to learn that part ;-) ). | ||
+ | |||
+ | * On one side, in the " | ||
+ | |||
+ | struct CefLogHandler { | ||
+ | | ||
+ | CefLogHandler() {}; | ||
+ | virtual ~CefLogHandler() {}; | ||
+ | |||
+ | // Log a message: | ||
+ | virtual void log(const std:: | ||
+ | }; | ||
+ | |||
+ | // Function used to set our log handler: | ||
+ | void setLogHandler(CefLogHandler* handler); | ||
+ | |||
+ | // Function used to actually log a message: | ||
+ | void handleLogMessage(unsigned int level, const std:: | ||
+ | |||
+ | }; | ||
+ | |||
+ | // Define a deBUG MESSAGE macro: | ||
+ | #define DEBUG_MSG(msg) { \ | ||
+ | std:: | ||
+ | os.precision(9); | ||
+ | os << std::fixed << msg; \ | ||
+ | cef:: | ||
+ | } | ||
+ | |||
+ | #define DEBUG_MSG2(msg) { \ | ||
+ | std:: | ||
+ | os.precision(9); | ||
+ | os << std::fixed << msg; \ | ||
+ | cef:: | ||
+ | }</ | ||
+ | |||
+ | This means that in my software I can then assign a **CefLogHandler** instance with an overload **log()** method to retrieve log messages originating from CEF directly into my software logging system, which make it all more consistent from my point of view. | ||
+ | |||
+ | * On the other side, the approach was not possible in the "GPU service process" | ||
+ | |||
+ | /* | ||
+ | Helper logging function used to log debug outputs to a file. | ||
+ | */ | ||
+ | void nvLOG(unsigned int level, const std:: | ||
+ | |||
+ | }; | ||
+ | |||
+ | // Define a DEBUG MESSAGE macro: | ||
+ | #define NV_LOG(msg) { \ | ||
+ | std:: | ||
+ | os.precision(9); | ||
+ | os << std::fixed << msg; \ | ||
+ | nv:: | ||
+ | } | ||
+ | |||
+ | #define NV_LOG2(msg) { \ | ||
+ | std:: | ||
+ | os.precision(9); | ||
+ | os << std::fixed << msg; \ | ||
+ | nv:: | ||
+ | } | ||
+ | |||
+ | #define NV_LOG3(msg) { \ | ||
+ | std:: | ||
+ | os.precision(9); | ||
+ | os << std::fixed << msg; \ | ||
+ | nv:: | ||
+ | }</ | ||
+ | |||
+ | Both of those logging system have a verbosity controlled by the **environement variable** " | ||
+ | - NV_CEF_LOG_LEVEL=0 => No log output at all | ||
+ | - NV_CEF_LOG_LEVEL=1 => Minimal log outputs (mainly errors if any) | ||
+ | - NV_CEF_LOG_LEVEL=2 => Maximum output level (errors and infos) | ||
+ | |||
+ | ===== Patch files ===== | ||
+ | |||
+ | So finally, here is a link to a github repo I just created to hold those patch files: this repo contains the **cef.sh** script with the functions described above, and all the modified CEF files in the src/ folder but no real file history (as all my source files are stored on a different [private] repo I have) | ||
+ | |||
+ | **Github repo**: https:// | ||
+ | |||
+ | => If you have any question or problem you can still post a comment here or at an issue on the github repo, or contact me on linkedin/ | ||
+ | |||
+ | |||
+ | ===== Additional notes ===== | ||
+ | |||
+ | During the work on this project I also took a lot of notes on what I was doing, what was working, what was not, etc: it's a bit messy, but it still contains valuable info if you need to know more. So you can **access those notes** on this page: [[public: | ||
+ | |||
+ | |||
+ | ===== 13/12/2017 - Update: Texture resource usage investigations ===== | ||
+ | |||
+ | I noticed during the initial usage tests for this patch that there seem to be a serious issue on the GL texture release process (textures are simply not released anymore if the requests for new frames are coming too quickly). I'm currently investigating this issue. | ||
+ | |||
+ | ~~DISCUSSION~~ | ||