# Direct GPU buffer copy in CEF

## Building CEF from sources

Build CEF directly from sources was less complicated than I anticipated: I meanly followed the instructions from the Master Build Quick Start guide. Also taking some notes from the official Branches and Building page.

Initial steps:

• Create the base folders:
d:\Projects\CEFBuild\automate
d:\Projects\CEFBuild\chromium_git
• Then extract to create the folder:
d:\Projects\CEFBuild\depot_tools
• From “d:\Projects\CEFBuild\depot_tools”, execute the script:
update_depot_tools.bat

In the end, I simply created a few helper batch files, placed in the chromium_git folder (note that my build root for CEF is: D:\Projects\CEFBuild):

1. First the update.bat file, used to retrieve the code updates:
set CEF_USE_GN=1
set GN_ARGUMENTS=--ide=vs2015 --sln=cef --filters=//cef/*
2. Then the create_cef_project.bat file, used to create the CEF Visual studio/Ninja project files:
cd D:\Projects\CEFBuild\chromium_git\chromium\src\cef

set CEF_USE_GN=1
set GN_ARGUMENTS=--ide=vs2015 --sln=cef --filters=//cef/*
call cef_create_projects.bat
3. Then the build_cef_release.bat to build the release version of CEF:
cd D:\Projects\CEFBuild\chromium_git\chromium\src
ninja -C out\Release_GN_x64 cef
4. And finally the make_distrib.bat file which will generate the binary_distrib folder inside D:\Projects\CEFBuild\chromium_git\chromium\src\cef:
REM cf. https://bitbucket.org/chromiumembedded/cef/wiki/BranchesAndBuilding.md

set CEF_VCVARS="D:\Apps\VisualStudio2015_CE\VC\bin\amd64\vcvars64.bat"

cd D:\Projects\CEFBuild\chromium_git\chromium\src\cef\tools
./make_distrib.bat --ninja-build --x64-build --allow-partial

One thing to note though is: by default on Windows the build system will attempt to build both x86 and x64 binaries. Yet, some dependencies where not available on my computer to build the x86 version of the library, so I had to manually disable that (before running create_cef_project.bat AFAIR) by tweaking the line of D:\Projects\CEFBuild\chromium_git\chromium\src\cef\tools\gn_args.py to only request the build of the x64 version:

if platform == 'linux':
use_sysroot = GetArgValue(args, 'use_sysroot')
if use_sysroot:
# Only generate configurations for sysroots that have been installed.
for cpu in ('x86', 'x64', 'arm'):
if LinuxSysrootExists(cpu):
supported_cpus.append(cpu)
else:
msg('Not generating %s configuration due to missing sysroot directory'
% cpu)
else:
supported_cpus = ['x64']
elif platform == 'windows':
# supported_cpus = ['x86', 'x64']
supported_cpus = ['x64']
elif platform == 'macosx':
supported_cpus = ['x64']
else:
raise Exception('Unsupported platform')

But now I realize there might be a more appropriate way to handle this by specifying the target_cpu command line argument somewhere maybe ? (to be investigated).

Note: Currently experimenting with CEF branch 3163

## Trying to automate header copy in make_distrib

The logic used the copy the headers during the make_distrib step is found in make_distrib.py around line 525:

# transfer common include files
transfer_gypi_files(cef_dir, cef_paths2['includes_common'], \
'include/', include_dir, options.quiet)
• So it sounds easy to tweak the list of headers that should be distributed: we need to edit the 'includes_common' member from cef_paths2, which is read from file ${CEF_SRC_DIR}/cef_paths2.gypi • OK when adding our files into the previous cef_paths2.gypi config files they are distributed as expected. ## Investigating current Off Screen Rendering (OSR) pipeline • Starts with CefRenderHandler::OnPaint() method • Then it seems this handler is mostly used in CefRenderWidgetHostViewOSR class. • Actually this class has an OnPaint() method which will access the handler. • CefRenderWidgetHostViewOSR::OnPaint(…) is called in CefCopyFrameGenerator::OnCopyFrameCaptureSuccess(…) • Previous method called in CopyFromCompositingSurfaceFinished and PrepareBitmapCopyOutputResult • We eventually call: cc::CopyOutputRequest::CreateRequest(…) or gl_helper→CropScaleReadbackAndCleanMailbox(…) • ⇒ We should check in CopyFromCompositingSurfaceHasResult(…) if we have either a texture result or a bitmap result (suspecting we have a texture when using GPU rendering and a bitmap when doing software rendering ?). • ⇒ So we should add a CEF logging mechanism to check this and report the information in our target app. • Also need to check content::ImageTransportFactory and viz::GLHelper classes ## Implement CEF logging capability We need to be able to retrieve some debug infos from the cef library in an integrated way: so we need to be able to assign a log handler inside the library, and then use this handle to output log messages if valid. Can we simply add a new class/method inside the source base ? Let's try that: building the LogHandler class and trying to use it in CefRenderWidgetHostViewOSR: • Trying to call build_cef_release.bat directly: link failed • Detected the changes in render_widget_host_view_osr.cc • But didn't detect the new source files log_handler.{h, cc} ? • (Linking is taking ages…): need to stop the process to get the outputs (actually needs to ensure the dos console has the focus!), and as expected our new function is not found: [3/7] LINK(DLL) libcef.dll libcef.dll.lib libcef.dll.pdb FAILED: libcef.dll libcef.dll.lib libcef.dll.pdb D:/Projects/CEFBuild/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../build/toolchain/win/tool_wrapper.py link-wrapper environment.x64 False link.exe /nologo /IMPLIB:./libcef.dll.lib /DLL /OUT:./libcef.dll /PDB:./libcef.dll.pdb @./libcef.dll.rsp libcef_static.lib(render_widget_host_view_osr.obj) : error LNK2019: unresolved external symbol "void __cdecl cef::handleLogMessage(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &)" (?handleLogMessage@cef@@YAXAEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z) referenced in function "private: void __cdecl CefCopyFrameGenerator::CopyFromCompositingSurfaceHasResult(class gfx::Rect const &,class std::unique_ptr<class cc::CopyOutputResult,struct std::default_delete<class cc::CopyOutputResult> >)" (?CopyFromCompositingSurfaceHasResult@CefCopyFrameGenerator@@AEAAXAEBVRect@gfx@@V?$unique_ptr@VCopyOutputResult@cc@@U?$default_delete@VCopyOutputResult@cc@@@std@@@std@@@Z)
./libcef.dll : fatal error LNK1120: 1 unresolved externals
• Now trying to rebuild the project files first with create_cef_project.bat: link failed
• (Linking still taking ages…) and then we get the same error as above.
• So we probably need to declare the new files somewhere then… yes: that will be in the src/cef/BUILD.gn file.
• Adding our new files in BUILD.gn: OK
• Called create_cef_project.bat
• Then called build_cef_release.bat
• OK this time the build is successful !

Now that we have a LogHandler support we should assign it on start of our software. How would we do that ?

• For now we added our functions/class in the libcef module, which is built as a static library.
• But then this static library is only linked into the libcef_dll shared library (?)
• CefInitialize(…) is implemented libcef\browser\context.cc: so at the same level as our log handler functions.
• So first let's move our LogHandler class header into the root include folder: (also removed log_handler.h from BUILD.gn) OK
• Now rebuilding: Failing: cannot find our cef_log_handler.h file: so the header location is incorrect ?
• Okay: so we keep the “log_handler.h” file next to log_handler.cc and we will manually provide a copy as cef_log_handler.h in our final include folder. Can now rebuild successfully.
• Setting up the new libcef dependency: manually installing our libcef/browser/log_handler.h header as “cef_log_handler.h”
• Now trying to rebuild our project: failed of course, we cannot find the function:
CEFManager.cpp.obj : error LNK2019: unresolved external symbol "void __cdecl cef::setLogHandler(class cef::CefLogHandler *)" (?setLogHandler@cef@@YAXPEAVCefLogHandler@1@@Z) referenced in function "public: void __cdecl cef::CEFManager::initialize(class cef::CEFManager::Traits &)" (?initialize@CEFManager@cef@@QEAAXAEAVTraits@12@@Z)
scCEF.dll : fatal error LNK1120: 1 unresolved externals

So we need to figure out what we are missing here and how to make our function available from the libcef library to the outside world.

• CefInitialize is marked as GLOBAL in libcef_dll_wrapper.cc and it will call cef_initialize() which is exported from libcef, and in turn calls CefInitialize from the browser/context.cc class.
• ⇒ OK, so we need an exported method too in libcef_dll.cc, let's try that:
1. Need to add our exported function declaration in include/capi/cef_app_capi.h:
///
// This function should be called on the main application thread to assign
// a log handler to the CEF module (used for investigations)
///
CEF_EXPORT void cef_set_log_handler(struct cef::CefLogHandler* handler);
2. So we need to use a struct in C now instead of a class in C++. We update the cef_log_handler.h header accordingly.
3. Then we implement and export our function cef_set_log_handler inside libcef_dll.cc:
CEF_EXPORT void cef_set_log_handler(cef::CefLogHandler* handler) {
cef::setLogHandler(handler);
}
4. And we also get rid of the internal header browser/log_handler.h ⇒ we use “include/cef_log_handler.h” directly instead in both log_handler.cc and render_widget_host_view_osr.cc
• Now rebuilding: OK
• Deploying in target project: noticed that our cef_app_capi.h doesn't contain our new method ⇒ copying it manually!
• Bingo ! we can now call our new cef_set_log_handler() function as expected in our target project.
• Now let's run the project and see if we get the expected outputs since we added in render_widget_host_view_osr.cc:
void CopyFromCompositingSurfaceHasResult(
const gfx::Rect& damage_rect,
std::unique_ptr<cc::CopyOutputResult> result) {
if (result->IsEmpty() || result->size().IsEmpty() ||
!view_->render_widget_host()) {
OnCopyFrameCaptureFailure(damage_rect);
return;
}

if (result->HasTexture()) {
DEBUG_MSG("NervCEF: CompositingSurface has texture!");
PrepareTextureCopyOutputResult(damage_rect, std::move(result));
return;
}

DCHECK(result->HasBitmap());
DEBUG_MSG("NervCEF: CompositingSurface has bitmap!");
PrepareBitmapCopyOutputResult(damage_rect, std::move(result));
}

⇒ We expect to get one or the other of those 2 debug outputs (or both ?)… well, instead we got a crash but it doesn't seem to be related to what we are changing here:

2017-09-15T10:19:33.735898 [Debug]            Calling CefInitialize...
[0915/111933.735:FATAL:libcef_dll_wrapper.cc(209)] Check failed: false.
• The code at that location is:
CEF_GLOBAL bool CefInitialize(const CefMainArgs& args,
const CefSettings& settings,
CefRefPtr<CefApp> application,
void* windows_sandbox_info) {
const char* api_hash = cef_api_hash(0);
if (strcmp(api_hash, CEF_API_HASH_PLATFORM)) {
// The libcef API hash does not match the current header API hash.
NOTREACHED();
return false;
}

// More stuff here
}
• Adding execution flag on all dlls and exes files (but this doesn't help):
chmod +x *.dll
chmod +x *.exe
• Copy all CEF resources: doesn't help
• But what we can realize here is that the error is due to an API hash mismatch! ⇒ So we need to copy all the header files from CEF again when building a binary distrib! Crap… doesn't work that easily: we actually modify the library CAPI, and those hashes are generated from these…:
//The API hash is created by analyzing CEF header files for C API type
// definitions. The hash value will change when header files are modified
// in a way that may cause binary incompatibility with other builds. The
// universal hash value will change if any platform is affected whereas the
// platform hash values will change only if that particular platform is
// affected.

Okay, so we are probably not too far from success, BUT we should avoid making changes to the include/capi folder, thus, not include our new exported function into cef_app_capi.h ⇒ This doesn't matter much: let's just defined it in a separated file!

• Rebuilt CEF with our special cef_nv_log_handler.h and cef_nv_exports.h headers: OK
• Installing as dependency for target project: ⇒ need to manually copy those new files. OK

Great! We now have the expected outputs:

2017-09-17T13:37:32.881686 [Debug]            NervCEF: CompositingSurface has texture!
2017-09-17T13:37:33.056642 [Debug]            NervCEF: CompositingSurface has texture!
2017-09-17T13:37:33.377407 [Debug]            NervCEF: CompositingSurface has texture!

## Investigating image copy pipeline

Now that we can output some debug outputs if need, let's try to clarify further the offscreen image copy pipeline.

First, let's check what kind of messages we got in the previous run test:

• We only get the message: CompositingSurface has texture! and never reach the has bitmap! part.
• So it seems correct that a “texture” source is used when the GPU acceleration is active, and a simple bitmap is used instead when the CPU generation is active.

⇒ It all comes down to the method: viz::GLHelper::CropScaleReadbackAndCleanMailbox(…)

Found the following description for this method in components/viz/common/gl_helper.h:

// Copies the block of pixels specified with |src_subrect| from |src_mailbox|,
// scales it to |dst_size|, and writes it into |out|.
// |src_size| is the size of |src_mailbox|. The result is in |out_color_type|
// format and is potentially flipped vertically to make it a correct image
// representation.  |callback| is invoked with the copy result when the copy
// operation has completed.
// Note that the texture bound to src_mailbox will have the min/mag filter set
// to GL_LINEAR and wrap_s/t set to CLAMP_TO_EDGE in this call. src_mailbox is
// assumed to be GL_TEXTURE_2D.

So our parameter “src_mailbox” is assumed to be a GL_TEXTURE_2D ⇒ sounds good for us! Now checking the method implementation in components/viz/common/gl_helper.cc:

const gpu::Mailbox& src_mailbox,
const gpu::SyncToken& sync_token,
const gfx::Size& src_size,
const gfx::Rect& src_subrect,
const gfx::Size& dst_size,
unsigned char* out,
const SkColorType out_color_type,
const base::Callback<void(bool)>& callback,
GLHelper::ScalerQuality quality) {
GLuint mailbox_texture = ConsumeMailboxToTexture(src_mailbox, sync_token);
dst_size, out, out_color_type, callback,
quality);
gl_->DeleteTextures(1, &mailbox_texture);

Checking the implementation oc ComsumeMailboxToTexture:

GLuint GLHelper::ConsumeMailboxToTexture(const gpu::Mailbox& mailbox,
const gpu::SyncToken& sync_token) {
if (mailbox.IsZero())
return 0;
if (sync_token.HasData())
WaitSyncToken(sync_token);
GLuint texture =
gl_->CreateAndConsumeTextureCHROMIUM(GL_TEXTURE_2D, mailbox.name);
return texture;
}

And finally we find for CreateAndConsumeTextureCHROMIUM (inside gpu/command_buffer/client/gles2_implementation.cc):

GLuint GLES2Implementation::CreateAndConsumeTextureCHROMIUM(
GLenum target, const GLbyte* data) {
GPU_CLIENT_LOG("[" << GetLogPrefix() << "] glCreateAndConsumeTextureCHROMIUM("
<< static_cast<const void*>(data) << ")");
const Mailbox& mailbox = *reinterpret_cast<const Mailbox*>(data);
DCHECK(mailbox.Verify()) << "CreateAndConsumeTextureCHROMIUM was passed a "
"mailbox that was not generated by "
"GenMailboxCHROMIUM.";
GLuint client_id;
GetIdHandler(SharedIdNamespaces::kTextures)->MakeIds(this, 0, 1, &client_id);
helper_->CreateAndConsumeTextureINTERNALImmediate(target,
client_id, data);
if (share_group_->bind_generates_resource())
helper_->CommandBufferHelper::Flush();
CheckGLError();
return client_id;
}

So it sounds like everytime we request a copy we start with creating a GL texture, then we crop/scale/readback on CPU memory, and finally we destroy the texture.

We continue following the CreateAndConsumeTextureINTERNALImmediate method (found in gles2_cmd_helper_autogen.h):

void CreateAndConsumeTextureINTERNALImmediate(GLenum target,
GLuint texture,
const GLbyte* mailbox) {
const uint32_t size =
gles2::cmds::CreateAndConsumeTextureINTERNALImmediate::ComputeSize();
gles2::cmds::CreateAndConsumeTextureINTERNALImmediate* c =
GetImmediateCmdSpaceTotalSize<
gles2::cmds::CreateAndConsumeTextureINTERNALImmediate>(size);
if (c) {
c->Init(target, texture, mailbox);
}
}

And from there, we call this init method (found in gles2_cmd_format_autogen.h):

void Init(GLenum _target, GLuint _texture, const GLbyte* _mailbox) {
target = _target;
texture = _texture;
}

So: we create a GL texture Id (with the call to MakeIds(…)), then we send a command to copy the mailbox data into our texture. So, I think when CreateAndConsumeTextureCHROMIUM() returns we should have a valid GL texture ID, which contains the imagery data we want to retrieve ?

Then the next question is: How can we retrieve this data into our own texture object on the GPU from a different GL or DirectX context ?

## Inter-context texture copy

### From GL to GL

⇒ Those links do not seem very optimistic, but then I found the GL_NV_copy_image extension page, which states that:

This extension enables efficient image data transfer between image
objects (i.e. textures and renderbuffers) without the need to bind
the objects or otherwise configure the rendering pipeline.  The
WGL and GLX versions allow copying between images in different
contexts, even if those contexts are in different sharelists or
even on different physical devices.

This sounds more interesting already. But then we have the context specification issue in wglCopyImageSubDataNV and glXCopyImageSubDataNV: we need to be able to provide the source/dest contexts as HGLRC or GLXContext objects.

So let's assume for a moment that this function is available (ie. we are using a recent nvidia GPU), and that we are on windows (ie. we are considering the HGLRC contexts): how could we retrieve the source context ?

• But we cannot find any HGLRC reference in Chromium sources… maybe that's just not how it works for OpenGL ES ?
• https://www.khronos.org/registry/EGL/sdk/docs/man/html/eglCreateContext.xhtml:
eglCreateContext creates an EGL rendering context for the current rendering API (as set with eglBindAPI) and returns a handle to the context. The context can then be used to render into an EGL drawing surface. If eglCreateContext fails to create a rendering context, EGL_NO_CONTEXT is returned.

If share_context is not EGL_NO_CONTEXT, then all shareable data in the context (as defined by the client API specification for the current rendering API) are shared by context share_context, all other contexts share_context already shares with, and the newly created context. An arbitrary number of rendering contexts can share data. However, all rendering contexts that share data must themselves exist in the same address space. Two rendering contexts share an address space if both are owned by a single process.
• The previous seems to indicate that interprocess data sharing with EGL is not working ?

### From GL to DirectX

⇒ Are we using “ANGLE” in CEF ? According to the previous forum post YES. So the rational here should be applicable: we should be able to:

1. Create a shared DirectX texture (DX11 is mentioned, but maybe it could also workd with DirectX9Ex ?)
2. Then we pass the shared handle to the CEF engine and use it to create a new EGL surface.
3. We bind the surface to a GL texture,
4. And we copy from our source mailbox onto that texture ?

This might all just sound like a plan

## Investigating old Hardware offscreen render support implementation

So what about the first accelerated paint implementation mentioned in this forum post ?

⇒ Not understanding much from this patch. But it seems to rely on the OpenGL API. I don't think we can really apply the logic available here in the new Chromium source code base. So let's instead focus on the DirectX/Angle interactions mentioned above.

## Figuring out how the CEF commands work

• If we take the case of the GenTextures method for instance, in th GLES2Implementation the actual generation of the texture IDs is done with the helper_ element which is a GLES2CmdHelper
• Calling GenTexturesImmediate(…) on this object will generate a gles2::cmds::GenTexturesImmediate command, which is initialized with the provided parameters.
• All the commands are defined in the file gles2_cmd_format_autogen.h
• Okay, great, we have this “GenTexturesImmediate” structure… but where do we actually use it then ?
• Found in ui/gl/angle_platform_impl.{h,cc} the function InitializeAnglePlatform(…) ⇒ maybe we could check if this is indeed called ? (And where ?)
• We call in in ui/gl/gl_surface_egl.cc when initializing our display…

Now trying to figure out where we create a GLSurfaceEGL class instance… Searching in all .cc files with:

$grep -rnw --include \*.cc . -e 'GLSurfaceEGL' ./components/exo/wayland/clients/client_base.cc:279: if (gl::GLSurfaceEGL::HasEGLExtension("EGL_EXT_image_flush_external") || ./components/exo/wayland/clients/client_base.cc:280: gl::GLSurfaceEGL::HasEGLExtension("EGL_ARM_implicit_external_sync")) { ./components/exo/wayland/clients/client_base.cc:283: if (gl::GLSurfaceEGL::HasEGLExtension("EGL_ANDROID_native_fence_sync")) { ./gpu/command_buffer/service/texture_definition.cc:150: EGLDisplay egl_display = gl::GLSurfaceEGL::GetHardwareDisplay(); ./gpu/ipc/service/direct_composition_child_surface_win.cc:45: : gl::GLSurfaceEGL(), ./gpu/ipc/service/direct_composition_surface_win.cc:97: if (!gl::GLSurfaceEGL::IsDirectCompositionSupported()) ./gpu/ipc/service/direct_composition_surface_win.cc:1005: : gl::GLSurfaceEGL(), ./gpu/ipc/service/gpu_init.cc:110: gl::GLSurfaceEGL::IsDirectCompositionSupported() && ./gpu/ipc/service/image_transport_surface_win.cc:54: if (gl::GLSurfaceEGL::IsDirectCompositionSupported()) { # (...) => more lines here. • Found interesting use case in gpu/command_buffer/service/texture_definition.cc:150: scoped_refptr<NativeImageBufferEGL> NativeImageBufferEGL::Create( GLuint texture_id) { EGLDisplay egl_display = gl::GLSurfaceEGL::GetHardwareDisplay(); EGLContext egl_context = eglGetCurrentContext(); DCHECK_NE(EGL_NO_CONTEXT, egl_context); DCHECK_NE(EGL_NO_DISPLAY, egl_display); DCHECK(glIsTexture(texture_id)); DCHECK(gl::g_driver_egl.ext.b_EGL_KHR_image_base && gl::g_driver_egl.ext.b_EGL_KHR_gl_texture_2D_image && gl::g_current_gl_driver->ext.b_GL_OES_EGL_image); const EGLint egl_attrib_list[] = { EGL_GL_TEXTURE_LEVEL_KHR, 0, EGL_IMAGE_PRESERVED_KHR, EGL_TRUE, EGL_NONE}; EGLClientBuffer egl_buffer = reinterpret_cast<EGLClientBuffer>(texture_id); EGLenum egl_target = EGL_GL_TEXTURE_2D_KHR; EGLImageKHR egl_image = eglCreateImageKHR( egl_display, egl_context, egl_target, egl_buffer, egl_attrib_list); if (egl_image == EGL_NO_IMAGE_KHR) { LOG(ERROR) << "eglCreateImageKHR for cross-thread sharing failed: 0x" << std::hex << eglGetError(); return NULL; } return new NativeImageBufferEGL(egl_display, egl_image); } • ⇒ This seems somewhat similar to the code we would like to execute. • We could check if the display is initialized in our client process ? (But I doubt it…) Or maybe we could send our shared texture handle into the GPUProcess to copy a given texture ? ## Server side command handling On the client side, it now seems relatively clear how the various GL commands are “posted”, but now we need to figure out how those commands are handled on the server side. • [Unrelated note]: gpu/ipc/service/direct_composition_child_surface_win.{h, cc}: contains a reference on a DirectX11 device. • This device is retrieved directly from ANGLE with: bool DirectCompositionChildSurfaceWin::Initialize(gl::GLSurfaceFormat format) { d3d11_device_ = gl::QueryD3D11DeviceObjectFromANGLE(); dcomp_device_ = gl::QueryDirectCompositionDevice(d3d11_device_); if (!dcomp_device_) return false; // More code here } ⇒ Focusing on the initial question: trying to find where we use the command ID kGenTexturesImmediate ? • grep -rnw –include \*.cc . -e 'kGenTexturesImmediate' ⇒ gives nothing • grep -rnw –include \*.h . -e 'kGenTexturesImmediate' ⇒ gives only one result • grep -rnw –include \*.cc . -e 'kCmdId' ⇒ gives nothing • grep -rnw –include \*.h . -e 'kCmdId' ⇒ only some not interesting entries. So it seems the Command ID is not how we retrieve the GenTexturesImmediate of command on the server/host side ? • Searching with: grep -rnw –include \*.h –include \*.cc . -e 'GenTexturesImmediate' • Found references in gles2_cmd_decoder_autogen.h: gles2_cmd_decoder.h defines an abstract GLES2Decoder class • A concrete class is GLES2DecoderPassthroughImpl (in gles2_cmd_decoder_passthrough.{h,cc}) • Found that in the decoder autogen file we have an handler for the GenTexturesImmediate, which is: error::Error GLES2DecoderImpl::HandleGenTexturesImmediate( uint32_t immediate_data_size, const volatile void* cmd_data) { const volatile gles2::cmds::GenTexturesImmediate& c = *static_cast<const volatile gles2::cmds::GenTexturesImmediate*>(cmd_data); GLsizei n = static_cast<GLsizei>(c.n); uint32_t data_size; if (!SafeMultiplyUint32(n, sizeof(GLuint), &data_size)) { return error::kOutOfBounds; } volatile GLuint* textures = GetImmediateDataAs<volatile GLuint*>(c, data_size, immediate_data_size); if (textures == NULL) { return error::kOutOfBounds; } auto textures_copy = base::MakeUnique<GLuint[]>(n); GLuint* textures_safe = textures_copy.get(); std::copy(textures, textures + n, textures_safe); if (!CheckUniqueAndNonNullIds(n, textures_safe) || !GenTexturesHelper(n, textures_safe)) { return error::kInvalidArguments; } return error::kNoError; } ⇒ From there, the GenTexturesHelper(…) method will call glGenTextures(): bool GLES2DecoderImpl::GenTexturesHelper(GLsizei n, const GLuint* client_ids) { for (GLsizei ii = 0; ii < n; ++ii) { if (GetTexture(client_ids[ii])) { return false; } } std::unique_ptr<GLuint[]> service_ids(new GLuint[n]); glGenTextures(n, service_ids.get()); for (GLsizei ii = 0; ii < n; ++ii) { CreateTexture(client_ids[ii], service_ids[ii]); } return true; } So this is where we call the gl functions: in the GLES2Decoder. Yet, how do we reach the HandleGenTexturesImmediate method ? • Not too surprisingly, we can find this method re-implemented in GLES2DecoderPassthroughImpl • The re-implementation is a bit different, but will eventually call glGenTextures() too. GLES2Decoder class cannot be instantiated because is is abstract. Yet we need to figure out what are all the possible derived classes and where they are instantiated now: • Seems that for the creation, we use the static method: GLES2Decoder* GLES2Decoder::Create( GLES2DecoderClient* client, CommandBufferServiceBase* command_buffer_service, ContextGroup* group) { if (group->gpu_preferences().use_passthrough_cmd_decoder) { return new GLES2DecoderPassthroughImpl(client, command_buffer_service, group); } return new GLES2DecoderImpl(client, command_buffer_service, group); } • So we can create the implementation either as GLES2DecoderPassthroughImpl or GLES2DecoderImpl • This depends on the config settings GpuPreferences::use_passthrough_cmd_decoder which is set to false by default. • The config entry can be set from content/public/browser/gpu_utils.cc:126 if the command line “use-passthrough-cmd-decoder” is provided. This all still doesn't tell us where HandleGenTexturesImmediate is effectively called… The method is not available to any public interface so it should be used only “internally” ? • The internal QueryManager seems to generate queries that should then be processed. • But the CommandsIssuedQuery class will just throw an exception in Process(…). My my my… The GLES2Decoder instance is create with a CommandBufferService object. ⇒ Found information on how to add a command from this page Found the CommandInfo struct in gles2_cmd_decoder.cc: // A struct to hold info about each command. struct CommandInfo { CmdHandler cmd_handler; uint8_t arg_flags; // How to handle the arguments for this command uint8_t cmd_flags; // How to handle this command uint16_t arg_count; // How many arguments are expected for this command. }; // A table of CommandInfo for all the commands. static const CommandInfo command_info[kNumCommands - kFirstGLES2Command]; OK, so now we need to figure out where this array is filled, and where we call the cmd_handler function! • command_info array populated with: const GLES2DecoderImpl::CommandInfo GLES2DecoderImpl::command_info[] = { #define GLES2_CMD_OP(name) \ { \ &GLES2DecoderImpl::Handle##name, cmds::name::kArgFlags, \ cmds::name::cmd_flags, \ sizeof(cmds::name) / sizeof(CommandBufferEntry) - 1, \ } \ , /* NOLINT */ GLES2_COMMAND_LIST(GLES2_CMD_OP) #undef GLES2_CMD_OP }; • ⇒ This is applying a macro operator on the list of commands described in gles2_cmd_ids_autogen.h ⇒ So here we have it: this is how the “server” side knows which handler should be executed for a given command • Access to the command_info array will then happen in the GLES2DecoderImpl::DoCommandsImpl(…) method: where we retrieve some commands from the command buffer, and then call the corresponding GL functions with the handlers. OK ## Sharing DirectX texture with ANGLE As mentioned above, it seems to be possible to have ANGLE write to ashared DirectX texture. So in our case we would need to follow those steps: ### 1. Create a shared DirectX texture We should already have the required support functions to do this ? • Our CEFOverlayRenderer class will create a simple “Color texture” on construction, could we get a shared handle for this ? ⇒ we just need to update the RenderEngine::createColorTexture() to support an optional bool shared parameter. Once this is done, we set the shared flag to tru for this texture on creation, and then we can expect to get a share handle from the DirectX context upon texture creation. • Then in our MXRenderHandler helper class, we could retrieve the share handle. But the retrieval should be done in the base CefRenderHandler class, with a method such as CefRenderHandler::getShareHandle(). Or we could use an indirect handle retriever such as with: struct SharedRenderSurfaceManager { virtual void* getShareHandle(CefRenderHandler* handler) = 0; }; CEF_EXPORT void cef_set_shared_render_surface_manager(SharedRenderSurfaceManager* manager); Then we can provide our own implementation of the SharedRenderSurfaceManager class on CEF start, just like with the LogHandler class above. Where do we use the getShareHandle() then ? • Can we retrieve the CefRenderHandler from CefCopyFrameGenerator::PrepareTextureCopyOutputResult ? • We receive a CefRenderWidgetHostViewOSR as view_ • From inside that view we can call: CefRefPtr<CefRenderHandler> handler = browser_impl_->GetClient()->GetRenderHandler(); • ⇒ So yes, we should be able to retrieve the RenderHandler pointer! • From there we can retrieve the share handle using the SharedRenderSurfaceManager: OK Then, we need to create an EGL surface based on this handle: EGLSurface surface = EGL_NO_SURFACE; EGLint pBufferAttributes[] = { EGL_WIDTH, width, EGL_HEIGHT, height, EGL_TEXTURE_TARGET, EGL_TEXTURE_2D, EGL_TEXTURE_FORMAT, EGL_TEXTURE_RGBA, EGL_NONE }; surface = eglCreatePbufferFromClientBuffer(mEglDisplay, EGL_D3D_TEXTURE_2D_SHARE_HANDLE_ANGLE, sharedHandle, mEglConfig, pBufferAttributes); if (surface == EGL_NO_SURFACE) { // error handling code } ⇒ Any chance we could execute the previous code in the CEF context ? • From the GLHelper class, the gl_ member is a gpu::gles2::GLES2Interface pointer. • Where do we create the GLHelper instance ? • It comes from the ImageTransportFactory instance. • The ImageTransportFactory should be a GpuProcessTransportFactory • The GpuProcessTransportFactory, will create the GLHelper with: viz::GLHelper* GpuProcessTransportFactory::GetGLHelper() { if (!gl_helper_ && !per_compositor_data_.empty()) { scoped_refptr<ContextProvider> provider = SharedMainThreadContextProvider(); if (provider.get()) gl_helper_.reset( new viz::GLHelper(provider->ContextGL(), provider->ContextSupport())); } return gl_helper_.get(); } • So from our ContextProvider we get a GLES2interface pointer with ContextGL(). • Is this GLES2Interface() somehow what we need to call an eglXXX() function ? • The GLES2Interface is provided in ContextProviderCommandBuffer as a gpu::gles2::GLES2Implementation: gpu::gles2::GLES2Interface* ContextProviderCommandBuffer::ContextGL() { DCHECK(bind_succeeded_); DCHECK(context_thread_checker_.CalledOnValidThread()); if (trace_impl_) return trace_impl_.get(); return gles2_impl_.get(); } ### 2. Request rendering on shared DX texture Once we have a valid share handle for our DirectX texture, then each time we have to refresh the offscreen rendered image, we could send a command to the GPUProcess, to request rendering from a given GL texture on our shared DX texture. Something like: copyOnDXSharedSurface(uint glTextureID, void* shareHandle). ### 3. Resources allocation/deallocation Before we can copy to our shared DirectX surface, we need to allocate the required resources for it: this can be done on the GPU process itself directly: if when we handle the copy, if the required resources for a given share handle are not found, then we create them. But then we have the question of the resources deallocation: when we are done rendering with a given browser, we should release those resources allocated on the GPU Process. For instance with a command such as: releaseDXSharedSurfaceResources(void* shareHandle). ### 4. Logging in GPU Process During development, we will have to get some outputs from the GPU Process itself to ensure everything is happening as expected. How can we get those outputs ? • Doesn't seem there is a very simple way to focus on our own outputs: so instead we could just build a minimal logging to file system and we can output messages from there in the GPU Process itself. ## First implementation stage Creating the nvLogger helper class, and trying to use it to log messages to file when creating the GLES command decoder: • We get a link error in that case: [6/13] LINK ui.service.exe ui.service.exe.pdb FAILED: ui.service.exe ui.service.exe.pdb D:/Projects/CEFBuild/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../build/toolchain/win/tool_wrapper.py link-wrapper environment.x64 False link.exe /nologo /OUT:./ui.service.exe /PDB:./ui.service.exe.pdb @./ui.service.exe.rsp service_sources.lib(gles2_cmd_decoder.obj) : error LNK2019: unresolved external symbol "void __cdecl cef::nvLOG(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &)" (?nvLOG@cef@@YAXAEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z) referenced in function "public: static class gpu::gles2::GLES2Decoder * __cdecl gpu::gles2::GLES2Decoder::Create(class gpu::gles2::GLES2DecoderClient *,class gpu::CommandBufferServiceBase *,class gpu::gles2::ContextGroup *)" (?Create@GLES2Decoder@gles2@gpu@@SAPEAV123@PEAVGLES2DecoderClient@23@PEAVCommandBufferServiceBase@3@PEAVContextGroup@23@@Z)
./ui.service.exe : fatal error LNK1120: 1 unresolved externals
• This is not so surprising. Let's instead move our logger code into the gpu service folder and reference it from here instead of from the cef higher level module.
• Adding our new nv_logger.{h,cc} in the service level BUILD.gn file.

Running the target project with this new version of CEF we have a cef_nv.log file generated! But the content is not what we expected:

nvLogger initialized.Destroying nvLogger.
• Fixing the final end line symbol: OK
• Ensuring we don't call the logger at all if we are not in the GPU Process: OK
• ⇒ What could have happened here is that the logger was called in the GPU process (and the file was written with the expected content), but then on close of the browser process, the “Destroying nvLogger” message was sent, and thus the file was re-written from the browser process).
• Rebuilding: OK
• Now our content is just a little bit better:
nvLogger initialized.Creating GLES2DecoderImpl instance.Creating GLES2DecoderImpl instance.Creating GLES2DecoderImpl instance.Creating GLES2DecoderImpl instance.
• Hmmm, well this content gets overriden again then! Not sure the build was done correctly actually… Rebuilding again: Arrff, just forgot to call make_distrib.bat actually! (you stupid manu… )
• After make_distrib.bat call the behavior is what we expect (except that the nvLogger instance “destruction” doesn't get logged but that's not too critical and depends on how the GPU process is stopped):
nvLogger initialized.
Creating GLES2DecoderImpl instance.
Creating GLES2DecoderImpl instance.
Creating GLES2DecoderImpl instance.
Creating GLES2DecoderImpl instance.
Creating GLES2DecoderImpl instance.
Creating GLES2DecoderImpl instance.
Creating GLES2DecoderImpl instance.
Creating GLES2DecoderImpl instance.
Creating GLES2DecoderImpl instance.
Creating GLES2DecoderImpl instance.
Creating GLES2DecoderImpl instance.

What would be interesting to add from that point now si the process ID / thread id / and log time for each message in that file.

• We have a Process class in base/process/process.h

⇒ OK, updated the code, and now we get those results:

2017-09-26 13:53:13.473 UTC [20408]{16984}: nvLogger initialized.
2017-09-26 13:53:13.473 UTC [20408]{16984}: Creating GLES2DecoderImpl instance.
2017-09-26 13:53:13.479 UTC [20408]{16984}: Creating GLES2DecoderImpl instance.
2017-09-26 13:53:13.482 UTC [20408]{16984}: Creating GLES2DecoderImpl instance.
2017-09-26 13:53:13.484 UTC [20408]{16984}: Creating GLES2DecoderImpl instance.
2017-09-26 13:53:32.438 UTC [20408]{16984}: Creating GLES2DecoderImpl instance.
2017-09-26 13:53:34.164 UTC [20408]{16984}: Creating GLES2DecoderImpl instance.
2017-09-26 13:53:34.170 UTC [20408]{16984}: Creating GLES2DecoderImpl instance.
2017-09-26 13:53:36.405 UTC [20408]{16984}: Creating GLES2DecoderImpl instance.
2017-09-26 13:53:36.443 UTC [20408]{16984}: Creating GLES2DecoderImpl instance.
2017-09-26 13:53:36.448 UTC [20408]{16984}: Creating GLES2DecoderImpl instance.
2017-09-26 13:53:37.010 UTC [20408]{16984}: Creating GLES2DecoderImpl instance.

So it seems all GLES2Decoder instances are created from the same process id (expected) and from the same thread: so this all good for us.

## Where to retrieve the rendered frame

• OnSwapCompositorFrame() is the key method executed when a frame is done rendering.
• We register a callback to get notified when this method is called with CefRenderWidgetHostViewOSR::RegisterGuestViewFrameSwappedCallback()
• When this callback is called, we call CefRenderWidgetHostViewOSR::InvalidateInternal() and this will request a copy of the generated frame.
• Then the CefCopyFrameGenerator is used to handle the frame request and dispatch.
• Updating CefRenderHandler adding the functions:
/**
* Check if shared handle should be used.
*/
virtual bool UseSharedHandle() const { return false; }

/**
* Retrieve the shared handle if applicable.
*/
virtual void* GetSharedHandle() const { return nullptr; }
• ⇒ Same problem as with the client: the RenderHandler pointer in Cef library is different from the one in SimCore. Maybe this object is copied ?
• Trying to make the functions pure virtuals ⇒ indeed we get an error now:
../../cef/tests/cefclient/browser/browser_window_osr_win.cc(19): error C2259: 'client::ClientHandlerOsr': cannot instantiate abstract class
../../cef/tests/cefclient/browser/browser_window_osr_win.cc(19): note: due to following members:
../../cef/tests/cefclient/browser/browser_window_osr_win.cc(19): note: 'bool CefRenderHandler::UseSharedHandle(void) const': is abstract
D:\Projects\CEFBuild\chromium_git\chromium\src\cef\include/cef_render_handler.h(64): note: see declaration of 'CefRenderHandler::UseSharedHandle'
../../cef/tests/cefclient/browser/browser_window_osr_win.cc(19): note: 'void *CefRenderHandler::GetSharedHandle(void) const': is abstract
D:\Projects\CEFBuild\chromium_git\chromium\src\cef\include/cef_render_handler.h(69): note: see declaration of 'CefRenderHandler::GetSharedHandle'
• ⇒ This is only for the unit testing code, so we just add default implementation in the ClientHandlerOsr::OsrDelegate class:
virtual bool UseSharedHandle() const { return false; }
virtual void* GetSharedHandle() const { return nullptr; }
• There should still be another error… OK, next error is:
D:\Projects\CEFBuild\chromium_git\chromium\src\cef\libcef_dll/ctocpp/ctocpp_ref_counted.h(110): error C2259: 'CefRenderHandlerCToCpp': cannot instantiate abstract class
• We need to update the class defined in “libcef_dll/ctocpp/render_handler_ctocpp.{h,cc}”
• ⇒ Should not modify those files by hand, should rather read the file: translator.README.txt
• Executing generation in “chromium/src/cef/tools”:
translator.bat
• ⇒ Same error (previous command reported that 0 files were written anyway)
• Adding the magic comment above the new functions:
/*--cef()--*/
• OK! Then also updated the comments for those functions. Now the translator will update some files:
///
// Check if shared handle should be used with this render handler.
///
/*--cef()--*/
virtual bool UseSharedHandle() { return false; }

///
// Return the shared handle for this renderhandler. If no shared handle
// is available then null is returned.
///
/*--cef()--*/
virtual void* GetSharedHandle() { return nullptr; }
• Then building will fail because of missing return value for GetSharedHandle() in “cef/libcef_dll/ctocpp/render_handler_ctocpp.cc(39)”
• Implementation was generated properly for UseSharedHandle:
bool CefRenderHandlerCToCpp::UseSharedHandle() const {
cef_render_handler_t* _struct = GetStruct();
if (CEF_MEMBER_MISSING(_struct, use_shared_handle))
return false;

// AUTO-GENERATED CONTENT - DELETE THIS COMMENT BEFORE MODIFYING

// Execute
int _retval = _struct->use_shared_handle(_struct);

// Return type: bool
return _retval ? true : false;
}

• We implement GetSharedHandle as:
void* CefRenderHandlerCToCpp::GetSharedHandle() const {
cef_render_handler_t* _struct = GetStruct();
if (CEF_MEMBER_MISSING(_struct, get_shared_handle))
return NULL;

void* handle = _struct->get_shared_handle(_struct);

return handle;
}
• Also updating the method render_handler_get_shared_handle from “render_handler_cpptoc.cc”:
void* CEF_CALLBACK
render_handler_get_shared_handle(const struct _cef_render_handler_t* self) {
DCHECK(self);
if (!self)
return 0;

// Execute
bool handle = CefRenderHandlerCppToC::Get(self)->GetSharedHandle();

// Return type: void*
return handle;
}
• Still we should not make the functions pure virtuals: because of the unit tests.
• Also needed to remove the const qualifiers on our new methods.
• ⇒ And still not working after rebuild. So let's really keep our new methods as pure virtual if you want to play it that way boy.
• Strangely, the RenderHandler pointer is always different when calling the log message… how could that be ?
• Figured out that the cpptoc interface was nnot behaving as expected: missing function for use_shared_handle.
• Then also realized that our libcef_dll module in the target project was not updated accordingly: we need to update that with the version generated by the translator tool.
• Then in CefCopyFrameGenerator::InternalGenerateCopyFrame() we create a CopyOutRequest
• Then eventually, we get back to the gl_helper→CropScaleReadbackAndCleanMailbox() call: This is the call that we should replace (render_widget_host_view_osr.cc line 204)
• We can create another function:
void GLHelper::CopyToSharedDXHandle(
const gpu::Mailbox& src_mailbox,
const gpu::SyncToken& sync_token,
void* sharedHandle) {
GLuint mailbox_texture = ConsumeMailboxToTexture(src_mailbox, sync_token);
gl_->NvCopyTextureToSharedHandle(mailbox_texture, sharedHandle);
gl_->DeleteTextures(1, &mailbox_texture);
}

• Note: it seems all texture copy operations are currently handled with the copy_texture_to_impl_ member: could use this interface instead ? (but this is not really necessary)
• So we should implement NvCopyTextureToSharedHandle(GLuint id, void* handle) as a new GL command: Added in “src/gpu/command_buffer/cmd_buffer_functions.txt”:
GL_APICALL void         GL_APIENTRY glNervCopyTextureToSharedHandle (GLuint texture_id, const GLvoid* handle);
• Need to be very careful on the previous function declaration otherwise it will not be found.
• Adding the function into _FUNCTION_INFO in “src/gpu/command_buffer/build_gles2_cmd_buffer.py”:
'NervCopyTextureToSharedHandle': {
'decoder_func': 'DoNervCopyTextureToSharedHandle',
'unit_test': False,
},
• Then needs to call the generation script from the “src” folder:
python gpu\command_buffer\build_gles2_cmd_buffer.py
• And we get a NotImplementedError when calling this script.
File "gpu\command_buffer\build_gles2_cmd_buffer.py", line 9848, in WriteCmdInit
self.type_handler.WriteImmediateCmdInit(self, f)
File "gpu\command_buffer\build_gles2_cmd_buffer.py", line 5447, in WriteImmediateCmdInit
raise NotImplementedError(func.name)
NotImplementedError: NervCopyTextureToSharedHandleImmediate
• Trying to use InitializeDiscardableTextureCHROMIUM as template. So its working with the function info:
'NervCopyTextureToSharedHandle': {
'type': 'Custom',
'decoder_func': 'DoNervCopyTextureToSharedHandle',
'unit_test': False,
'impl_func': False,
'client_test': False,
},
• Our new function doesn't seem to be anywhere in the GLES2 Command decoder class… hmmm.
• Our function should really be in gles2_cd_decoder.cc, trying to add the “extension” flag in the function info ⇒ NO this is incorrect: we should implement the function here ourself.
• We have the ID generated for NervCopyTextureToSharedHandle in “common/gles2_cmd_ids_autogen.h”
• Now trying to build anyway… It should fail because we don't have the function GLES2DecoderImpl::HandleNervCopyXXX() yet.
• Obversing the following error:
FAILED: ui.service.exe ui.service.exe.pdb
service_sources.lib(gles2_cmd_decoder.obj) : error LNK2019: unresolved external symbol "private: enum gpu::error::Error __cdecl gpu::gles2::GLES2DecoderImpl::HandleNervCopyTextureToSharedHandle(unsigned int,void const volatile *)" (?HandleNervCopyTextureToSharedHandle@GLES2DecoderImpl@gles2@gpu@@AEAA?AW4Error@error@3@IPEDX@Z) referenced in function "void __cdecl dynamic initializer for 'private: static struct gpu::gles2::GLES2DecoderImpl::CommandInfo const * const gpu::gles2::GLES2DecoderImpl::command_info''(void)" (??__E?command_info@GLES2DecoderImpl@gles2@gpu@@0QBUCommandInfo@123@B@@YAXXZ)
service_sources.lib(gles2_cmd_decoder_passthrough.obj) : error LNK2019: unresolved external symbol "private: enum gpu::error::Error __cdecl gpu::gles2::GLES2DecoderPassthroughImpl::HandleNervCopyTextureToSharedHandle(unsigned int,void const volatile *)" (?HandleNervCopyTextureToSharedHandle@GLES2DecoderPassthroughImpl@gles2@gpu@@AEAA?AW4Error@error@3@IPEDX@Z) referenced in function "void __cdecl dynamic initializer for 'private: static struct gpu::gles2::GLES2DecoderPassthroughImpl::CommandInfo const * const gpu::gles2::GLES2DecoderPassthroughImpl::command_info''(void)" (??__E?command_info@GLES2DecoderPassthroughImpl@gles2@gpu@@0QBUCommandInfo@123@B@@YAXXZ)
gles2_implementation.lib(gles2_implementation.obj) : error LNK2001: unresolved external symbol "public: virtual void __cdecl gpu::gles2::GLES2Implementation::NervCopyTextureToSharedHandle(unsigned int,void const *)" (?NervCopyTextureToSharedHandle@GLES2Implementation@gles2@gpu@@UEAAXIPEBX@Z)
./ui.service.exe : fatal error LNK1120: 3 unresolved externals
[688/961] CXX obj/third_party/WebKit/Source/bindings/modules/v8/bindings_modules_impl_0/V8VRLayer.obj
ninja: build stopped: subcommand failed.
• The NervCopyTextureToSharedHandle structure is defined gles2_cmd_format_autogen.h
• Currently the struct seems to pass the share handle into shared memory ⇒ We should avoid this by using a uint64_t type instead of void*: OK
• After adding the 3 functions below, the compilation works:
error::Error GLES2DecoderImpl::HandleNervCopyTextureToSharedHandle(
uint32_t immediate_data_size,
const volatile void* cmd_data) {
const volatile gles2::cmds::NervCopyTextureToSharedHandle& c =
*static_cast<const volatile gles2::cmds::NervCopyTextureToSharedHandle*>(
cmd_data);
GLuint texture_id = c.texture_id;
void* handle = (void*)(c.shared_handle());
NV_LOG("GLES2DecoderImpl: received shared handle "<<(const void*)handle<<" for source texture_id: "<<texture_id);

return error::kNoError;
}

error::Error GLES2DecoderPassthroughImpl::HandleNervCopyTextureToSharedHandle(
uint32_t immediate_data_size,
const volatile void* cmd_data) {
const volatile gles2::cmds::NervCopyTextureToSharedHandle& c =
*static_cast<const volatile gles2::cmds::NervCopyTextureToSharedHandle*>(
cmd_data);
GLuint texture_id = c.texture_id;
void* handle = (void*)(c.shared_handle());
NV_LOG("GLES2DecoderPassthroughImpl: received shared handle "<<(const void*)handle<<" for source texture_id: "<<texture_id);

return error::kNoError;
}

<sxh cpp>void GLES2Implementation::NervCopyTextureToSharedHandle(
GLuint texture_id, GLuint64 shared_handle) {
helper_->NervCopyTextureToSharedHandle(texture_id, shared_handle);

}</sxh>

• Now time to try to call one of these in our render to texture pipeline: updated the render_view_osr method PrepareTextureCopyOutputResult:
DCHECK(result->HasTexture());
base::ScopedClosureRunner scoped_callback_runner(
base::Bind(&CefCopyFrameGenerator::OnCopyFrameCaptureFailure,
weak_ptr_factory_.GetWeakPtr(), damage_rect));

const gfx::Size& result_size = result->size();
DEBUG_MSG("Texture size is: "<<result_size.width()<<"x"<<result_size.height());

// We don't allocate any SkBitmap below:
#if 0
SkIRect bitmap_size;
if (bitmap_)
bitmap_->getBounds(&bitmap_size);

if (!bitmap_ || bitmap_size.width() != result_size.width() ||
bitmap_size.height() != result_size.height()) {
// Create a new bitmap if the size has changed.
bitmap_.reset(new SkBitmap);
bitmap_->allocN32Pixels(result_size.width(), result_size.height(), true);
if (bitmap_->drawsNothing())
return;
}
#endif

content::ImageTransportFactory* factory =
content::ImageTransportFactory::GetInstance();
viz::GLHelper* gl_helper = factory->GetGLHelper();
if (!gl_helper)
return;

// We do not need the pixels below:
#if 0
uint8_t* pixels = static_cast<uint8_t*>(bitmap_->getPixels());
#endif

viz::TextureMailbox texture_mailbox;
std::unique_ptr<cc::SingleReleaseCallback> release_callback;
result->TakeTexture(&texture_mailbox, &release_callback);
DCHECK(texture_mailbox.IsTexture());
if (!texture_mailbox.IsTexture())
return;

ignore_result(scoped_callback_runner.Release());

// We don't call CropScaleReadback method here:
#if 0
texture_mailbox.mailbox(), texture_mailbox.sync_token(), result_size,
gfx::Rect(result_size), result_size, pixels, kN32_SkColorType,
base::Bind(
&CefCopyFrameGenerator::CopyFromCompositingSurfaceFinishedProxy,
weak_ptr_factory_.GetWeakPtr(), base::Passed(&release_callback),
damage_rect, base::Passed(&bitmap_)),
viz::GLHelper::SCALER_QUALITY_FAST);
#endif

CefRefPtr<CefRenderHandler> handler = view_->browser_impl()->GetClient()->GetRenderHandler();
if(!handler.get()) {
DEBUG_MSG("NervCEF: Invalid render handler: not performing texture copy.");
OnCopyFrameCaptureCompletion(false);
return;
}

if(!handler->UseSharedHandle()) {
DEBUG_MSG("NervCEF: Not using shared handle.");
OnCopyFrameCaptureCompletion(false);
return;
}

void* handle = handler->GetSharedHandle();
if(handle == nullptr) {
DEBUG_MSG("NervCEF: ignoring null shared handle.");
OnCopyFrameCaptureCompletion(false);
return;
}

// Here we call another function to copy the texture to the shared handle:
DEBUG_MSG("NervCEF: using shared handle: "<<(const void*)handle);

gl_helper->NervCopyMailboxToSharedHandle(texture_mailbox.mailbox(), texture_mailbox.sync_token(), handle);
OnCopyFrameCaptureCompletion(false);
• Great!: with that change above (and implementation of the helper method in GLHelper), we get the expected outputs on the GPU service side (ie. “received shared handle xxxx for source texture_id: yyy”)
• We can probably render the texture on our surface with a code similar to:
bool RenderToTexture::DrawTextureBox()
{
glDisable(GL_LIGHTING);

glMatrixMode(GL_PROJECTION);

glMatrixMode(GL_MODELVIEW);

glTranslatef(-1.0f, -1.0f, 0.5f);
glScalef(2.0f, 2.0f, 1.0f);

static VERTTYPE Vertices[] =
{
0.0f, 1.0f, 0.0f,
0.0f, 0.0f, 0.0f,
1.0f, 0.0f, 0.0f,
1.0f, 1.0f, 0.0f
};
static VERTTYPE TexCoords[] =
{
0.0f, 1.0f,
0.0f, 0.0f,
1.0f, 0.0f,
1.0f, 1.0f
};

glEnableClientState(GL_VERTEX_ARRAY);
glEnableClientState(GL_TEXTURE_COORD_ARRAY);
glVertexPointer(3, VERTTYPEENUM, 0, (VERTTYPE *)&Vertices);
glTexCoordPointer(2, VERTTYPEENUM, 0, (VERTTYPE *)&TexCoords);

glBindTexture(GL_TEXTURE_2D, m_hTexture);
glDrawArrays(GL_TRIANGLE_FAN, 0, 4);

glDisableClientState(GL_VERTEX_ARRAY);
glDisableClientState(GL_TEXTURE_COORD_ARRAY);

return true;
}

## Getting the copy operation to work

• So we now have all the elements in place… But still we get a black image rendered in our project (ie. nothing written on this surface from the CEF service process). How could that be ?
• It could be related to the size of the texture ? nope
• To the binding to a texture in the sample ? nope
• To the context config ? Nope
• To the display we use in the CEF process ? Nope
• ⇒ I even tried to embed a test window in the CEF service process, with its own hwnd and EGLDisplay! And the result of that is that the window will not display at all until we close the software: at that time, all the commands are executed in a row.
• ⇒ So it seems we have a command buffer issue: our commands are stacked but not executed in a timely fashion… Need to clarify that.
• Note: When it comes to using texture, checking GLES2DecoderImpl::DeleteTexturesHelper it would seem that the client texture id is not the real ID and instead this is mapped to a Texture ref on the service side.
• Trying to call glFlush() in gl_helper.cc and CheckGLError() from gles2_implementation.cc but I doubt this is it. ⇒ GREATTTT ! This actually works!
• I rather think we need to tweak the function generation ⇒ maybe using a custom handler type ? ⇒ We need to be able to generate the Immediate version of the function. ⇒ no need to go that way for now.
• Okay, stay cool. Now is a good time to make a backup of our state on the second CEF build setup.
• Continuing on our path: I tried to create a new context sharing resources with the current context to be able to access the texture data: this is failing with a EGL_BAD_MATCH error.
• Then trying to use the current context to do the rendering directly: this is not rendering anything on the pbuffer.
• So what's next ? What is this BAD_MATCH error ?
• Note: Maybe we can just call helper_→CommandBufferHelper::Flush(); instead of a glFlush() command ?
• Note: The actual implementation in GLES2DecoderImpl::DoBindTexture is much more complex that anticipated.
• So let's consider creating our context from the decoder base context:
// We can retrieve the context with:
gl::GLContext* GetGLContext();
// And the context group with:
ContextGroup* GetContextGroup();

// From the GLContext, we can retrieve the share group:
// share_group = GLContext::share_group();

// Then with this share group we can try to create a shared context:
nvContext = eglCreateContext(egl_display, nvConfig, sgroup->GetHandle(), contextAttribs);

• Now, maybe we can already check if our decoder is using a GLContext that has a share_group ? Retrieval of the context happens in GLES2DecoderImpl::Initialize(…)
• So, it seems our GLES2Decoder is initialized in “gpu/ipc/service/gpu_command_buffer_stub.cc” in the GpuCommandBufferStub::Initialize(…) method.
• In this method, whatever we do, the context is eventually created with a call to gl::init::CreateGLContext(…) giving it the share group from the current GPu channel.
• CreateGLContext implementation is provided in “ui/gl/init/gl_factory_win.cc”
• When using offscreen rendering the surface format come from the surface created with gl::init::CreateOffscreenGLSurface(…)
• Then those settings are tweaked a little bit, and eventually the default context surface is created with gl::init::CreateOffscreenGLSurfaceWithFormat(…)
• And I'm now suspecting we are actually creating a “surfaceless” context now…
• Correct! This is the kind of outputs we get from the service nv_logger system:
2017-11-16 19:11:32.913 UTC [808]{12132}: nvLogger initialized.
2017-11-16 19:11:32.913 UTC [808]{12132}: Reached: CreateOffscreenGLSurfaceWithFormat() function.
2017-11-16 19:11:32.914 UTC [808]{12132}: CreateOffscreenGLSurfaceWithFormat: Using EGLGLES2 SurfacelessEGL implementation
2017-11-16 19:11:33.042 UTC [808]{12132}: Reached: CreateOffscreenGLSurfaceWithFormat() function.
2017-11-16 19:11:33.043 UTC [808]{12132}: CreateOffscreenGLSurfaceWithFormat: Using EGLGLES2 SurfacelessEGL implementation
2017-11-16 19:11:33.043 UTC [808]{12132}: Creating GLES2DecoderImpl instance.
2017-11-16 19:11:33.043 UTC [808]{12132}: GpuCommandBufferStub: using offscreen rendering.
• So it seems this could be it. And we should try to prevent the creation of such a surfaceless context.
• ⇒ Ok, so tried to prevent using surfaceless context, and this is completely failing (freezing on our project start): so it doesn't seem to be a good idea. Moreother, looking at the definition of what surfaceless context are (cf. EGL_KHR_surfaceless_context) it doesn't seem to be a problem to use them and still be able to render on a pbuffer.
• Now, trying to use the current context again, but this time we open Gl error checking, and we have an error code:
Stage7 (error code: 1282)
• “Stage7” is reached after execution of the command:
glVertexAttribPointer(posLoc, 3, GL_FLOAT, GL_FALSE, 5 * sizeof(GLfloat), vVertices);
• Why that ? ⇒ No idea.
• Anyway, tried to remove the part where we draw the texture, and in that case we don't have any error, but we get absolutely nothing rendered on the pbuffer surface ?
• ⇒ It might be an idea to try to create our context with the same attribs as those provided in gl_context_egl.cc ?
• So it seems that yes, we can create our context from the shared base, if we use the settings from the gl_context_egl.cc file: basically, this means our context needs to be created with a request for GLES 3.0 instead of version 2.0.
• ⇒ Let's retest the base implementation again with GL ES 2.0 and no shared base using:
EGLint contextAttribs[] = {EGL_CONTEXT_CLIENT_VERSION, 2, EGL_NONE, EGL_NONE};
nvContext = eglCreateContext(egl_display, nvConfig, nullptr, contextAttribs);
• OK: making some progress: testing GL ES 2.0 context is OK. Also tested upgrading to GL ES 3, and this is still OK
• Okay so it seems we now have the same kind of issues as before when calling glVertexAttribPointer
• ⇒ I suspect this could be due to the fact that we might have a buffer bound already.
• According to the khronos documentation page:
pointer
Specifies a pointer to the first generic vertex attribute in the array. If a non-zero buffer is currently bound to the GL_ARRAY_BUFFER target, pointer specifies an offset of into the array in the data store of that buffer. The initial value is 0.
• So we can try to un bind the current: buffer with:
GLint curBuffer = 0;
glGetIntegerv(GL_ARRAY_BUFFER_BINDING, &curBuffer);

CHECK(glIsBuffer(curBuffer)==GL_TRUE);

// We bind the buffer to 0;
glBindBuffer(GL_ARRAY_BUFFER, 0);

// Here we do our rendering.

// Then we restore the bound buffer:
glBindBuffer(GL_ARRAY_BUFFER, curBuffer);

• Okay, so from the previous run we can notice an error stating that the current buffer is not a valid buffer…
• So got confirmation that the previous buffer is 0, but still, this is not working as expected ⇒ we should now try to create a buffer ourself and assign it with the data we need:
// during init:

glGenBuffersARB(1, &nvVertBuffer);
glBindBuffer(GL_ARRAY_BUFFER, nvVertBuffer);
glBufferData(GL_ARRAY_BUFFER, size_of_buffer, vVertices, GL_STATIC_DRAW);

• OK: so we could get ride of the erreur on glVertexAttribPointer, good, but now the call to glDrawElements is also failing…
• It seems we might need to build the ELEMENT_ARRAY_BUFFER too, with the list of indices, let's try that.
• Great: finally getting some texture to get displayed! Yet for the moment, it seems only 1 pixel from our texture is used: Not sure the texture coordinates are correct… And indeed, the vertex attrib pointer call for the texture coords was most probably incorrect with:
glVertexAttribPointer(1, 2, GL_FLOAT, GL_FALSE, 5 * sizeof(GLfloat), reinterpret_cast<const void*>(3));
// Should rather be:
glVertexAttribPointer(1, 2, GL_FLOAT, GL_FALSE, 5 * sizeof(GLfloat), reinterpret_cast<const void*>(3*sizeof(GLfloat)));
• Wonderfull!: finally getting somewhere! Could now display the test texture with transparency support!!
• ⇒ Now trying to use the actual texture with overlay content… (and crossing fingers very hard…)

## Origin of the Compositor texture

Currently it seems we are generating a new texture each time the compositor has a new output to provide: we need to investigate what is actually the source of this generation and thus see if we can use the root texture directly (if any ?)

• When we reach the PrepareTextureCopyOutputResult() function we already have a CopyOutputResult object, where does this come from ?
• This result object is passed to CopyFromCompositingSurfaceHasResult first.
• The request originate from InternalGenerateCopyFrame:
std::unique_ptr<cc::CopyOutputRequest> request =
cc::CopyOutputRequest::CreateRequest(base::Bind(
&CefCopyFrameGenerator::CopyFromCompositingSurfaceHasResult,
weak_ptr_factory_.GetWeakPtr(), damage_rect));

request->set_area(gfx::Rect(view_->GetPhysicalBackingSize()));
view_->GetRootLayer()->RequestCopyOfOutput(std::move(request));
• So the CopyOutputRequest is probably just a simple structure to pass to copy parameters (callback, target size, etc), and the actually request is probably made in the call: GetRootLayer()→RequestCopyOfOutput(…)
• It seems the “Root layer” is a simple solid color layer (except on MACOSX ?):
root_layer_.reset(new ui::Layer(ui::LAYER_SOLID_COLOR));
• RequestCopyOfOutput(…) will only push the request on a list.

## Texture memory issues

• Currently investigating how the TextureRef and Texture objects are released exactly in the TextureManager class: currently it seems we never get to the point where Textures are actually released in this call:
void Texture::RemoveTextureRef(TextureRef* ref, bool have_context) {
NV_LOG2("In Texture::RemoveTextureRef for "<<owned_service_id_<<", have_context="<<(have_context ? "true" : "false"));

if (memory_tracking_ref_ == ref) {
GetMemTracker()->TrackMemFree(estimated_size());
memory_tracking_ref_ = NULL;
}
size_t result = refs_.erase(ref);
DCHECK_EQ(result, 1u);
NV_LOG2("Left texture refs size: "<<refs_.size());
for(auto& lref: refs_) {
NV_LOG2("Left refs: "<<(const void*)(*refs_.begin()));
}

if (refs_.empty()) {
if (have_context) {
NV_LOG2("Deleting service texture: "<<owned_service_id_);
glDeleteTextures(1, &owned_service_id_);
}
delete this;
} else if (memory_tracking_ref_ == NULL) {
// TODO(piman): tune ownership semantics for cross-context group shared
// textures.
memory_tracking_ref_ = *refs_.begin();
GetMemTracker()->TrackMemAlloc(estimated_size());
}
}

• ⇒ We always have 1 ref left when we try to release the Texture. But we have at least 2 options here:
• We can try to force delete this second ref somehow (but this would imply to modify the texture class interface, so we should probably avoid this)
• We can trace where the call to AddTextureRef occurs to know where we created this additional reference ⇒ let's try this option.
• While trying to recompile CEFfrom sources, now noticed that the code injection in libcef_dll.cc is broken: so instead, moving our function definition in libcef_dll2.cc (which doesn't seem to be autogenerated):
#include "include/cef_nv_exports.h"

// ... (more lines here) ...

CEF_EXPORT void cef_set_log_handler(cef::CefLogHandler* handler) {
cef::setLogHandler(handler);
}
• Now also trying to trace where AddTextureRef is called for a given TextureRef:
DCHECK(refs_.find(ref) == refs_.end());
refs_.insert(ref);
// Trace where we are:
NV_LOG2("Adding textureref " << (const void*)(ref) << " on texture with service id "<< owned_service_id_ << ": "<< base::debug::StackTrace().ToString());

if (!memory_tracking_ref_) {
memory_tracking_ref_ = ref;
GetMemTracker()->TrackMemAlloc(estimated_size());
}
}
• Trying to get the stacktrace will just kill the cef engine completely (too slow mechanism): but just trying to get some insights we have the log results:
2018-01-09 08:14:27.557 UTC [2300]{11140}: GenTexturesHelper: Creating service texture: 1009
2018-01-09 08:14:27.557 UTC [2300]{11140}: Adding textureref 0000010E093B00A0 on texture with service id 1009
2018-01-09 08:14:27.557 UTC [2300]{11140}: TextureManager: num textures: 775
2018-01-09 08:14:27.569 UTC [2300]{11140}: Adding textureref 0000010E093AF8C0 on texture with service id 1009
2018-01-09 08:14:27.569 UTC [2300]{11140}: TextureManager: num textures: 3
2018-01-09 08:14:27.569 UTC [2300]{11140}: GLES2DecoderImpl: received shared handle 00000000C0000202 for source texture_id: 679
2018-01-09 08:14:27.569 UTC [2300]{11140}: GLES2DecoderImpl: Drawing from service texture_id: 1009
2018-01-09 08:14:27.569 UTC [2300]{11140}: GLES2DecoderImpl: Releasing texture unit infos.
2018-01-09 08:14:27.569 UTC [2300]{11140}: GLES2DecoderImpl: Deleting texture with client id=679 (service_id=1009)
2018-01-09 08:14:27.569 UTC [2300]{11140}: In TextureRef destructor with force_context_lost=false, manager have_context:true
2018-01-09 08:14:27.569 UTC [2300]{11140}: In Texture::RemoveTextureRef for 1009, have_context=true
2018-01-09 08:14:27.569 UTC [2300]{11140}: Left texture refs size: 1
2018-01-09 08:14:27.569 UTC [2300]{11140}: Left refs: 0000010E093B00A0
2018-01-09 08:14:27.569 UTC [2300]{11140}: GLES2DecoderImpl: Done copying to shared handle surface.
• So just after the call to GenTexturesHelper, we add a TextureRef, then we add another one: the later is eventually removed, but not the first one: need to clarify what this is exactly.
• So we first call:
std::unique_ptr<GLuint[]> service_ids(new GLuint[n]);
glGenTextures(n, service_ids.get());
for (GLsizei ii = 0; ii < n; ++ii) {
NV_LOG2("GenTexturesHelper: Creating service texture: "<<service_ids[ii]);
CreateTexture(client_ids[ii], service_ids[ii]);
}
return true;
• Then we reach the CreateTexture function:
// Creates a Texture for the given texture.
TextureRef* CreateTexture(
GLuint client_id, GLuint service_id) {
return texture_manager()->CreateTexture(client_id, service_id);
}
• Then we get into TextureManager::CreateTexture:
TextureRef* TextureManager::CreateTexture(
GLuint client_id, GLuint service_id) {
DCHECK_NE(0u, service_id);
scoped_refptr<TextureRef> ref(TextureRef::Create(
this, client_id, service_id));
std::pair<TextureMap::iterator, bool> result =
textures_.insert(std::make_pair(client_id, ref));
DCHECK(result.second);
return ref.get();
}
• ⇒ This is where we create a first TextureRef which is inserted into the textures_ map.
• Then it seems we replicate the exact same process (except that the texture service id is only allocated once)
• Also, the textureManager reports 2 different values for the “num textures”: as if we had 2 texture managers…
• ⇒ So maybe we should report the address of the manager to confirm that:
scoped_refptr<TextureRef> TextureRef::Create(TextureManager* manager,
GLuint client_id,
GLuint service_id) {
TextureRef* ref = new TextureRef(manager, client_id, new Texture(service_id));
NV_LOG2("Created textureref " << (const void*)(ref) << " on texture with service id "<< service_id << " for manager "<<<(const void*)manager);
return ref;
}
• But just assume that multiple managers call this Create function. Then we also create multiple instances of the base Texture object in the process ? So how can we get 2 different refs on a single Texture instance ??
• What about the memory_tracking_ref_ member in the Texture class ?
• Or could it be due to the call to ConsumeMailboxToTexture on the client side ? ⇒ could be creating the texture on the mailboxmanager, and then keeping a reference on it ?
• ⇒ we might be reaching the line (in GLES2DecoderImpl::DoCreateAndConsumeTextureINTERNAL):
texture_ref = texture_manager()->Consume(client_id, texture);
• OK: this seems consistent, we then reach the function:
TextureRef* TextureManager::Consume(
GLuint client_id,
Texture* texture) {
DCHECK(client_id);
scoped_refptr<TextureRef> ref(new TextureRef(this, client_id, texture));
bool result = textures_.insert(std::make_pair(client_id, ref)).second;
DCHECK(result);
return ref.get();
}
• ⇒ So in the code above, we create a new TextureRef based on an existing Texture ⇒ 2 refs on that texture object.
• So from that perspective, our texture is generated with the line:
Texture* texture =
static_cast<Texture*>(group_->mailbox_manager()->ConsumeTexture(mailbox));
• Got those outputs for the texture manager display:
2018-01-09 12:28:27.570 UTC [5012]{220}: Deleting service texture: 958
2018-01-09 12:28:27.570 UTC [5012]{220}: Adding textureref 0000023DE5FB2950 on texture 0000023DE2DB7E30 with service id 265
2018-01-09 12:28:27.570 UTC [5012]{220}: TextureManager 0000023DD68C8D60: num textures: 777
2018-01-09 12:28:27.570 UTC [5012]{220}: GenTexturesHelper: Creating service texture: 958
2018-01-09 12:28:27.570 UTC [5012]{220}: Adding textureref 0000023DE5FB2470 on texture 0000023DE2F97720 with service id 958
2018-01-09 12:28:27.570 UTC [5012]{220}: TextureManager 0000023DD68C8D60: num textures: 778
2018-01-09 12:28:27.570 UTC [5012]{220}: Created textureref 0000023DE5FB2470 on texture with service id 958 for manager 0000023DD68C8D60
2018-01-09 12:28:27.585 UTC [5012]{220}: Adding textureref 0000023DE5FB29B0 on texture 0000023DE2F97720 with service id 958
2018-01-09 12:28:27.585 UTC [5012]{220}: TextureManager 0000023DE2DF9100: num textures: 3
2018-01-09 12:28:27.585 UTC [5012]{220}: GLES2DecoderImpl: received shared handle 0000000040001342 for source texture_id: 649
2018-01-09 12:28:27.585 UTC [5012]{220}: GLES2DecoderImpl: Drawing from service texture_id: 958
2018-01-09 12:28:27.585 UTC [5012]{220}: GLES2DecoderImpl: Releasing texture unit infos.
2018-01-09 12:28:27.585 UTC [5012]{220}: GLES2DecoderImpl: Deleting texture with client id=649 (service_id=958)
2018-01-09 12:28:27.585 UTC [5012]{220}: Deleting TextureRef with force_context_lost=false, manager have_context:true for manager 0000023DE2DF9100
2018-01-09 12:28:27.585 UTC [5012]{220}: Removing TextureRef 0000023DE5FB29B0 from texture 0000023DE2F97720 with service id 958, have_context=true
2018-01-09 12:28:27.585 UTC [5012]{220}: Left texture refs size: 1
2018-01-09 12:28:27.586 UTC [5012]{220}: Left refs: 0000023DE5FB2470
2018-01-09 12:28:27.586 UTC [5012]{220}: GLES2DecoderImpl: Done copying to shared handle surface.
• So, this is all getting pretty messing with texture references/manager/etc. And thinking about it, we don't need any of these: all we really need is an access to the service_id encapsulated into the mailbox… So we could rather jsut pass the mailbox directly!
• OK: now using the mailbox directly and got ride of the memory leak issue.

## TODO

• Copy all the files containing new references on the NervCopyTextureToSharedHandle key word (?)

## Discussion

Enter your comment. Wiki syntax is allowed: