Feature #792


Try adding opengl to cairo in Launcher

Added by Hammel almost 4 years ago. Updated 9 months ago.

In Progress
Target version:
Start date:
14 Aug 2020
Due date:
% Done:


Estimated time:
03 - Medium


This is an experiment to see if we can speed up the UI.

Some references:
  2. Cairo git on for the 1.16 release in Buildroot
  5. - uses cairo opengl backend (but it's C++)
  6. - cairo source - look in src/ for gl backend support
  7. - old but good discussion on using GL with Cairo.
    1. Buildroot needs Mesa added with vc4 support to get Cairo with GL.
  8. Update to VC library to allow X11 EGL on RPi
  10. GTK gl-slider (git)

Might want to bump to latest release (1.17.2) from Buildroot's 2020.02.4 LTS current (1.16.0)


Move-EGL_DISPMANX_WINDOW_T-to-separate-file.diff (3.5 KB) Move-EGL_DISPMANX_WINDOW_T-to-separate-file.diff Patch to move EGL_DISPMANX_WINDOW_T to it's own file Hammel, 06 Sep 2023 11:03

Related issues

Related to PiBox - Infrastructure #1003: Cleanup buildroot configClosedHammel04 Jul 2023

Actions #1

Updated by Hammel almost 4 years ago

  • Description updated (diff)
Actions #2

Updated by Hammel over 2 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 10

Some research on how to use this tonight.

  1. The EGL backend for Cairo gets enabled in Buildroot if libgles is enabled.
  2. libgles is part of the VC package installed to /opt/vc/, specifically in /opt/vc/lib.
  3. /opt/vc/lib is included in /etc/

So by enabling libgles this should just work. There is a Gallium vc4 entry for Raspberry Pi under Mesa3D (Target Packages->Graphics libs/apps->Mesa3d) but this doesn't set the required Buildroot variable. However, enabling rpi-userland (Target Packages->Hardware Handling->rpi-userland) will do the trick.

That package downloads the same things I pull from the userland, but it uses a specific git hash. I'll need to patch the package with the latest updates to get as close to the latest version I've been using.

I'll have to work out what gets installed and how to merge it with what I do now. One thing I can do is change the to set the install prefix to be /opt/vc, which would map what I do, then I can copy in my stuff as usual.


Based on this discussion I probably don't need to configure rpi-userland if I configure the vc4/v3d options for Mesa, which I can get to if I have board-specific Buildroot configs (see comments below).

Actions #3

Updated by Hammel about 1 year ago

  • Priority changed from High to Urgent
  • Target version changed from 2.0 - Harkonnen to 3.0 - Corrino
Actions #4

Updated by Hammel about 1 year ago

  • Description updated (diff)
Actions #5

Updated by Hammel 10 months ago

Actions #6

Updated by Hammel 10 months ago

See RM #1003. This shows that I need separate Buildroot configs per HW type in order to enable options based on processor type.

Actions #7

Updated by Hammel 10 months ago

  • Description updated (diff)
Actions #8

Updated by Hammel 10 months ago

  • Priority changed from Urgent to Immediate
  • Severity changed from 01 - Critical to 02 - High
Actions #9

Updated by Hammel 10 months ago

From RM #1003:

I'd like to enable a Mesa driver for each RPi build. But that can't be enabled by default because of the Target Options->Target Architecture Variant option set in the single Buildroot config. This field needs to be set based on the processor.

Hardware Setting Mesa Driver
RPi 1 ARM1176JZF-S VC4
RPi 2 Cortex-A7/Cortex-A53 VC4
RPi 3 Cortex-A53 VC4
RPi 4 Cortex-A72 V3D
RPi Zero ARM1176JZF-S VC4
RPi Zero 2W Cortex-A53 VC4

Note that the RPi2 with the Cortex-A7 has not been available since 2016, so building for the Cortex-A53 (like the Rpi3) is reasonable.

It's unclear if the Cortex-A53 builds will work with the Cortex-A7, but I think they will. Since Buildroot is NOT generating the kernel or toolchain, it just uses this option to identify what other Target Package options to enable for selection. So I should just be able to have the three configs: rpi1, rpi2 (which includes rpi3) and rpi4.

So I first need to dup the buildroot config for each hardware build (rpi, rpi2 and rpi4). Then update them for the hardware platform. Then enable the correct OpenGL support.

I'm going to start with RPI3, as that seems the most likely to have the best opengl support. I need to review how to enable RPi-Userland in Buildroot and then integrate it into to PiBox (dropping the GLES build since that duplicates RPi-Userland, assuming they both build from master).

Actions #10

Updated by Hammel 10 months ago

This is being tested under the rpi3 sandbox. Currently a full, new 3.0 build is ongoing.

When that completes, I can work on dup'ing the config file and configuring for rpi2 hardware.

Actions #11

Updated by Hammel 10 months ago

  • % Done changed from 10 to 20

Updated config for rpi2 is building. Once this completes and I put it on hardware, I should have a unit test to validate that cairo is using the GLES backend.

Actions #12

Updated by Hammel 10 months ago

  • Description updated (diff)
Actions #13

Updated by Hammel 10 months ago

  • Description updated (diff)
Actions #14

Updated by Hammel 10 months ago

  • Description updated (diff)

Buildroot build completes successfully, but dev system build fails:

make -C libs/vgfont
arm-unknown-linux-gnueabi-gcc -I/home/mjhammel/src/ximba/raspberrypi3/src/../bld/buildroot-2023.02.2/output/staging/usr/include -I/home/mjhammel/src/ximba/raspberrypi3/src/../bld/buildroot-2023.02.2/output/staging/usr/include/freetype2 -DUSE_VCHIQ_ARM -D__RASPBERRYPI__ -I/home/mjhammel/src/ximba/raspberrypi3/src/../bld/buildroot-2023.02.2/output/target/usr/include/freetype2 -I/home/mjhammel/src/ximba/raspberrypi3/src/../bld/buildroot-2023.02.2/output/target/usr/include -I/home/mjhammel/src/ximba/raspberrypi3/src/../bld/buildroot-2023.02.2/output/target/usr/include/arm-linux-gnueabi -I/home/mjhammel/src/ximba/raspberrypi3/src/../bld/buildroot-2023.02.2/output/target/opt/vc/include/ -I/home/mjhammel/src/ximba/raspberrypi3/src/../bld/buildroot-2023.02.2/output/target/opt/vc/include/interface/vcos/pthreads -I/home/mjhammel/src/ximba/raspberrypi3/src/../bld/buildroot-2023.02.2/output/target/opt/vc/include/interface/vmcs_host/linux -I./ -I/home/mjhammel/src/ximba/raspberrypi3/src/../bld/buildroot-2023.02.2/output/target/opt/vc/src/hello_pi/libs/ilclient -I/home/mjhammel/src/ximba/raspberrypi3/src/../bld/buildroot-2023.02.2/output/target/opt/vc/src/hello_pi/libs/vgfont -I/home/mjhammel/src/ximba/raspberrypi3/src/../bld/buildroot-2023.02.2/output/target/opt/vc/src/hello_pi/libs/revision -g -c font.c -o font.o -Wno-deprecated-declarations
In file included from font.c:56:
graphics_x_private.h:53:4: error: unknown type name 'EGL_DISPMANX_WINDOW_T'
   53 |    EGL_DISPMANX_WINDOW_T egl_win;
      |    ^~~~~~~~~~~~~~~~~~~~~
make[3]: *** [../../Makefile.include:12: font.o] Error 1
make[2]: *** [ all] Error 2
make[1]: *** [configs/ buildroot-rpitools] Error 2
make: *** [configs/ .buildroot] Error 2

This may require adding some library paths that weren't there previously now that EGL is enabled or disabling the vgfont build in src/buildroot/

Or take a look at this change:

Which leads me to this:

$ find . -type f -exec grep -l "EGL_DISPMANX_WINDOW_T" {} \;

So maybe include/EGL is not included in

Actions #15

Updated by Hammel 10 months ago

There is a patch (attached to this RM now) that might fix this build problem. It must be applied to the gles tree before building gles because that is where the source for the comes from when it's built by the Buildroot build.

However, if this patch does not fix the problem I might just drop building the rpitools since I don't really need them.

Actions #16

Updated by Hammel 10 months ago

  • % Done changed from 20 to 40

The patch did fix the build problems with rpitools.

Now I need to build a system (other than the dev system) and see if Cairo picked up the opengl backend, thus making the UI more responsive even on low end RPis.

And I still need a test to validate that is actually happening.

Actions #17

Updated by Hammel 10 months ago

Not done with building yet - I was missing the Mesa->OpenGL ES API support in the Buildroot config. Without it, cairo is built without opengl support. I can this by looking at <buildroot>/output/build/cairo/config.log:

$ ./configure --target=arm-buildroot-linux-gnueabihf --host=arm-buildroot-linux-gnueabihf --build=x86_64-pc-linux-gnu --prefix=/usr --exec-prefix=/usr --sysconfdir=/etc --localstatedir=/var --program-prefix= --disable-gtk-doc --disable-gtk-doc-html --disable-doc --disable-docs --disable-documentation --with-xmlto=no --with-fop=no --disable-dependency-tracking --enable-ipv6 --disable-nls --disable-static --enable-shared --enable-trace=no --enable-interpreter=no --disable-directfb --enable-ft --enable-gobject --disable-gl --disable-glesv2 --disable-vg --enable-xlib --enable-xcb --with-x --enable-xlib-xrender --enable-ps --enable-pdf --enable-png --enable-script --enable-svg --disable-tee --disable-xml

If I rebuild cairo with the OpenGL ES API enabled it looks like this.

$ ./configure --target=arm-buildroot-linux-gnueabihf --host=arm-buildroot-linux-gnueabihf --build=x86_64-pc-linux-gnu --prefix=/usr --exec-prefix=/usr --sysconfdir=/etc --localstatedir=/var --program-prefix= --disable-gtk-doc --disable-gtk-doc-html --disable-doc --disable-docs --disable-documentation --with-xmlto=no --with-fop=no --disable-dependency-tracking --enable-ipv6 --disable-nls --disable-static --enable-shared --enable-trace=no --enable-interpreter=no --disable-directfb --enable-ft --enable-gobject --disable-gl --enable-glesv2 --disable-vg --enable-xlib --enable-xcb --with-x --enable-xlib-xrender --enable-ps --enable-pdf --enable-png --enable-script --enable-svg --disable-tee --disable-xml

But this fails to build:

checking whether cairo's OpenGLESv2 surface backend feature could be enabled... no (glesv2.pc nor OpenGL ES 2.0 headers not found)
configure: error:  OpenGLESv2 surface backend feature could not be enabled
make[2]: *** [package/ /home/mjhammel/src/ximba/raspberrypi3/bld/buildroot-2023.02.2/output/build/cairo-1.16.0/.stamp_configured] Error 1
make[1]: *** [configs/ .buildroot] Error 2
make: *** [configs/ buildroot-rebuild] Error 2

This might be because I need to rebuild Mesa to pickup the API update in the Buildroot config.

Actions #18

Updated by Hammel 10 months ago

  • Description updated (diff)

Rebuilding Mesa fixed the rebuild of Cairo by adding in the GLES API.

So now I can see the Cairo build includes OpenGL support. One thing I need to do in launcher is switch to this generic algorithm for making use of the GL backend:

/* create a 480x800 window gl surface in cairo */
cairo_surface_t *window_surface = cairo_gl_surface_create_for_window (device,
GDK_WINDOW_XID (gtk_widget_get_window(widget)), 480, 800);
/* once you done drawing, swap buffer */
cairo_gl_surface_swapbuffers (window_surface);

When launcher calls cairo_create() I can callcairo_surface_get_type() and check the return value against the cairo_surface_type_t enum to see if it reports CAIRO_SURFACE_TYPE_GL. If it does, then I've done everything I can do.

A StackOverflow discussion points to gl_surface_source example test which is an example of getting a GL surface for use in X.

Actions #19

Updated by Hammel 10 months ago

  • Description updated (diff)
Actions #20

Updated by Hammel 10 months ago

Did a test build with media system. This never booted fully into firstboot. I suspect something is wrong with the config.txt. I can disable the new items at the top and try again to see if it boots. The config.txt changes were never tested on rpi2 so they seem the most likely cause of the boot issues.

I should note that the toolchain and Buildroot builds are using the same ARM architecture settings. The kernel is using a Broadcom chip that should be specific to rpi2/rpi3 but I still need to verify that.

Actions #21

Updated by Hammel 10 months ago

Tried with a 2nd SD card but the same problem showed up.

Cleaned the complete build tree and restarted. Ran into some errors in Buildroot, like libpibox needing to add tslib as a dependency in and php failing with a "nothing" pragma directive that is not supported on arm-v8-a (which is what is being built for rpi2/rpi3). I've fixed those problems and Buildroot is running again. Waiting on it to complete.

Actions #22

Updated by Hammel 10 months ago

Rebuilding everything fixed the boot problem. The RPi3 boots to just fine. RPi2, however, fails to start I'll look into that later.

First I want to check if the Media System boots on RPi3 and, if so, does having Cairo using the openGL backend help with drawing in the Launcher.

Actions #23

Updated by Hammel 10 months ago

  • % Done changed from 40 to 50

It boots fine and it's a little faster. The next test would be to run it on an RPi3 with pisentry to see if PiNet's animation is better.

Actions #24

Updated by Hammel 10 months ago

Works fine on RPi3 with animation on PiNet. I might be able to speed up the animation even, but that's a task for another day (see RM #1032).

Next will be to check that the launcher is actually using the opengl backend. I might need to get an opengl surface for use by the cairo drawing routines.

I also need to determine why RPi2 exits. The log doesn't show anything terrible but it is missing the glx extension. That may be the mesa->OpenGL GLX option since I don't see any extensions as options available from Buildroot. Running S99UI manually:

root(tty0)$ /etc/init.d/S99UI start
Starting PiBox UI...
modprobe: FATAL: Module g2d_23 not found in directory /lib/modules/6.1.51-v7+
Illegal instruction
/usr/bin/xinit: connection to X server lost

waiting for X server to shut down (II) Server terminated successfully (0). Closing log file.

This suggests that the server does not work on this RPi2. Checking /etc/pibox-version.
$ cat /etc/pibox-version
VERSION: PiBox 2.0
HW: rpi2
BUILD: Thu Sep  7 19:52:33 UTC 2023
MODEL: Raspberry Pi 2 Model B Rev 1.1

And according to wikipedia the Rev 1.1 uses the A7 chip, not the A53. So that may explain why probably doesn't run.

So I can either build for rpi, rpi2 Rev1.1, rpi2 Rev1.2/rpi3 and rpi4 or I can just fallback to A7 for all rpi2 and rpi3's. RPi2 (A53 version) will be EOL in 2026 so it's still worth supporting, but the Rev 1.1 may not be worth maintaining. It's also worth noting that RPi 3 is becoming available again after the supply chain issues but RPi 2 is not. So maybe I should just update all RPi 2's to RPi 3's. Also, the RPI 4 is now supported by PiBox and is available in quantity at $10 more than the RPi 3.

Also, the Pi Zero 2W is the same as the RPi3. So just dropping RPi2 would be the simplest solution, though I could build it with the older RPi1 models and see if that works.

Actions #25

Updated by Hammel 10 months ago

RPi1 does not boot on RPi2. I only have 1 RPi2 rev 1.1 (I have one RPi2 rev1.2 that DOES work with the RPi3 build) now anyway, so maybe it's just time to retire RPi2 rev1.1 and move to just RPi1, RPi3 and RPi4. I'll order an RPi3 replacement for the RPi2 and that will be the end of the problem.

Actions #26

Updated by Hammel 10 months ago

  • Description updated (diff)
Actions #27

Updated by Hammel 10 months ago

  • Description updated (diff)
Actions #28

Updated by Hammel 10 months ago

  • Description updated (diff)
Actions #29

Updated by Hammel 10 months ago

This gl-cairo-cube example is likely to be the code I need to review to get Cairo working with GTK+ and OpenGL. However, it requires adding GtkGlExt as a new package to Buildroot so that apps can have direct access to it.

I grabbed the source and put it under src/ximba/work/gl-cairo-cube. It doesn't build out of the box. I managed to get gtk2 to work with it but it's not finding gtk/gtkgl.h.

Actions #30

Updated by Hammel 10 months ago

That code expects gtk+3 and gdk3. So that isn't going to work for me.

Actions #31

Updated by Hammel 10 months ago

I might be able to do this with SDL instead of Cairo (or with Cairo).

If you get an SDL window in the GtkDrawingArea and use the SDL window with Cairo (see gl-cairo-simple) that might work. Although gl-cairo-simple is using GL to draw and I want to use Cairo to draw into OpenGL, for speed improvements and offloading display stuff from the CPU.

Actions #32

Updated by Hammel 10 months ago

This one seems to work if I use either of

window_id = (void*)(intptr_t)gdk_x11_drawable_get_xid(gdk_window);
window_id = (void*)(intptr_t)GDK_WINDOW_XWINDOW(gdk_window);

instead of

window_id = (void*)(intptr_t)GDK_WINDOW_XID(gdk_window);

I would then need to get the SDL surface for use with Cairo.

Actions #33

Updated by Hammel 10 months ago

Getting back to straight Cairo/GTK/GDK (no SDL), I've found that you can do this in Cairo.

        cairo_surface_t *surface =
            cairo_image_surface_create (CAIRO_FORMAT_ARGB32, 240, 80);
        cairo_t *cr =
            cairo_create (surface);

To get a surface that we know is OpenGL, we can use this example which uses the undocumented GL backend API cairo_gl_surface_create() along with a GLX API. This is, in fact, the exact example I was looking for that will definitively get me an OpenGL backend for Cairo drawings in the Gtk widget.

static cairo_surface_t*
create_source_surface(int width, int height)

    int rgba_attribs[] = {
        GLX_RED_SIZE, 1,
        GLX_GREEN_SIZE, 1,
        GLX_BLUE_SIZE, 1,
        GLX_ALPHA_SIZE, 1,
        None };

    XVisualInfo *visinfo;
    GLXContext ctx;
    struct closure *arg;
    cairo_device_t *device;
    cairo_surface_t *surface;
    Display *dpy;

    dpy = XOpenDisplay(NULL);
    if (dpy == NULL)
        return NULL;

    visinfo = glXChooseVisual (dpy, DefaultScreen (dpy), rgba_attribs);
    if (visinfo == NULL) {
        XCloseDisplay (dpy);
        return NULL;
    ctx = glXCreateContext (dpy, visinfo, NULL, True);
    XFree (visinfo);

    if (ctx == NULL) {
        XCloseDisplay (dpy);
        return NULL;
    arg = (struct closure*)(malloc (sizeof (struct closure)));
    arg->dpy = dpy;
    arg->ctx = ctx;
    device = cairo_glx_device_create (dpy, ctx);
    if (cairo_device_set_user_data (device,
                                    (cairo_user_data_key_t *) cleanup,
        cleanup (arg);
        return NULL;

    surface = cairo_gl_surface_create (device,
                                       width, height);
    cairo_device_destroy (device);
    return surface;

Now I need to integrate this example to launcher, including verification of the cairo_surface_type_t returned from cairo_surface_get_type() being CAIRO_SURFACE_TYPE_GL.


I tried to build this example on Debian but it won't build because Debian is missing cairo-gl.h. However, that header is available in the 1.16 release in the Buildroot staging tree (it's under /usr/include/cairo/cairo-gl.h). So I can create an app package of this example to build against an RPi staging tree as a proof of concept that this code would work if ported to Launcher and/or PiNet.

Actions #34

Updated by Hammel 10 months ago

Ported to an app build. That worked. Copied to a RPi 2 v1.2 and ran it. It works! It doesn't do anything, but the window opens to black and there are no errors on the command line.

Next step, add the code to test if it's an GL backend. That can written to stdout for this test. It would be nice to have the window do something too.

Actions #35

Updated by Hammel 10 months ago

This doesn't actually work. The sample program fails to get an XVisualInfo pointer in create_source_surface(). It's not clear why this fails but it's a call to glxChooseVisual() that fails.

An alternative method is available from an SDL github archive in the SDL_x11opengl.c:X11_GL_CreateContext() function.

Actions #36

Updated by Hammel 10 months ago

This simply isn't working. Another option, for animations at least, is to use dispmanx to overlay an animation over the app. For example, create a block for the animation in GTK and then use the dimensions of that block to launch an external dispmanx program to do the animation, with keyboard input staying with the GTK app.

Some references:
Actions #37

Updated by Hammel 10 months ago

  • % Done changed from 50 to 60

I believe mixing the hello_tiger code (from gles repo) and Eye of Pi code can get me something like what the existing animation in PiNet does. Also, dispmanx has a check for RPi4 which I'll need because RPi4 doesn't support dispmanx. That means the code using Cairo in PiNet animation will be used instead on RPi4.

It's questionable if any of this is necessary now since
  1. Launcher is reasonably fast as it is now on all boards.
  2. DispmanX isn't supported on RPi4 which means going forward I wouldn't be able to use it anyway (on new boards).

But I might want faster redraws of the launcher icons if the list of icons grows. I have no plans to add apps that need an icon at this point, only modifications to existing apps. So maybe the idea of adding EGL (outside of the existing gles component) isn't required after all. After all, the gles sample applications build without adding EGL to Buildroot.

Correction: It seems that if I run in 32bit mode that the gles samples may run with DispmanX. I just need to try them and see if they work.

Actions #38

Updated by Hammel 9 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 60 to 100

The animation demos won't run on RPi4 however the hello_dispmanx DOES work. However, the demos are running on fb0 so the TFT doesn't show the display. I ran the hello_tiger on rpi3 with no hdmi and a TFT connected but nothing is displayed, but the RPi4 had both hdmi and TFT (with a Media System installed, so the TFT wouldn't work anyway) and the hello_dispmanx shows on the hdmi.

There is a whole discussion on why this happens on the rpi forums, but the gist is that it's not worth the effort, for the following reasons:

  1. There will be very little displayed on the TFTs that requires high frame rate refreshes, other than
    1. omxplayer, for webcams, but that's already working at 30fps just fine.
    2. pinet animation, which may just need to be updated to run faster - ie the algorithm is the problem, not the cpu or display.

rpi-fbcp, used with the TFT displays, works fine for omxplayer at 30fps. I might get as much as 40 ro 50fps, but it's unclear if a user would notice the difference. But the fact that omxplayer does work with rpi-fbcp but the demos do not means there is additional work just to make it show up on the TFT.

So while I can add some DispmanX support for non-TFT systems, it really isn't worth the effort right now. I just don't have the use cases for it.

I tested musicfe's carousel with the Cairo + EGL update and compared to a build without Cairo+EGL and there is little difference on an RPi 3. The only way to make the carousel work better would be to switch to a DispmanX interface. That seems like unnecessary work for now. I can probably speed it up a little by improving the algorithm or even pre-allocating the icons.

So there doesn't seem to be much immediate benefit of adding OpenGL support without making the apps more complex, which would violate my own KISS requirements.

So I'm dropping this feature from 3.0.

Closing issue.

Actions #39

Updated by Hammel 9 months ago

  • Status changed from Closed to In Progress
  • Priority changed from Immediate to Normal
  • % Done changed from 100 to 70
  • Severity changed from 02 - High to 03 - Medium

Re-opening (but lowering priority/severity) after discovering there are overlays that can be enabled that might help with OpenGL/DispmanX usage.
See <archive>/firmware/master/boot/overlays/README (that's in the download archive for rpifw). It describes all the overlays.

I need to retry all of this with the correct overlays enabled.


Also available in: Atom PDF