Overlay application with electron

[helgeerbe]

Hi,

does anybody of you have first hand experience with electron?

I'm wondering, if it is easy to create simple overlay applications with a transparent background. This could be

• a clock
• weather forecast
• reminder
• buttons for a touch screen

So the application could run above PictureFrame

[jgodfrey]

I do not, but creating an overlay for the frame is something I have on my list as well... Though, I had assumed I'd do it directly in the picture frame app - but I haven't really started any experimentation. I do wonder if an entirely separate app (running in its own app framework) on top of the frame might be a bit heavy for (at least some) RPis, but it's an interesting thought...

[sapnho]

That would be really neat. I had thought about a weather forecast screen as well which you could activate by voice control or a webcam view up on the mountain.

[paddywwoof]

It's possible to make X11 windows "see through" so they will run in front of other windows. But the only instance I'm aware of this being used is pi3d! i.e. you can use glx=True.

If you are using X then you can create a small window and launch it after the picture frame and it will be in front - however if you accidentally "miss" a button on the touch screen then you will swap focus to the picture frame and the button window or clock etc will disappear. [EDIT actually I'm pretty sure it's possible to force windows to the top in X]

On Pis prior to 4 you can run picture frame more efficiently without X just using bcmhost and this has the option of setting layers for the output from different apps. I doubt that electron will work without X (in fact it will probably require chrome or suchlike, certainly node server) but I've never used it so don't really know.

Several people has made overlays for the picture frame for clocks, weather, touch screens etc but in the pi3d code. Currently this pi3d/pi3d_demos#63

[helgeerbe]

I gave electron a try. It's pretty cool.

This is the clock code:

<main>
        <p id="uhr">
</p>
</main>

<style>
  p {
        font-size: 3em;
        margin: 1em auto;
  }
</style>

<script>
'use strict';
(function () {
        function uhrzeit() {
                var jetzt = new Date(),
                        h = jetzt.getHours(),
                        m = jetzt.getMinutes(),
                        s = jetzt.getSeconds();
                m = fuehrendeNull(m);
                s = fuehrendeNull(s);
                document.getElementById('uhr')
                        .innerHTML = h + ':' + m + ':' + s;
                setTimeout(uhrzeit, 500);
        }

        function fuehrendeNull(zahl) {
                zahl = (zahl < 10 ? '0' : '') + zahl;
                return zahl;
        }
        document.addEventListener('DOMContentLoaded', uhrzeit);
}());
</script>

[helgeerbe]

You can open a transparent browser window, an run any webpage you like. chrome is used for the rendering.

[sapnho]

And how does it integrate with Pi3D?

[jgodfrey]

chrome is used for the rendering.

I assume when you say "chrome", you mean the Chromium components (or maybe CEF (chromium embedded framework)) and not the actual Google Chrome browser, right?

[helgeerbe]

no, its just the rendering engine. Electron makes also use of nodejs. So pictureframe could simply exec predefined program.js files. Terminating could be sending a signal like SIGTERM.

Starting and stopping via mqtt/webinterface should be easy.

Backdraw is that, electron requires X. You can put a transparent/invisable window over the picture.

[jgodfrey]

no, its just the rendering engine.

Right, so Chromium. And, now that I poke around on the site, I see Chromium mentioned on the main page... Nice.

[helgeerbe]

Just a simple scetch.
PictureFrame is launching invisable Browser Window.
Dynamic Content is send from PictureFrame to browser (Ajax?)

                                             start
                                               |
                                               |
+------------+                                 V
|            | start
|            +<------------- +-------------------+
|  electron  | sub process   |                   |
|            |               | python web server |
| (basically |     http      |                   |
|  browser)  | <-----------> | (business logic)  |
|            | communication |                   |
|            |               | (all html/css/js) |
|            |               |                   |
+------------+               +-------------------+

[paddywwoof]

If you start the basic http server then presumably you can 'just' fetch stuff in a slightly more elegant way than is done now. https://github.com/helgeerbe/picframe/blob/main/picframe/html/pf_functions.js

Chromium (with v8 along with X) are reasonably large bits of software does it all work smoothly?

[helgeerbe]

Actual I tested it on my ubuntu vm. But I will give it a try on my raspberry 3B.
The average load on my Pi3 is 0.5 with peeks to 1. So only 1 core out of 4 is used for PictureFrame. So 3 cores are idling now. Of course it would depend on the javascript code you will run. But a simple digital/analog clock, or a weather forecast wouldn't consume to much cpu power. At least this is my expectation.

[helgeerbe]

On Pi3 I have to run xcompmgr a composite manger. And have to start electron with the following command line arguments --enable-transparent-visuals --disable-gpu

This works fine. The CPU load for my clock example is not really higher.

@paddywwoof But on the PI3 I was not able to place the window on top of Picture Frame. On my VM it was not a problem to put transparent window in the foreground. Any tips? I tried glx true and false.

[paddywwoof]

On Pi's prior to 4 pi3d doesn't use X, so you can move the display layer down but all the X windows are in the same layer so pi3d is under the desktop. In order to work with electron it would have to be made to run in an X window. I've not tried that for a few years but the changes to cope with the Pi4 might have made it possible... But it might not be.

[helgeerbe]

Finally I got it working.

1. Use FAKE KMS
2. Don't enable composite in raspi-config it starts xcompmngr -aR This is not working. I can't find a R option in the man pages. Run xcompmngr & with no arguments from command line.
3. For electron only --disable-gpu is needed. Without this option, background is not transparent.

It doesn't matter if glx is true or false. If you use legacy gl driver, PictureFrame is always the top window.

Load is around 0.8 (5 min. average). Memory usage jumps from below 20% up to below 50%.

It's nice to have an animated clock on the frame.

I have to find out. How to deliver (based on commands to PictureFrame) different content, to switch between clock, weather forecast, open street map showing location of picture.....

Thinking of something like a plugin structure for PictureFrame to easy extend browser content (e. g. on a touch screen hit empty screen and blend in buttons to show next, prev image)

[paddywwoof]

Ah yes, well done. pi3d checks to see if there is a line starting vc4 in /proc/modules and for Pi3 at least that obviously works. Not sure that Pi2 or zero will do that.

GLX would only be needed if picframe was running on top and had transparent areas, but it's the other way round.

You should be able to poll the picframe http server for json info in the same way that the basic html page does to get responses back from Controller methods. You would have to run it in a js style repeat loop with appropriate setTimeout.

[helgeerbe]

If you like to test. https://github.com/helgeerbe/picframe-overlay

[jgodfrey]

@helgeerbe - I'm curious regarding the advantages you see to this solution as compared to just making a simple (optional) overlay panel as part of the picture frame app itself? On the surface, it seems like a lot of extra machinery (and management of that machinery), to accomplish something that should be rather straight-forward as a part of the app itself.

I haven't looked this over well (and I certainly haven't tried it), so maybe there really is something going on here that would be difficult to accomplish inside the picframe app itself - I'm not sure.

Thanks for any insights...

[helgeerbe]

Hi @jgodfrey . I started with the idea to have some widgets running on top of picframe. Controlled by (start/stop) picframe. I guess it would quite easy to blend in some simple overlays. E. g. clock, weather forecast. But then I was thinking about the user we have. Some might like a digital clock, some would like an analog clock. Overlay should be located upper right corner. Centered in the middle of the frame....

So I started thinking about to make it more universal. Having a plugin structure, where everybody could create his own widget, or tweak an existing one to his needs. Then I read the discussion about controlling the picture frame over a touch screen.

I thought a browser interface would be great. With html, css, and javascript you can do and create every widget you like. But you don't have to think, how to render the widget, how to implement buttons, how to handle user interaction.

When I checked browsers I had to learn, that it seems not possible to make the background transparent.
Then I came across electron. It offers a main controller process to communicate with picframe to launch the reqeuested overlays as simple html files. So every new widget is a simple html, css, js file. If you like, a widget could use node.js inernally, so you can open websockets or subscribe to a mqtt interface (e. g indicating that I ave a call on my answering machine). Or based, on gps info from the current picture you can show a open street map and open an wikipedia article about the location. (OK, just dreaming....)

Backdraw. This is not a lightweight solution. It requires X. But it runs multipel platforms. And createing a new widget is as easy as creating a webpage..

Even to implement a touch gui would be easy. e. g. tapping at the botton of the frame you will blend in some control buttons. Clicking the button will make a webcall to picframe (interface is already implemented) to move to the next picture, tun shuffle on/off ...

[helgeerbe]

Just came across pywebview

pywebview is a lightweight cross-platform wrapper around a webview component that allows to display HTML content in its own native GUI window. You may think of as Electron for Python (minus huge executable sizes). It gives you power of web technologies in your desktop application, hiding the fact that GUI is browser based. You can use pywebview either with a lightweight web framework like Flask or Bottle or on its own with a two way bridge between Python and DOM.

But I don't know, if it is possible to make the background transparent.

[helgeerbe]

Last but not least. Picframe on Pi4 is running as root. So it might be a good idea to run the wigets (which could connect to the world) in different process space with normal user rights.

[jgodfrey]

Thanks for the input Helge - I appreciate it.