Abusing the Sixel support in Windows Terminal for fun and no-profit

[mrange]

Hi.

Since Sixel support was added in Windows Terminal by @j4james and @DHowett I have had fun tinkering with it.

That culminated in a Windows Terminal demo I submitted for a demo competition this weekend (got 2nd). I thought some here would appreciate this odd use case for Sixel Support in Windows Terminal :)

https://youtu.be/004ng70swbY

(Sixeling starts at about 1:15 if you don't want to see the classic terminal effects).

[j4james]

I cannot express how mind-blowingly awesome this is! Reminds me of the days when I used to download demos from a BBS over a painfully slow modem connection. Congratulations on your placing in the competition!

I just want to point out that much of the credit for our Sixel support is due to @lhecker, who implemented the DirectX renderer, as well as the conpty passthrough functionality, without which we wouldn't have been able to get Sixel in Windows Terminal.

[lhecker]

Without exaggerating, this is the nicest thing I've ever seen anyone build on top of something I've worked on in my career. Thank you so much!

I'm surprised that it works as well as it does. I did not really spend any time on intentional performance optimizations for Sixels or its integration into our text renderer. What makes me incredibly happy as well is seeing the last animation at the end: https://www.youtube.com/watch?v=004ng70swbY&t=196s
I spent a lot of time fine tuning those builtin glyphs and even now I'm not fully happy with them yet. But seeing it used in such a beautiful way makes me just so so happy.

---

By the way, and this is only loosely related (to those builtin glyphs), but I recently attempted to implement a tiled, deferred text renderer using a compute shader. My goal is to fully solve the glyph overlap issue.

It works! I don't know yet when I can finish it and submit a PR as I'm currently working on a yet-to-be-announced project, but hopefully it should be within this year.

[mrange]

Glad you liked it. It was fun pushing Windows Terminal sixel capacity to the limit. One challenge was finding effects that fit "in the budget". The "cubes" contains lots of details but there are large areas of black to reduce the sixel count. The full screen tunneling effects have areas of the same color to reduce sixel count.

One thing I wanted to do was mixing text and sixels but it caused alot of flickering on some machines I tried. Not the average use case I suppose :)

Will you incorporate that compute shader into Windows Terminal? I suppose it will help performance but does it also solve the glyph overlap issue in general?

Just FYI, the last effect uses Wave function collapse to generate the grid and in case you like to see some undocumented and hastily written code you find it here: https://github.com/mrange/impulse-gerp-2025/blob/main/src/terminal/effect3.cpp

[j4james]

One thing I wanted to do was mixing text and sixels but it caused alot of flickering on some machines I tried.

@mrange FYI, one trick you could possibly use to avoid flickering is double-buffering/page-flipping. Basically you do your compositing of Sixel and text on a background page, and then only flip that page to the foreground when it's ready.

You first need to disable page cursor coupling with DECPCCM so you can switch to another page without it becoming visible. Then use PPA to change the active page and begin writing your new content there. Once the new frame is ready, you re-enable DECPCCM to make that page visible.

And if you need to reuse content between frames, you can copy from one page to another with a DECCRA sequence.

[mrange]

Ahh, thanks this is very interesting. If it works there might be a sixel demo 2 :)

[lhecker]

Will you incorporate that compute shader into Windows Terminal? I suppose it will help performance but does it also solve the glyph overlap issue in general?

My hope is to make it the next rendering approach for Windows Terminal. It's similar to the current one and doesn't require too many changes. It'll raise the requirement for the D3D11 renderer to Shader Model 5 though.

I don't think it'll make it faster. I expect it to be roughly the same on most hardware, since the current approach is already extremely lightweight (<200us frametime (>5000 FPS) for 3600 glyphs on my PC) and the new approach should be similar. But of course I'm hoping to be wrong and that it saves some more power on mobile GPUs. 😄

It should solve the glyph overlap issue in every case since we'll have the full snapshot of all glyphs and the entire viewport state available in the compute shader call. (The downside is that we have to restart the rendering from scratch if the glyph atlas runs out of space and must be recreated. Right now it can just incrementally build the image since we make use of the Output Merger.)

[lhecker]

Oh and, the flickering is caused because you exceed the buffer size of the pipe and so you get in-between frames rendered. The issue can be solved as @j4james suggested it.
Another solution, one which is not available in Windows Terminal yet, is to use the synchronized update sequence: #8331. Now that ConPTY is passthrough, it should be moderately easy to implement by calling into the renderer from inside the VT dispatcher and tell it to stop rendering for up to 100ms or so, until the restriction gets lifted again.

[DHowett]

I cannot overstate how cool this is. Seriously, thank you for sharing it!