Clay is a UI layout library written in C, and this video gives a high level explanation of how the flexbox-like layout algorithm in Clay works.
The library can be found at https://github.com/nicbarker/clay
If you're not familiar with Clay, a detailed introductory video can be found here:
https://youtu.be/DYWTw19_8r4
If you would like to learn a bit more about how trees are implemented, see this video:
https://youtu.be/KwBuV7YZido
The paper on constraint solvers: https://arxiv.org/pdf/1401.1031
Facebook's "Yoga" layout library which was mentioned: https://github.com/facebook/yoga