Sounds like quite the sophisticated setup. But, I guess, very well worth it, if you can "compile" the books in the end. Thanks for the insights.
It is quite complex, but after setting it up I can focus on content completely, since typesetting will be done automatically by Pandoc/LaTeX.
It has its own shortcomings, but it can be really satisfying to use.
