This is my hacky fix that did manage to align them correctly:
Basically adds up the width of the characters in a line, and if its even or uneven it adds an additional 0.5f to offset it.
I'm sure I'm ignoring a bunch of use cases that you may understand better, but for now this kind of fixes my issue :) Hopefully it can be added as a feature!
Sorry to bother and thanks for the help! If i do find issues with this I may be back :D
EDIT: Im still not there cause spaces screw me over, but im getting there haha
EDIT2 : Nevermind, it was something else! It still works!