I looked into it some more, and it looks like weblate is messing with file permissions every time it updates one of the language files..
After looking into it a bit, I just set up a cron job that fixes file permissions every 5 minutes. So, there might be a 5-minute period between "new translations being committed" and them being available, which is not ideal, but it's much better than them remaining accessible for days!