I found the issue. Apparently when you release shift and it reduces your speed to normal, if you entered a building or activated something while sprinting, it would activate twice, taking speed from you indefinitely.
This is now fixed for the next update :)