Yeah it is the trailer itself. I have tried and tried and tried to figure out why it does that, but so far I havent been able to figure it out.. I will update if I figure it out at some point!
I haven't dug into it, but could the mass be to low causing it, or could it be in the friction?