Well It does not get called less, it just gets called at constant intervals. Every Physics step is calculated in fixedupdate. Since you try to do a physics call it should go in fixedupdate.
In normal / late update the call interval can vary depending on the framerate thus a collider may not be correctly moved according to phyiscs yet.