That's very strange, every "building" is a giant 3d cube with a huge box collider, basically you're going through the box collider. Don't know why.
I know how to solve, I had to let the colliders only in the top part of the boxes, but I had no time for that.