I feel like you were moving character by adding to its position (it is like teleporting character), and that can cause problems because character can end up in wall's collider, wall collider will try to push player back making it look like you are bouncing. I suggest to use addForce or at least changing body's velocity each frame. With that player will collide with walls smoothly without bouncing.