Hi Ubojan!
What I did was instead of using the physics system to detect the grabbing of the levers I write my own. When grab strength is above 0.9 I check if a lever is inside the hand area and, if so, the lever gets to a grabbed state and adjust its rotation to match the hand position, always lerping to keep things smooth. I personally like to avoid using physics unnecessarily since the reactions of rigidbodies are too unpredictable.
Thanks for the textures tip. For this jam I just had time to make a quick prototype and the art side was hugely overlooked. :)
Cheers!