I would say the randomness of the rolls is chaos enough. I think giving some control to the player would feel fairer and more compelling, and a higher score more rewarding, but in the end it is your game and your decision. Other people might feel differently.
If you want to remove these 50/50 questions, be careful because there are other hidden conditions, for example "Is the next roll greater than 3 (whether the 3 is fixed or was rolled in a previous turn)", as if I say yes, I win with 4, 5 or 6 and loose with 1, 2 and 3. The opposite is true if I say no. Something similar happens with lower than 4. Good luck!