In my opinion, right now the problem with energy management=time management system isn't how little energy gets depleted in travel, it's how there's no real consequence even if we hit 0 energy across the party. All it does is stop my character from opening treasure chest (20 without lockpick and 5 with) and using special attacks, which is negated by having a character with about 6 magic affinity and acid spit special attack since that move doesn't cost energy.
-33 damage sounds like a lot on paper, but with a party fully trained, modded with enhanced muscle and decked out with hammer it is still very easy to steamroll through most enemy encounter in mid to high level areas. Only bandit encampment encounter will give you trouble if you have no high magic affinity character who can acid spit (insanely high damage with a high magic affinity character, apply -armor debuff without energy cost, I feel it'll get nerfed in next build) but that's about it. And assuming you are at the point where you can safely travel through the marsh you already have high enough awareness to sneak through this encounter so once again there's no consequence.
Personally I think it might just be better to cut the lose, and embrace the idea there's no real way to limit how a min/max type of player want to adventure per day. A huge revamp on how to limit adventure a player can do this stage in the development will more likely unbalance the game and make it not fun to play. Once again, just my humble opinion.