I'm fairly certain I was asked this before somewhere and I'm sure I had a reason to remove the highest... but that was a little while ago and I've slept since then.
Youre right though, if you removed either the highest dice equal or lower than the result, or the lowest dice equal or higher, it does sound on paper like it would make more sense. Feel free to do any of the above, I doubt it will unbalance the game too much!