It's just a personal preference, I think.
I'm fine with games being 'games' rather than 'simulations', and taking liberties with realism... object management is something I can usually happily do without... which is why I'm usually okay having an unlimited inventory. What you can walk around with, in most games, is pretty impossible, even with an inventory limit! :)
However... there have been occasions when I've implemented an inventory limit, or when I've though they particularly worked well in a game... If the limit is there as part of the puzzles... perhaps requiring clever ways of porting objects between locations... then I think that's fine.
The majority of games I've played, during my almost forty years of adventure playing, have had inventory limits. They've never put me off a game.