Personally I like games that offers multiple types of challenges to the player and to take mental notes of the destinations is an extra layer of challenge. When you start to sort the order of the destinations mentally you also start to decrease the amount of stops you need to make.
I realize that stopping at every floor is one of the first things any one would do in this game (my wife also did that on our play testing sessions). I believe that's because it is the starting point of the short learning curve this game offers. Play a few more runs and try to increase your highscore and you will change your strategy entirely.