They do actually care - if you are slow they turn red and you get fewer points for transporting them. They don't leave because, like, they still need to get where they're going. Thanks for playing!
Edit to add: I considered having a "ugh I'm taking the stairs" mechanic but I didn't have time to implement it!