Thanks for replying!
I assume it was the timeout of the last villager in the hut then. Maybe I just didn't understand what was going on. It was just that I couldn't do anything for a bit because no one was in my queue, but a blue timer was there.