Kind of hard to guess from that, but try restarting Steam or attempting to connect again after 15-20 seconds - if Steam takes too long to establish the connection to someone, you'll get a connection timeout, but subsequent attempts should take less time.