You need high affection before he shows you the library, the thing that tripped me up is chasing after Tyson after orlando's confession. You need to stay and talk to Hoss instead.