It's all tied to affection. It's a fairly high requirement, but essentially all it comes down to is you being able to romance the lion enough before he shows you the library.
Alright thanks! I'll try to do that.
Edit: Finally got it after 2 more retries! It took up like 7 save files in total to get it, worth it.
