Yeah, it's just a lot of luck and time really, probably the most annoying thing you have to do in the game. You only need to do it once though since after you get black key you're golden. I recommend raiding that trash room and just leaving a trail for him to follow.