Actually scratch that for me, I was just being stupid and didn't go to the front door of the manor to leave. Now I got the heart icon.
