If you walk around to the castle wall area, you can walk back around and drop down to where he is. so no, its not an offlimits area
