Like Spedu said, the quest isn't finished yet, but to avoid failing it in the first place, you need to talk to the knight near the entrance inside the townhall in Safen, and hand him the 'lost key'. You have to do this before you leave Logtown (getting the cutscene with the priates at the crossroad means you've already gone too far), because by then, Sword will no longer be in Safen. You can check the walkthrough for details.