Nah there's no other way to remove it, unless you want to edit your save manually to remove it (although that can break your save if you mess anything up, resulting in starting over). Also, you can throw away the key after you acquire it; and no other merchant besides the Blacksmith seems to sell it in the game; so be careful not to throw it away once you acquire it.
Yeah the Arena is in the "Town of Monsters"; which is the next town, and past the Mermaid. Depending on the RNG of your PC's Encounter spawns (when creating your PC) you might have to defeat her or sex her to pass (unless your PC has high enough Charisma), so it might be advisable to just level up in the starting area for a bit.
I'd suggest not buying meat tbh; sleeping at the inn (in both the Town of Salijam and the Town of Monsters) yields more food per gold than buying meat. There's also the Dryad encounters which if your PC has high enough Charisma they'll be given food.
If you can level up your PC in the starting area (and get lucky with cache spawns in that area as well), and get sufficient food from the inn; you'll be able to traverse to the Town of Monsters (even if you have to defeat the Mermaid to get to the town it won't lock you out of sexing her later), and can then get the 100 gold you need (although I'd suggest being around level 10+ before attempting to fight in the Arena; pref 15-17). Your PC will need the Sunderer perk to negate armor, and pretty decent Strength (usually 5 or 6 Strength should suffice) to get out of Grapples.
But yeah, as JessHorserage said, if you go to the Brothel Madame and start her first quest you can request your reward to be more gold; which will instantly yield 100 gold upfront before even completing the quest.