I've pretty much never had to worry too much about 'installing' because I avoid that whole process hehe!
The main issue, I guess would be the save game folder, stored somewhere. But at the moment, most of my games have the old school system of 'level codes', so no need to saved games (yes, somewhat frustrating considering it's a bit restrictive) but the game can just be removed by deleting the folder.
Or am I doing it wrong? :/