Yeah, I agree with your thoughts here. The concept does a lot of the heavy lifting. I could imagine a fully fleshed out game where this concept could really shine.
