Do you mean you don't want the player and enemy to be able to touch each other?
If I understand correctly, I think you may want to just delete the IgnoreCollision part and replace it with something like:
if (//whatever specific way you used to check the layer)
{
return;
}
Or I might still be misunderstanding your problem.