Do you have normal maps also assigned?
Yep.
Since I have multiple sprites that change depending on the player input, everytime the sprite changes, I get the sprite name and add an "_n" then use asset get index to change the normal map to the sprite that is named the same as sprite index with a "_n" after it. I did notice that when I updated the material map in this way (instead of just setting it to sprite index in the variable defs) the result was much closer to the original set up I had with material sprites, but it wasn't exact. I might be able to tweak the metalic/roughness set numbers to get it to where it was.