The code you posted seems to make sense. The interesting thing here is how you compute sheildcheck, and you didn't include that bit of code...
Also, just to get the obvious out of the way: obj_damageapply still does HP damage and not shield damage, right? (So you didn't accidentally change that one as well when you changed all the variables)
(Also note that your two sheildcheck checks are >0 and <1 so both of them will occur for values between zero and one! Is this intentional?)
trg here is the target monster object and we've already checked that it's alive, so you really only need to check if amp_read_var(trg.amp_id,amp_SHIELDHP) is greater than zero: if so, deal shield damage, else deal regular damage.