I now believe that the issue is that the first else-statement gets executed regardless of which if-statement it is in. That means that nested if-statements are not possible in this way. I ended up solving it with a procedure instead. I should clarify that this is a loop script.
procedure trySpawn
willSpawn = 90
diceRoll = RANDOM(1, 100)
map.spawnPoints=0
if $diceRoll > $willSpawn {
posX = RANDOM(0,64)
posY = RANDOM(0,64)
entity spawnat "bat_angry" $posX $posY 0
map.spawnedEnemies++
}
end
if $map.spawnPoints > $map.spawnCost {
call trySpawn
} else {
map.spawnPoints++
}
status "spawned Enemies: $map.spawnedEnemies"