Thanks! That was my first ever attempt at Godot and I only had 2 evening.
That was super fun to learn something new and taking part in jam was a huge motivation booster!
I'm using queue_free() but the issue comes from where I use it.
Walls are RigidBody2D with Polygon2D and CollisionPolygon2D inside.
Currently when walls are fractured I use queue_free() on the polygon (that gets split into fragments). Simply using get_parent().queue_free() solves the issue (because whole wall including CollisionPolygon2D gets removed).