If nobody sees you do it, nobody will go hostile. That being said, if you're continuously able to kill villagers next to their neighbors, there probably is a bug. That or your stealth is insanely high.
