Yes, the universe is big enough to have intelligent life, so that we've found billions of habitable planets out there as you said, at this point according to all this vast amount of possibilities we should've already found at least a single clue about another civilization elsewhere, but still no evidence, why? We don't know, thus the Fermi Paradox.
Whatever unlikelyness you attribute to it due to lack of data
It's not lack of data, I already gave you two possibilities in my previous posts. One of each is the Great Filter, which points out that intelligent civilizations usually face catastrophes over period of times that annihilates all civilizations in that planet, be it nuclear war, global warming, asteroids collision, death of a star, and so on...
As for being dangerous, assuming they existed and they are militarily more advanced than us, yes that's more dangerous, but my point is not a "if" scenario, my point is the likelihood of that scenario, and yes comparing the likelihood of human killing themselves and alien killing us, humans are for more dangerous.