You are mixing two different things. It was claimed that we are alone or that it is probable that we are alone. Alone in this context means, we are the only intelligent species. At all.
I argued against that, because the univserse is big. And even if you guess very pessimistic, the numbers are against that proposition. We know that life can happen, we know that intelligence can happen. If it can happen, it will happen.
But the universe being big also means two other things. It could be that the next life bearing planet is far away. And that we would be unable to see it.
The only thing the fermi thingy shows, is, that there is not an expanding colonizing tech civ near to us in time and space to have colonized us. It is neither a contradiction nor a paradoxon.
But as I previously wrote, even if you would lower the chance for an intelligent species down to 1% per galaxy, you would still have millions of those species. And none might be in the nearest 50 or so galaxies, including our own. For all we know it even the colonizing thing could be happening in Andromada but not (yet) in the Milky Way .
In my opinion the biggest chance lowering thing is, that we are among the first, due to there not being enough heavy matter for rock planets to form.
So you can easily shift the goal post to, we might be the first intelligent species capable of leaving their planet.