You can still pursue all girls, but the more story progresses in the future, the more you will lose out if you don't narrow it down. It's not a harem for a reason.
