It might depend on what Python libraries you have installed on your Python instance. If you have numpy, scipy, etc., they can get pretty big! If it’s just pygame it’s usually manageable.
There’s a way to exclude specific libraries in the pyinstaller config file, or you can set up a fresh Python environment through virtualenv or similar.
Hope that helps!