The mini game happens literally at the end of the current build. It's the job the twins give you.
And they appear for me during night encounters at various locations. They appear randomly though. You can still fight them at the Library when you book a room.