Hi GeekyTime.
So, there's a bunch of issues happening all at once. I have noticed a significant change in tracking quality since firmware v23 hit.
A really good base test is just to load a default oculus home environment, disable the guardian, and just walk around carefully to see if/where tracking falls apart. Up until two weeks ago, the only problematic areas of my own apartment were dark rooms, and near large mirrors... but i've been noticing new issues recently, nothing game-breaking for me, but the tracking will jump to a slightly of a different position, usually snapping back to the correct alignment once i take a step out of the problematic area.
Its frustrating because this device was SOLID up until the recent firmware. Not only tracking, i have frame skips and other issues across almost all apps right now. So, fingers crossed, we can just wait until Oculus fixes a few issues on their end. Your device should be able to track reliably in a wide range of different rooms setups, lighting setups, and right now it isn't functioning as well as it should. Open gymnasiums, large outdoor fields, rooms with huge windows.. none of this should be causing problems. The open floor plan layout might be the real issue. For my understanding of how the quest tracks, its looks for points of contrast, edges and other visual markers when figuring out its alignment. I had to help another user who had a problem similar to yours, her issue was a single wall in her living room (plain white, large wall with no details/pictures/decoration) would cause tracking to freak out when she was standing right on front of it. Ultimately, she ended up just mapping a layout that didn't include this area near the wall, but still included most of the living area and a few connected rooms.
So, I'm assuming your problem is similar to hers, but because your place is even more 'open', there's a bunch of different areas where the cameras dont have enough visual info to get good tracking data. If you do the tests without loading into my app, i wanna know if you can always identify the same areas the give you trouble?
You've said you tried the app setup in smaller areas.. is this still within the open floor plan environment? Do you have a normal playspace that you use for playing games, and does the app setup work if you just create a single room mapped in that area? Trying to figure out if it is (a) something different about your space, (b) something different about your hardware, or (c) something about the recent firmware. Sounds like a combo of A and C.
I wonder, if you want to test a really hacky 'fix', could you just make a bit of a mess? Like, clutter up your open floor plan space with a few 'things' in the corners, on table surfaces, just anything to add more visual contrast points in different parts, does this affect the tracking quality? If you do this in a single room, just put stuff in the corners and along the edges of the wall, can you convert it into a 'trackable' space?
We can discuss further, and it would probably help to see a quick photo of the space... so email me at curiousvrlabs@gmail.com and we'll go from there.
Sorry you're having so much trouble, REALLY REALLY hope i can get you past this hurdle and you can experience the neat stuff that i've got going on in there.