Really clever idea that I think could have some potential for iteration.
At the moment I think the way you interact with the dungeon is a bit too simple. There should be more reason to switch between the modes. The best way to play atm is to get a really short path to the dungeon and then just rush through it while skipping all the enemies. The actual dungeon and the combat is honestly the one part that I enjoy the least in this game, but if there was a way to tie it in more with the match 3 puzzles than I think there is potential here.
Great proof of concept tho and extremely unique.