It seems more of a refresh or memory issue as the first time I got this was after I interacted with everything in the starting house at least once or twice just before the tutorial fight but this happened more often when a fight is drawn out, as it occurred the most during the first fight with the silkworms at the bridge when I was goofing around trying to get caught by them and used taunt for most of my turns. I hope this helps with figuring it out.
On a more positive note I love how your characters interact with the world around them and each other as it adds real depth to them and makes them feel alive which is something that is very rare in games like this so keep it up.