Thanks.
Turn out the bug is due to FindGameObjectWithTag function, now it is fixed. Iwill avoid using that from now on.
