It means the lowest of all dice rolled, not just your dice. In the example you gave, no, it would not cross off an ammunition because it's not the lowest roll - the lowest roll is a 3.
Note that if there are "doubles" for the lowest, then you would cross it off. For example, if the rolls were reversed – the opponent rolled a 5 and 4, and you rolled two three's, or if you rolled a 5 archery and 3 bow, and your opponent rolled two threes – then you would lose the ammunition because three is the lowest die roll, and the bow rolled that value.
Also, note that the "lowest die roll" still counts even if there is a tie for highest and dice get removed from consideration for applying effects or generating rerolls. If you, say, roll two fives, and the opponent rolled a 2 and a 1 and those dice are removed, your bow still didn't roll the "lowest" of all die rolls, even though all that are left are two five's and one of them is your bow; the lowest roll was still a one. (Another way to think of it is that the comparison is made before you remove dice due to matches for highest roll.)