Hearing that your issues were on the same track intrigued me so I did a little bit of sleuthing.
The error I was getting in my browser's console was "The buffer passed to decodeAudioData contains invalid content which cannot be decoded successfully."; this does not help a lot but confirms that the issue is with specific files being unreadable for Firefox (or at least the AudioContext part of Firefox, as opening the file by itself in a new tab works like a charm).
Using the sample code from MDN's documentation (https://developer.mozilla.org/en-US/docs/Web/API/BaseAudioContext/decodeAudioDat...) I managed to reproduce the issue locally, which definitely excludes the issue being from your code (yay !). I still do not know exactly what went wrong with those files, but simply re-encoding them with a newer version of the encoder (Lavf57.56.100 => Lavf58.29.100) fixed the issue for me (I did that using ffmpeg, ffmpeg -i ladiesandgs.webm ladiesandgs_fixed.webm) made the previously unplayable files work just fine in Firefox.
The files I noticed were affected and confirmed fixed with this workaround are "Ladies and G's", "Take a amen break", "Heatwave" and "Memories".