Hmmm... so the energy is like a food source or maybe a catalyst to charge a spell for those that can wield that energy. So does that mean the source of energy could represent a specific energy source to charge a specific spell or does that mean that any energy source is consistant to any magic?
This is my random theory based on what you added granted it might be still be off by a lot on what this energy source can be:
If they collect some sort of energy such as an earth based creatures and earth based living beings(something like golems for example granted I think golems wouldn't exactly be living creatures but could be some anomaly that gain sentient life from nature or summoned familiar or artifical life created by some elemental user as some forbidden taboo on creating false life by some means of sacrifice whether from their life force or to some physical extraction from another energy source(elemental user or even a crystalized version of elementals) maybe? Nature energy would have a basic mixture that is dominant in different terrains like forests having a strong abundance of water and earth properties/affinities while places with fire energy would be more from volcanic or even heated places like vents under the water making a special case of fire/water. Basically the strength and complexity based on terrain having a high level of energy would be a diffcult situtation but trying to extract it would be considered an advanced process for only high leveled elementals or having two opposing elemental specialist working together to gain energy from an advanced area because if the elemental didn't have mastery over more than two energies and try to obtain it alone then extraction wouldn't work well for those of a single specialty even when it has their specialty entwined together. I'm basically saying that you have to get the whole package deal of both energies or the risk of losing it all especially if the energy they have trouble absorbing could counteract with absorbing their main energy specialty so they would be exposing their very weakness to the energies in a basic battle of disruption and absoption of the elemental's energy so the terrain would be considered such a cruel mistress eating away at your energy instead of them harnessing the energy or maybe the elemental just become some mindless rogue causing havoc from craving energy from any source they can get to until nothing is left because their very core is being eaten away unless they constantly feed because their control of energy isn't consistant(the elemental user would have control to contain their energy just like a balloon holding water but for a mindless elemental they would be a balloon with holes which energy doesn't substain well and even moreso if used against living beings or at any point). Taboo user would have the same effects but instead of being mindless they can freely absorb energy but at the cost of going through some sort of ritual like having to sacrificing specific things like those crystalized elementals of each source and somehow obtain a source of light and dark energy on a day of something like an eclipse or some alignment of all energies at their peak(such as the crystalized elemental used as a storage for energy from some day or month that is strongest for a specific element and for each elemental that has been crystalized will have specific colors of speciality and is depended on how bright the crystal shines so if the crystal is dull it lack energy and if it is corroded it means that it has to go through a purification which could come from light element specialist if its possible to revert the effects or setting it through a blood ritual to corride the energy even more. Maybe emotions could play a part to whether the purity could be for an elemental? The stronger the negative emotion the darker the color gets or it could go the other way the more positive the emotion the brighter it gets. kinda like if a water elemental's specialty is water but if they failed to collect energy they become crystalized but the last emotion they felt was one of regret/revenge then the bluish color would darken but if they felt hopeful/selfless then it would shine bright blue. So in order to have use of forbidden element uses would require both colors(light blue and dark blue) which could convert light and darkness as the source of energy they may need if there wouldn't be any light/dark elementals but are derived by emotions or worse case for gaining darkness could lead to the blood of the corrupted elemental since they already empty husks runnng amok and for the light energy would be a little harder to obtain but finding a source of a blossom of life of a elemental becoming such as a fledgingly awakening to their new elemental specialty basically someone who has yet to become an elemental will have a null based energy which hasn't manifested yetand have to go through a right of passage such as a pilgrimage, meditation, surviving hard climates, or even physical type of endurance like holding their hand out in flames or being dunked in to freezing water given on requirements on specific criteria the fledgingly seems to react to such a reactions to stones glowing or being born on a day where wind is at its peak to name a few and are given confirmation from their intiation to the ritual if they really quialify for that said energy control and if they can't then they die from failing to become an elemental. So thoses wanting to have control over life must exceed their own weakness by sacrificing others to gain unimaginable power over every elemental specialist and can take control of elementals abilities and even disrupt the very fabric of time to an extent if they can extract each element's potential by delving into the plains of what we see in the picture of light and darkness so basically being able to fiddle around in the world of light and dark as a way to change pieces of fate but at a cost of being exposed to a chance of being destroyed if they linger within the world as they rewrite or erase existence. So in order to change fate or hold time in stagnantion wouldcome at the cost of exposure and being weakened within the realm where beings beyond the elementals derive from. So if you try to change anything you may very well erase your existance and be expunged from both the physical and spiritual plains of energy. So not only will you gain strong powers but you may also end up being the weakest if used it wrong. So limits are playing into not being consumed and not severing your powers for good physically and spiritually not being erased from existence as well as the changes made which may as well erase those that have been created or shifted an anomaly in the world.