I am not sure if this is a bug or not, but you can easily level gems, even if you can’t use them. All you need to do is place them in a socket on a higher level character. As they higher level character kills mobs, the gems will level very fast as well.
However there is one thing to consider, which is the higher mana costs when you raise the gems. If you do raise the gems past the amount of mana which the lower level character has or is able to harness, then the low level character will be unable to use the gems or have to take a mana potion after each use.