Because its own energy is getting weaker and weaker. The reason why music magicians disappear is that after they leave the world of music magic, their own energy will become weaker and weaker until they disappear. Before leaving the world of music magic, the music magicians drank the water in the pool mixed with the juice of the Sleeping Flower that Rias stole from Gemini, causing memory confusion. Only Rias became a conscious person.