Theory: Gem civilization is led by four high ranked diamonds. All the other gems serve under one or the other depending on their color. We already know about Yellow Diamond, and it’s theorized that Rose Quartz was once Pink Diamond or possibly was a high ranked subordinate of Pink Diamond. If that’s the case, then the other diamonds likely represent the CMYK or RGB color model-
CMYK - The four ink colors (cyan/magenta/yellow/black) used to produce every other color in full color printing
RGB - the three colors of light (Red/green/blue) used to display every other color in electronics such as computer screens and digital projection
The gems bodies are literally made of light, so it’s more likely that they follow the RGB model. Gems with the overlap/secondary colors are higher ranked then gems of the original red/blue/green. With white as the penultimate color because it’s a combination of all the rest. This explains why Yellow had control over different colored gems - red and green combine into yellow.
Diamonds have domain over their component colors.
If there are four diamonds based on the secondary colors in RGB, Their armies/minions would be drawn from their base colors (two out of the three).
Yellow Diamond: Red & Green
Cyan(Blue) Diamond: Blue & Green
Pink Diamond: Red & Blue
White Diamond: All or Nothing
(this post got a bit long so more under the cut)