With Micha's help I got better results, but stepping back I think this was the wrong approach. Mostly because the distance between hues does not necessarily line up with what you perceive as a different hue. Which I kind of knew, but now I really understand the limits of.