Have you tried just converting one channel and not multiplexing the ADC? I have had issues before because a previous channel precharged the ADC and would see the same kind of "bias" voltage even though the channel being converted is grounded.