Hi,
The MAX6126 comes in various versions but I suspect you're using the 5V one, since you're feeding in voltages above 4.096V I'm guessing you're using the 5V version of the MAX6126 - is that correct?

Your numbers don't quite add up.... With a VRef of 5V an input of 3V should give you a reading of (1024/5) * 3 = 614, you're getting 678. Have you followed the datasheet for the voltage reference regarding supply and output bypassing and have you verified that the VRef voltage actually IS what you expect it to be?

/Henrik.