The first thing I would do is check your input sine wave and make sure it always exceeds the 2.5V ref level with enough margin for the op-amp. That would be the first thing to suspect when doing this kind of conversion. Another thing to check is to make sure your sine wave offset level does not drift over time. Also check the response of the op-amp over your frequency range.