I've run into this problem before. The ~ operator returns a 16 bit result. If you store the result into your one byte var, the upper word is truncated thus removing the uneccesary bits.

I suspect you were doing something like:
HSEROUT ["0-5=", ~Result+1]

If so, that would cause exactly what you describe. The pseudo code posted however does not suffer from this problem.