Not sure how to explain it at the moment, but the values above 255 would have to have 256 subtracted out from them (add this back in when you do the PWM statement) to keep it all below bytes. Let me try to explain this.

The other math is just for your lookup steps. first table of 256 values (your numbers) works the way you have it. To lookup the next 256 your adc value will be 256-511, so subtract 256 from this value to give you the approporiate position in the lookup table. Use your data for the second lookup table. The only catch is when the values in the lookup table jump above 255. I would simply subtract 256 from the values you calculated and use those new numbers in the lookup table. Then in your PWM statement if the ADC valaue is above 490 (the point in your table that goes above 255) add 256 to the final duty value. Hope this makes some sense, maybe I am overlooking something.