Is it only external Xtals/resonators that throw the internal timing off?
no not at all.

the only time it fails is when DEFINE OSC does not match the real osc frequency.