Silly question, but have you tied the unused inputs appropriately? You may be getting noise resetting things or sending garbage on the SPI port...