>>This works fine, but I think it's automatically turning my lower 4 pins into outputs?

No - it does not affect the lower 4 pins if they are inputs ... the value on the inputs are what they are and not affected by the writes to the port. Writing to the port also does not affect the TRIS register unless you have errors in your code.