It was the internal pull up and I didn't have the TRISIO registers set up.