Can not say exactly but FAT32 should be a little slower, larger sectors to read I guess.

Can you have the data on the card in CSV format instead of line by line? Be a little less reading.

Might post your code, maybe someone will see something?