I do not have a good answer at the moment. I mostly use SD card for logging and yes the FAT file system can be a bit slow.

How often is the display updated in your app?