After some search using a few different keywords yielded to the answers I was looking for from @tkaiser here. From that discussion, it would appear that the SD card speed was limited for stability reason.
Hi, Is there any reason why the card is only initialized as high speed SDHC on this board? On my phone, it is initialized as ultra high speed SDR104 SDHC. I'd like to know if there's anything I need to do to maximize the card's performance on this board. Thanks.