Hello everyone,
So, I guess we all love fast storage. The faster the better.
The general rule is usually as follows: HDDs < SSDs < PCIe / M.2 drives
But even so, one could say that there are times when we are actually limited by the protocols in use, and not the speed that current storage technologies can achieve. I tried to break them down a bit (or a lot, like, completely) and find a setup that would theoretically offer the fastest storage possible, taking EVERYTHING into consideration.
Firstly, let me state all the important numbers:
- Sata 3: 6 Gigabit/s % (10bit/1byte) = 600 Megabyte/s
The Sata protocol uses a 10 bit per 1 byte encoding, meaning 2 bits of overhead for every byte.
The Sata bus uses a serial communication through the AHCI protocol, meaning that only one command can be sent at a time. The transmission time of the command is of course much less that the execution time, so you can have a command queue, allowing an ssd to have simultaneous read/writes on different chips, unlike hdds.
Still, the sum of reads + writes cannot be over 600MB/s
- Sata Express: 16 Gigabit/s % (10bit/1byte) = 1,6 Gigabyte/s
Again, 10bit per byte protocol. However, the sata bus is not nearly enough to reach 1,6 Gigabyte per second with a serial communication. To overcome the limitation, the NVMe protocol was created replacing the AHCI and featuring an insase amount of queues and commands per queue (65,536 command queues with 65,536 commands per queue). Yup, that will be enough for quite some time.
So now we have: reads + writes = 1,6Gigabyte/s
- PCIe 3, x1 Lane: 8 Gigabits/s % (130bit/16byte) = 984,6.... ~ 985Megabyte/s
Unlike PCIe 1 & 2, the PCI 3 protocol uses a 130 bit per transfer encoding, meaning 128 bits(16bytes) + 2 bits of overhead per transfer. Also, this is unidirectional, meaning we can have 985MB/s reads AND 985MB/s writes simultaneously.
Now, lets start talking about drives:
- Sata 3 SSDs: 600MB/s read+write through a sata 3 port, pretty straightforward
- Sata Express SSDs: 1,6GB/s read+write through a sata express port
- mSata SSDs: Also known as M.2 Sata SSDs. 1,6GB/s read+write through an M.2 sata port and utilizing 2 PCIe 3 lanes, but controlled by a sata controller. Two PCIe 3 lanes can handle 1,97GB/s unidirectional, but since they are controlled by a sata express controller, that acts as a limiting factor. In other words, NOT AN M.2 but a SataEx SSD, just with a different connector.
- PCIe 3 SSDs: Utilizing the NVMe protocol, the PCIe 3 encoding(130bit/16byte) and using 4 lanes, they can achieve 3,94GB/s unidirectional.
- M.2 NVMe SSDs: The same with the PCIe 3 SSDs, but with a different connector and form factor. Meaning 3,94GB/s unidirectional. Now thats a true M.2 SSD.
- U.2 SSDs: Again the same with the above 2, 4 PCIe lanes and NVMe, but different connector, taking the disk off the motherboard. Also 3,94GB/s unidirectional.
Now, lets see what is the most efficient communication method between these and the CPU:
There are CPU lanes, and chipset lanes. In order to keep this short, I will just say that the most recent CPUs communicate with their Chipset using a speed equal to 4 PCIe 3 lanes, thus 3,94GB/s unidirectional. That bandwidth is then shared by all the devices and peripherals connected to the chipset, which may offer more than 4 lanes under the claim that not all devices will use their full bandwidth at the same time.
USB controller, Sata controller, HD Audio, LAN adapter, and perhaps some M.2 and PCIe ports all sharing 4 main lanes. Pretty sad. So at this point lets assume that all available speed goes to the Sata controller.
About Sata disks:
Sata disks need a Sata controller, which is always on the Chipset.
Depending on the Sata controller, communication with the Chipset is done either through a single channel no matter how many Sata ports there are available, or through multiple channels.
Meaning that on Z270, one Sata 3 perfect disk will have 600MB/s, and 2 Sata 3 same disks in raid 0 will still have 600MB/s. It has only one 6Gb/s channel.
https://www.intel.com/content/www/us/en/products/chipsets/desktop-chipsets/z270.html (block diagram)
On X99, each Sata 3 port can handle 6Gb/s individually
https://www.intel.com/content/www/us/en/products/chipsets/desktop-chipsets/x99.html (block diagram)
So, with the right choice of platform, one can hit 3,94GB/s from Sata (not unidirrectional due to controller) under perfect conditions, hardly the case though.
About PCIe 3 disks:
PCIe 3 disks can either connect directly to the CPU lanes, or indirectly through the Chipset's lanes.
- Throught Chipset: 3,94GB/s unidirectional at best, no matter the number of drives.
This Gigabyte motherboard is proof of that, featuring 3 PCIe 3 SSDs x4 lanes each on raid 0 and hitting scores no more than that.
However, PCIe and Sata drives' speed will sature to the same speed, meaning you cant use both at full speed at the same time. Under realistic conditions, no matter the setup (raid or not), those speeds will never be reached.
https://www.gigabyte.com/Motherboard/GA-Z270X-Gaming-9-rev-10#kf
- Directly to the CPU: Now this is the ideal setup. The more lanes available for use, the more speed you can squeeze out. Examples of this are:
- The AM4 platform, with up to 24 CPU lanes (16 to GPU + 4 to M.2 directly to CPU + 4 to chipset)
- The X99 platform, with up to 40 CPU lanes
- The X299 platform, with up to 44 CPU lanes
The AM4 platform seperates the M.2 (and the USB/PS2 peripherals, if anyone is interested) from the chipset and as such from the other devices, allowing for the full speed utilization of the M.2 along with the Sata disks/LAN Adapter/Onboard Audio at the same time.
The two intel enthusiast platforms can allow for some insane speeds if combined with top-end PCIe SSDs all connected to the CPU, especially if raided through software raid or a raid controller.
Speaking of which, I would love to see a PCIe 3 x16 NVMe raid controller that can host up to 4 M.2 NVMe SSDs. Now that would be the definition of fast storage. Oh wait, I found one: xD
http://www.highpoint-tech.com/USA_new/CS-product_nvme_raid_controller.htm
Anyways, I would really like to hear your opinions on this matter and any complains or corrections you may have on my research. In case I forgot anything, I will make sure to update the post.
Also, it would be really useful if we could make a list of the currenty available top performing drives in each disk category, which are capable of or nearly hitting their specifications limit. Thats what Im planning to do now after finishing this theoritical reasearch, but any help would be greatly appreciated.
So, I guess we all love fast storage. The faster the better.
The general rule is usually as follows: HDDs < SSDs < PCIe / M.2 drives
But even so, one could say that there are times when we are actually limited by the protocols in use, and not the speed that current storage technologies can achieve. I tried to break them down a bit (or a lot, like, completely) and find a setup that would theoretically offer the fastest storage possible, taking EVERYTHING into consideration.
Firstly, let me state all the important numbers:
- Sata 3: 6 Gigabit/s % (10bit/1byte) = 600 Megabyte/s
The Sata protocol uses a 10 bit per 1 byte encoding, meaning 2 bits of overhead for every byte.
The Sata bus uses a serial communication through the AHCI protocol, meaning that only one command can be sent at a time. The transmission time of the command is of course much less that the execution time, so you can have a command queue, allowing an ssd to have simultaneous read/writes on different chips, unlike hdds.
Still, the sum of reads + writes cannot be over 600MB/s
- Sata Express: 16 Gigabit/s % (10bit/1byte) = 1,6 Gigabyte/s
Again, 10bit per byte protocol. However, the sata bus is not nearly enough to reach 1,6 Gigabyte per second with a serial communication. To overcome the limitation, the NVMe protocol was created replacing the AHCI and featuring an insase amount of queues and commands per queue (65,536 command queues with 65,536 commands per queue). Yup, that will be enough for quite some time.
So now we have: reads + writes = 1,6Gigabyte/s
- PCIe 3, x1 Lane: 8 Gigabits/s % (130bit/16byte) = 984,6.... ~ 985Megabyte/s
Unlike PCIe 1 & 2, the PCI 3 protocol uses a 130 bit per transfer encoding, meaning 128 bits(16bytes) + 2 bits of overhead per transfer. Also, this is unidirectional, meaning we can have 985MB/s reads AND 985MB/s writes simultaneously.
Now, lets start talking about drives:
- Sata 3 SSDs: 600MB/s read+write through a sata 3 port, pretty straightforward
- Sata Express SSDs: 1,6GB/s read+write through a sata express port
- mSata SSDs: Also known as M.2 Sata SSDs. 1,6GB/s read+write through an M.2 sata port and utilizing 2 PCIe 3 lanes, but controlled by a sata controller. Two PCIe 3 lanes can handle 1,97GB/s unidirectional, but since they are controlled by a sata express controller, that acts as a limiting factor. In other words, NOT AN M.2 but a SataEx SSD, just with a different connector.
- PCIe 3 SSDs: Utilizing the NVMe protocol, the PCIe 3 encoding(130bit/16byte) and using 4 lanes, they can achieve 3,94GB/s unidirectional.
- M.2 NVMe SSDs: The same with the PCIe 3 SSDs, but with a different connector and form factor. Meaning 3,94GB/s unidirectional. Now thats a true M.2 SSD.
- U.2 SSDs: Again the same with the above 2, 4 PCIe lanes and NVMe, but different connector, taking the disk off the motherboard. Also 3,94GB/s unidirectional.
Now, lets see what is the most efficient communication method between these and the CPU:
There are CPU lanes, and chipset lanes. In order to keep this short, I will just say that the most recent CPUs communicate with their Chipset using a speed equal to 4 PCIe 3 lanes, thus 3,94GB/s unidirectional. That bandwidth is then shared by all the devices and peripherals connected to the chipset, which may offer more than 4 lanes under the claim that not all devices will use their full bandwidth at the same time.
USB controller, Sata controller, HD Audio, LAN adapter, and perhaps some M.2 and PCIe ports all sharing 4 main lanes. Pretty sad. So at this point lets assume that all available speed goes to the Sata controller.
About Sata disks:
Sata disks need a Sata controller, which is always on the Chipset.
Depending on the Sata controller, communication with the Chipset is done either through a single channel no matter how many Sata ports there are available, or through multiple channels.
Meaning that on Z270, one Sata 3 perfect disk will have 600MB/s, and 2 Sata 3 same disks in raid 0 will still have 600MB/s. It has only one 6Gb/s channel.
https://www.intel.com/content/www/us/en/products/chipsets/desktop-chipsets/z270.html (block diagram)
On X99, each Sata 3 port can handle 6Gb/s individually
https://www.intel.com/content/www/us/en/products/chipsets/desktop-chipsets/x99.html (block diagram)
So, with the right choice of platform, one can hit 3,94GB/s from Sata (not unidirrectional due to controller) under perfect conditions, hardly the case though.
About PCIe 3 disks:
PCIe 3 disks can either connect directly to the CPU lanes, or indirectly through the Chipset's lanes.
- Throught Chipset: 3,94GB/s unidirectional at best, no matter the number of drives.
This Gigabyte motherboard is proof of that, featuring 3 PCIe 3 SSDs x4 lanes each on raid 0 and hitting scores no more than that.
However, PCIe and Sata drives' speed will sature to the same speed, meaning you cant use both at full speed at the same time. Under realistic conditions, no matter the setup (raid or not), those speeds will never be reached.
https://www.gigabyte.com/Motherboard/GA-Z270X-Gaming-9-rev-10#kf
- Directly to the CPU: Now this is the ideal setup. The more lanes available for use, the more speed you can squeeze out. Examples of this are:
- The AM4 platform, with up to 24 CPU lanes (16 to GPU + 4 to M.2 directly to CPU + 4 to chipset)
- The X99 platform, with up to 40 CPU lanes
- The X299 platform, with up to 44 CPU lanes
The AM4 platform seperates the M.2 (and the USB/PS2 peripherals, if anyone is interested) from the chipset and as such from the other devices, allowing for the full speed utilization of the M.2 along with the Sata disks/LAN Adapter/Onboard Audio at the same time.
The two intel enthusiast platforms can allow for some insane speeds if combined with top-end PCIe SSDs all connected to the CPU, especially if raided through software raid or a raid controller.
Speaking of which, I would love to see a PCIe 3 x16 NVMe raid controller that can host up to 4 M.2 NVMe SSDs. Now that would be the definition of fast storage. Oh wait, I found one: xD
http://www.highpoint-tech.com/USA_new/CS-product_nvme_raid_controller.htm
Anyways, I would really like to hear your opinions on this matter and any complains or corrections you may have on my research. In case I forgot anything, I will make sure to update the post.
Also, it would be really useful if we could make a list of the currenty available top performing drives in each disk category, which are capable of or nearly hitting their specifications limit. Thats what Im planning to do now after finishing this theoritical reasearch, but any help would be greatly appreciated.
Facebook
Twitter
Instagram