What is NVMe?

NVMe is a software protocol to connect non-volatile mass storage devices via PCI Express hardware in­ter­faces. The ab­bre­vi­a­tion NVMe stands for “non-volatile memory express”. A non-volatile memory is a storage medium which contains contents that are not lost when the power is switched off.

NVMe enables fast data access even for per­sis­tent memory tech­nol­o­gy. But how does it work, and what other ad­van­tages does NVM Express offer?

Tra­di­tion­al­ly, a dis­tinc­tion has been made between volatile main memory (random access memory, RAM) of a computer and non-volatile mass storage. While the RAM of a computer loses its contents when it is switched off, contents of the non-volatile mass storage device are retained. Following the use of punch cards during the early days of computer tech­nol­o­gy and the sub­se­quent wide­spread tape storage systems, rotating hard disks (hard disk drives, HDDs) have become the standard device for mass storage.

The sep­a­ra­tion into main memory and hard disk was ac­com­pa­nied by dras­ti­cal­ly different access times. The time it took to access se­quen­tial­ly stored data in RAM was about a hundredth of the time it took to read the same data from a rotating hard disk. Only with the wide­spread use of mobile devices and the as­so­ci­at­ed emergence of solid-state drives (SSD) was this dynamic extended.

Now, SSDs are widely installed in laptop and desktop computers and are referred to as con­tin­u­ous storage media. The data storage is com­plete­ly elec­tron­ic, free from moving parts – from the processor register, to the L1 and L2 caches, to the SSD mass storage device. However, data access to SSDs has largely been reliant on protocols optimized for rotating HDDs such as Serial ATA (SATA).

NVM Express is the first protocol specif­i­cal­ly developed for non-rotating, SSD-based mass storage devices. It succeeds the long-es­tab­lished software in­ter­faces SATA and AHCI. SATA in par­tic­u­lar, which was initially used for the internal com­mu­ni­ca­tion with SSDs, quickly developed into a bot­tle­neck: Modern SSDs were able to achieve a much higher data through­put than the SATA protocol allowed. With the non-volatile memory express interface, which has been specif­i­cal­ly optimized for SSD media, SSDs can exploit their full potential.

The existing PCI Express (Pe­riph­er­al Component In­ter­con­nect Express, PCIe) standard is prefer­ably used to connect SSD mass storage devices to the mainboard. Graphics cards and other hardware, for example, are connected via PCIe. PCIe is a physical interface. As a protocol, NVMe presents as a logical coun­ter­part that is optimized for non-volatile SSDs. This means that a single driver on the operating system side is suf­fi­cient to address all NVM Express com­pat­i­ble SSDs by different man­u­fac­tur­ers.

A software protocol is used to structure digital com­mu­ni­ca­tion, i.e. the exchange of data. The protocol defines a set of commands that determine the language used between in­di­vid­ual com­mu­ni­ca­tion partners. The non-volatile memory express protocol makes do with a lean in­struc­tion set com­pris­ing only 13 commands.

Only ten admin commands are mandatory:

• Create I/O Sub­mis­sion Queue
• Delete I/O Sub­mis­sion Queue
• Create I/O Com­ple­tion Queue
• Delete I/O Com­ple­tion Queue
• Get Log Page
• Identify
• Abort
• Set Features
• Get Features
• Async Event Requests

There are also three I/O commands that are required:

• Read
• Write
• Flush

The modern hard drive has several rotating disks. Two read/write heads are usually used per disk. The read/write heads are all attached to a single, movable read/write arm. Due to this physical arrange­ment, neither HDDs nor their protocols are optimized for mass parallel access to data.

The well-es­tab­lished SATA protocol has a single command queue to execute hard disk commands one after the other. The non-volatile memory express protocol, on the other hand, supports 64,000 queues, with up to 64,000 commands per queue. This ensures a high degree of data access par­al­lelism, which leads to high data through­put rates and low latencies.

Compared to HDDs, SSDs do not read and write data se­quen­tial­ly; the data access to SSDs is just as fast at every position as it is with random access main memory.

Protocol commands with NVMe are created and processed according to the following pattern:

1. Host transmits I/O Sub­mis­sion Queues
2. Con­troller recalls I/O Sub­mis­sion Queues
3. Con­troller runs I/O Sub­mis­sion Queues
4. Con­troller transmits I/O Com­ple­tion Queues
5. Host receives I/O Com­ple­tion Queues

NVM Express is ad­van­ta­geous because it uses the existing PCIe standard and the optimized protocol for modern solid-state storage. The lean protocol command set, tailored for the operation of SSDs, leads to low overhead when reading and writing data. The direct use of PCIe means there is no need for a protocol trans­la­tor, which in turn results in lower latencies.

Because command queues are processed in parallel, SSDs connected to NVMe can develop their full per­for­mance potential. NVMe devices can also be connected via multi-pathing in pro­fes­sion­al hosting and high-per­for­mance computing en­vi­ron­ments. Here, the NVMe device is connected via multiple, parallel data lines within the computer in­fra­struc­ture. If a line fails, the device can still be accessed. NVMe devices also support hot swapping; if a storage medium fails com­plete­ly, it can be exchanged during operation.

NVM Express is used in almost all computer systems nowadays. From laptop and desktop computers to servers and work­sta­tions to su­per­com­put­ers: fast SSD storage media connected via NVMe are installed every­where. The different form factors used include:

• PCI Express cards
• M.2 cards
• SFF (small form factor) or 2,5" devices with U.2 con­nec­tion
• E1 or NGSFF (next gen­er­a­tion small form factor)

Another in­ter­est­ing de­vel­op­ment of the non-volatile memory express protocol is “NVMe over Fabrics”. Here, instead of a direct physical con­nec­tion, a network con­nec­tion is used to address NVMe solid-state memory via the network. In this way, col­lec­tions of SSD media can be im­ple­ment­ed providing enormous amounts of storage with short access times.