Memory Vendors Move Up the Stack to Optimize Storage

As NVMe continues to mature and help storage systems get the most from flash-based SSDs, memory vendors are looking a little higher up the stack for more optimization gains.

Recognizing that existing storage engines are designed for the spinning disk era, Micron Technology recently unveiled its heterogeneous-memory storage engine (HSE) aimed at not only getting more from SSDs but also other storage-class memory (SCM), including 3D XPoint. The company’s open source HSE will enable developers using all-flash infrastructure to customize or enhance code for its unique use cases and take advantage of the performance and reduced latency of next-generation nonvolatile media, said Steve Moyer, vice president of storage software engineering at Micron.

“We’re a huge fan of NVMe,” he said in a telephone interview with EE Times. “It’s eliminated a lot of the inefficiencies from the traditional storage protocols and allowed the lowest layers of the operating system stack — the blocks — to operate very efficiently.” However, there’s more work to be done one level up to make the storage engine use the media more effectively, said Moyer. “One of those aspects is the read and write paths through the stack, eliminating anything that’s unnecessary so you can get the lowest latencies possible.”

Micron’s HSE recognizes that legacy storage engines weren’t designed to work with flash-based SSDs and other SCMs such as 3D XPoint and is designed to manage multiple classes of storage. (Source: Micron)

Micron’s HSE was designed with massive-scale databases in mind, he said, such as those with billions of key counts, terabytes of data, and thousands of concurrent operations, while also having the ability to exploit multiple classes of media, including QLC 3D NAND flash and 3D XPoint. The platform is designed to take advantage of next-generation NVMe, which has become a data center standard for fast, low-latency SSD connectivity, as well as other emerging technologies such as NVMe sets, zoned namespaces, LBA streams, and open channel.

One area in particular that Micron’s HSE focuses on is the write amplification that occurs on host side of the storage stack as to improve the endurance, which is important for getting the most out of more cost effective SSDs, such as those using low cost/high capacity QLC flash.

The company is already working with partners using the popular NoSQL Mongo DB database to run production workloads through its storage stack using the HSE, and Moyer said depending on the workload, they’re able to realize three to six times better throughput than they were getting with their “classic” storage engine, as well significant reduction in tail latency and write amplification as data flows through the complete stack. “The NVMe device gives very low latency, high throughput access to the flash media, but all that does is begin to expose the weaknesses in the host storage stack above it,” he said. “If you want the end customer to get the most of these great SSDs then you’ve got to start looking at optimizing the rest of the stack.”

The HSE platform is designed to be extensible to new interfaces and new storage devices, including other SCM besides flash such as 3D Xpoint. The platform is delivered as an embeddable key-value database with code being maintained Micron in a GitHub repository. Aside from its integration capabilities with MongoDB, it can be integrated with other NoSQL databases and object stores and is designed to be extensible to new interfaces and new storage devices.

Kioxia America’s SEF unit is designed to deliver flash-based storage across a PCIe connection. (Source: Kioxia)

Micron is not the only company that has moved up the stack to better optimize SSDs and other SCMs. Facebook has developed its own open source storage engine, MyRocks, for example. But although Micron’s HSE is also open source, it’s not a standard, noted Thomas Coughlin, president of Coughlin and Associates, and there are other approaches to optimizing flash. Kioxia America recently introduced its Software-Enabled Flash (SEF), a technology that combines software flexibility, host control, and flash native semantics into a flash native API and purpose-built controller to make flash easier to manage and deploy. “They probably should start talking with each other if they want to have open source standards, otherwise they won’t be standard,” he said.

While SSD endurance has improved significantly over the last several years, due in part to the advent of 3D NAND, storage engines such as Micron’s HSE reduce the right amplification because there’s less interaction with the flash cells. “You’re not writing on the flash sales as often,” said Coughlin. Offering a storage engine to work higher up the stack with databases and object storage systems is also advantageous from a business perspective. “There’s more margin if you can do that.”

The post Memory Vendors Move Up the Stack to Optimize Storage appeared first on EE Times Asia.

from EE Times Asia https://ift.tt/30zioqf