For a long time, enterprise object storage was positioned for archive and backup workloads—hardly the most exciting task, yet certainly a critical and necessary part of a data infrastructure. Life in this scenario was easy; the primary concern was to build for large sequential streaming workloads and optimize for availability, durability, and scalability of capacity.
But then a new challenge arrived, prompting a seismic shift for this state of equilibrium: modern cloud workloads and generative AI.
Both of these workloads demand high performance and low latency and are more typically suited to network-attached storage (NAS) and storage area network (SAN) environments. However, the scalability required to store millions, potentially billions, of files with detailed enriched metadata helped to position object storage as the preferred destination.
Over the last few years, companies have gained a mature understanding of building modern cloud architectures at scale using native cloud services such as Amazon S3, and are now looking to bring these capabilities to their own hybrid clouds. This created a demand for S3-compatible object storage combined with data management in a broader range of environments, including on-premises, at the edge, within containerized environments, and within the public cloud providers themselves. This is the backbone of providing application and data portability and a major imperative for companies to consider when assessing their options for hosting unstructured data.
Arise All-Flash Architectures
The primary responses to these new performance demands were the addition of flash to existing vendors’ storage offerings and the launch of all-flash, NVMe-based offerings from new challengers in the market. These impressive innovations helped drive greater adoption of object storage across AI/ML and data lake environments.
Fast Isn’t Everything
Going fast is certainly important; however, the top vendors in this space were also able to apply enterprise management capabilities to these new architectures, including replication, ransomware protection, full S3 protocol compatibility, and robust partner certifications to ensure compatibility with existing customer investments. Some vendors now even offer certifications in ML. One example is PyTorch, which is one of the leading frameworks for developing and training ML models.
Object Storage Solutions
While vendors reacted with new architectures, customer demands have placed an emphasis on additional data management capabilities in order to reduce the overall costs of higher-performance hardware.
It’s true that not all data needs to go fast all the time. In fact, it’s common for data that demanded higher performance earlier in its lifecycle to be accessed less frequently as it ages. Vendors offering automatic storage optimization based on data access profiling can enable the movement of this data to more commercially viable tiers, freeing up performant hardware for newer, high-value data without losing the manageability of those objects within a single namespace. Many vendors now support extending these storage tiers to public cloud providers, so customers can benefit from both performance and cloud-scale capacity within a single management plane.
As object storage is used for a growing number of mission-critical applications, ransomware protection at the storage layer is increasingly important. Vendors investing in these innovations are well-placed to satisfy these requirements for the year ahead.
With so many vendors in this space, I recommend that prospective customers develop a clear understanding of their business requirements for unstructured data and, more importantly, the differences in the way vendors implement their features and architectures for object storage.
Next Steps
To learn more, take a look at GigaOm’s object storage Key Criteria and Radar reports. These reports provide a comprehensive overview of the market, outline the criteria you’ll want to consider in a purchase decision, and evaluate how a number of vendors perform against those decision criteria.
If you’re not yet a GigaOm subscriber, sign up here.
