Ever-changing data retention guidelines, along with the new demands of AI, are dramatically changing organisational storage needs. As regulations change and get stricter by the day, data retention periods are growing exponentially.
Because there is a direct one-to-one relationship between retention and storage, organisations must become more strategic about storage.
How long is a piece of string?
When it comes to data retention for compliance with regulatory requirements and internal guidelines, data stewards are taking no chances.
What we're seeing from a retention point of view is that data has become a never-ending requirement that changes constantly. There are so many new regulatory amendments, new rules, governance, risk and compliance frameworks being introduced within companies that data stewards are unsure if they can get rid of it or not.
Many are opting for a keep it forever scenario − just in case it needs to be recoverable under a future framework. That then creates a need to constantly increase storage − be it on tape, storage devices or in the cloud.
This growth in storage requirements is not predictable, because many factors influencing data growth and retention periods are external and often out of the organisation’s control.
Many are opting for a keep it forever scenario − just in case it needs to be recoverable under a future framework.
In most companies, retention of data is no longer an IT decision but rather a business decision, and every department needs to be involved in the process of understanding how long the data needs to be retained for.
While some departments over-inflate how long they really need to keep data for, others underestimate it, and so we see organisations putting a blanket requirement for data retention in place − which is often indefinitely, and not necessarily a true view of what’s needed.
AI impacts storage requirements
AI is also driving significant change in storage and data retention requirements. Most companies are either looking at AI, or have already embarked on an AI journey, which brings with it the requirement to access data.
AI relies on historical data to build those algorithms, meaning companies now have to keep their data for longer and make it readily accessible for AI models.
With AI, it's necessary to have high-performance storage capable of handling the performance requirements that the AI workloads require to operate optimally.
Organisations must consider metrics like IOPS, performance metrics and benchmarks around similar workloads in other environments. So, it is important that storage vendors have those numbers available so they can indicate the kind of performance organisations can expect for specific workloads and datasets.
Getting the storage balance right
Businesses face a number of challenges in provisioning the right levels of storage at the right cost. The longer they keep the data, the more cost they will incur, but if they fail to keep data, they face future compliance and business risks.
They must weigh up their storage requirements, ensuring they align with both the organisation’s current and future requirements while aligning to a budget.
Organisations will be unlikely to retain archived data in fast, high-performance storage. That high-performance storage is often better suited to newer and mission-critical data that needs to be accessed within a shorter window, quite regularly.
Historical, archived data is better suited to a capacity type archive-tier storage, which comes at a lower cost.
However, the caveat is that while cheaper storage is acceptable for long-term retention of data, it still must be reliable and secure. Putting data on ‘cheap and cheerful’ storage that isn't as secure as it could be raises risks of that data being corrupted or preyed on by ransomware or malware.
Both long-term storage and high-performance storage for mission-critical data and AI must be fit for purpose, reliable, secure and maintained regularly. Companies should also consider immutable storage, which gives them an extra level of security.
While many businesses are trying to get the balance right and many vendors are promoting best practice approaches to storage, a lot of decisions around storage still come down to cost.
In some organisations, people are unsure about their decision-making so they do nothing at all. In others, companies are trying to adopt best practice, but they find that storage requirements and data retention best practice are constantly moving goalposts − they must attempt to provision for the unknown and hope their decisions will meet future storage requirements.
To future-proof their storage for longer data retention, companies should be critical around what they need from a performance perspective and from a security perspective, and they must ensure their storage can scale on demand to cater for future demands around backup policy, governance and risk compliance, legislation, or AI.
Share