Doing performance testing right

Infinidat often visits prospects who want to replace their incumbent storage arrays with another storage system, due to pricing, flexibility, ease of use etc. Its prospects want to be sure that the performance of the new solution will be at least the same as current solution or better, to support growth, speed up business applications and make end users happier. Because of it, companies are testing, sometimes frequently, storage solutions for different business cases. Performance testing is a not a usual practice for local IT teams in most cases, so some recommendations could be helpful.

Infinidat thinks that the best way to test performance is to run real applications on it. Sometimes it is possible thanks to infrastructure features (SAN virtualisers, hypervisors, volume managers, existing testing environment). Doing that way of testing, gives ability to understand how storage behaves in real life which hardly could be simulated, for example, single threaded application generates not much IO but if we are not aware of it and emulate it with many threads, we will get much better results than in real life. For such kind of applications, to get the most from active multi controller system, parallelism should be added on DB or application level.

But unfortunately it is rather rare an option, due to the things such as no-risk approach, inability to get or put equipment onsite, security rules, lack of precious time to do that etc. The next option is to use the real applications with simulated user workload. It is a good option, especially if application has specific, well known patterns, usually batch jobs. Batch jobs have well known completion time and it is rather easy to compare times in different environments. What could stop doing that - you still need to have equipment onsite or provide real data to the vendor (data could be anonymised, but it is again time and money), specialists are spending time, and you need a method to simulate user interaction or buy special software for it. Companies with huge IT budgets, big telcos, banks, do that because it allows them to run their business in a better way and save money, but for most companies it is overkill.

So nowadays the most common way to test storage performance is running synthetic tests using tools like Vdbench and iorate in a controlled environment, often remotely in Infinidat lab - it is cheap, quick and needs little interaction with the prospect. The biggest problem there is how results are related to what will be in the field.

To have this correlation you can, for example, collect performance metrics from the existing arrays and try to simulate it on a new cool storage It is a good way but you need to dig in it thoroughly as not all important metrics could be digested, like storage tiers, sub LUNs workload distribution, workload by IO sizes, even cache hits. After understanding what is going on, done in cooperation with Infinidat professionals, profiles could be written down to emulate on test box. Usually peak workload is emulated but it is not always a good idea. Frequently peak workload differs significantly from usual workload in cache hits, read percentage, IO size etc. So it is better to emulate two kinds of workload - typical and peak, separately. Emulating peak workload for long periods of time like tens of minutes, even hours does not reflect real life because peaks are short and short bursts of writes could be kept easily in write cache and then destaged, but doing that for a long time will fill it up and performance will be degraded. It can be easily checked by running the same synthetic load on existing system and see if it could sustain peak performance for a long time.

But what if you are testing not for the existing application but for something which does not exists in your company? You can ask the application vendor for profiles to run but probably the better way is to ask a friend! You can ask folks in other companies who implemented similar things, also you could ask Infinidat or other vendors what it sees for such apps in the field and do your approximation from it.

So after running synthetic tests and taking into account that you cannot emulate 100% right, what else you can do to improve accuracy? You can run tests on storage from different vendors and understand how they are related to each other, also you can run the same tests on your array to see how performance numbers are related to something you know, and the latter is more connected to real life.

But frankly speaking, it is hard to emulate anyway - you could be proposed tenfold cache size than your existing box but if you did synthetic testing with cache hit set the same as your existing box, you probably will not see any difference in test numbers between 16GB and 1TB of cache, also most algorithms to maximise cache usage hardly work for synthetics, but for Infinidat we excel in it with 100+ patents and tons of RAM. So go for real app tests or try your best with Vdbench!