Some tips for Microbenchmarking in Java
Microbenchmarking is one way of performance testing a small part of your product. Perhaps you wan’t to compare the difference between two similar algorithm implementations. The goal however is to make sure that you test the right thing.
So what can affect your output and what can you do about it?
For one, the JVM options of the production environment vs local environment. Another issue that can come up is GC. But the biggest culprit is most likely the compiler.
Assuming you start your program with the same options then here are my three things I try and remember when microbenchmarking in order to give me a good enough picture of the performance.
Here are my three tips.
Number 1: Warm up
Warm up simply means that you run your method(s) some number of times before actually measuring the execution time. This is done to allow the code to be compiled so that you aren’t measuring the compilation time. It also allows other optimizations (there are plenty) to be performed that makes the output more likely to match the execution time in production. One of those optimizations is method inlining which I’ve written about in another article.
Number 2: Average
Run the method, let’s say 3 times and look at those numbers and perhaps take the average. This is just to ensure you don’t see a spike due to a minor/major GC.
Number 3: “Use” the result
Finally, ensure that you use the result in some way so that the compiler does not make your code into a no-op. Depending on your test setup, one way to do that is to write the result to a volatile instance variable. Just make sure you aren’t writing too often or that will end up affecting the time of your test.
So these are my main things I think about when microbenchmarking. Hope you get some value out of this as well!
