July 18, 2013 - by Joyent Team
How common are latency outliers or multimodal latency? When either are present, summary statistics like the average are inadequate, as it doesn't show details of the latency distribution that you really need to know.
To answer this question, Brendan Gregg surveyed hundreds of systems across the Joyent cloud, and used tests to quantify their occurrence: over 95% had six sigma outliers, and at least 20% had multiple modes. He also included detailed visualizations of the entire datasets to show what these numbers are based on, so that you can really see for yourself.
These visualizations are a new type called frequency trails, which he developed to show the location of outliers. This isn't his first visualization for latency distributions: several years ago he created latency heat maps with Bryan Cantrill, which are now used in many performance analysis products (including Joyent Cloud Analytics!).
Last week, Brendan wrote three blogs detailing this work. In Detecting Outliers, Brendan uses frequency trails to explain outliers, and introduces a six sigma test to help you detect them.
Next, Brendan studied multimodal latency. In Modes and Modality, Brendan similarly visualizes and creates a test for detection of multiple modes.
Finally, he explains What the Mean Really Means through a visualization that illustrates how the average fails to reflect data skewed by outliers or latency divided between multiple modes.