February 05, 2013 - by rachelbalik
Brendan Gregg on “Thinking Methodically About Performance.”
When your friends and family come to you with the flood of technology questions you’ve come accept as an occupational hazard, you know how to ask the critical, mission-saving questions such as, “is it plugged in” and “have you tried turning it on?”
But when it comes to solving your own technology problems issues, the solution -- while it may be simple -- is rarely obvious. Brendan Gregg, lead performance engineer at Joyent, recently explored the complexity of systems-performance management in a post called “Thinking Methodically About Performance” for the Association of Computing Machinery. In the post, Brendan discusses the ways in which we try, and sometimes fail, to resolve performance issues for complex systems that may be “composed of databases, Web servers, load balancers, and custom applications, all running upon operating systems—either bare-metal or virtual.” And this is before adding the issues around hardware.
Brendan explains that in addition to all these literal moving parts, there are a number of other reasons why performance issues can be difficult to pinpoint. Solving these problems often requires interdisciplinary expertise -- assessing why components that function in isolation, break down as part of a greater whole. As Brendan writes:
“Performance issues can be complex and mysterious, providing little or no clue to their origin. In the absence of a starting point—or a methodology to provide one—performance issues are often analyzed randomly: guessing where the problem may be and then changing things until it goes away. While this can deliver results—if you guess correctly—it can also be time-consuming, disruptive, and may ultimately overlook certain issues.”
Still, many people attempt to tackle these problems either without a methodology, or with one that still leaves much to be desired. At Joyent, we employ what’s called the USE (utilization, saturation, and errors) Method - invented by Brendan - which comes into play after the problem has been stated, and provides a framework for quickly identifying systemic bottlenecks.
In his post, Brendan explores the shortcomings of other methodologies and offers a robust explanation of how to employ the USE methodology by listing, examining and then discovering the metrics that will be relevant in a cloud computing environment. He also provides succinct recommendations for how to examine existing software resources.
Read the whole post, Thinking Methodically About Performance, on the ACM website and keep your eyes on the Joyent Twitter feed for updates on Brendan’s forthcoming book "Systems Performance - Enterprise and the Cloud", due in 2013 from Prentice Hall, which will cover the USE Method and much more.
This article also appears in the full in the Communication of the ACM, Volume 56, No. 2.