Kernel Crash Analysis and Debugging

Duration: 4 days

Format: Lecture 30%, Lab Work 70%


This course teaches the tools, techniques, and strategies that can be used to debug and analyze system crashes, hangs, and performance issues on the SmartOS Operating System. The course is heavily concentrated on lab work. Students will be given various problems, and asked to use the techniques covered in lecture to solve the problems. Tools include DTrace, mdb, and kmdb. The class will examine problems related to logic errors, memory corruption, synchronization problems, stack corruption, memory leaks, boot problems, and others. The course will also cover architecture specific details needed to do debugging on X86/X64.


  • Use DTrace, mdb, and kmdb to examine system and application bugs, including crashes and hangs.
  • Determine which tools to use for specific types of problems.
  • Describe what happens on a system panic.
  • Describe how threads can hang, and how to determine the root cause of this.
  • Describe stack frames and stack operation on 32-bit and 64-bit X86.
  • Use mdb to examine crashed (and working) systems.
  • Use kmdb to determine causes of system hangs.
  • Use DTrace to see the flow of control leading up to a problem, as well as for general debugging.
  • Use DTrace to examine performance issues, and to drill-down to find the root cause.

Intended Audience

This course is designed for system engineers/analyists and kernel programmers. Students may include developers and support engineers. While the course concentrates on kernel issues, application developers can also benefit.


A SmartOS/Illumos/Solaris Internals course should be taken prior to this course. To do the labwork, students will need to know the C programming language.

Topic Outline

Module 1: Introduction to Panics and Hangs

  • What is a Panic?
  • Causes of Panics
  • What happens on Panic?
  • Panic Dump Actions
  • Panic Dump Contents
  • The dumpadm Command
  • The savecore Command
  • Hangs
  • References

Module 2: Preliminary Analysis Information Gathering

Module 3: Analysis and Debugging Tools

  • mdb
  • kmdb
  • DTrace
  • zdb
  • Which tool to use
  • Lab

Module 4: mdb

  • Running the mdb command
  • Syntax
  • Examples of the “/”, “?”, and “=” Commands dcmds and walkers
  • Lots of examples
  • Kernel Panic Example
  • Lab

Module 5: DTrace

  • How DTrace works
  • Debugging Example
  • The D Language Actions
  • Built-in Variables
  • Anonymous Tracing, Speculative Tracing, and Destructive Actions
  • Examining post-mortem DTrace records using mdb
  • Tunable variables and troubleshooting DTrace problems
  • Lab

Module 6: kmdb

  • Differences between kmdb and mdb
  • Examining system hangs with kmdb
  • Booting with kmdb
  • Lab

Module 7: zdb

  • Examining on-disk ZFS format with zdb

Module 8: X86 Architecture Overview

  • Function call mechanism
  • Stack Frames
  • Optimizations
  • Description of Assembler Language Problems that can occur with stacks
  • Lab

Module 9: Implementing dcmds and walkers

  • Adding functionality to mdb
  • Lab Work


See the course calendar. From there you can also also fill out the request form to let us know of other courses or dates you would be interested in.

Cancellation Policy

See cancellation policy.

Sign up now for Instant Cloud Access Get Started