Advanced Crash Dump Analysis

The Advanced Crash Dump Analysis course provides students with essential skills to accomplish more detailed crash dump analysis than getting a stack trace and matching it to bugs. This course teaches students what information to review to determine the cause of a panic, what causes bad traps to occur, and if there is corrupt data or a hardware issue. The course also presents information on analyzing user-forced crashes and live crash dumps. The latter part of the course includes information about the SPARC architecture and assembly language and explains how this information can be used to gather additional information from crash dumps.

Students who can benefit from this course:

• System Administrators who have some crash dump analysis experience and need more detailed information.

Audience
Course Topics Obtaining and Identifying Crash Dumps

• Estimate the size of a crash dump
• Describe how to get information from a hard hang system
• Identify the types of crashes
• Describe how to work with partial or corrupt crash dump information

SPARC Architecture

• Describe the SPARC architecture implementations
• Describe instruction pipelining
• Describe SPARC architecture data types
• Describe the organization of data in memory
• Describe SPARC architecture registers and their usage
• Describe the use of register windows
• Explain major components of SPARC architecture instructions

Analyzing Bad Trap Panics

• Describe how traps occur and what events they handle
• Describe how bad trap panics occur
• Describe useful information in bad trap messages
• Describe how to relate stack data to function prototypes
• Find the data that caused the trap
• Describe how to determine if a crash is related to a hardware problem

Analyzing Unique Panic String Panics

• Describe common types of unique panic string panics
• Describe how to get information about kernel memory corruption
• Analyze kernel heap corruption panics
• Analyze unique panic string panics caused by locks
• Analyze file system corruption panics

Analyzing User-Forced Crash Dumps

• Describe how to detect a hung system
• Describe common causes of system hangs
• Recognize normal stacks
• Diagnose hang problems created by memory shortages, locks, or hardware
• Diagnose hang problems related to the clock
• Describe procedures for analyzing difficult hangs

Analyzing Live Crash Dumps

• Run debugging tools on a live system
• Describe the risks of modifying information on a live system
• Create a live crash dump
• Describe what information is not likely to be valid in a live crash dump file
• Diagnose an unkillable process
• Diagnose a memory leak

SPARC Assembly Language

• Describe assembly language conventions
• Describe SPARC architecture instruction types
• Identify arithmetic and logical instructions
• Describe the use of the sethi instruction
• Identify load and store instructions
• Describe alternate address spaces
• Describe floating point instructions
• Identify synthetic instructions

SPARC Architecture Stacks and Arguments

• Describe simple argument passing
• Describe stack formats
• Describe complex argument passing
• Verify argument validity
• Describe how local variables are accessed
• Describe the characteristics of leaf functions
• Describe what a tail call is and how it effects the stack

SPARC Architecture Traps and Interrupts

• Describe the purposes of traps
• Describe how the UltraSPARC(R) processor handles traps
• Describe trap levels
• Describe watchdog resets and RED state exceptions
• Analyze RED state output
• Analyze XIR data
• Describe how interrupts are processed
• Analyze crashes at trap levels greater than zero

Matching Assembly Language to C Code

• Describe the differences between optimized and unoptimized code
• Describe how variables are accessed
• Describe the assembly instruction that indicates a function call
• Find the line of C code that corresponds to or includes that assembly instruction (given a system dump and an instruction in a function)

Course Objectives

• Obtain and identify crash dumps
• Describe the Sparc architecture features necessary for analyzing crash dumps
• Analyze bad trap panics well enough to identify the corrupt data
• Analyze common unique panic string panics
• Analyze user-forced crash dumps
• Analyze live crash dumps
• Describe the SPARC assembly language instructions
• Describe SPARC architecture argument passing and stack contents
• Describe SPARC trap and interrupt handling
• Match an assembly language location in a crash dump to a line of kernel code