How to get Crash or Hang dump? ~ Vinay's Blog

I have various methods/tools to collect crash/ hang dump. This article is aimed to consolidate and provide all the related information in a single nutshell.

Target Users : The Reader is familiar with Windows OS (XP & Vista)

Wondering what crash is & how to get crash dump! Here U go!!

A crash in computing is a condition where a program (either an application or part of the operating system) stops performing its expected function and also stops responding to other parts of the system. Often the offending program may simply appear to freeze.

A crash can either be a software fault or system crash. A software fault occurs whenever an exception is left unhandled. Upon software fault the OS stops the application from proceeding further.

System Crash or Blue screens on NT-based Windows systems are usually caused by poorly-written device drivers or malfunctioning hardware. Blue screens can also be caused by physical faults such as faulty memory, power supplies, overheating of computer components, or hardware running beyond its specification limits.

Various Methods for collecting crash dump

Below are the few methods to collect crash information.

Using Windbg
Using Adplus
Using ‘Create Dump option in Vista
Using ‘Problem Reports & Solutions’ in Vista
Getting dump on BSOD
From Keystroke
Dump on Heap corruption

Generate crash dump using Windbg as default post-mortem debugger

Install Windbg before installing the product.
Set Windbg as the default post-mortem debugger
Launch command prompt (as admin in Vista)
Go to the windbg installed path (C:\Program Files\Debugging Tools for Windows\)
Type “windbg.exe –I” and hit ‘Enter’
You will see the message “Windbg is set as default post mortem debugger”

Now install the product and continue with your testing. Whenever crash occurs, windbg opens automatically. The below command is used to collect the crash.

.dump /ma <path>\<filename>.dmp

Where, <path> is the path where you want to save the crash dump and <filename> is the name of the file.

Generate crash dump using Windbg after crash occurs (not set any post-mortem debugger)

Install Windbg
From the crash window, see which process is crashed
From the task manager, identify the process Id
Make sure you have not closed the crash window
Launch windbg from “Start > Programs > Debugging Tools for Windows > WinDbg”
Select File > Attach to a Process
Select the Process for which you want to collect the crash dump and click OK
Type the command .dump /ma <path>\<filename>.dmp in the launched window

Generate Crash/hang dump using Adplus (not set any post-mortem debugger)

Install Windbg
Identify the process for which you want to generate the dump
Make a note of the process id from task manager
From command prompt (run as ‘admin’ for Vista), go to C:\Program Files\Debugging Tools for Windows\
Type “adplus.vbs” and hit ‘Enter’
In the launched dialogs click ‘Ok’
In the newly launched command window, you can use any of the below commands
1. ‘adplus –crash –p <pid>’
2. ‘adplus –crash –pn <pname>’
3. ‘adplus –hang –p <pid>’
4. ‘adplus –hang –pn <pname>’
Click ‘Ok’ in the launched dialogs
cdb.exe will be running
Now perform the test steps you want to do (you want to make a crash/hang)

Once the process crashed the dump will be automatically generated as a dated folder in “C:\Program Files\Debugging Tools for Windows\”

If the cdb.exe doesn’t end, you can use Ctl+C to quit the cdb.exe window and can get the dump

Generate dump using ‘Create Dump’ option in Vista

Identify the process for which you want to collect the dump
Go to Task Manager > Right click the process > click ‘Create Dump’
The dump will be created in %temp% folder

Generate dump using ‘Problem Reports & Solutions’ in Vista

Windows Vista introduced the “Problem Reports and Solutions” control panel application. This dialog will show all detected application errors on the user’s machine. When a crash is encountered, we can leverage this built-in utility to get more information about what went wrong, and how to get more information to help resolve the issue.

When this event is recorded, there are three possible scenarios

The crash or hang was recorded and submitted
The crash or hang was recorded, but not yet submitted
The crash or hang was recorded, but no information was captured.

The first two can provide us with valuable information to troubleshoot the issue. The third can give us a general idea, but no specific information about the exact issue the user reported.

Windows Error Reporting configuration is stored in the following registry key in Vista

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting

User based settings such as Block list are stored at:

HKEY_CURRENT_USER\Software\Microsoft\ Windows\Windows Error Reporting

Problem Reports and Solutions

This dialog is available via Start / Control Panel / Problem Reports and Solutions, or under Start / Control Panel / System and Maintenance / Problem Reports and Solutions (depending on whether the user has “classic view” configured or not). From there, click on “View Problem History”, and look for instances of applications. Double click an entry of interest, and select the “Copy to Clipboard” link, and provide the info. The data will tell us how much information is available, without any other files.

Case 1: WER data Submitted

In this case, the error report has already been submitted. The “Copy to Clipboard” data will include the bucket ID. Usually, in this case, the event files are removed from the local computer, as they have already been submitted. We must look up the crash using the bucket ID provided.

Case 2: WER data Available Locally

In other cases, the report has not yet been sent to WinQual. In this case, we can generally collect the files locally. The files that are available will also be included with the “Copy to clipboard” link at the bottom of the screen is used.

Once the “View Copy” of files link is clicked, an explorer window in the temp directory will be opened. The files present in the temp directory will definitely give more information on crash and hence this can be given to the developer for analysis.

Case 3: WER data not available or Submitted

Unfortunately, in some cases there’s no data available but the exception code, modules names and versions, and offset of the crash. In this case, the best we can do is dig through the data on winqual in an attempt to find it, but it’s not guaranteed the correct crash will be found. The screen shot below is of a managed application.

Blue Screens

In many cases, reports of bug checks (BSODs) will also appear on this screen, allowing this approach to be used when a user experiences a system crash and suspects it may be related to specific software. The approach may be used to collect information about this type of crash as for the application hangs or crashes described above.

Common Issues

In some cases, multiple crashes can appear at the same module & offset. Having a WinQual bucket ID or simply a module offset is sometimes not sufficient to describe the exact crash the user encountered.

Note:

1. When a process is added to the Block list of ‘Problem Reports and Solutions’ you will not get the WER Report for that process.

This can be checked by looking at the Block List at,

Control Panel > Classic View > ‘Problem Reports and Solutions’ > Change Settings > Advanced Settings > Block List

2. Starting with Windows Server 2008 and Windows Vista with Service Pack 1 (SP1), Windows Error Reporting (WER) can be configured so that full user-mode dumps are collected and stored locally after a user-mode application crash.

For more information on this refer http://msdn.microsoft.com/en-us/library/bb787181.aspx

Generate Dump on BSOD

Whenever BSOD occurs the memory dump will be collected automatically at

%SystemRoot%MEMORY.DMP

But the default dump is a small memory dump. In order to get the complete memory dump or kernel memory dump

Go to System Properties > Advanced > Startup and Recovery > Settings
From Write debugging information, choose the one which you want
Click Ok.
Proceed with your testing.

Generate Memory Dump using keyboard

Following are the steps to generate the dump using keyboard:

1. To enable keyboard crash dumping, we need to enable a regkey in regedit. This setting varies w.r.t the keyboard type we use in our machine.

To enable the feature on a computer that uses a PS/2 keyboard, follow these steps:

Start Registry Editor.
Locate the following registry subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\i8042prt\Parameters

On the Edit menu, click Add Value, and then add the following registry entry:

Name: CrashOnCtrlScroll
Data Type: REG_DWORD
Value: 1

Exit Registry Editor and then restart the computer.

Note: Restart the computer after this regkey setting is must for PS/2 Keyboard.

To enable the feature on a computer that uses a USB keyboard, install the hotfix that is mentioned in the "Windows Server 2003 resolution" subsection of the article at http://support.microsoft.com/kb/244139

To make sure that the feature is enabled on a computer that uses a USB keyboard, follow these steps:

Start Registry Editor.
Locate the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\kbdhid\Parameters
Make sure that the following registry entry is enabled:
Name: CrashOnCtrlScroll
Data Type: REG_DWORD
Value: 1
Exit Registry Editor.

Note: For USB keyboard, Restart is not required. Instead unplug the keyboard and plug it again after this regkey setting.

2. After setting this regkey, select the appropriate memory dump file option using the below steps:

Right-click My Computer and then click Properties.
Click the Advanced tab, and then click the Startup and Recovery button.
Click Write Debugging Information and then click to select Complete Memory Dump, Kernel Memory Dump, or Small Memory Dump.

3. When you get the hang, just hold down the CTRL key and pressing the SCROLL LOCK key two times. You must use the CTRL key on the right side of the spacebar.

You can see a Blue screen (as you see during BSOD) wherein it shows the progress of collecting the crash dumps.

Machine will be restarted after this process and you can get the memory dump in "C:\Windows\MEMORY.DMP"

For detailed info, visit this page http://support.microsoft.com/kb/244139

Getting heap information in case of Heap Corruption

Enable WinDbg as Post-Mortem debugger using the command - Windbg -I
Enable Full Page Heap Verification for the process (In case of Heap Corruption). Use the below command gflags.exe -p /enable <Image File> /full
When the crash occurs, type the following in the Windows Debugger Command Window .dump /ma <Dump_Path\File_Name.dmp>
As the Page Heap is enabled, we will get complete details of Stack+Heap in the Dump. (Check the Size of the Dump .... It should be approx. 200 MB +).
When the dump is created, disable the heap for the particular application. gflags.exe -p /disable <Image File>

Note: Make sure the gflags is enabled before the corresponding process (for which you are trying to generate the dump) starts. In other words, kill the process for which you want to get the heap information, enable gflags for that process, start the process, reproduce the crash and get the dump.

For more information on crash dump you can refer http://msdn.microsoft.com/en-us/library/cc265901.aspx

Set Symbols for Debugging

Private symbols files have all information in them for the debugger to resolve data (local variables, structure type information, functions, source file name/line info...).
Public symbols files are essentially stripped versions of private symbols that usually just contain the function information so that stack traces work. Sometimes limited type information is included so that debugger extensions can work.
Export symbols are the symbols contained inside the binary, so you have these without a .pdb. They contain the public/exported function names.

1) Download the XP/Vista symbols from http://www.microsoft.com/whdc/DevTools/Debugging/symbolpkg.mspx

2) Install the symbols

3) Set the symbol path in windbg via,

Start > Programs > Debugging Tools for Windows > Windbg > File > Symbol File Path

Windbg commands to get the first level info

Get Stack Trace using !analyze -v
Task Tree using tlist.exe -t
Services active in each process using tlist.exe -s
Command Line Information using tlist.exe -c
Complete Verbose Information using tlist.exe –v

Remote Kernel Debugging

Sometimes if you want to set a breakpoint in the windows kernel and the kernel hit the breakpoint, then the whole system would freeze, because kernel controls (among other stuff) process management, thread switching, etc. In such situations use of two computers to do a remote kernel debugging would be a better option. Below are the steps to perform Remote Kernel debugging using a serial cable or USB cable or Firewire (IEEE 1394).

Consider “Machine 1”, which is the computer that you want to debug and ‘Machine 2”, which is the computer running Windbg. First, connect these two computers using Serial cable or USB cable or Firewire
Now from Machine 2, start Windbg > File > Kernel Debug & select COM or USB or 1394 according to the connection you have made.
Specify the values in the corresponding tab as below and click OK to the Kernel Debugging window
- COM
  - Baud rate – 115200 (mostly)
  - Port – COM1 or COM2
  - You can check the ‘Pipe’ option if you want to debug a Vmware running in Machine 2
- USB
  - Specify the target name (any name, but the same should be used in Machine 1)
- 1394
  - Specify the Channel (a numeric value, but the same should be used in Machine 1)
Now windbg in Machine 2 will wait until you configure Machine 1 to go into debug mode
From Machine 2, enter into safe mode with networking.
- XP
  - From command prompt go to C:\Windows\System32 folder
  - Type, bootcfg.exe /debug ON /baud 115200 /PORT COM1 /id 1
  - Use corresponding commands for USB or 1394
- Vista
  - From command prompt go to C:\Windows\System32 folder
  - Type, bcdedit /dbgsettings DebugType [debugport:Port] [baudrate:Baud]
  - Ex, for serial debugging (using COM1 at 115200bps) it is: bcdedit /dbgsettings serial debugport:1 baudrate:115200
  - After that, enable debugging by using the below command. bcedit /debug on
  - Restart the Machine 1, if everything was configured correctly, you will see Machine 1 stop during boot, and Machine 2's windbg will print a message saying that the connection was established.
Now reproduce the crash
From Machine 1 , use the below command to get the crash dump. .dump /ma <path\filename>.dmp

Jan 3, 2010