Tru64 UNIX
Kernel Debugging
Part Number: AA-RH99A-TE
July 1999
Product Version:
Tru64 UNIX Version 5.0 or higher
This manual explains how to use tools to debug a kernel and analyze a
crash dump of the Tru64 UNIX (formerly DIGITAL UNIX) operating
system. Also, this manual explains how to write extensions to the kernel
debugging tools.
Compaq Computer Corporation
Houston, Texas
Download from Www.Somanuals.com. All Manuals Search And Download.
Contents
About This Manual
1 Introduction to Kernel Debugging
1.1
1.2
1.3
1.4
Linking a Kernel Image for Debugging ............................
1–1
1–3
1–3
1–5
Debugging Kernel Programs ........................................
Debugging the Running Kernel .....................................
Analyzing a Crash Dump File ......................................
2 Kernel Debugging Utilities
2.1
The dbx Debugger ....................................................
Invoking the dbx Debugger for Kernel Debugging ..........
Debugging Stripped Images ....................................
Specifying the Location of Loadable Modules for Crash
Dumps .............................................................
Examining Memory Contents ..................................
Printing the Values of Variables and Data Structures .....
Displaying a Data Structure Format .........................
Debugging Multiple Threads ...................................
Examining the Exception Frame ..............................
Examining the User Program Stack ..........................
Extracting the Preserved Message Buffer ....................
Debugging on SMP Systems ....................................
The kdbx Debugger ...................................................
Beginning a kdbx Session .......................................
The kdbx Debugger Commands ................................
Using kdbx Debugger Extensions .............................
Displaying the Address Resolution Protocol Table .....
Performing Commands on Array Elements .............
Displaying the Buffer Table ...............................
Displaying the Callout Table and Absolute Callout
2–2
2–2
2–3
2.1.1
2.1.2
2.1.3
2–4
2–5
2–6
2–6
2–7
2.1.4
2.1.5
2.1.6
2.1.7
2.1.8
2.1.9
2.1.10
2.1.11
2.2
2.2.1
2.2.2
2.2.3
2.2.3.1
2.2.3.2
2.2.3.3
2.2.3.4
2–7
2–8
2–10
2–10
2–12
2–12
2–13
2–15
2–16
2–16
2–18
Table ...........................................................
Casting Information Stored in a Specific Address .....
Displaying Machine Configuration .......................
Converting the Base of Numbers .........................
Displaying CPU Use Statistics ............................
2–18
2–19
2–19
2–20
2–20
2.2.3.5
2.2.3.6
2.2.3.7
2.2.3.8
Contents iii
Download from Www.Somanuals.com. All Manuals Search And Download.
2.2.3.9
2.2.3.10
2.2.3.11
2.2.3.12
2.2.3.13
2.2.3.14
2.2.3.15
2.2.3.16
2.2.3.17
2.2.3.18
2.2.3.19
2.2.3.20
2.2.3.21
2.2.3.22
2.2.3.23
2.2.3.24
2.2.3.25
2.2.3.26
2.2.3.27
2.2.3.28
2.2.3.29
2.2.3.30
2.2.3.31
2.2.3.32
2.2.3.33
2.3
Disassembling Instructions ................................
Displaying Remote Exported Entries ....................
Displaying the File Table ..................................
Displaying the udb and tcb Tables ........................
Performing Commands on Lists ..........................
Displaying the lockstats Structures ......................
Displaying lockinfo Structures ............................
Displaying the Mount Table ...............................
Displaying the Namecache Structures ...................
Displaying Processes’ Open Files .........................
Converting the Contents of Memory to Symbols .......
Displaying the Process Control Block for a Thread ....
Formatting Command Arguments ........................
Displaying the Process Table ..............................
Converting an Address to a Procedure name ...........
Displaying Sockets from the File Table ..................
Displaying a Summary of the System Information ....
Displaying a Summary of Swap Space ...................
Displaying the Task Table .................................
Displaying Information About Threads ..................
Displaying a Stack Trace of Threads .....................
Displaying a u Structure ...................................
Displaying References to the ucred Structure ..........
Removing Aliases ............................................
Displaying the vnode Table ................................
The kdebug Debugger ................................................
Getting Ready to Use the kdebug Debugger .................
Invoking the kdebug Debugger ................................
Diagnosing kdebug Setup Problems ...........................
Notes on Using the kdebug Debugger ........................
The crashdc Utility ...................................................
2–21
2–21
2–21
2–22
2–22
2–24
2–25
2–26
2–27
2–27
2–28
2–28
2–28
2–29
2–30
2–30
2–30
2–31
2–31
2–32
2–32
2–33
2–34
2–36
2–36
2–37
2–39
2–41
2–42
2–44
2–44
2.3.1
2.3.2
2.3.3
2.3.4
2.4
3 Writing Extensions to the kdbx Debugger
3.1
3.2
Basic Considerations for Writing Extensions .....................
Standard kdbx Library Functions ..................................
Special kdbx Extension Data Types ...........................
Converting an Address to a Procedure Name ...............
Getting a Representation of an Array Element .............
Retrieving an Array Element Value ...........................
Returning the Size of an Array ................................
Casting a Pointer to a Data Structure ........................
3–1
3–2
3–2
3–3
3–4
3–4
3–6
3–6
3.2.1
3.2.2
3.2.3
3.2.4
3.2.5
3.2.6
iv Contents
Download from Www.Somanuals.com. All Manuals Search And Download.
3.2.7
3.2.8
3.2.9
Checking Arguments Passed to an Extension ...............
Checking the Fields in a Structure ............................
Setting the kdbx Context .......................................
Passing Commands to the dbx Debugger .....................
Dereferencing a Pointer .........................................
Displaying the Error Messages Stored in Fields ............
Converting a Long Address to a String Address ............
Freeing Memory ..................................................
Passing Commands to the kdbx Debugger ...................
Getting the Address of an Item in a Linked List ............
Passing an Extension to kdbx ..................................
Getting the Next Token as an Integer ........................
Getting the Next Token as a String ...........................
Displaying a Message ...........................................
Displaying Status Messages ....................................
Exiting from an Extension ......................................
Reading the Values in Structure Fields ......................
Returning a Line of kdbx Output ..............................
Reading an Area of Memory ....................................
Reading the Response to a kdbx Command ..................
Reading Symbol Representations ..............................
Reading a Symbol’s Address ....................................
Reading the Value of a Symbol .................................
Getting the Address of a Data Representation ..............
Converting a String to a Number ..............................
Examples of kdbx Extensions .......................................
Compiling Custom Extensions ......................................
Debugging Custom Extensions .....................................
3–7
3–7
3–8
3–9
3–9
3.2.10
3.2.11
3.2.12
3.2.13
3.2.14
3.2.15
3.2.16
3.2.17
3.2.18
3.2.19
3.2.20
3.2.21
3.2.22
3.2.23
3.2.24
3.2.25
3.2.26
3.2.27
3.2.28
3.2.29
3.2.30
3.2.31
3.3
3–10
3–10
3–11
3–11
3–13
3–14
3–14
3–15
3–16
3–16
3–17
3–17
3–18
3–18
3–19
3–20
3–20
3–21
3–21
3–22
3–22
3–35
3–36
3.4
3.5
4 Crash Analysis Examples
4.1
4.2
Guidelines for Examining Crash Dump Files ....................
Identifying a Crash Caused by a Software Problem .............
Using dbx to Determine the Cause of a Software Panic ...
Using kdbx to Determine the Cause of a Software Panic ..
Identifying a Hardware Exception .................................
Using dbx to Determine the Cause of a Hardware Error ..
Using kdbx to Determine the Cause of a Hardware Error
Finding a Panic String in a Thread Other Than the Current
Thread ..................................................................
Identifying the Cause of a Crash on an SMP System ...........
4–1
4–2
4–2
4–3
4–4
4–4
4–7
4.2.1
4.2.2
4.3
4.3.1
4.3.2
4.4
4–8
4–9
4.5
Contents
v
Download from Www.Somanuals.com. All Manuals Search And Download.
A Output from the crashdc Command
Index
Examples
3–1
3–2
3–3
3–4
3–5
Template Extension Using Lists ....................................
Extension That Uses Linked Lists: callout.c .....................
Template Extensions Using Arrays ................................
Extension That Uses Arrays: file.c .................................
Extension That Uses Global Symbols: sum.c .....................
3–23
3–24
3–27
3–28
3–34
Figures
2–1
Using a Gateway System During Remote Debugging ...........
The dbx Address Modes ..............................................
2–38
2–5
Tables
2–1
vi Contents
Download from Www.Somanuals.com. All Manuals Search And Download.
About This Manual
This manual provides information on the tools used to debug a kernel and
analyze a crash dump file of the Tru64™ UNIX (formerly DIGITAL UNIX)
operating system. It also explains how to write extensions to the kernel
debugging tools. You can use extensions to display customized information
from kernel data structures or a crash dump file.
Audience
This manual is intended for system programmers who write programs that
use kernel data structures and are built into the kernel. It is also intended
for system administrators who are responsible for managing the operating
system. System programmers and administrators should have in-depth
knowledge of operating system concepts, commands, and utilities.
New and Changed Features
The following list describes changes that have been made to this manual
for Tru64 UNIX Version 5.0:
•
The former Chapter 4, Managing Crash Dumps, has been deleted and
its contents have been moved to the System Administration manual.
All information on that subject is now in one manual. The System
Administration manual was chosen because many aspects of managing
crash dumps (such as storage considerations and default settings) are
handled by a system administrator, often during system installation.
•
Crash dumps are now compressed by default and are stored in
compressed crash dump files. These are named vmzcore.n to
differentiate them from the uncompressed vmcore.n files. Starting with
Version 5.0, all the Tru64 UNIX debugging tools can read vmzcore.n as
well as vmcore.n files. Examples throughout this manual have been
updated to show use of vmzcore.n files.
•
•
When debugging a crash dump with dbx or kdbx, you can examine the
call stack of the user program whose execution precipitated the kernel
crash. For more information, see Section 2.1.9.
If a loadable kernel module was moved to another location after a kernel
crash, you can specify the directory path where dbx should look for the
module. For more information, see Section 2.1.3.
About This Manual vii
Download from Www.Somanuals.com. All Manuals Search And Download.
Organization
This manual consists of four chapters and one appendix:
Chapter 1
Chapter 2
Chapter 3
Introduces the concepts of kernel debugging and
crash dump analysis.
Describes the tools used to debug kernels and
analyze crash dump files.
Describes how to write a kdbx debugger extension. This
chapter assumes you have purchased and installed a Tru64
UNIX Source Kit and so have access to source files.
Chapter 4
Provides background information useful for and examples
of analyzing crash dump files.
Contains example output from the crashdc utility.
Appendix A
Related Documents
For additional information, refer to the following manuals:
•
The Alpha Architecture Reference Manual describes how the operating
system interfaces with the Alpha hardware.
•
The Alpha Architecture Handbook gives an overview of the Alpha
hardware architecture and describes the 64-bit Alpha RISC (Reduced
Instruction Set Computing) instruction set.
•
•
•
The Installation Guide and Installation Guide — Advanced Topics
describe how to install your operating system.
The System Administration manual provides information on managing
and monitoring your system, including managing crash dumps.
The Programmer’s Guide provides information on the tools, specifically
the dbx debugger, for programming on the Tru64 UNIX operating
system. This manual also provides information about creating
configurable kernel subsystems.
•
The Writing Kernel Modules manual discusses how to code kernel
modules (single binary images) that can be statically loaded as part of
the /vmunix kernel or dynamically loaded into memory, that enhance
the functionality of the Unix kernel.
Icons on Tru64 UNIX Printed Manuals
The printed version of the Tru64 UNIX documentation uses letter icons on
the spines of the manuals to help specific audiences quickly find the manuals
that meet their needs. (You can order the printed documentation from
Compaq.) The following list describes this convention:
viii About This Manual
Download from Www.Somanuals.com. All Manuals Search And Download.
G
S
Manuals for general users
Manuals for system and network administrators
Manuals for programmers
P
R
Manuals for reference page users
Some manuals in the documentation help meet the needs of several
audiences. For example, the information in some system manuals is also
used by programmers. Keep this in mind when searching for information
on specific topics.
The Documentation Overview provides information on all of the manuals in
the Tru64 UNIX documentation set.
Reader’s Comments
Compaq welcomes any comments and suggestions you have on this and
other Tru64 UNIX manuals.
You can send your comments in the following ways:
•
•
Fax: 603-884-0120 Attn: UBPG Publications, ZKO3-3/Y32
Internet electronic mail: [email protected]
A Reader’s Comment form is located on your system in the following
location:
/usr/doc/readers_comment.txt
Please include the following information along with your comments:
•
The full title of the manual and the order number. (The order number
appears on the title page of printed and PDF versions of a manual.)
•
The section numbers and page numbers of the information on which
you are commenting.
•
•
The version of Tru64 UNIX that you are using.
If known, the type of processor that is running the Tru64 UNIX software.
The Tru64 UNIX Publications group cannot respond to system problems
or technical support inquiries. Please address technical questions to your
local system vendor or to the appropriate Compaq technical support office.
Information provided with the software media explains how to send problem
reports to Compaq.
Conventions
The following conventions are used in this manual:
About This Manual ix
Download from Www.Somanuals.com. All Manuals Search And Download.
%
$
A percent sign represents the C shell system prompt.
A dollar sign represents the system prompt for the
Bourne, Korn, and POSIX shells.
#
A number sign represents the superuser prompt.
% cat
Boldface type in interactive examples indicates
typed user input.
file
Italic (slanted) type indicates variable values,
placeholders, and function argument names.
[| ]
{| }
In syntax definitions, brackets indicate items that
are optional and braces indicate items that are
required. Vertical bars separating items inside
brackets or braces indicate that you choose one item
from among those listed.
.
.
.
A vertical ellipsis indicates that a portion of an
example that would normally be present is not
shown.
cat(1)
A cross-reference to a reference page includes
the appropriate section number in parentheses.
For example, cat(1) indicates that you can find
information on the cat command in Section 1 of
the reference pages.
Ctrl/x
This symbol indicates that you hold down the
first named key while pressing the key or mouse
button that follows the slash. In examples, this
key combination is enclosed in a box (for example,
Ctrl/C ).
x
About This Manual
Download from Www.Somanuals.com. All Manuals Search And Download.
1
Introduction to Kernel Debugging
Kernel debugging is a task normally performed by systems engineers writing
kernel programs. A kernel program is one that is built as part of the kernel
and that references kernel data structures. System administrators might
also debug the kernel in the following situations:
•
•
•
A process is hung or stops running unexpectedly
The need arises to examine, and possibly modify, kernel parameters
The system itself hangs, panics, or crashes
This manual describes how to debug kernel programs and the kernel. It also
includes information about analyzing crash dump files.
In addition to the information provided here, tracing a kernel problem can
require a basic understanding of one or more of the following technical areas:
•
The hardware architecture
See the Alpha Architecture Handbook for an overview of the Alpha
hardware architecture and a description of the 64-bit Alpha RISC
instruction set.
•
The internal design of the operating system at a source code and data
structure level
See the Alpha Architecture Reference Manual for information on how the
Tru64 UNIX operating system interfaces with the hardware.
This chapter provides an overview of the following topics:
•
Linking a kernel image prior to debugging for systems that are running
a kernel built at boot time. (Section 1.1)
•
•
•
Debugging kernel programs (Section 1.2)
Debugging the running kernel (Section 1.3)
Analyzing a crash dump file(Section 1.4)
1.1 Linking a Kernel Image for Debugging
By default, the kernel is a statically linked image that resides in the file
/vmunix. However, your system might be configured so that it is linked
at bootstrap time. Rather than being a bootable image, the boot file is a
Introduction to Kernel Debugging 1–1
Download from Www.Somanuals.com. All Manuals Search And Download.
text file that describes the hardware and software that will be present on
the running system. Using this information, the bootstrap linker links the
modules that are needed to support this hardware and software. The linker
builds the kernel directly into memory.
You cannot directly debug a bootstrap-linked kernel because you must supply
the name of an image to the kernel debugging tools. Without the image, the
tools have no access to symbol names, variable names, and so on. Therefore,
the first step in any kernel debugging effort is to determine whether your
kernel was linked at bootstrap time. If the kernel was linked at bootstrap
time, you must then build a kernel image file to use for debugging purposes.
The best way to determine whether your system is bootstrap linked or
statically linked is to use the file command to test the type of file from
which your system was booted. If your system is a bootstrap-linked system,
it was booted from an ASCII text file; otherwise, it was booted from an
executable image file. For example, issue the following command to
determine the type of file from which your system was booted:
#/usr/bin/file ‘/usr/sbin/sizer -b‘
/etc/sysconfigtab: ascii text
The sizer -b command returns the name of the file from which the system
was booted. This file name is input to the file command, which determines
that the system was booted from an ASCII text file. The output shown in the
preceeding example indicates that the system is a bootstrap-linked system.
If the system had been booted from an executable image file named vmunix,
the output from the file command would have appeared as follows:
vmunix:COFF format alpha executable or object module
not stripped
If your system is running a bootstrap-linked kernel, build a kernel image
that is identical to the bootstrap-linked kernel your system is running, by
entering the following command:
# /usr/bin/ld -o vmunix.image ‘/usr/sbin/sizer -m‘
The output from the sizer -m command is a list of the exact modules and
linker flags used to build the currently running bootstrap-linked kernel.
This output causes the ld command to create a kernel image that is identical
to the bootstrap-linked kernel running on your system. The kernel image is
written to the file named by the -o flag, in this case the vmunix.image file.
Once you create this image, you can debug the kernel as described in this
manual, using the dbx, kdbx, and kdebug debuggers. When you invoke
the dbx or kdbx debugger, remember to specify the name of the kernel
image file you created with the ld command, such as the vmunix.image
file shown here.
When you are finished debugging the kernel, you can remove the kernel
image file you created for debugging purposes.
1–2 Introduction to Kernel Debugging
Download from Www.Somanuals.com. All Manuals Search And Download.
1.2 Debugging Kernel Programs
Kernel programs can be difficult to debug because you normally cannot
control kernel execution. To make debugging kernel programs more
convenient, the system provides the kdebug debugger. The kdebug
debugger is code that resides inside the kernel and allows you to use the dbx
debugger to control execution of a running kernel in the same manner as
you control execution of a user space program. To debug a kernel program
in this manner, follow these steps:
1. Build your kernel program into the kernel on a test system.
2. Set up the kdebug debugger, as described in Section 2.3.
3. Issue the dbx -remote command on a remote build system, supplying
the pathname of the kernel running on the test system.
4. Set breakpoints and enter dbx commands as you normally would.
Section 2.1 describes some of the commands that are useful during
kernel debugging. For general information about using dbx, see the
Programmer’s Guide.
The system also provides the kdbx debugger, which is designed especially
for debugging kernel code. This debugger contains a number of special
commands, called extensions, that allow you to display kernel data
structures in a readable format. Section 2.2 describes using kdbx and its
extensions. (You cannot use the kdbx debugger with the kdebug debugger.)
Another feature of kdbx is that you can customize it by writing your own
extensions. The system contains a set of kdbx library routines that you can
use to create extensions that display kernel data structures in ways that are
meaningful to you. Chapter 3 describes writing kdbx extensions.
1.3 Debugging the Running Kernel
When you have problems with a process or set of processes, you can attempt
to identify the problem by debugging the running kernel. You might also
invoke the debugger on the running kernel to examine the values assigned
to system parameters. (You can modify the value of the parameters using
the debugger, but this practice can cause problems with the kernel and
should be avoided.)
You use the dbx or kdbx debugger to examine the state of processes running
on your system and to examine the value of system parameters. The kdbx
debugger provides special commands, called extensions, that you can use to
display kernel data structures. (Section 2.2.3 describes the extensions.)
To examine the state of processes, you invoke the debugger (as described in
Section 2.1 or Section 2.2) using the following command:
Introduction to Kernel Debugging 1–3
Download from Www.Somanuals.com. All Manuals Search And Download.
# dbx -k /vmunix /dev/mem
This command invokes dbx with the kernel debugging flag, −k, which
maps kernel addresses to make kernel debugging easier. The /vmunix and
/dev/mem parameters cause the debugger to operate on the running kernel.
Once in the dbx environment, you use dbx commands to display process IDs
and trace execution of processes. You can perform the same tasks using the
kdbx debugger. The following example shows the dbx command you use to
display process IDs:
(dbx) kps
PID
00000
00001
00014
00016
COMM
kernel idle
init
kloadsrv
update
.
.
.
If you want to trace the execution of the kloadsrv daemon, use the dbx
command to set the $pid symbol to the process IDof the kloadsrv daemon.
Then, enter the t command:
(dbx) set $pid = 14
(dbx) t
thread_block() ["/usr/sde/build/src/kernel/kern/sched_prim.c":1623, 0xfffffc0000\
43d77c]
>
0
1
mpsleep(0xffffffff92586f00, 0x11a, 0xfffffc0000279cf4, 0x0, 0x0) ["/usr/sde/build\
/src/kernel/bsd/kern_synch.c":411, 0xfffffc000040adc0]
2
sosleep(0xffffffff92586f00, 0x1, 0xfffffc000000011a, 0x0, 0xffffffff81274210) ["/usr/sde\
/build/src/kernel/bsd/uipc_socket2.c":654, 0xfffffc0000254ff8]
sosbwait(0xffffffff92586f60, 0xffffffff92586f00, 0x0, 0xffffffff92586f00, 0x10180) ["/usr\
/sde/build/src/kernel/bsd/uipc_socket2.c":630, 0xfffffc0000254f64]
soreceive(0x0, 0xffffffff9a64f658, 0xffffffff9a64f680, 0x8000004300000000, 0x0) ["/usr/sde\
/build/src/kernel/bsd/uipc_socket.c":1297, 0xfffffc0000253338]
recvit(0xfffffc0000456fe8, 0xffffffff9a64f718, 0x14000c6d8, 0xffffffff9a64f8b8,\
3
4
5
0xfffffc000043d724) ["/usr/sde/build/src/kernel/bsd/uipc_syscalls.c":1002,\
0xfffffc00002574f0]
6
recvfrom(0xffffffff81274210, 0xffffffff9a64f8c8, 0xffffffff9a64f8b8, 0xffffffff9a64f8c8,\
0xfffffc0000457570) ["/usr/sde/build/src/kernel/bsd/uipc_syscalls.c":860,\
0xfffffc000025712c]
7
orecvfrom(0xffffffff9a64f8b8, 0xffffffff9a64f8c8, 0xfffffc0000457570, 0x1, 0xfffffc0000456fe8)\
["/usr/sde/build/src/kernel/bsd/uipc_syscalls.c":825, 0xfffffc000025708c]
syscall(0x120024078, 0xffffffffffffffff, 0xffffffffffffffff, 0x21, 0x7d) ["/usr/sde\
/build/src/kernel/arch/alpha/syscall_trap.c":515, 0xfffffc0000456fe4
_Xsyscall(0x8, 0x12001acb8, 0x14000eed0, 0x4, 0x1400109d0) ["/usr/sde/build\
8
9
/src/kernel/arch/alpha/locore.s":1046, 0xfffffc00004486e4]
(dbx) exit
Often, looking at the trace of a process that is hanging or has unexpectedly
stopped running reveals the problem. Once you find the problem, you can
modify system parameters, restart daemons, or take other corrective actions.
For more information about the commands you can use to debug the running
kernel, see Section 2.1 and Section 2.2.
1–4 Introduction to Kernel Debugging
Download from Www.Somanuals.com. All Manuals Search And Download.
1.4 Analyzing a Crash Dump File
If your system crashes, you can often find the cause of the crash by using
dbx or kdbx to debug or analyze a crash dump file.
The operating system can crash because one of the following occurs:
•
•
•
Hardware exception
Software panic
Hung system
When a system hangs, it is often necessary to force the system to create
dumps that you can analyze to determine why the system hung. The
System Administration manual describes the procedure for forcing a
crash dump of a hung system.
•
Resource exhaustion
The system crashes or hangs because it cannot continue executing. Normally,
even in the case of a hardware exception, the operating system detects
the problem. (For example a machine-checking routine might discover a
hardware problem and begin the process of crashing the system.) In general,
the operating system performs the following steps when it detects a problem
from which it cannot recover:
1. It calls the system panic function.
The panic function saves the contents of registers and sends the panic
string (a message describing the reason for the system panic) to the
error logger and the console terminal.
If the system is a Symmetric Multiprocessing (SMP) system, the panic
function notifies the other CPUs in the system that a panic has
occurred. The other CPUs then also execute the panic function and
record the following panic string:
cpu_ip_intr: panic request
Once each CPU has recorded the system panic, execution continues only
on the master CPU. All other CPUs in the SMP system stop execution.
2. It calls the system boot function.
The boot function records the stack.
3. It calls the dump function.
The dump function copies core memory into swap partitions and the
system stops running or the reboot process begins. Console environment
variables control whether the system reboots automatically. (The
System Administration manual describes these environment variables.)
Introduction to Kernel Debugging 1–5
Download from Www.Somanuals.com. All Manuals Search And Download.
At system reboot time, the copy of core memory saved in the swap partitions
is copied into a file, called a crash dump file. You can analyze the crash
dump file to determine what caused the crash. By default, the crash dump is
a partial (rather than full) dump and is in compressed form. For complete
information about managing crash dumps and crash dump files, including
how to change default settings, see the System Administration manual. For
examples of analyzing crash dump files, see Chapter 4.
1–6 Introduction to Kernel Debugging
Download from Www.Somanuals.com. All Manuals Search And Download.
2
Kernel Debugging Utilities
The Tru64 UNIX system provides several tools you can use to debug the
kernel and kernel programs. The Ladebug debugger (available as an option)
is also capable of debugging the kernel.
This chapter describes three debuggers and a utility for analyzing crash
dumps:
•
The dbx debugger, which is described for kernel debugging in Section 2.1.
(For general dbx user information, see the Programmer’s Guide.)
You can use the dbx debugger to display the values of kernel variables
and kernel structures. However, you must understand the structures
and be prepared to follow the address links to find the information you
need. You cannot use dbx alone to control execution of the running
kernel, for example by setting breakpoints.
•
The kdbx debugger, which is described in Section 2.2.
The kdbx debugger is an interface to dbx that is tailored specifically
to debugging kernel code. The kdbx debugger has knowledge of the
structure of kernel data and so displays kernel data in a readable format.
Also, kdbx is extensible, allowing you to create commands that are
tailored to your kernel-debugging needs. (Chapter 3 describes how to
tailor the kdbx debugger.) However, you cannot use dbx command line
editing features when you use the kdbx debugger.
•
•
The kdebug debugger, which is described in Section 2.3.
The kdebug debugger is a kernel-debugging program that resides
inside the kernel. Working with a remote version of the dbx debugger,
the kdebug debugger allows you to set breakpoints in and control the
execution of kernel programs and the kernel.
The crashdc utility, which is described in Section 2.4.
The crashdc utility is a crash dump analysis tool. This utility is useful
when you need to determine why the system is hanging or crashing.
The sections that follow describe how to use these tools to debug the kernel
and kernel programs.
Kernel Debugging Utilities 2–1
Download from Www.Somanuals.com. All Manuals Search And Download.
______________________
Note _______________________
Starting with Tru64 UNIX Version 5.0, all the previously
mentioned tools can be used with compressed (vmzcore.n) and
uncompressed (vmcore.n) crash dump files. Older versions of
these tools can read only vmcore.n files. If you are using an
older version of a tool, use the expand_dump utility to produce
a vmcore.n file from a vmzcore.n file. For more information
about compressed and uncompressed crash dump files, see
expand_dump(8) and the System Administration manual.
2.1 The dbx Debugger
The dbx debugger is a symbolic debugger that allows you to examine,
modify, and display the variables and data structures found in stripped or
nonstripped kernel images.
The following sections describe how to invoke the dbx debugger for kernel
debugging (Section 2.1.1) and how to use its commands to perform tasks
such as the following:
•
•
Debugging stripped images (Section 2.1.2)
Specifying the location of loadable modules for crash dumps
(Section 2.1.3)
•
•
Examining memory contents (Section 2.1.4)
Displaying the values of kernel variables, and the value and format of
kernel data structures (Section 2.1.5)
•
•
•
•
•
•
Displaying the format of a data structure (Section 2.1.6)
Debugging multiple threads (Section 2.1.7)
Examining the exception frame (Section 2.1.8)
Examining the user program stack (Section 2.1.9)
Extracting the preserved message buffer (Section 2.1.10)
Debugging on SMP systems (Section 2.1.11)
For more information on dbx, see the Programmer’s Guide.
2.1.1 Invoking the dbx Debugger for Kernel Debugging
To debug kernel code with the dbx debugger, you use the −k flag. This flag
causes dbx to map memory addresses. When you use the dbx −k command,
the debugger operates on two separate files that reflect the current state of
the kernel that you want to examine. These files are as follows:
•
The disk version of the executable kernel image
2–2 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
•
The system core memory image
These files may be files from a running system, such as /vmunix and
/dev/mem, or dump files, such as vmunix.n and vmzcore.n (compressed)
or vmcore.n (uncompressed). By default, crash dump files are created in
the /var/adm/crash directory (see the System Administration manual).
______________________
Note _______________________
You might need to be the superuser (root login) to examine the
running system or crash dump files produced by savecore.
Whether you need to be the superuser depends on the directory
and file protections for the files you attempt to examine with
the dbx debugger.
Use the following dbx command to examine the running system:
# dbx −k /vmunix /dev/mem
Use a dbx command similar to the following to examine a compressed or
uncompressed crash dump file, respectively:
# dbx −k vmunix.1 vmzcore.1
# dbx −k vmunix.1 vmcore.1
The version number (.1, in this example) is determined by the value
contained in the bounds file, which is located in the same directory as the
dump files.
2.1.2 Debugging Stripped Images
By default, the kernel is compiled with a debugging flag that does not strip
all of the symbol table information from the executable kernel image. The
kernel is also partially optimized during the compilation process by default.
If the kernel or any other file is fully optimized and stripped of all symbol
table information during compilation, your ability to debug the file is greatly
reduced. However, the dbx debugger provides commands to aid you in
debugging stripped images.
When you attempt to display the contents of a symbol during a debugging
session, you might encounter messages such as the following:
No local symbols.
Undefined symbol.
Inactive symbol.
These messages might indicate that you are debugging a stripped image.
To see the contents of all symbols during a debugging session, you can leave
the debugging session, rebuild all stripped modules (but do not strip them),
and reenter the debugging session. However, on certain occasions, you might
Kernel Debugging Utilities 2–3
Download from Www.Somanuals.com. All Manuals Search And Download.
want to add a symbol table to your current debugging session rather than
end the session and start a new one. To add a symbol table to your current
debugging session, follow these steps:
1. Go to a window other than the one in which the debugger is running,
or put the debugger in the background, and rebuild the modules for
which you need a symbol table.
2. Once the modules build correctly, use the ostrip command to strip a
symbol table out of the resulting executable file. For example, if your
executable file is named kernel_program, issue a command such as
the following one:
% /usr/ucb/ostrip -t kernel_program
The -t flag causes the ostrip command to produce two files. One,
named kernel_program, is the stripped executable image. The other,
named kernel_program.stb, contains the symbol table information
for the kernel_program module. (For more information about the
ostrip command, see ostrip(1).)
3. Return to the debugging session and add the symbol table file by issuing
the dbx command stbadd as follows:
dbx> stbadd kernel_program.stb
You can specify an absolute or relative pathname on the stbadd
command line.
Once you issue this command, you can display the contents of symbols
included in the symbol table just as if you had built the module you
are debugging without stripping.
You can also delete symbol tables from a debugging session using the dbx
command stbdel. For more information about this command, see dbx(1).
2.1.3 Specifying the Location of Loadable Modules for Crash Dumps
When a crash dump occurs, the location of any loadable modules used
by the kernel is recorded in the crash dump file, enabling dbx to find the
modules. If the version of a loadable module that was running when the
crash occurred is moved to a different location, dbx will not find it. You can
specify the directory path where dbx should look for loadable modules by
using any one of the following methods (see dbx(1) for complete details):
•
On the dbx command line, specify the directory path with the
-module_path option. For example:
# dbx -k vmunix.1 vmzcore.1 -module_path /project4/mod_dir
•
Before invoking dbx, set the environment variable DBX_MODULE_PATH.
For example:
# setenv DBX_MODULE_PATH /project4/mod_dir
2–4 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
•
During the dbx session, if you want to load a module dynamically, first
set the $module_path dbx variable and then use the addobj command
to load the module, as in the following example:
(dbx) set $module_path /project4/mod_dir
(dbx) addobj kmodC
To verify that modules are being loaded from the correct location, turn on
verbose module-loading using any one of the following methods:
•
•
Specify the -module_verbose dbx command option.
Set the DBX_MODULE_VERBOSE environment variable to any integer
value.
•
Set the $module_verbose dbx variable to a nonzero value.
2.1.4 Examining Memory Contents
To examine memory contents with dbx, use the following syntax:
address/count[mode]
The count argument specifies the number of items that the debugger
displays at the specified address, and the mode argument determines how
dbx displays memory. If you omit the mode argument, the debugger uses
the previous mode. The initial default mode is X (hexadecimal). Table 2–1
lists the dbx address modes.
Table 2–1: The dbx Address Modes
Mode
Description
b
c
Displays a byte in octal.
Displays a byte as a character.
Displays a short word in decimal.
Displays a long word in decimal.
Displays a single precision real number.
Displays a double precision real number.
Displays machine instructions.
Displays data in typed format.
Displays a short word in octal.
Displays a long word in octal.
d
D
f
g
i
n
o
O
s
x
X
Displays a string of characters that ends in a null.
Displays a short word in hexadecimal.
Displays a long word in hexadecimal.
Kernel Debugging Utilities 2–5
Download from Www.Somanuals.com. All Manuals Search And Download.
The following examples show how to use dbx to examine kernel images:
(dbx) _realstart/X
fffffc00002a4008: c020000243c4153e
(dbx) _realstart/i
[_realstart:153, 0xfffffc00002a4008] subq
(dbx) _realstart/10i
sp, 0x20, sp
[_realstart:153, 0xfffffc00002a4008] subq
[_realstart:154, 0xfffffc00002a400c] br
sp, 0x20, sp
r1, 0xfffffc00002a4018
[_realstart:156, 0xfffffc00002a4010] call_pal
[_realstart:157, 0xfffffc00002a4014] bgt
[_realstart:171, 0xfffffc00002a4018] ldq
[_realstart:172, 0xfffffc00002a401c] stq
[_realstart:177, 0xfffffc00002a4020] bis
[_realstart:178, 0xfffffc00002a4024] bis
[_realstart:179, 0xfffffc00002a4028] bis
[_realstart:181, 0xfffffc00002a402c] bis
0x4994e0
r31, 0xfffffc00002a3018
gp, 0(r1)
r31, 24(sp)
r16, r31, r9
r17, r31, r10
r18, r31, r11
r19, r31, r12
2.1.5 Printing the Values of Variables and Data Structures
You can use the print command to examine values of variables and data
structures. The print command has the following syntax:
print expression
p expression
For example:
(dbx) print utsname
struct {
sysname = "OSF1"
nodename = "system.dec.com"
release = "V5.0"
version = "688.2"
machine = "alpha"
}
Note that dbx has a default alias of p for print:
(dbx) p utsname
2.1.6 Displaying a Data Structure Format
You can use the whatis command to display the format for many of the
kernel data structures. The whatis command has the following syntax:
whatis type name
The following example displays the itimerval data structure:
(dbx) whatis struct itimerval
struct itimerval {
struct timeval {
int tv_sec;
int tv_usec;
} it_interval;
2–6 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
struct timeval {
int tv_sec;
int tv_usec;
} it_value;
};
2.1.7 Debugging Multiple Threads
You can use the dbx debugger to examine the state of the kernel’s threads
with the querying and scoping commands described in this section. You
use these commands to show process and thread lists and to change the
debugger’s context (by setting its current process and thread variables)
so that a stack trace for a particular thread can be displayed. Use these
commands to examine the state of the kernel’s threads:
print $tid
print $pid
where
Display the thread ID of the current
thread
Display the process ID of the current
process
Display a stack trace for the current
thread
tlist
Display a list of kernel threads for the
current process
kps
Display a list of processes (not available
when used with kdebug)
set $pid=process_id
Change the context to another process (a
process IDof 0 changes context to the
kernel)
tset thread_id
Change the context to another thread
Displays the stack trace for all threads.
tstack
2.1.8 Examining the Exception Frame
When you work with a crash dump file to debug your code, you can use
dbx to examine the exception frame. The exception frame is a stack frame
created during an exception. It contains the registers that define the state
of the routine that was running at the time of the exception. Refer to the
Kernel Debugging Utilities 2–7
Download from Www.Somanuals.com. All Manuals Search And Download.
/usr/include/machine/reg.h header file to determine where registers
are stored in the exception frame.
The savedefp variable contains the location of the exception frame. (Note
that no exception frames are created when you force a system to dump, as
described in the System Administration manual.) The following example
shows an example exception frame:
(dbx) print savedefp/33X
ffffffff9618d940: 0000000000000000 fffffc000046f888
ffffffff9618d950: ffffffff86329ed0 0000000079cd612f
ffffffff9618d960: 000000000000007d 0000000000000001
ffffffff9618d970: 0000000000000000 fffffc000046f4e0
ffffffff9618d980: 0000000000000000 ffffffff9618a2f8
ffffffff9618d990: 0000000140012b20 0000000000000000
ffffffff9618d9a0: 000000014002ee10 0000000000000000
ffffffff9618d9b0: 00000001400075e8 0000000140026240
ffffffff9618d9c0: ffffffff9618daf0 ffffffff8635af20
ffffffff9618d9d0: ffffffff9618dac0 00000000000001b0
ffffffff9618d9e0: fffffc00004941b8 0000000000000000
ffffffff9618d9f0: 0000000000000001 fffffc000028951c
ffffffff9618da00: 0000000000000000 0000000000000fff
ffffffff9618da10: 0000000140026240 0000000000000000
ffffffff9618da20: 0000000000000000 fffffc000047acd0
ffffffff9618da30: 0000000000901402 0000000000001001
ffffffff9618da40: 0000000000002000
2.1.9 Examining the User Program Stack
When debugging a crash dump with dbx, you can examine the call stack of
the user program whose execution precipitated the kernel crash. To examine
a crash dump and also view the user program stack, you must invoke dbx
using the following command syntax:
dbx -k vmunix.n vm[z]core.n path/user-program
The version number (n) is determined by the value contained in the
bounds file, which is located in the same directory as the dump files. The
user-program parameter specifies the user program executable.
The crash dump file must contain a full crash dump. For information on
setting system defaults for full or partial crash dumps, see the System
Administration manual. You can use the assign command in dbx, as shown
in the following example, to temporarily specify a full crash dump. This
setting stays in effect until the system is rebooted.
# dbx -k vmunix.3
dbx version 5.0
.
.
.
(dbx) assign partial_dump=0
2–8 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
To specify a full crash dump permanently so that this setting remains in
effect after a reboot, use the patch command in dbx, as shown in the
following example:
(dbx) patch partial_dump=0
With either command, a partial_dump value of 1 specifies a partial dump.
The following example shows how to examine the state of a user program
named test1 that purposely precipitated a kernel crash with a syscall
after several recursive calls:
# dbx -k vmunix.1 vmzcore.1 /usr/proj7/test1
dbx version 5.0
Type ’help’ for help.
stopped at [boot:1890 ,0xfffffc000041ebe8]
Source not available
warning: Files compiled -g3: parameter values probably wrong
(dbx) where
0 boot() ["../../../../src/kernel/arch/alpha/machdep.c":1890,
1
>
0xfffffc000041ebe8]
1 panic(0xfffffc000051e1e0, 0x8, 0x0, 0x0, 0xffffffff888c3a38)
["../../../../src/kernel/bsd/subr_prf.c":824, 0xfffffc0000281974]
2 syscall(0x2d, 0x1, 0xffffffff888c3ce0, 0x9aa1e00000000, 0x0)
["../../../../src/kernel/arch/alpha/syscall_trap.c":593, 0xfffffc0000423be4]
3 _Xsyscall(0x8, 0x3ff8010f9f8, 0x140008130, 0xaa, 0x3ffc0097b70)
["../../../../src/kernel/arch/alpha/locore.s":1409, 0xfffffc000041b0f4]
4 __syscall(0x0, 0x0, 0x0, 0x0, 0x0) [0x3ff8010f9f4]
5 justtryme(scall = 170, cpu = 0, levels = 25) ["test1.c":14,
0x120001310]
6 recurse(inbox = (...)) ["test1.c":28, 0x1200013c4]
7 recurse(inbox = (...)) ["test1.c":30, 0x120001400]
8 recurse(inbox = (...)) ["test1.c":30, 0x120001400]
9 recurse(inbox = (...)) ["test1.c":30, 0x120001400]
.
.
.
30 recurse(inbox = (...)) ["test1.c":30, 0x120001400]
31 main(argc = 3, argv = 0x11ffffd08) ["test1.c":52, 0x120001518]
(dbx) up 8
recurse: 30
(dbx) print r
struct {
2
if (r.a[2] > 0) recurse(r);
3
a = {
[0] 170
[1] 0
[2] 2
[3] 0
.
.
.
(dbx) print r.a[511]
4
25
(dbx)
1
The where command displays the kernel stack followed by the user
program stack at the time of the crash. In this case, the kernel stack
has 4 activation levels; the user program stack starts with the fifth level
and includes several recursive calls.
Kernel Debugging Utilities 2–9
Download from Www.Somanuals.com. All Manuals Search And Download.
2
3
The up 8 command moves the debugging context 8 activation levels up
the stack to one of the recursive calls within the user program code.
The print r command displays the current value of the variable r,
which is a structure of array elements. Full symbolization is available
for the user program, assuming it was compiled with the -g option.
4
The print r.a[511] command displays the current value of array
element 511 of structure r.
2.1.10 Extracting the Preserved Message Buffer
The preserved message buffer (pmsgbuf) contains information such as
the firmware version, operating system version, pc value, and device
configuration. You can use dbx to extract the preserved message buffer from
a running system or dump files. For example:
(dbx) print *pmsgbuf
struct {
msg_magic = 405601
msg_bufx = 1537
msg_bufr = 1537
msg_bufc = "Alpha boot: available memory from 0x7c6000 to 0x6000000
Tru64 UNIX V5.0; Sun Jan 03 11:20:36 EST 1999
physical memory = 96.00 megabytes.
available memory = 84.57 megabytes.
using 360 buffers containing 2.81 megabytes of memory
tc0 at nexus
scc0 at tc0 slot 7
asc0 at tc0 slot 6
rz1 at scsi0 target 1 lun 0 (LID=0) (DEC
rz2 at scsi0 target 2 lun 0 (LID=1) (DEC
rz3 at scsi0 target 3 lun 0 (LID=2) (DEC
rz4 at scsi0 target 4 lun 0 (LID=3) (DEC
RZ25
RZ25
RZ26
RRD42
(C) DEC 0700)
(C) DEC 0700)
(C) DEC T384)
(C) DEC 4.5d)
tz5 at scsi0 target 5 lun 0 (DEC
scsi1 at tc0 slot 7
TLZ06
(C)DEC 0374)
fb0 at tc0 slot
1280X1024
8
ln0: DEC LANCE Module Name: PMAD-BA
ln0 at tc0 slot
7
.
.
.
2.1.11 Debugging on SMP Systems
Debugging in an SMP environment can be difficult because an SMP system
optimized for performance keeps the minimum of lock debug information.
The Tru64 UNIX system supports a lock mode to facilitate debugging SMP
locking problems. The lock mode is implemented in the lockmode boot
time system attribute. By default, the lockmode attribute is set to a value
between 0 and 3, depending upon whether the system is an SMP system and
whether the RT_PREEMPTION_OPT attribute is set. (This attribute optimizes
system performance.)
2–10 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
For debugging purposes, set the lockmode attribute to 4. Follow these steps
to set the lockmode attribute to 4:
1. Create a stanza-formatted file named, for example, generic.stanza
that appears as follows:
generic:
lockmode=4
The contents of this file indicate that you are modifying the lockmode
attribute of the generic subsystem.
2. Add the new definition of lockmode to the /etc/sysconfigtab
database:
# sysconfigdb -a -f generic.stanza generic
3. Reboot your system.
Some of the debugging features provided with lockmode set to 4 are as
follows:
•
Automatic lock hierarchy checking and minimum spl checking when
any kernel lock is acquired (assuming a lockinfo structure exists
for the lock class in question). This checking helps you find potential
deadlock situations.
•
•
Lock initialization checking.
Additional debug information maintenance, including information about
simple and complex locks.
For simple locks, the system records an array of the last 32 simple locks
which were acquired on the system (slock_debug). The system creates
a slock_debug array for each CPU in the system.
For complex locks, the system records the locks owned by each thread in
the thread structure (up to eight complex locks).
To get a list of the complex locks a thread is holding use these commands:
# dbx -k /vmunix
(dbx) print thread->lock_addr
{
[0] 0xe4000002a67e0030
[1] 0xc3e0005b47ff0411
[2] 0xb67e0030a6130048
[3] 0xa67e0030d34254e5
[4] 0x279f0200481e1617
[5] 0x4ae33738a7730040
[6] 0x477c0101471c0019
[7] 0xb453004047210402
}
(dbx) print slock_debug
{
Kernel Debugging Utilities 2–11
Download from Www.Somanuals.com. All Manuals Search And Download.
[0] 0xfffffc000065c580
[1] 0xfffffc000065c780
}
•
Lock statistics are recorded to allow you to determine what kind of
contention you have on a particular lock. Use the kdbx lockstats
extension as shown in the following example to display lock statistics:
# kdbx /vmunix
(kdbx) lockstats
Lockstats
li_name
cpu count
tries
misses %misses waitsum
waitmax waitmin trmax
=========== ===================== === ====== ========== ======= ====== ============ ======= ======= ======
k0x00657d40
k0x00653400
k0x00657d80
k0x00653440
k0x00657dc0
k0x00653480
k0x00657e00
inode.i_io_lock
nfs_daemon_lock
nfs_daemon_lock
lk_lmf
1
0
1
0
1
0
1
0
1
1784
1
74268
1936 2.61
110533
500
0
0
0
0
0
0
0
0
6
0
0
0
0
0
0
0
0
10
0
0
0
0
0
0
0
7
0
0
2
3
5
0
0
0
0
0
0
0
0
0
0
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0
0
0
0
0
0
0
0
1
1
1
1
1
40
40
lk_lmf
procfs_global_lock
procfs_global_lock
k0x006534c0 procfs.pr_trace_lock
k0x00657e40 procfs.pr_trace_lock
2.2 The kdbx Debugger
The kdbx debugger is a crash analysis and kernel debugging tool; it serves
as a front end to the dbx debugger. The kdbx debugger is extensible,
customizable, and insensitive to changes to offsets and field sizes in
structures. The only dependencies on kernel header files are for bit
definitions in flag fields.
The kdbx debugger has facilities for interpreting various symbols and kernel
data structures. It can format and display these symbols and data structures
in the following ways:
•
In a predefined form as specified in the source code modules that
currently accompany the kdbx debugger
•
As defined in user-written source code modules according to a
standardized format for the contents of the kdbx modules
All dbx commands (except signals such as Ctrl/P) are available when you
use the kdbx debugger. In general, kdbx assumes hexadecimal addresses for
commands that perform input and output.
As with dbx, you can use kdbx to examine the call stack of the user program
whose execution precipitated a kernel crash (see Section 2.1.9).
The sections that follow explain using kdbx to debug kernel programs.
2.2.1 Beginning a kdbx Session
Using the kdbx debugger, you can examine the running kernel or dump files
created by the savecore utility. In either case, you examine an object file
and a core file. For running systems, these files are usually /vmunix and
2–12 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
/dev/mem, respectively. By default, crash dump files are created in the
/var/adm/crash directory (see the System Administration manual).
Use the following kdbx command to examine a running system:
# kdbx −k /vmunix /dev/mem
Use a kdbx command similar to the following to examine a compressed or
uncompressed crash dump file, respectively:
# kdbx −k vmunix.1 vmzcore.1
# kdbx −k vmunix.1 vmcore.1
The version number (.1 in this example) is determined by the value contained
in the bounds file, which is located in the same directory as the dump files.
To examine a crash dump file and also view the call stack of the user
program whose execution precipitated the kernel crash, you must invoke
kdbx using the following command syntax:
kdbx -k vmunix.n vm[z]core.n path/user-program
For more information, see Section 2.1.9.
When you begin a debugging session, kdbx reads and executes the
commands in the system initialization file /var/kdbx/system.kdbxrc.
The initialization file contains setup commands and alias definitions. (For
a list of kdbx aliases, see the kdbx(1) reference page.) You can further
customize the kdbx environment by adding commands and aliases to:
•
•
•
The /var/kdbx/site.kdbxrc file
This file contains customized commands and alias definitions for a
particular system.
The ~/.kdbxrc file
This file contains customized commands and alias definitions for a
specific user.
The ./.kdbxrc file
This file contains customized commands and alias definitions for a
specific project. This file must reside in the current working directory
when kdbx is invoked.
2.2.2 The kdbx Debugger Commands
The kdbx debugger provides the following commands:
alias [name] [command-string]
Sets or displays aliases. If you omit all arguments, alias displays all
aliases. If you specify the variable name, alias displays the alias for
name, if one exists. If you specify name and command-string, alias
establishes name as an alias for command-string.
Kernel Debugging Utilities 2–13
Download from Www.Somanuals.com. All Manuals Search And Download.
context proc | user
Sets context to the user’s aliases or the extension’s aliases. This
command is used only by the extensions.
coredata start_address end_address
Dumps, in hexadecimal, the contents of the core file starting at
start_address and ending before end_address.
dbx command-string
Passes the command-string to dbx. Specifying dbx is optional; if
kdbx does not recognize a command, it automatically passes that
command to dbx. See the dbx(1) reference page for a complete
description of dbx commands.
help [-long] [args]
Prints help text.
pr [flags] [extensions] [arguments]
Executes an extension and gives it control of the kdbx session until it
quits. You specify the name of the extension in extension and pass
arguments to it in arguments.
−debug
Causes kdbx to display input to and output
from the extension on the screen.
−pipe in_pipe
Used in conjunction with the dbx debugger
for debugging extensions. See Chapter 3 for
information on using the −pipe flag.
out_pipe
−print_output
Causes the output of the extension to be
sent to the invoker of the extension without
interpretation as kdbx commands.
−redirect_output
Used by extensions that execute other
extensions to redirect the output from the
called extensions; otherwise, the user receives
the output.
−tty
Causes kdbx to communicate with the
subprocess through a terminal line instead
of pipes. If you specify the −pipe flag, proc
ignores it.
2–14 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
print string
Displays string on the terminal. If this command is used by an
extension, the terminal receives no output.
quit
Exits the kdbx debugger.
source [-x] [file(s)]
Reads and interprets files as kdbx commands in the context of the
current aliases. If the you specify the −x flag, the debugger displays
commands as they are executed.
unalias name
Removes the alias, if any, from name.
The kdbx debugger contains many predefined aliases, which are defined in
the kdbx startup file /var/kdbx/system.kdbxrc.
2.2.3 Using kdbx Debugger Extensions
In addition to its commands, the kdbx debugger provides extensions. You
execute extensions using the kdbx command pr. For example, to execute the
arp extension, you enter this command:
kdbx> pr arp
Some extensions are provided with your Tru64 UNIX system and reside
in the /var/kdbx directory. Aliases for each of these extensions are also
provided that let you omit the pr command from an extension command line.
Thus, another way to execute the arp extension is to enter the following
command:
kdbx> arp
This command has the same effect as the pr arp command.
You can create your own kdbx extensions as described in Chapter 3.
For extensions that display addresses as part of their output, some use a
shorthand notation for the upper 32-bits of an address to keep the output
readable. The following table lists the notation for each address type.
Notation
Address Type
virtual
Replaces
ffffffff
fffffffe
Example
v
v0x902416f0
e0x12340000
e
virtual
Kernel Debugging Utilities 2–15
Download from Www.Somanuals.com. All Manuals Search And Download.
Notation
Address Type
Replaces
fffffc00
00000000
Example
k
u
?
k0x00487c48
u0x86406200
?0x3782cc33
kseg
user space
Unrecognized or
random type
The sections that follow describe the kdbx extensions that are supplied
with your system.
2.2.3.1 Displaying the Address Resolution Protocol Table
The arp extension displays the contents of the address resolution protocol
(arp) table. The arp extension has the following form:
arp [−]
If you specify the optional hyphen (−), arp displays the entire arp table;
otherwise, it displays those entries that have nonzero values in the
iaddr.s_addr and at_flags fields.
For example:
(kdbx) arp
NAME
BUCK SLOT
IPADDR
ETHERADDR MHOLD TIMER FLAGS
=================== ==== ==== ============ =============== ===== ===== =====
sys1.zk3.dec.com
sys2.zk3.dec.com
sys3.zk3.dec.com
11
18
31
0
0
0
16.140.128.4
16.140.128.1
16.140.128.6 8.0.2b.24.23.64
170.0.4.0.91.8
0.0.c.1.8.e8
0
0
0
450
194
539
3
3
103
2.2.3.2 Performing Commands on Array Elements
The array_action extension performs a command action on each element
of an array. This extension allows you to step through any array in the
operating system kernel and display specific components or values as
described in the list of command flags.
This extension has the following format:
array_action "type" length start_address [ flags] command
The arguments to the array_action extension are as follows:
"type "
The type of address of an element in the specified
array.
length
The number of elements in the specified array.
start_address
The address of an array. The address can be
specified as a variable name or a number. The
more common syntax or notation used to refer
2–16 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
to the start_address is usually of the form
&arrayname[0].
flags
If the you specify the −head flag, the next argument
appears as the table header.
If the you specify the −size flag, the next argument
is used as the array element size; otherwise, the size
is calculated from the element type.
If the you specify the −cond flag, the next argument
is used as a filter. It is evaluated by dbx for
each array element, and if it evaluates to TRUE,
the action is taken on the element. The same
substitutions that are applied to the command are
applied to the condition.
command
The kdbx or dbx command to perform on each
element of the specified array.
______________________
Note _______________________
The kdbx debugger includes several aliases, such as
file_action, that may be easier to use than using the
array_action extension directly.
Substitutions similar to printf can be performed on the command for each
array element. The possible substitutions are as follows:
Description
Conversion Character
%a
%c
Address of element
Cast of address to pointer to
array element
%i
%s
%t
Index of element within the array
Size of element
Type of pointer to element
For example:
(kdbx) array_action "struct kernargs *" 11 &kernargs[0] p %c.name
0xfffffc00004737f8 = "askme"
0xfffffc0000473800 = "bufpages"
0xfffffc0000473810 = "nbuf"
0xfffffc0000473818 = "memlimit"
0xfffffc0000473828 = "pmap_debug"
Kernel Debugging Utilities 2–17
Download from Www.Somanuals.com. All Manuals Search And Download.
0xfffffc0000473838 = "syscalltrace"
0xfffffc0000473848 = "boothowto"
0xfffffc0000473858 = "do_virtual_tables"
0xfffffc0000473870 = "netblk"
0xfffffc0000473878 = "zalloc_physical"
0xfffffc0000473888 = "trap_debug"
(kdbx)
2.2.3.3 Displaying the Buffer Table
The buf extension displays the buffer table. This extension has the
following format:
buf [ addresses -free -all]
|
If you omit arguments, the debugger displays the buffers on the hash list.
If you specify addresses, the debugger displays the buffers at those addresses.
Use the −free flag to display buffers on the free list. Use the −all flag to
display first buffers on the hash list, followed by buffers on the free list.
For example:
(kdbx) buf
BUF
MAJ
MIN
BLOCK COUNT SIZE RESID VNO
FWD
BACK
FLAGS
=========== === ===== ====== ===== ===== ===== =========== =========== =========== ===========
Bufs on hash lists:
v0x904e1b30
v0x904e21f8
v0x904e46c8
v0x904e9ef0
v0x904df758
v0x904eb538
v0x904e5930
v0x904eae70
v0x904f3ec8
8
8
8
8
8
8
8
8
8
2
54016 8192 8192
0
0
0
0
0
0
0
0
0
v0x902220d0 v0x904f23a8 v0x904e1d20 write cache
v0x90279800 v0x904e3748 v0x904e22f0 write cache
v0x90220fa8 v0x904e22f0 v0x904e23e8 read cache
v0x90221560 v0x904f2b68 v0x904e66c0 read cache
v0x90220fa8 v0x904eac80 v0x904df378 write cache
v0x90221560 v0x904ec990 v0x904eb440 read
v0x90221560 v0x904f3fc0 v0x904ec5b0 read cache
v0x90221560 v0x904df378 v0x904e08c8 write cache
v0x90220fa8 v0x904dff18 v0x904e1560 write cache
1025 131722 1024 8192
1025 107952 2048 8192
2050 199216 8192 8192
1025 107968 8192 8192
2050 223840 8192 8192
2050 379600 8192 8192
2050 625392 2048 8192
1025
18048 8192 8192
.
.
.
(kdbx)
2.2.3.4 Displaying the Callout Table and Absolute Callout Table
The callout extension displays the callout table. This extension has the
following format:
callout
For example:
(kdbx) callout
Processor:
0
Current time (in ticks):
615421360
FUNCTION
=============================
realitexpire
ARGUMENT
============ ============
k0x008ab220
k0x005d98e0
TICKS(delta)
30772
36541
wakeup
2–18 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
wakeup
k0x0187a220
k0x010ee950
k0x0132f220
k0x01069950
k0x01bba950
374923
376286
40724481
80436086
82582849
thread_timeout
thread_timeout
realitexpire
thread_timeout
The abscallout extension displays the absolute callout table. This table
contains callout entries with the absolute time in fractions of seconds. This
extension has the following format:
abscallout
For example:
(kdbx)abscallout
Processor:
0
FUNCTION
ARGUMENT
SECONDS
=============================
psx4_tod_expire
psx4_tod_expire
psx4_tod_expire
psx4_tod_expire
psx4_tod_expire
psx4_tod_expire
psx4_tod_expire
psx4_tod_expire
psx4_tod_expire
psx4_tod_expire
=========== =============
k0x01580808 86386.734375
k0x01580840 172786.734375
k0x01580878 259186.734375
k0x015808b0 345586.718750
k0x015808e8 431986.718750
k0x01580920 518386.718750
k0x01580958 604786.750000
k0x01580990 691186.750000
k0x015809c8 777586.750000
k0x01580a00 863986.750000
2.2.3.5 Casting Information Stored in a Specific Address
The cast extension forces dbx to display part of memory as the specified
type and is equivalent to the following command:
dbx print *((type ) address )
The cast extension has the following format:
cast address type
For example:
(kdbx) cast 0xffffffff903e3828 char
’^@’
2.2.3.6 Displaying Machine Configuration
The config extension displays the configuration of the machine. This
extension has the following format:
Kernel Debugging Utilities 2–19
Download from Www.Somanuals.com. All Manuals Search And Download.
config
For example:
(kdbx) config
Bus #0 (0xfffffc000048c6a0): Name - "tc" Connected to - "nexus"
Config 1 - tcconfl1
Config 2 - tcconfl2
Controller "scc" (0xfffffc000048c970)
(kdbx)
2.2.3.7 Converting the Base of Numbers
The convert extension converts numbers from one base to another. This
extension has the following format:
convert [-in [ 8 10 16] ] [-out [ 2
8
| |
10 16] ] [ args]
|
|
|
The −in and −out flags specify the input and output bases, respectively. If
you omit −in, the input base is inferred from the arguments. The arguments
can be numbers or variables.
For example:
(kdbx) convert -in 16 -out 10 864c2a14
2253138452
(kdbx)
2.2.3.8 Displaying CPU Use Statistics
The cpustat extension displays statistics about CPU use. Statistics
displayed include percentages of time the CPU spends in the following states:
•
•
•
•
•
Running user level code
Running system level code
Running at a priority set with the nice() function
Idle
Waiting (idle with input or output pending)
This extension has the following format:
cpustat [ -update n] [ -cpu n]
The −update flag specifies that kdbx update the output every n seconds.
The −cpu flag controls the CPU for which kdbx displays statistics. By
default, kdbx displays statistics for all CPUs in the system.
For example:
(kdbx) cpustat
Cpu
User (%)
Nice (%) System (%) Idle (%)
Wait (%)
===== ========== ========== ========== ========== ==========
0
1
0.23
0.21
0.00
0.00
0.08
0.06
99.64
99.68
0.05
0.05
2–20 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
2.2.3.9 Disassembling Instructions
The dis extension disassembles some number of instructions. This
extension has the following format:
dis start-address [ num-instructions]
The num-instructions, argument specifies the number of instructions
to be disassembled. The start-address argument specifies the starting
address of the instructions. If you omit the num-instructions argument,
1 is assumed.
For example:
(kdbx) dis 0xffffffff864c2a08 5
[., 0xffffffff864c2a08]
[., 0xffffffff864c2a0c]
[., 0xffffffff864c2a10]
[., 0xffffffff864c2a14]
[., 0xffffffff864c2a18]
call_pal
call_pal
ldg
bgt
call_pal
0x20001
0x800000
$f18, -13304(r3)
r31, 0xffffffff864c2a14
0x4573d0
(kdbx)
2.2.3.10 Displaying Remote Exported Entries
The export extension displays the exported entries that are mounted
remotely. This extension has the following format:
export
For example:
(kdbx) export
ADDR EXPORT
MAJ MIN
INUM
GEN MAP FLAGS PATH
================== === ===== ===== ========== ==== ===== =================
0xffffffff863bfe40
0xffffffff863bfdc0
0xffffffff863bfe00
0xffffffff863bfe80
8
8
8
8
4098
2050 67619
2050 15263
2
1308854383
736519799
731712009
731270099
-2
-2
-2
-2
0 /cdrom
0 /usr/users/user2
0 /usr/staff/user
0 /mnt
1024
6528
2.2.3.11 Displaying the File Table
The file extension displays the file table. This extension has the following
format:
file [ addresses]
If you omit the arguments, the extension displays file entries with nonzero
reference counts; otherwise, it displays the file entries located at the
specified addresses.
For example:
(kdbx) file
Addr
Type Ref Msg Fileops
f_data
Cred Offset Flags
Kernel Debugging Utilities 2–21
Download from Www.Somanuals.com. All Manuals Search And Download.
=========== ==== === === ======= =========== =========== ====== =====
v0x90406000 file
v0x90406058 file
v0x904060b0 file
v0x90406108 file
v0x90406160 file
v0x904061b8 sock
v0x90406210 file
v0x90406268 file
v0x904062c0 file
v0x90406318 file
v0x90406370 sock
4
1
1
2
2
2
1
1
3
2
2
0
0
0
0
0
vnops v0x90259550 v0x863d5540
vnops v0x9025b5b8 v0x863d5e00
vnops v0x90233908 v0x863d5d60
vnops v0x90233908 v0x863d5d60
vnops v0x90228d78 v0x863d5b80
68 r w
4096 r
0 r
602 w
904 r
0 sockops v0x863b5c08 v0x863d5c20
0 r w
0
0
0
0
vnops v0x90239e10 v0x863d5c20
vnops v0x90245140 v0x863d5c20
vnops v0x90227880 v0x863d5900
vnops v0x90228b90 v0x863d5c20
2038 r
301 w a
23 r w
856 r
0 sockops v0x863b5a08 v0x863d5c20
0 r w
.
.
.
2.2.3.12 Displaying the udb and tcb Tables
The inpcb extension displays the udb and tcb tables. This extension has
the following format:
inpcb [-udp] [-tcp] [ addresses]
If you omit the arguments, kdbx displays both tables. If you specify the −udp
flag or the −tcp flag, the debugger displays the corresponding table.
If you specify the address argument, the inpcb extension ignores the −udp
and −tcp flags and displays entries located at the specified address.
For example:
(kdbx) inpcb -tcp
TCP:
Foreign Host
0.0.0.0
FPort
Local Host LPort
Socket
PCB Options
0 0.0.0.0
47621 u0x00000000 u0x00000000
1451 v0x8643f408 v0x863da408
1020 v0x8643fc08 v0x863da208
514 v0x8643ac08 v0x8643d008
1450 v0x863fba08 v0x863dad08
1021 v0x86431e08 v0x86414708
514 v0x86412808 v0x8643ce08
1449 v0x86436608 v0x86415e08
1448 v0x86431808 v0x863daa08
system.dec.com
system.dec.com
system.dec.com
system.dec.com
system.dec.com
system.dec.com
system.dec.com
system.dec.com
6000 comput.dec.com
998 comput.dec.com
999 comput.dec.com
6000 comput.dec.com
1008 comput.dec.com
1009 comput.dec.com
6000 comput.dec.com
6000 comput.dec.com
.
.
.
0.0.0.0
0.0.0.0
0.0.0.0
0.0.0.0
0.0.0.0
0 0.0.0.0
0 0.0.0.0
0 0.0.0.0
0 0.0.0.0
0 0.0.0.0
806 v0x863e3e08 v0x863dbe08
793 v0x863d1808 v0x8635a708
0 v0x86394408 v0x8635b008
1024 v0x86394208 v0x8635b108
111 v0x863d1e08 v0x8635b208
2.2.3.13 Performing Commands on Lists
The list_action extension performs some command on each element of a
linked list. This extension provides the capability to step through any linked
list in the operating system kernel and display particular components. This
extension has the following format:
2–22 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
list_action " type" next-field end-addr start-addr [ flags] command
The arguments to the list_action extension are as follows:
"type "
The type of an element in the specified list.
next-field
end-addr
The name of the field that points to the next element.
The value of the next field that terminates the list.
If the list is NULL-terminated, the value of the
end-addr argument is zero (0). If the list is circular,
the value of the end-addr argument is equal to the
start-addr argument.
start_addr
flags
The address of the list. This argument can be a
variable name or a number address.
Use the −head header flag to display the header
argument as the table header.
Use the −cond arg flag to filter input as specified
by arg. The debugger evaluates the condition for
each array element, and if it evaluates to true,
the action is taken on the element. The same
substitutions that are applied to the command are
applied to the condition.
command
The debugger command to perform on each element
of the list.
The kdbx debugger includes several aliases, such as procaddr, that might
be easier than using the list_action extension directly.
The kdbx debugger applies substitutions in the same style as printf
substitutions for each command element. The possible substitutions are as
follows:
Description
Conversion Character
%a
%c
Address of an element
Cast of an address to a pointer
to a list element
%i
%n
%t
Index of an element within the list
Name of the next field
Type of pointer to an element
Kernel Debugging Utilities 2–23
Download from Www.Somanuals.com. All Manuals Search And Download.
For example:
(kdbx) list_action "struct proc *" p_nxt 0 allproc p \
%c.task.u_address.uu_comm %c.p_pid
"list_action" 1382
"dbx" 1380
"kdbx" 1379
"dbx" 1301
"kdbx" 1300
"sh" 1296
"ksh" 1294
"csh" 1288
"rlogind" 1287
.
.
.
2.2.3.14 Displaying the lockstats Structures
The lockstats extension displays the lock statistics contained in the
lockstats structures. Statistics are kept for each lock class on each CPU
in the system. These structures provide the following information:
•
•
•
•
•
•
•
•
•
•
The address of the structure
The class of lock for which lock statistics are being recorded
The CPU for which the lock statistics are being recorded
The number of instances of the lock
The number of times processes have tried to get the lock
The number of times processes have tried to get the lock and missed
The percentage of time processes miss the lock
The total time processes have spent waiting for the lock
The maximum amount of time a single process has waited for the lock
The minimum amount of time a single process has waited for the lock
The lock statistics recorded in the lockstats structures are dynamic.
This extension is available only when the lockmode system attribute is
set to 4.
This extension has the following format:
lockstats -class name -cpu number -read -sum -total -update n
|
|
|
|
|
If you omit all flags, lockstats displays statistics for all lock classes on all
CPUs. The following describes the flags you can use:
2–24 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
−class name
−cpu number
Displays the lockstats structures for the specified
lock class. (Use the lockinfo command to display
information about the names of lock classes.)
Displays the lockstats structures for the specified
CPU.
−read
−sum
Displays the reads, sleeps attributes, and waitsums
or misses.
Displays summary data for all CPUs and all lock
types.
−total
Displays summary data for all CPUs.
−update n
Updates the display every n seconds.
For example:
(kdbx) lockstats
Lockstats li_name
=========== ==================== === ====== ========== ======= ======= ============ ======= ======= ========
cpu count
tries
misses %misses waitsum
waitmax waitmin trmax
k0x00657d40
k0x00653400
k0x00657d80
k0x00653440
k0x00657dc0
k0x00653480
k0x00657e00
inode.i_io_lock
nfs_daemon_lock
nfs_daemon_lock
lk_lmf
1
0
1
0
1
0
1
0
1
1784
1
74268
1936 2.61
110533
500
0
0
0
0
0
0
0
0
6
0
0
0
0
0
0
0
0
10
0
0
0
0
0
0
0
0
7
0
0
2
3
5
0
0
0
0
0
0
0
0
0
0
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0
0
0
0
0
0
0
0
1
1
1
1
1
40
40
lk_lmf
procfs_global_lock
procfs_global_lock
k0x006534c0 procfs.pr_trace_lock
k0x00657e40 procfs.pr_trace_lock
.
.
.
2.2.3.15 Displaying lockinfo Structures
The lockinfo extension displays static lock class information contained
in the lockinfo structures. Each lock class is recorded in one lockinfo
structure, which contains the following information:
•
•
•
•
•
The address of the structure
The index into the array of lockinfo structures
The class of lock for which information is provided
The number of instances of the lock
The lock flag, as defined in the /sys/include/sys/lock.h header file
This extension is available only when the lockmode system attribute is
set to 4.
Kernel Debugging Utilities 2–25
Download from Www.Somanuals.com. All Manuals Search And Download.
This extension has the following format:
lockinfo [ -class name ]
The −class flag allows you to display the lockinfo structure for a
particular class of locks. If you omit the flag, lockinfo displays the
lockinfo structures for all classes of locks.
For example:
(kdbx) lockinfo
Lockinfo
Index
li_name
li_count li_flgspl
================== ===== =========================== ========== =========
xfffffc0000652030
0xfffffc0000652040
0xfffffc0000652050
0xfffffc0000652060
0xfffffc0000652070
0xfffffc0000652080
0xfffffc0000652090
0xfffffc00006520a0
0xfffffc00006520b0
0xfffffc00006520c0
0xfffffc00006520d0
0xfffffc00006520e0
0xfffffc00006520f0
0xfffffc0000652100
0xfffffc0000652110
0xfffffc0000652120
3
cfg_subsys_lock
subsys_tbl_lock
inode.i_io_lock
nfs_daemon_lock
lk_lmf
21
1
4348
1
1
1
40
0
1
16
1
1
64
1
1
0xd0
0xc0
0x90
0xc0
0xc0
0xc0
0xc0
0xc0
0xc0
0xc0
0xc0
0xc0
0xc0
0xc0
0xc0
0xd0
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
procfs_global_lock
procfs.pr_trace_lock
procnode.prc_ioctl_lock
semidq_lock
semid_lock
undo_lock
msgidq_lock
msgid_lock
pgrphash_lock
proc_relation_lock
pgrp.pg_lock
20
2.2.3.16 Displaying the Mount Table
The mount extension displays the mount table, and has the following format:
mount [-s] [ address]
The −s flag displays a short form of the table. If you specify one or more
addresses, kdbx displays the mount entries named by the addresses.
For example:
(kdbx) mount
MOUNT
MAJ
MIN
VNODE
ROOTVP
TYPE
PATH
FLAGS
=====
=========== ===== ===== ============ =========== ====
v0x8196bb30
loc
========================
/
8
0
NULL v0x8a75f600 ufs
v0x8196a910
v0x8196aae0
v0x8196acb0
v0x8196ae80
v0x8196b050
v0x8196b220
ro
v0x8a62de00 v0x8a684e00 nfs
v0x8a646800 v0x8a625400 nfs
v0x8a684800 v0x8a649400 nfs
v0x8a67ea00 v0x8a774800 nfs
v0x8a67c400 v0x8a767800 nfs
v0x8a651800 v0x8a781000 nfs
/share/cia/build/alpha.dsk5
/share/xor/build/agosminor.dsk1 ro
/share/buffer/build/submits.dsk2 ro
/share/cia/build/goldos.dsk6
/usr/staff/alpha1/user
/usr/sde
ro
ro
v0x8196b3f0
loc
v0x8196b5c0
loc
v0x8196b790
loc
v0x8196b960
8
8
8
0
2050
v0x8a61ca00 v0x8a77fe00 ufs
v0x8a61c000 v0x8a79c200 ufs
v0x8a5c4800 v0x8a760600 ufs
/usr3
/usr2
/usr
7
6
0
v0x8a5c5000 NULL
procfs /proc
2–26 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
2.2.3.17 Displaying the Namecache Structures
The namecache extension displays the namecache structures on the system,
and has the following format:
namecache
For example:
(kdbx) namecache
namecache
nc_vp
nc_vpid nc_nlen
nc_dvp
nc_name
=========== =========== ======= ======= ============ =============
v0x9047b2c0 v0x9021f4f8
v0x9047b310 v0x9021e988
v0x9047b360 v0x9021e5b8
v0x9047b3b0 v0x9021e7a0
v0x9047b400 v0x9021ed58
v0x9047b4a0 v0x9021f128
v0x9047b4f0 v0x9021f310
v0x9047b540 v0x9021fab0
v0x9047b590 v0x9021f6e0
v0x9047b5e0 v0x9021eb70
v0x9047b630 v0x9021f310
v0x9047b6d0 v0x9021fc98
v0x9047b720 v0x9021fe80
v0x9047b770 v0x90220068
v0x9047b810 v0x90220250
v0x9047b8b0 v0x90220438
v0x9047b900 v0x90220620
v0x9047b950 v0x90220808
v0x9047b9a0 v0x902209f0
v0x9047b9f0 v0x90220bd8
24
0
0
199
0
0
0
20
0
28
34
0
0
0
0
0
0
0
0
0
4
11
2
3
4
4
7
3
7
3
3
7
2
3
8
4
5
v0x9021e5b8 sbin
v0x9021e7a0 swapdefault
v0x9021e7a0 ..
v0x9021e5b8 dev
v0x9021eb70 rz1g
v0x9021e7a0 init
v0x9021e5b8 upgrade
v0x9021e5b8 etc
v0x9021f4f8 inittab
v0x9021e5b8 var
v0x9021e5b8 usr
v0x9021eb70 console
v0x9021e7a0 sh
v0x9021f4f8 nls
v0x9021e7a0 bcheckrc
v0x9021e7a0 fsck
v0x9021f4f8 fstab
v0x9021e7a0 ufs_fsck
v0x9021eb70 rz1a
v0x9021eb70 rrz1a
8
4
5
.
.
.
2.2.3.18 Displaying Processes’ Open Files
The ofile extension displays the open files of processes and has the
following format.
ofile [ -proc address -pid pid -v]
|
|
If you omit arguments, ofile displays the files opened by each process. If
you specify −proc address or −pid pid the extension displays the open
files owned by the specified process. The −v flag displays more information
about the open files.
For example:
(kdbx) ofile -pid 1136 -v
Proc=0xffffffff9041e980
ADDR_FILE f_cnt ADDR_VNODE V_TYPE V_TAG USECNT V_MOUNT
=========== ===== =========== ====== ====== ====== =========== ====== =====
pid= 1136
INO# QSIZE
v0x90408520
v0x90408520
v0x90408520
v0x90408368
27 v0x902c1390 VCHR VT_UFS
27 v0x902c1390 VCHR VT_UFS
27 v0x902c1390 VCHR VT_UFS
3
3
3
v0x863abab8
v0x863abab8
v0x863abab8
1103
1103
1103
0
0
0
1
v0x9026e6b8 VDIR VT_UFS
18
v0x863ab728 64253
512
Kernel Debugging Utilities 2–27
Download from Www.Somanuals.com. All Manuals Search And Download.
2.2.3.19 Converting the Contents of Memory to Symbols
The paddr extension converts a range of memory to symbolic references
and has the following format:
paddr address number-of-longwords
The arguments to the paddr extension are as follows:
address
The starting address.
number-of-longwords
The number of longwords to display.
For example:
(kdbx) paddr 0xffffffff90be36d8 20
[., 0xffffffff90be36d8]: [h_kmem_free_memory_:824, 0xfffffc000037f47c] 0x0000000000000000
[., 0xffffffff90be36e8]: [., 0xffffffff8b300d30] [hardclock:394, 0xfffffc00002a7d5c]
[., 0xffffffff90be36f8]: 0x0000000000000000 [., 0xffffffff863828a0]
[., 0xffffffff90be3708]: [setconf:133, 0xfffffc00004949b0] [., 0xffffffff90be39f4]
[., 0xffffffff90be3718]: 0x00000000000004e0 [thread_wakeup_prim:858, 0xfffffc0000328454]
[., 0xffffffff90be3728]: 0x0000000000000001 0xffffffff0000000c
[., 0xffffffff90be3738]: [., 0xffffffff9024e518] [hardclock:394, 0xfffffc00002a7d5c]
[., 0xffffffff90be3748]: 0x00000000004d5ff8 0xffffffffffffffd4
[., 0xffffffff90be3758]: 0x00000000000bc688 [setconf:133, 0xfffffc00004946f0]
[., 0xffffffff90be3768]: [thread_wakeup_prim:901, 0xfffffc00003284d0]
0x000003ff85ef4ca0
2.2.3.20 Displaying the Process Control Block for a Thread
The pcb extension displays the process control block for a given thread
structure located at thread_address. The extension also displays the
contents of integer and floating-point registers (if nonzero).
This extension has the following format:
pcb thread_address
For example:
(kdbx) pcb 0xffffffff863a5bc0
Addr pcb
v0x90e8c000
sp
ksp
v0x90e8fb88
usp
0x0
pc
ps
0x5
0xfffffc00002dc110
ptbr
0x2ad4
pcb_physaddr
0x55aa000
0xffffffff90e8fb88
r9
0xffffffff863a5bc0
r10 0xffffffff863867a0
r11 0xffffffff86386790
r13 0x5
2.2.3.21 Formatting Command Arguments
The printf extension formats one argument at a time to work around the
dbx debugger’s command length limitation. It also supports the %s string
2–28 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
substitution, which the dbx debugger’s printf command does not. This
extension has the following format:
printf format-string [ args]
The arguments to the printf extension are as follows:
format-string
args
A character string combining literal characters with
conversion specifications.
The arguments for which you want kdbx to display
values.
For example:
(kdbx) printf "allproc = 0x%lx" allproc
allproc = 0xffffffff902356b0
2.2.3.22 Displaying the Process Table
The proc extension displays the process table. This extension has the
following format:
proc [ address]
If you specify an address, the proc extension displays only the proc
structures at that address; otherwise, the extension displays all proc
structures.
For example:
(kdbx) proc
.
.
.
Addr
PID
PPID PGRP UID
NICE SIGCATCH P_SIG
Event
Flags
=========== ===== ===== ===== ===== ==== ======== ======== =========== ============
v0x8191e210
v0x8197cd80
v0x8198a210
v0x819a8d80
v0x819a8210
0
1
13
120
122
0
0
1
1
1
1
1
0
1
13
0
0
0
0
0
0 00000000 00000000
0 207a7eff 00000000
0 00002000 00000000
0 00086001 00000000
0 00004001 00000000
0 00081000 00000000
0 20006003 00000000
0 00080000 00000000
0 00007efb 00000000
0 00004007 00000000
0 00000000 00000000
0 00000000 00000000
0 01880003 00000000
NULL in sys
NULL in pagv exec
NULL in pagv
NULL in pagv
NULL in pagv
120
122
5267 1138
131
v0x81a14210 5249
v0x819b6210 131
NULL in pagv exec
NULL in pagv
0
v0x81a18d80 5266 5267 5267 1138
v0x81a2ed80 5267 4938 5267 1138
v0x81a42d80 5268 5266 5267 1138
v0x81a18210 5270 5273 5267 1138
v0x8198ed80 5273 5266 5267 1138
v0x81a0ad80 5276 5279 5276 1138
in pagv ctty exec
NULL in pagv exec
NULL in pagv exec
NULL in pagv exec
NULL in pagv exec
NULL in pagv exec
NULL
v0x81a26d80 5278 5249 5278 1138
in pagv ctty exec
0 00080002 00000000
NULL
v0x819f2d80 5279
v0x81a14d80 5281
v0x81a3cd80 5287 5281 5287 1138
in pagv ctty exec
1
1
5267 1138
5267 1138
0 00081000 00000000
0 00081000 00000000
0 01880003 00000000
NULL in pagv exec
NULL in pagv exec
NULL
Kernel Debugging Utilities 2–29
Download from Www.Somanuals.com. All Manuals Search And Download.
v0x81a28210 5301 5276 5301 1138
in pagv ctty exec
0 00080002 00000000
NULL
v0x819aad80
v0x8197c210 6346
v0x819c4210
:
195
1
1
1
195
6346
0
0
0
0
0 00080628 00000000
0 00004006 00000000
0 00086efe 00000000
NULL in pagv
NULL in pagv exec
NULL in pagv
204
2.2.3.23 Converting an Address to a Procedure name
The procaddr extension converts the specified address to a procedure
name. This extension has the following format:
procaddr [ address ]
For example:
(kdbx) procaddr callout.c_func
xpt_pool_free
2.2.3.24 Displaying Sockets from the File Table
The socket extension displays those files from the file table that are sockets
with nonzero reference counts. This extension has the following format:
socket
For example:
(kdbx) socket
Fileaddr
Sockaddr
Type
PCB
Qlen Qlim Scc Rcc
=========== =========== ===== =========== ==== ==== === ====
v0x904061b8 v0x863b5c08 DGRAM v0x8632dc88
v0x90406370 v0x863b5a08 DGRAM v0x8632db08
v0x90406478 v0x863b5808 DGRAM v0x8632da88
v0x904064d0 v0x863b5608 DGRAM v0x8632d688
v0x904065d8 v0x863b5408 DGRAM v0x8632dc08
v0x90406630 v0x863b5208 DGRAM v0x8632d588
v0x904067e8 v0x863b4208 DGRAM v0x8632d608
v0x90406840 v0x863b4008 DGRAM v0x8632d788
v0x904069a0 v0x8641f008 STRM v0x8632c808
v0x90406aa8 v0x863b4c08 STRM v0x8632d508
v0x90406bb0 v0x863b4e08 STRM v0x8632da08
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
2
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
.
.
.
2.2.3.25 Displaying a Summary of the System Information
The sum extension displays a summary of system information and has the
following format:
sum
For example:
2–30 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
(kdbx) sum
Hostname : system.dec.com
cpu: DEC3000 - M500
Boot-time: Tue Nov 3 15:01:37 1992
Fri Nov 6 09:59:00 1998
avail: 1
Time:
Kernel : OSF1 release 1.2 version 1.2 (alpha)
(kdbx)
2.2.3.26 Displaying a Summary of Swap Space
The swap extension displays a summary of swap space and has the following
format:
swap
For example:
(kdbx) swap
Swap device name
Size
In Use
Free
-------------------------------- ---------- ---------- ----------
/dev/rz3b
/dev/rz2b
131072k
16384p
131072k
16384p
32424k
4053p
8k
98648k Dumpdev
12331p
131064k
1p
16383p
-------------------------------- ---------- ---------- ----------
Total swap partitions:
2
262144k
32768p
32432k
4054p
229712k
28714p
(kdbx)
2.2.3.27 Displaying the Task Table
The task extension displays the task table. This extension has the following
format:
task [ proc_address ]
If you specify addresses, the extension displays the task structures named
by the argument addresses; otherwise, the debugger displays all tasks.
For example:
(kdbx) task
.
.
.
Task Addr
Ref Threads
Map
Swap_state Utask Addr Proc Addr
Pid
=========== === ======= =========== ========== =========== =========== ======
v0x8191e000 17
15 v0x808f7ef0 INSWAPPED v0x8191e3b0 v0x8191e210
1 v0x808f7760 INSWAPPED v0x8197cf20 v0x8197cd80
1 v0x808f7550 INSWAPPED v0x8198a3b0 v0x8198a210
1 v0x808f7340 INSWAPPED v0x819a8f20 v0x819a8d80
1 v0x808f7290 INSWAPPED v0x819a83b0 v0x819a8210
1 v0x819f1ad0 INSWAPPED v0x81a143b0 v0x81a14210
1 v0x808f6fd0 INSWAPPED v0x819b63b0 v0x819b6210
1 v0x819f1a20 INSWAPPED v0x81a18f20 v0x81a18d80
1 v0x819f1340 INSWAPPED v0x81a2ef20 v0x81a2ed80
1 v0x819f1080 INSWAPPED v0x81a42f20 v0x81a42d80
1 v0x819f1970 INSWAPPED v0x81a183b0 v0x81a18210
1 v0x808f74a0 INSWAPPED v0x8198ef20 v0x8198ed80
0
1
13
120
122
v0x8197cb70
v0x8198a000
v0x819a8b70
v0x819a8000
v0x81a14000
v0x819b6000
v0x81a18b70
v0x81a2eb70
v0x81a42b70
v0x81a18000
v0x8198eb70
3
3
3
3
3
3
3
3
3
3
3
5249
131
5266
5267
5268
5270
5273
Kernel Debugging Utilities 2–31
Download from Www.Somanuals.com. All Manuals Search And Download.
v0x81a0ab70
v0x81a26b70
v0x819f2b70
v0x81a14b70
v0x81a3cb70
v0x81a28000
v0x819aab70
v0x8197c000
v0x819c4000
3
3
3
3
3
3
3
3
3
1 v0x819f1ce0 INSWAPPED v0x81a0af20 v0x81a0ad80
1 v0x819f1760 INSWAPPED v0x81a26f20 v0x81a26d80
1 v0x819f1e40 INSWAPPED v0x819f2f20 v0x819f2d80
1 v0x819f1b80 INSWAPPED v0x81a14f20 v0x81a14d80
1 v0x819f11e0 INSWAPPED v0x81a3cf20 v0x81a3cd80
1 v0x819f1550 INSWAPPED v0x81a283b0 v0x81a28210
1 v0x808f71e0 INSWAPPED v0x819aaf20 v0x819aad80
1 v0x808f76b0 INSWAPPED v0x8197c3b0 v0x8197c210
1 v0x808f6e70 INSWAPPED v0x819c43b0 v0x819c4210
5276
5278
5279
5281
5287
5301
195
6346
204
.
.
.
2.2.3.28 Displaying Information About Threads
The thread extension displays information about threads and has the
following format:
thread [ proc_address ]
If you specify addresses, the thread extensions displays thread structures
named by the addresses; otherwise, information about all threads is
displayed.
For example:
(kdbx) thread
Thread Addr Task Addr
Proc Addr
Event
pcb
state
=========== =========== =========== =========== =========== =====
v0x8644d690 v0x8637e440 v0x9041e830 v0x86420668 v0x90f50000 wait
v0x8644d480 v0x8637e1a0 v0x9041eec0 v0x86421068 v0x90f48000 wait
v0x863a17b0 v0x86380ba0 v0x9041db10 v0x8640e468 v0x90f30000 wait
v0x863a19c0 v0x86380e40 v0x9041d9c0 v0x8641f268 v0x90f2c000 wait
v0x8644dcc0 v0x8637ec20 v0x9041e6e0 v0x8641fc00 v0x90f38000 wait
v0x863a0520 v0x8637f400 v0x9041ed70 v0x8640ea00 v0x90f3c000 wait
v0x863a0310 v0x8637f160 v0x9041e980 u0x00000000 v0x90f44000 run
v0x863a2410 v0x863818c0 v0x9041dc60 v0x8640f268 v0x90f18000 wait
v0x863a15a0 v0x86380900 v0x9041d480 v0x8641ec00 v0x90f24000 wait
.
.
.
2.2.3.29 Displaying a Stack Trace of Threads
The trace extension displays the stack of one or more threads. This
extension has the following format:
trace [ thread_address... -k -u -a]
|
|
|
If you omit arguments, trace displays the stack trace of all threads. If you
specify a list of thread addresses, the debugger displays the stack trace of
the specified threads. The following table explains the trace flags:
−a
Displays the stack trace of the active thread on each CPU
Displays the stack trace of all kernel threads
−k
2–32 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
−u
Displays the stack trace of all user threads
For example:
(kdbx) trace
*** stack trace of thread 0xffffffff819af590 pid=0 ***
0 thread_run(new_thread = 0xffffffff819af928)
>
["../../../../src/kernel/kern/sched_prim.c":1637, 0xfffffc00002f9368]
1 idle_thread() ["../../../../src/kernel/kern/sched_prim.c":2717,
0xfffffc00002fa32c]
*** stack trace of thread 0xffffffff819af1f8 pid=0 ***
>
0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1455,
0xfffffc00002f9084]
1 softclock_main() ["../../../../src/kernel/bsd/kern_clock.c":810,
0xfffffc000023a6d4]
.
.
.
*** stack trace of thread 0xffffffff819fc398 pid=0 ***
>
0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1471,
0xfffffc00002f9118]
1 vm_pageout_loop() ["../../../../src/kernel/vm/vm_pagelru.c":375,
0xfffffc0000395664]
2 vm_pageout() ["../../../../src/kernel/vm/vm_pagelru.c":834,
0xfffffc00003961e0]
.
.
.
*** stack trace of thread 0xffffffff819fce60 pid=2 ***
>
0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1471,
0xfffffc00002f9118]
1 msg_dequeue(message_queue = 0xffffffff819a5970, max_size = 8192,
option = 0, tout = 0, kmsgptr = 0xffffffff916e3980)
["../../../../src/kernel/kern/ipc_basics.c":884, 0xfffffc00002e8b54]
2 msg_receive_trap(header = 0xfffffc00005bc150, option = 0, size =
8192, name = 0, tout = 0)
["../../../../src/kernel/kern/ipc_basics.c":1245, 0xfffffc00002e92a4]
3 msg_receive(header = 0xfffffc00005be150, option = 6186352, tout
=
0) ["../../../../src/kernel/kern/ipc_basics.c":1107, 0xfffffc00002e904c]
4 ux_handler() ["../../../../src/kernel/builtin/ux_exception.c":221,
0xfffffc000027269c]
*** stack trace of thread 0xffffffff81a10730 pid=13 ***
>
0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1471,
0xfffffc00002f9118]
1 mpsleep(chan = 0xffffffff819f3270
=
"H4\237\201\377\377\377\377^X0\237\201\377\377\377\377^ ^YR", pri =
296, wmesg = 0xfffffc000042f5e0
=
"\200B\260\300B\244KA\340\3038F]\244\377, timo = 0,
lockp = (nil), flags = 0)
["../../../../src/kernel/bsd/kern_synch.c":341, 0xfffffc0000250250]
2 sigsuspend(p = 0xffffffff81a04278, args = 0xffffffff9170b8a8,
retval = 0xffffffff9170b898)
.
.
.
2.2.3.30 Displaying a u Structure
The u extension displays a u structure. This extension has the following
format:
Kernel Debugging Utilities 2–33
Download from Www.Somanuals.com. All Manuals Search And Download.
u [ proc-addr]
If you omit arguments, the extension displays the u structure of the
currently running process.
For example:
(kdbx) u ffffffff9027ff38
procp
ar0
comm
args
0x9027ff38
0x90c85ef8
cfgmgr
g
B*
ü
u_ofile_of: 0x86344e30 u_pofile_of: 0x86345030
0 0xffffffff902322d0
1 0xffffffff90232278
2 0xffffffff90232278
3 0xffffffff90232328
4 0xffffffff90232380 Auto-close
5 0xffffffff902324e0
sizes
u_outime
sigs
29 45 2 (clicks)
0
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
40
sigmask
0 fffefeff fffefeff fffefeff
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0 fffefeff
0 fffefeff
0
0
0
0
0
0
0
0
sigonstack
oldmask
sigstack
cdir rdir
timers
2000
0
901885b8
193248
0
0
start
acflag
0
723497702
(kdbx)
2.2.3.31 Displaying References to the ucred Structure
The ucred extension displays all instances of references to ucred structures.
This extension has the following format:
ucred [ -proc -uthread -file -buf -ref addr -check addr checkall]
|
|
|
|
|
|
If you omit all flags, ucred displays all references to ucred structures. The
following describes the flags you can specify:
-proc
Displays all ucreds referenced by the proc
structures
2–34 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
-uthread
Displays all ucreds referenced by the uthread
structures
-file
Displays all ucreds referenced by the file structures
Displays all ucreds referenced by the buf structures
Displays all references to a given ucred
-buf
-ref address
-check address
-checkall
Checks the reference count of a particular ucred
Checks the reference count of all ucreds, with
mismatches marked by an asterisk (*)
For example:
(kdbx) ucred
ADDR OF UCRED
ADDR OF Ref
Ref Type cr_ref cr_uid cr_gid cr_ruid
=================== ================== ======== ====== ====== ====== =======
0xffffffff863d4960
0xffffffff8651fb80
0xffffffff86525c20
0xffffffff86457ea0
0xffffffff86457ea0
0xffffffff8651b5e0
0xffffffff8651efa0
0xffffffff90420f90
0xffffffff9041e050
0xffffffff90420270
0xffffffff90421380
0xffffffff9041f6a0
0xffffffff9041f010
0xffffffff9041e1a0
proc
proc
proc
proc
proc
proc
proc
3
18
2
4
4
0
0
0
1
1
1
15
15
1
0
0
0
1139
1139
0
1139
1139
0
2
2
1138
10
1138
.
.
.
0xffffffff863d4960
0xffffffff8651fb80
0xffffffff86525c20
0xffffffff86457ea0
0xffffffff86457ea0
0xffffffff8651b5e0
0xffffffff8651efa0
0xffffffff90fb82e0 uthread
0xffffffff90fbc2e0 uthread
0xffffffff90fb02e0 uthread
0xffffffff90f882e0 uthread
0xffffffff90f902e0 uthread
0xffffffff90fc02e0 uthread
0xffffffff90fac2e0 uthread
3
18
2
4
4
0
0
0
1
1
1
15
15
1
0
0
0
1139
1139
0
1139
1139
0
2
2
1138
10
1138
.
.
.
0xffffffff863d5c20
0xffffffff863d5b80
0xffffffff863d5c20
0xffffffff863d5b80
0xffffffff86456000
0xffffffff863d5c20
0xffffffff90406790
0xffffffff904067e8
0xffffffff90406840
0xffffffff90406898
0xffffffff904068f0
0xffffffff90406948
file
file
file
file
file
file
16
7
16
7
15
16
0
0
0
0
0
0
0
0
15
0
0
0
0
0
1139
0
1139
0
.
.
.
(kdbx) ucred -ref 0xffffffff863d5a40
ADDR OF UCRED ADDR OF Ref
=================== ================== ======== ====== ====== ====== =======
Ref Type cr_ref cr_uid cr_gid cr_ruid
0xffffffff863d5a40
0xffffffff863d5a40
0xffffffff863d5a40
0xffffffff863d5a40
0xffffffff9041c0d0
0xffffffff90ebc2e0 uthread
0xffffffff90406f78
0xffffffff90408730
proc
4
4
4
4
0
0
0
0
0
0
0
0
0
0
0
0
file
file
(kdbx) ucred -check 0xffffffff863d5a40
ADDR OF UCRED cr_ref Found
Kernel Debugging Utilities 2–35
Download from Www.Somanuals.com. All Manuals Search And Download.
=================== ====== =======
0xffffffff863d5a40
4
4
2.2.3.32 Removing Aliases
The unaliasall extension removes all aliases, including the predefined
aliases. This extension has the following format:
unaliasall
For example:
(kdbx) unaliasall
2.2.3.33 Displaying the vnode Table
The vnode extension displays the vnode table and has the following format:
vnode [ -free -all -ufs -nfs -cdfs -advfs -fs address -u uid -g
|
|
|
|
|
|
|
|
gid -v]
|
If you omit flags, vnode displays ACTIVE entries in the vnode table.
(ACTIVE means that usecount is nonzero.) The following describes the
flags you can specify:
-free
Displays INACTIVE entries in the vnode table
-all
Prints ALL (both ACTIVE and INACTIVE) entries
in the vnode table
-ufs
Displays all UFS entries in the vnode table
Displays all NFS entries in the vnode table
Displays all CDFS entries in the vnode table
Displays all ADVFS entries in the vnode table
Displays the vnode entries of a mounted file system
Displays vnode entries of a particular user
Displays vnode entries of a particular group
-nfs
-cdfs
-advfs
-fs address
-u uid
-g gid
-v
Displays related inode, rnode, or cdnode
information (used with -ufs, -nfs, or -cdfs only)
2–36 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
For example:
(kdbx) vnode
ADDR_VNODE V_TYPE V_TAG USECNT V_MOUNT
=========== ====== ====== ====== ===========
v0x9021e000 VBLK VT_NON
v0x9021e1e8 VBLK VT_NON
v0x9021e3d0 VBLK VT_NON
v0x9021e5b8 VDIR VT_UFS
v0x9021e7a0 VDIR VT_UFS
v0x9021ed58 VBLK VT_UFS
v0x9021ef40 VBLK VT_NON
v0x9021f128 VREG VT_UFS
v0x9021f310 VDIR VT_UFS
v0x9021f8c8 VREG VT_UFS
v0x9021fe80 VREG VT_UFS
v0x902209f0 VDIR VT_UFS
v0x90220fa8 VBLK VT_UFS
v0x90221190 VBLK VT_NON
v0x90221560 VREG VT_UFS
1
k0x00467ee8
83 v0x863abab8
k0x00467ee8
34 v0x863abab8
1
1
1
1
3
1
1
1
1
9
1
1
v0x863abab8
v0x863abab8
k0x00467ee8
v0x863abab8
v0x863abab8
v0x863abab8
v0x863abab8
v0x863abab8
v0x863abab8
k0x00467ee8
v0x863abab8
v0x90221748 VBLK VT_UFS 3153 v0x863abab8
.
.
.
(kdbx) vnode -nfs -v
ADDR_VNODE V_TYPE V_TAG USECNT V_MOUNT
FILEID MODE UID GID QSIZE
=========== ====== ====== ====== =========== ====== ====== ==== ==== ======
v0x90246820 VDIR VT_NFS
v0x902471a8 VDIR VT_NFS
v0x90247578 VDIR VT_NFS
v0x90247948 VDIR VT_NFS
v0x9026d1c0 VDIR VT_NFS
v0x9026e8a0 VDIR VT_NFS
v0x9026ea88 VDIR VT_NFS
v0x90272788 VDIR VT_NFS
v0x902fd080 VREG VT_NFS
v0x902ff888 VREG VT_NFS
v0x90326410 VREG VT_NFS
1
1
1
1
1
1
1
1
1
1
1
v0x863ab560 205732 40751 1138
v0x863ab398 378880 40755 1138
23
10
0
2048
5120
1024
512
512
512
v0x863ab1d0
2
40755
0
v0x863ab008 116736 40755 1114
0
v0x863ab1d0 14347 40755
v0x863aae40 40755
v0x863ab1d0 36874 40755
v0x863ab1d0 67594 40755
0
0
0
0
10
10
10
10
2
512
512
v0x863ab1d0 49368 100755 8887 177 455168
v0x863ab1d0 49289 100755 8887 177 538200
v0x863aae40 294959 100755
3
4 196608
.
.
.
(kdbx) vnode -ufs -v
ADDR_VNODE V_TYPE V_TAG USECNT V_MOUNT
INODE# MODE UID GID QSIZE
=========== ====== ====== ====== =========== ====== ====== ==== ==== ======
v0x9021e5b8 VDIR VT_UFS
v0x9021e7a0 VDIR VT_UFS
v0x9021ed58 VBLK VT_UFS
v0x9021f128 VREG VT_UFS
v0x9021f310 VDIR VT_UFS
v0x9021f8c8 VREG VT_UFS
v0x9021fe80 VREG VT_UFS
v0x902209f0 VDIR VT_UFS
v0x90220fa8 VBLK VT_UFS
v0x90221560 VREG VT_UFS
34 v0x863abab8
2
40755
0
0
0
3
3
3
3
0
0
3
0
0
0
0
1024
2560
0
1
1
3
1
1
1
1
9
1
v0x863abab8
v0x863abab8
v0x863abab8
v0x863abab8
v0x863abab8
v0x863abab8
v0x863abab8
v0x863abab8
v0x863abab8
1088 40755
1175 60600
7637 100755
8704 40755
7638 100755
7617 100755
9792 41777
1165 60600
7635 100755
1184 60600
4 147456
4
4
4 196608
10
512
90112
512
0
0
4 245760
0
v0x90221748 VBLK VT_UFS 3151 v0x863abab8
0
.
.
.
2.3 The kdebug Debugger
The kdebug debugger allows you to debug running kernel programs. You
can start and stop kernel execution, examine variable and register values,
Kernel Debugging Utilities 2–37
Download from Www.Somanuals.com. All Manuals Search And Download.
and perform other debugging tasks, just as you would when debugging user
space programs.
The ability to debug a running kernel is provided through remote debugging.
The kernel code you are debugging runs on a test system. The dbx debugger
runs on a remote build system. The debugger communicates with the
kernel code you are debugging over a serial communication line or through
a gateway system. You use a gateway system when you cannot physically
connect the test and build systems. Figure 2–1 shows the connections
needed when you use a gateway system.
Figure 2–1: Using a Gateway System During Remote Debugging
Network
Build System
Gateway System
Test System
Serial Line
dbx Debugger
Kernel Code
ZK−0974U−R
As shown in Figure 2–1, when you use a gateway system, the build system is
connected to it using a network line. The gateway system is connected to
the test system using a serial communication line.
Prior to running the kdebug debugger, the test, build, and gateway systems
must meet the following requirements:
•
The test system must be running Tru64 UNIX Version 2.0 or higher,
must have the Kernel Debugging Tools subset loaded, and must have the
Kernel Breakpoint Debugger kernel option configured.
•
The build system must be running Tru64 UNIX Version 2.0 or higher and
must have the Kernel Debugging Tools subset loaded. Also, this system
must contain a copy of the kernel code you are testing and, preferably,
the source used to build that kernel code.
•
The gateway system must be running Tru64 UNIX Version 2.0 or higher
and must have the Kernel Debugging Tools subset loaded.
2–38 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
To use the kdebug debugger, you must set up your build, gateway, and
test systems as described in Section 2.3.1. Once you complete the setup,
you invoke dbx as described in Section 2.3.2 and enter commands as you
normally would. Refer to Section 2.3.3 if you have problems with the setup
of your remote kdebug debugging session.
2.3.1 Getting Ready to Use the kdebug Debugger
To use the kdebug debugger, you must do the following:
1. Attach the test system and the build (or gateway) system.
To attach the serial line between the test and build (or gateway)
systems, locate the serial line used for kernel debugging. In general, the
correct serial line is either /dev/tty00 or /dev/tty01. For example,
if you have a DEC 3000 family workstation, kdebug debugger input and
output is always to the RS232C port on the back of the system. By
default, this port is identified as /dev/tty00 at installation time.
If your system is an AlphaStation or AlphaServer system with an ace
console serial interface, the system uses one of two serial ports for
kdebug input and output. By default, these systems use the COMM1
serial port (identified as /dev/tty00) when operating as a build or
gateway system. These systems use the COMM2 serial port (identified
as /dev/tty01) when operating as the test system.
To make it easier to connect the build or gateway system and the test
system for kernel debugging, you can modify your system setup. You
can change the system setup so that the COMM2 serial port is always
used for kernel debugging whether the system is operating as a build
system, a gateway system, or a test system.
To make COMM2 the serial port used for kernel debugging on
AlphaStations and AlphaServers, modify your /etc/remote file.
On these systems, the default kdebug debugger definition in the
/etc/remote file appears as follows:
kdebug:dv=/dev/tty00:br#9600:pa=none:
Modify this definition so that the device is /dev/tty01 (COMM2),
as follows:
kdebug:dv=/dev/tty01/br#9600:pa=none:
2. On the build system, install the Product Authorization Key (PAK) for
the Developer’s kit (OSF-DEV), if it is not already installed. For the
gateway and tests systems, the OSF-BASE license PAK is all that
is needed. For information about installing PAKs, see the Software
License Management guide.
3. On the build system, modify the setting of the $kdebug_host,
$kdebug_line, or $kdebug_dbgtty as needed.
Kernel Debugging Utilities 2–39
Download from Www.Somanuals.com. All Manuals Search And Download.
The $kdebug_host variable is the name of the gateway system. By
default, $kdebug_host is set to localhost, assuming no gateway
system is being used.
The $kdebug_line variable selects the serial line definition to use in
the /etc/remote file of the build system (or the gateway system, if one
is being used). By default, $kdebug_line is set to kdebug.
The $kdebug_dbgtty variable sets the terminal on the gateway system
to display the communication between the build and test systems, which
is useful in debugging your setup. To determine the terminal name to
supply to the $kdebug_dbgtty variable, enter the tty command in the
correct window on the gateway system. By default, $kdebug_dbgtty
is null.
For example, the following $HOME/.dbxinit file sets the
$kdebug_host variable to a system named gatewy:
set $kdebug_host="gatewy"
4. Recompile kernel files, if necessary.
By default, the kernel is compiled with only partial debugging
information. Occasionally, this partial information causes kdebug to
display erroneous arguments or mismatched source lines. To correct
this, recompile selected source files on the test system specifying the
CDEBUGOPTS=−g argument.
5. Make a backup copy of the kernel running on the test system so that
you can restore that kernel after testing:
# mv /vmunix /vmunix.save
6. Copy the kernel to be tested to /vmunix on the test system and reboot
the system:
# cp vmunix.test /vmunix
# shutdown -r now
7. If you are debugging on an SMP system, set the lockmode system
attribute to 4 on the test system, as follows:
a. Create a stanza-formatted file named, for example
generic.stanza, that appears as follows:
generic:
lockmode = 4
This file indicates that you are modifying the lockmode attribute
in the generic subsystem.
b. Use the sysconfigdb command to add the contents of the file to
the /etc/sysconfigtab database:
# sysconfigdb -a -f generic.stanza generic
c. Reboot your system.
2–40 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
Setting this system attribute makes debugging on an SMP system easier.
For information about the advantages provided see Section 2.1.11.
8. Set the OPTIONS KDEBUG configuration file option in your test kernel.
To set this option, run the doconfig command without flags, as shown:
# doconfig
Choose KERNEL BREAKPOINT DEBUGGING from the kernel options
menu when it is displayed by doconfig. Once doconfig finishes
building a new kernel, copy that kernel to the /vmunix file and reboot
your system. For more information about using the kernel options menu
to modify the kernel, see the System Administration manual.
2.3.2 Invoking the kdebug Debugger
You invoke the kdebug debugger as follows:
1. Invoke the dbx debugger on the build system, supplying the pathname
of the test kernel. Set a breakpoint and start running dbx as follows:
# dbx -remote vmunix
dbx version 5.0
Type ’help’ for help.
main: 602 p = &proc[0];
(dbx) stop in main
[2] stop in main
(dbx) run
Note that you can set a breakpoint anytime after the execution of the
kdebug_bootstrap() routine. Setting a breakpoint prior to the
execution of this routine can result in unpredictable behavior.
You can use all valid dbx flags with the -remote flag and define entries
in your $HOME/.dbxinit file as usual. For example, suppose you
start the dbx session in a directory other than the one that contains
the source and object files used to build the vmunix kernel you are
running on the test system. In this case, use the -I command flag or
the use command in your $HOME/.dbxinit file to point dbx to the
appropriate source and object files. For more information, see dbx(1)
and the Programmer’s Guide.
2. Halt the test system and, at the console prompt (three right angle
brackets), set the boot_osflags console variable to contain the k
option, and then boot the system. For example:
>>> set boot_osflags "k"
>>> boot
Once you boot the kernel, it begins executing. The dbx debugger will
halt execution at the breakpoint you specified, and you can begin issuing
Kernel Debugging Utilities 2–41
Download from Www.Somanuals.com. All Manuals Search And Download.
dbx debugging commands. See Section 2.1, the dbx(1) reference page, or
the Programmer’s Guide for information on dbx debugging commands.
If you are unable to bring your test kernel up to a fully operational mode,
you can reboot the halted system running the generic kernel, as follows:
>>> set boot_osflags "S"
>>> set boot_file "/genvmunix"
>>> boot
Once the system is running, you can run the bcheckrc script manually
to check and mount your local file systems. Then, copy the appropriate
kernel to the root (/) directory.
When you are ready to resume debugging, copy the test kernel to
/vmunix and reset the console variables and boot the system, as follows:
>>> set boot_osflags "k"
>>> set boot_file "/vmunix"
>>> boot
When you have completed your debugging session, reset the console
variables on the test system to their normal values, as follows:
>>> set boot_osflags "A"
>>> set boot_file "/vmunix"
>>> set auto_action boot
You might also need to replace the test kernel with a more reliable kernel.
For example, you should have saved a copy of the vmunix file that is
normally used to run the test system. You can copy that file to /vmunix and
shut down and reboot the system:
# mv /vmunix.save /vmunix
# shutdown -r now
2.3.3 Diagnosing kdebug Setup Problems
If you have completed the kdebug setup as described in Section 2.3.2 and
it fails to work, refer to the following list for help in diagnosing and fixing
the setup problem:
•
Determine whether the serial line is attached properly and then use the
tip command to test the connection.
Once you determine that the serial line is attached properly, log on to
the build system (or the gateway system if one is being used) as root
and enter the following command:
# tip kdebug
If the command does not return the message connected, another
process, such as a print daemon, might be using the serial line port that
you have dedicated to the kdebug debugger. To remedy this condition,
do the following:
2–42 Kernel Debugging Utilities
Download from Www.Somanuals.com. All Manuals Search And Download.
–
–
Check the /etc/inittab file to see if any processes are using that
line. If so, disable these lines until you finish with the kdebug
session. See the inittab(4) reference page for information on
disabling lines.
Examine your /etc/remote file to determine which serial line is
associated with the kdebug label. Then, use the ps command to see
if any processes are using the line. For example, if you are using the
/dev/tty00 serial port for your kdebug session, check for other
processes using the serial line with the following command:
# ps agxt00
If a process is using tty00, either kill that process or modify the
kdebug label so that a different serial line is used.
If the serial line specified in your /etc/remote file is used as the
system’s serial console, do not kill the process. In this case, use
another serial line for the kdebug debugger.
–
Determine whether any unused kdebugd gateway daemons are
running with the following command:
# ps agx | grep kdebugd
After ensuring the daemons are unused, kill the daemon processes.
•
•
If the test system boots to single user or beyond, then kdebug has not
been configured into the kernel as specified in Section 2.3.1. Ensure that
the boot_osflags console environment variable specifies the k flag
and try booting the system again:
>>> set boot_osflags k
>>> boot
Be sure you defined the dbx variables in your $HOME/.dbxinit file
|