Linux Kernel Debugging

Effectively debug kernel modules, device drivers, and the kernel itself by gaining a solid understanding of powerful open source tools and advanced kernel debugging techniques

Key Features

• Fully understand how to use a variety of kernel and module debugging tools and techniques using examples
• Learn to expertly interpret a kernel Oops and identify underlying defect(s)
• Use easy-to-look up tables and clear explanations of kernel-level defects to make this complex topic easy

Book Description

The Linux kernel is at the very core of arguably the world’s best production-quality OS. Debugging it, though, can be a complex endeavor.

Linux Kernel Debugging is a comprehensive guide to learning all about advanced kernel debugging. This book covers many areas in-depth, such as instrumentation-based debugging techniques (printk and the dynamic debug framework), and shows you how to use Kprobes. Memory-related bugs tend to be a nightmare – two chapters are packed with tools and techniques devoted to debugging them. When the kernel gifts you an Oops, how exactly do you interpret it to be able to debug the underlying issue? We’ve got you covered. Concurrency tends to be an inherently complex topic, so a chapter on lock debugging will help you to learn precisely what data races are, including using KCSAN to detect them. Some thorny issues, both debug- and performance-wise, require detailed kernel-level tracing; you’ll learn to wield the impressive power of Ftrace and its frontends. You’ll also discover how to handle kernel lockups, hangs, and the dreaded kernel panic, as well as leverage the venerable GDB tool within the kernel (KGDB), along with much more.

By the end of this book, you will have at your disposal a wide range of powerful kernel debugging tools and techniques, along with a keen sense of when to use which.

What you will learn

• Explore instrumentation-based printk along with the powerful dynamic debug framework
• Use static and dynamic Kprobes to trap into kernel/module functions
• Catch kernel memory defects with KASAN, UBSAN, SLUB debug, and kmemleak
• Interpret an Oops in depth and precisely identify it's source location
• Understand data races and use KCSAN to catch evasive concurrency defects
• Leverage Ftrace and trace-cmd to trace the kernel flow in great detail
• Write a custom kernel panic handler and detect kernel lockups and hangs
• Use KGDB to single-step and debug kernel/module source code

Who this book is for

This book is for Linux kernel developers, module/driver authors, and testers interested in debugging and enhancing their Linux systems at the level of the kernel. System administrators who want to understand and debug the internal infrastructure of their Linux kernels will also find this book useful. A good grasp on C programming and the Linux command line is necessary. Some experience with kernel (module) development will help you follow along.

Table of Contents

1. A General Introduction to Debugging Software
2. Approaches to Kernel Debugging
3. Debug via Instrumentation – printk and Friends
4. Debug via Instrumentation – Kprobes
5. Debugging Kernel Memory Issues – Part 1
6. Debugging Kernel Memory Issues – Part 2
7. Oops! Interpreting the Kernel Bug Diagnostic
8. Lock Debugging
9. Tracing the Kernel Flow
10. Kernel Panic, Lockups, and Hangs
11. Using Kernel GDB (KGDB)
12. A Few More Kernel Debugging Approaches

Kaiwan N. Billimoria taught himself BASIC programming on his dad's IBM PC back in 1983. He was programming in C and Assembly on DOS until he discovered the joys of Unix, and by around 1997, Linux! Kaiwan has worked on many aspects of the Linux system programming stack, including Bash scripting, system programming in C, kernel internals, device drivers, and embedded Linux work. He has actively worked on several commercial/FOSS projects. His contributions include drivers to the mainline Linux OS and many smaller projects hosted on GitHub. His Linux passion feeds well into his passion for teaching these topics to engineers, which he has done for well over two decades now. He's also the author of Hands-On System Programming with Linux, Linux Kernel Programming (and its Part 2 book) and Linux Kernel Debugging. It doesn't hurt that he is a recreational ultrarunner too.