第5章 - 障害解析

↑ | ← | →

Kernel Panic (5-7)

カーネルクラッシュダンプの設定

  1. パッケージ導入
    kexec-tools ... kdump取得用
    crash, kernel-debuginfo ... kdump 解析用
    kernel-debuginfo は /etc/yum.repo.d/CentOS-Debuginfo.rep の enabled=1 とする 必要がある。
  2. カーネルパラメータ修正
    /etc/grub.conf を修正し、kernel パラメータに crashkernel=64M@16M を追加
    kdump用メモリを物理アドレスの 64MB 目から、16MB確保。
  3. 再起動
    再起動し、kernel パラメータが修正されていることを確認。
    	$ cat /proc/cmdline
	ro root=/dev/VolGroup00/LogVol00 crashkernel=64M@16M
	$ grep -i crash /proc/iomem
	  01000000-04ffffff : Crash kernel
  4. kdump 起動
    kdumpを起動。また自動実行されるよう調整
    	# /etc/init.d/kdump start
	# chkconfig kdump on
カーネルクラッシュの再現方法 
# echo c > /proc/sysrq-trigger		# クラッシュ実行
	sysrq-trigger : c crash / p register dump / m memory info / t task info
  * この直後、システムが停止、crash kernel がダンプを作成し、再起動

# ls -lR /var/crash
/var/crash:
total 4
drwxr-xr-x 2 root root 4096 Oct 25 23:55 2012-10-25-23:54

/var/crash/2012-11-25-23:54:
total 190580
-r-------- 1 root root 469238176 Oct 25 23:55 vmcore
クラッシュダンプの解析

生成された vmcore は crash により解析できる。
前提として kernel-debuginfo がインストールされている事。
# crash システムマップ　　DEBUGINFO　　vmcore 

# uname -a
Linux vm01.ycos.net 2.6.18-308.el5 #1 SMP Tue Feb 21 20:05:41 EST 2012 i686 i686 i386 GNU/Linux
# crash /boot/System.map-2.6.18-308.el5 /usr/lib/debug/lib/modules/2.6.18-308.16.1.el5.centos.plus/vmlinux /var/crash/2012-11-25-23\:54/vmcore

crash 5.1.8-1.el5.centos
Copyright (C) 2002-2011  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...

  SYSTEM MAP: /boot/System.map-2.6.18-308.el5
DEBUG KERNEL: /usr/lib/debug/lib/modules/2.6.18-308.16.1.el5.centos.plus/vmlinux (2.6.18-308.16.1.el5.centos.plus)
    DUMPFILE: /var/crash/2012-11-25-23:54/vmcore
        CPUS: 1
        DATE: Sun Nov 25 23:54:14 2012
      UPTIME: 00:01:50
LOAD AVERAGE: 0.80, 0.45, 0.17
       TASKS: 93
    NODENAME: vm01.ycos.net
     RELEASE: 2.6.18-308.el5
     VERSION: #1 SMP Tue Feb 21 20:05:41 EST 2012
     MACHINE: i686  (2873 Mhz)
      MEMORY: 511.9 MB
       PANIC: "SysRq : Trigger a crashdump"
         PID: 2335
     COMMAND: "bash"
        TASK: de777550  [THREAD_INFO: d72e3000]
         CPU: 0
       STATE: TASK_RUNNING (SYSRQ)

crash> log
Linux version 2.6.18-308.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-52)) #1 SMP Tue Feb 21 20:05:41 EST 2012
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000010000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000001fff0000 (usable)
 BIOS-e820: 000000001fff0000 - 0000000020000000 (ACPI data)
 BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
0MB HIGHMEM available.
511MB LOWMEM available.
Using x86 segment limits to approximate NX protection
On node 0 totalpages: 131056
  DMA zone: 4096 pages, LIFO batch:0
  Normal zone: 126960 pages, LIFO batch:31
DMI 2.5 present.
DMI: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Using APIC driver default
ACPI: RSDP (v002 VBOX                                  ) @ 0x000e0000
ACPI: XSDT (v001 VBOX   VBOXXSDT 0x00000001 ASL  0x00000061) @ 0x1fff0030
ACPI: FADT (v004 VBOX   VBOXFACP 0x00000001 ASL  0x00000061) @ 0x1fff00f0
ACPI: SSDT (v001 VBOX   VBOXCPUT 0x00000002 INTL 0x20100528) @ 0x1fff0240
ACPI: DSDT (v001 VBOX   VBOXBIOS 0x00000002 INTL 0x20100528) @ 0x00000000
crash> quit

5-7. panic 処理方法のまとめ
	see. http://iktaka.dyndns.org/node/128

5-8. core dump の実験
core は mem.c を使って再現可能、Segmentation falt