https://blog.csdn.net/petter7226/article/details/52024042?utm_source=blogxgwz7

 

环境centos6.5x86_64,这台web给数据采集用的,主要是抓取数据.启动过错出错信息:
Call Trace:
[<ffffffff802bf8db>] out_of_memory+0x8b/0x203
[<ffffffff8020f8a2>] __alloc_pages+0x27f/0x308
[<ffffffff8025d117>] getnstimeofday+0x10/0x28
[<ffffffff80213836>] __do_page_cache_readahead+0xc6/0x1ab
[<ffffffff802140ff>] filemap_nopage+0x14c/0x360
[<ffffffff80208ebc>] __handle_mm_fault+0x443/0x1348
[<ffffffff8020622a>] hypercall_page+0x22a/0x1000
[<ffffffff8026769a>] do_page_fault+0xf7b/0x12e0
[<ffffffff8026ef4b>] monotonic_clock+0x35/0x7b
[<ffffffff80262dd3>] thread_return+0x6c/0x113
[<ffffffff8026082b>] error_exit+0x0/0x6e
DMA per-cpu:
cpu 0 hot: high 186, batch 31 used:26
cpu 0 cold: high 62, batch 15 used:58
cpu 1 hot: high 186, batch 31 used:35
cpu 1 cold: high 62, batch 15 used:49
DMA32 per-cpu: empty
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages:        5760kB (0kB HighMem)
Active:46859 inactive:24599 dirty:0 writeback:0 unstable:0 free:1440 slab:28082 mapped-file:5 mapped-anon:76709 pagetables:390527
DMA free:5760kB min:5800kB low:7248kB high:8700kB active:187436kB inactive:98396kB present:2105344kB pages_scanned:328111276 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 16*4kB 68*8kB 2*16kB 4*32kB 2*64kB 2*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5760kB
DMA32: empty
Normal: empty
HighMem: empty
45 pagecache pages
Swap cache: add 14188563, delete 14188552, find 585539/2267924, race 4109+888
Free swap  = 0kB
Total swap = 4192956kB
重启后查看message错误日志:
Dec  9 12:09:33 SN7x kernel: INFO: task httpd:31276 blocked for more than 120 seconds.
Dec  9 12:09:33 SN7x kernel: “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
Dec  9 12:09:33 SN7x kernel: httpd         D 011d5cb2df1c1910     0 31276   6472         31284 31269 (NOTLB)
Dec  9 12:09:33 SN7x kernel:  ffff88005e587d28  0000000000000286  0000000050c40dea  ffff88007f67d678
Dec  9 12:09:33 SN7x kernel:  0000000000000008  ffff8800614cf820  ffffffff804f4b80  0000000000003d62
Dec  9 12:09:33 SN7x kernel:  ffff8800614cfa08  ffffffff00000001
Dec  9 12:09:33 SN7x kernel: Call Trace:
Dec  9 12:09:34 SN7x kernel:  [<ffffffff802b653d>] delayacct_end+0x5d/0x86
Dec  9 12:09:34 SN7x kernel:  [<ffffffff80263a7e>] __mutex_lock_slowpath+0x60/0x9b
Dec  9 12:09:35 SN7x kernel:  [<ffffffff80263ac8>] .text.lock.mutex+0xf/0x14
Dec  9 12:09:35 SN7x kernel:  [<ffffffff802cb8e0>] remove_exclusive_swap_page+0x10b/0x116
Dec  9 12:09:35 SN7x kernel:  [<ffffffff80222073>] generic_file_aio_write+0x4e/0xc1
Dec  9 12:09:35 SN7x kernel:  [<ffffffff8804c1b6>] :ext3:ext3_file_write+0x16/0x91
Dec  9 12:09:35 SN7x kernel:  [<ffffffff80218705>] do_sync_write+0xc7/0x104
Dec  9 12:09:35 SN7x kernel:  [<ffffffff802676cd>] do_page_fault+0xfae/0x12e0
Dec  9 12:09:35 SN7x kernel:  [<ffffffff8029c60a>] autoremove_wake_function+0x0/0x2e
Dec  9 12:09:35 SN7x kernel:  [<ffffffff802172b2>] vfs_write+0xce/0x174
Dec  9 12:09:35 SN7x kernel:  [<ffffffff80217aff>] sys_write+0x45/0x6e
Dec  9 12:09:36 SN7x kernel:  [<ffffffff802602f9>] tracesys+0xab/0xb6

查询结果:
httpd:31276 blocked for more than 120 seconds.  httpd进程阻塞了120s,查了下:
This is a know bug. By default linux uses up to 40% of the available memory for file system caching.

原理:linux会设置40%的可用内存用来做系统cache,当flush数据时这40%内存中的数据由于和IO同步问题导致超时(120s)简单讲就是设置在文件 /etc/sysctl.conf中加入 “vm.dirty_ratio=10″

目前处理方案:
更改maxclients,将其进程数的最大值设大点或
临时处理更改了下内核参数有待后续观察:
‘echo 0 > /proc/sys/kernel/hung_task_timeout_secs.
在进程处理于可中断的睡眠状态S 时,进程要等待 如 套接字、信号量被挂起,进入到对应事件等待队列中的的这个时间太长,占用了队列空间使进程长时间处于阻塞状态,影响了进程队列的有序调度
————————————————
版权声明:本文为CSDN博主「ugomi」的原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接及本声明。
原文链接:https://blog.csdn.net/petter7226/article/details/52024042

发表评论

您的电子邮箱地址不会被公开。 必填项已用*标注