Я написал программу на C, которая иногда умирает через несколько дней. Он работает на встроенном оборудовании, поэтому трудно правильно отладить проблему (нет локального gdb, нет valgrind, но у меня есть strace).
Он не генерирует файл ядра, когда он умирает, даже если используется ulimit -c unlimited.
Когда он умирает, все, что отображается на консоли, «убивается». Логи из самой программы не помогают.
Я подозреваю, что переполнение буфера, переполнение памяти (отсутствует free) или проблема многопоточности.
Я не использую обработчик сигнала в коде (это может помочь?). Откуда это kill -9???
Я пробовал следующее:
$ ./MyProg
killed
$ time -v ./MyProg
Command terminated by signal 9
Command being timed: "./MyProg"
User time (seconds): 762.04
System time (seconds): 1360.74
Percent of CPU this job got: 2%
Elapsed (wall clock) time (h:mm:ss or m:ss): 23h 4m 23s
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 0
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 40
Minor (reclaiming a frame) page faults: 29567
Voluntary context switches: 4742276
Involuntary context switches: 187702
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
$ cat /proc/PID/smaps # Shortly before the crash
10000000-10042000 r-xp 00000000 00:0b 203162716 /root/MyProg
Size: 264 kB
Rss: 224 kB
Pss: 224 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 224 kB
Private_Dirty: 0 kB
Referenced: 120 kB
10052000-10055000 rwxp 00042000 00:0b 203162716 /root/MyProg
Size: 12 kB
Rss: 12 kB
Pss: 12 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 12 kB
Referenced: 12 kB
10055000-1706a000 rwxp 10055000 00:00 0 [heap]
Size: 114772 kB
Rss: 114716 kB
Pss: 114716 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 114716 kB
Referenced: 114716 kB
30000000-30005000 r-xp 00000000 00:0b 135513112 /lib/ld-uClibc-0.9.29.so
Size: 20 kB
Rss: 20 kB
Pss: 1 kB
Shared_Clean: 20 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 20 kB
30005000-30006000 rw-p 30005000 00:00 0
Size: 4 kB
Rss: 4 kB
Pss: 4 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 4 kB
Referenced: 4 kB
30014000-30015000 r--p 00004000 00:0b 135513112 /lib/ld-uClibc-0.9.29.so
Size: 4 kB
Rss: 4 kB
Pss: 4 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 4 kB
Referenced: 4 kB
30015000-30016000 rwxp 00005000 00:0b 135513112 /lib/ld-uClibc-0.9.29.so
Size: 4 kB
Rss: 4 kB
Pss: 4 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 4 kB
Referenced: 4 kB
30016000-30027000 r-xp 00000000 00:0b 135513124 /lib/libm-0.9.29.so
Size: 68 kB
Rss: 12 kB
Pss: 1 kB
Shared_Clean: 12 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 12 kB
30027000-30036000 ---p 30027000 00:00 0
Size: 60 kB
Rss: 0 kB
Pss: 0 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 0 kB
30036000-30037000 r--p 00010000 00:0b 135513124 /lib/libm-0.9.29.so
Size: 4 kB
Rss: 4 kB
Pss: 0 kB
Shared_Clean: 4 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 4 kB
30037000-30038000 rwxp 00011000 00:0b 135513124 /lib/libm-0.9.29.so
Size: 4 kB
Rss: 4 kB
Pss: 4 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 4 kB
Referenced: 4 kB
30038000-30043000 r-xp 00000000 00:0b 135513129 /lib/libpthread-0.9.29.so
Size: 44 kB
Rss: 44 kB
Pss: 44 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 44 kB
Private_Dirty: 0 kB
Referenced: 20 kB
30043000-30052000 ---p 30043000 00:00 0
Size: 60 kB
Rss: 0 kB
Pss: 0 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 0 kB
30052000-30053000 r--p 0000a000 00:0b 135513129 /lib/libpthread-0.9.29.so
Size: 4 kB
Rss: 4 kB
Pss: 4 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 4 kB
Referenced: 4 kB
30053000-30058000 rwxp 0000b000 00:0b 135513129 /lib/libpthread-0.9.29.so
Size: 20 kB
Rss: 8 kB
Pss: 8 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 8 kB
Referenced: 8 kB
30058000-3005a000 rwxp 30058000 00:00 0
Size: 8 kB
Rss: 0 kB
Pss: 0 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 0 kB
3005a000-3005b000 r-xp 00000000 00:0b 135513131 /lib/librt-0.9.29.so
Size: 4 kB
Rss: 4 kB
Pss: 2 kB
Shared_Clean: 4 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 0 kB
3005b000-3006a000 ---p 3005b000 00:00 0
Size: 60 kB
Rss: 0 kB
Pss: 0 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 0 kB
3006a000-3006b000 r--p 00000000 00:0b 135513131 /lib/librt-0.9.29.so
Size: 4 kB
Rss: 4 kB
Pss: 2 kB
Shared_Clean: 4 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 0 kB
3006b000-3006c000 rwxp 00001000 00:0b 135513131 /lib/librt-0.9.29.so
Size: 4 kB
Rss: 4 kB
Pss: 4 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 4 kB
Referenced: 4 kB
3006c000-30079000 r-xp 00000000 00:0b 135513120 /lib/libgcc_s.so.1
Size: 52 kB
Rss: 28 kB
Pss: 21 kB
Shared_Clean: 8 kB
Shared_Dirty: 0 kB
Private_Clean: 20 kB
Private_Dirty: 0 kB
Referenced: 20 kB
30079000-30088000 ---p 30079000 00:00 0
Size: 60 kB
Rss: 0 kB
Pss: 0 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 0 kB
30088000-30089000 rwxp 0000c000 00:0b 135513120 /lib/libgcc_s.so.1
Size: 4 kB
Rss: 4 kB
Pss: 4 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 4 kB
Referenced: 4 kB
30089000-300d0000 r-xp 00000000 00:0b 135513132 /lib/libuClibc-0.9.29.so
Size: 284 kB
Rss: 188 kB
Pss: 22 kB
Shared_Clean: 180 kB
Shared_Dirty: 0 kB
Private_Clean: 8 kB
Private_Dirty: 0 kB
Referenced: 164 kB
300d0000-300df000 ---p 300d0000 00:00 0
Size: 60 kB
Rss: 0 kB
Pss: 0 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 0 kB
300df000-300e0000 r--p 00046000 00:0b 135513132 /lib/libuClibc-0.9.29.so
Size: 4 kB
Rss: 4 kB
Pss: 4 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 4 kB
Referenced: 4 kB
300e0000-300e1000 rwxp 00047000 00:0b 135513132 /lib/libuClibc-0.9.29.so
Size: 4 kB
Rss: 4 kB
Pss: 4 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 4 kB
Referenced: 4 kB
300e1000-300e6000 rwxp 300e1000 00:00 0
Size: 20 kB
Rss: 16 kB
Pss: 16 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 16 kB
Referenced: 16 kB
7f3fc000-7f400000 rwxp 7f3fc000 00:00 0
Size: 16 kB
Rss: 16 kB
Pss: 16 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 16 kB
Referenced: 16 kB
7faf8000-7fb0d000 rwxp 7ffeb000 00:00 0 [stack]
Size: 84 kB
Rss: 12 kB
Pss: 12 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 12 kB
Referenced: 12 kB
Из / var / log / messages:
user.warn kernel: dropbear invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
user.warn kernel: Call Trace:
user.warn kernel: show_stack+0x50/0x184 (unreliable)
user.warn kernel: oom_kill_process+0x54/0x1ac
user.warn kernel: out_of_memory+0x1a8/0x1dc
user.warn kernel: __alloc_pages+0x24c/0x2dc
user.warn kernel: __do_page_cache_readahead+0xc4/0x220
user.warn kernel: filemap_fault+0x150/0x37c
user.warn kernel: __do_fault+0x6c/0x40c
user.warn kernel: do_page_fault+0x274/0x3ec
user.warn kernel: handle_page_fault+0xc/0x80
user.warn kernel: Mem-info:
user.warn kernel: DMA per-cpu:
user.warn kernel: CPU 0: hi: 42, btch: 7 usd: 31
user.warn kernel: Active:29872 inactive:194 dirty:0 writeback:0 unstable:0
user.warn kernel: free:356 slab:1392 mapped:84 pagetables:65 bounce:0
user.warn kernel: DMA free:1424kB min:1440kB low:1800kB high:2160kB active:119488kB inactive:776kB present:130048kB pages_scanned:194475 all_unreclaimable? yes
user.warn kernel: lowmem_reserve[]: 0 0 0
user.warn kernel: DMA: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1424kB
user.warn kernel: 204 total pagecache pages
user.warn kernel: Free swap: 0kB
user.warn kernel: 32768 pages of RAM
user.warn kernel: 0 pages of HIGHMEM
user.warn kernel: 1236 free pages
user.warn kernel: 770 reserved pages
user.warn kernel: 112 pages shared
user.warn kernel: 0 pages swap cached
user.err kernel: Out of memory: kill process 206 (MyProg) score 623 or a child
user.err kernel: Killed process 206 (MyProg)
Что еще я могу попробовать? Спасибо