Мы наблюдаем увеличение использования памяти нашими экземплярами ec2 с течением времени.
Через две недели мы должны перезагрузить наши системы.
На этих машинах запускаются некоторые док-контейнеры. Давайте посмотрим
через 'free -m' через 14 дней (я сейчас остановил демон докера):
$free -m
total used free shared buffers cached
Mem: 7977 7852 124 0 4 573
-/+ buffers/cache: 7273 703
Swap: 0 0 0
Теперь я запускаю 'ps_mem':
Private + Shared = RAM used Program
124.0 KiB + 64.5 KiB = 188.5 KiB agetty
140.0 KiB + 60.5 KiB = 200.5 KiB acpid
180.0 KiB + 41.5 KiB = 221.5 KiB rngd
200.0 KiB + 205.5 KiB = 405.5 KiB lvmpolld
320.0 KiB + 89.5 KiB = 409.5 KiB irqbalance
320.0 KiB + 232.5 KiB = 552.5 KiB lvmetad
476.0 KiB + 99.5 KiB = 575.5 KiB auditd
624.0 KiB + 105.5 KiB = 729.5 KiB init
756.0 KiB + 72.5 KiB = 828.5 KiB crond
292.0 KiB + 622.5 KiB = 914.5 KiB udevd (3)
560.0 KiB + 377.0 KiB = 937.0 KiB mingetty (6)
1.0 MiB + 194.5 KiB = 1.2 MiB ntpd
1.1 MiB + 256.0 KiB = 1.4 MiB dhclient (2)
2.5 MiB + 103.5 KiB = 2.6 MiB rsyslogd
3.1 MiB + 259.0 KiB = 3.4 MiB sendmail.sendmail (2)
3.0 MiB + 609.0 KiB = 3.6 MiB sudo (2)
3.6 MiB + 1.6 MiB = 5.2 MiB bash (5)
2.9 MiB + 4.3 MiB = 7.2 MiB sshd (9)
14.5 MiB + 413.5 KiB = 14.9 MiB dmeventd
---------------------------------
45.4 MiB
=================================
Теперь я пытаюсь выделить новую память с помощью инструмента «стресс» (http://people.seas.harvard.edu/~apw/stress/):
$ stress --vm 1 --vm-bytes 1G --timeout 10s --verbose
stress: info: [11120] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
stress: dbug: [11120] using backoff sleep of 3000us
stress: dbug: [11120] setting timeout to 10s
stress: dbug: [11120] --> hogvm worker 1 [11121] forked
stress: dbug: [11121] allocating 1073741824 bytes ...
stress: FAIL: [11121] (494) hogvm malloc failed: Cannot allocate memory
stress: FAIL: [11120] (394) <-- worker 11121 returned error 1
stress: WARN: [11120] (396) now reaping child worker processes
stress: FAIL: [11120] (451) failed run completed in 0s
==> «стресс» не может выделить 1G новой памяти.
Но я не понимаю, куда ушла вся моя память?!
А вот и вывод «top» (аналог ps_mem):
Tasks: 107 total, 1 running, 66 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8168828k total, 8045784k used, 123044k free, 5656k buffers
Swap: 0k total, 0k used, 0k free, 589372k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2030 root 20 0 102m 17m 5812 S 0.0 0.2 1:21.36 dmeventd
11145 root 20 0 82664 6604 5752 S 0.0 0.1 0:00.00 sshd
11130 root 20 0 183m 4472 3824 S 0.0 0.1 0:00.00 sudo
18339 ec2-user 20 0 114m 3896 1744 S 0.0 0.0 0:00.08 bash
2419 root 20 0 241m 3552 1188 S 0.0 0.0 2:07.85 rsyslogd
11146 sshd 20 0 80588 3440 2612 S 0.0 0.0 0:00.00 sshd
11131 root 20 0 112m 3288 2924 S 0.0 0.0 0:00.00 bash
17134 root 20 0 117m 3084 2008 S 0.0 0.0 0:00.00 sshd
17148 ec2-user 20 0 112m 2992 2620 S 0.0 0.0 0:00.01 bash
2605 root 20 0 85496 2776 1064 S 0.0 0.0 0:21.44 sendmail
2614 smmsp 20 0 81088 2704 1208 S 0.0 0.0 0:00.17 sendmail
15228 root 20 0 112m 2632 2228 S 0.0 0.0 0:00.02 bash
1 root 20 0 19684 2376 2068 S 0.0 0.0 0:01.91 init
2626 root 20 0 118m 2276 1644 S 0.0 0.0 0:02.45 crond
2233 root 20 0 9412 2244 1748 S 0.0 0.0 0:00.49 dhclient
11147 root 20 0 15364 2176 1856 R 0.0 0.0 0:00.00 top
2584 ntp 20 0 113m 2128 1308 S 0.0 0.0 0:49.60 ntpd
Где используется эта проклятая 7273 МБ памяти?
cat / proc / meminfo
MemTotal: 8168828 kB
MemFree: 129736 kB
MemAvailable: 567464 kB
Buffers: 5116 kB
Cached: 585504 kB
SwapCached: 0 kB
Active: 476920 kB
Inactive: 130228 kB
Active(anon): 22340 kB
Inactive(anon): 80 kB
Active(file): 454580 kB
Inactive(file): 130148 kB
Unevictable: 17620 kB
Mlocked: 17620 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 16 kB
Writeback: 0 kB
AnonPages: 34088 kB
Mapped: 14668 kB
Shmem: 80 kB
Slab: 5625876 kB
SReclaimable: 142240 kB
SUnreclaim: 5483636 kB
KernelStack: 2016 kB
PageTables: 4384 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4084412 kB
Committed_AS: 109856 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 7286784 kB
DirectMap2M: 1101824 kB
Выход плиты
Active / Total Objects (% used) : 8445426 / 11391340 (74.1%)
Active / Total Slabs (% used) : 533926 / 533926 (100.0%)
Active / Total Caches (% used) : 78 / 101 (77.2%)
Active / Total Size (% used) : 5033325.10K / 5414048.91K (93.0%)
Minimum / Average / Maximum Object : 0.01K / 0.47K / 9.44K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
3216990 525372 16% 0.09K 76595 42 306380K kmalloc-96
3101208 3101011 99% 1.00K 219166 32 7013312K kmalloc-1024
2066976 2066841 99% 0.32K 86124 24 688992K taskstats
1040384 1039935 99% 0.03K 8128 128 32512K kmalloc-32
1038080 1037209 99% 0.06K 16220 64 64880K kmalloc-64
516719 516719 100% 2.09K 113785 15 3641120K request_queue
223356 22610 10% 0.57K 7977 28 127632K radix_tree_node
52740 39903 75% 0.13K 1758 30 7032K kernfs_node_cache
Теперь я перезагрузил машину и сделал "perf kmem record --caller". Через несколько секунд мне пришлось отменить, потому что файл data.perf был уже более 1 ГБ лагера. Я сделал «perf kmem stat --caller», и вот вывод:
---------------------------------------------------------------------------------------------------------
Callsite | Total_alloc/Per | Total_req/Per | Hit | Ping-pong | Frag
---------------------------------------------------------------------------------------------------------
dm_open+2b | 240/8 | 120/4 | 30 | 0 | 50,000%
match_number+2a | 120/8 | 60/4 | 15 | 0 | 50,000%
rebuild_sched_domains_locked+dd | 72/8 | 36/4 | 9 | 0 | 50,000%
dm_btree_del+2b | 40960/4096 | 20720/2072 | 10 | 0 | 49,414%
sk_prot_alloc+7c | 86016/2048 | 44016/1048 | 42 | 0 | 48,828%
hugetlb_cgroup_css_alloc+29 | 2560/512 | 1320/264 | 5 | 0 | 48,438%
blk_throtl_init+2a | 15360/1024 | 8040/536 | 15 | 0 | 47,656%
bpf_int_jit_compile+6e | 40960/8192 | 21440/4288 | 5 | 0 | 47,656%
mem_cgroup_css_alloc+2f | 10240/2048 | 5360/1072 | 5 | 2 | 47,656%
alloc_disk_node+32 | 30720/2048 | 16560/1104 | 15 | 0 | 46,094%
mem_cgroup_css_alloc+166 | 5120/1024 | 2800/560 | 5 | 2 | 45,312%
blkcg_css_alloc+3b | 2560/512 | 1400/280 | 5 | 0 | 45,312%
kobject_uevent_env+be | 1224704/4096 | 698464/2336 | 299 | 0 | 42,969%
uevent_show+81 | 675840/4096 | 385440/2336 | 165 | 0 | 42,969%
blkg_alloc+3c | 40960/1024 | 23680/592 | 40 | 0 | 42,188%
dm_table_create+34 | 7680/512 | 4560/304 | 15 | 0 | 40,625%
journal_init_common+34 | 30720/2048 | 18360/1224 | 15 | 0 | 40,234%
throtl_pd_alloc+2b | 56320/1024 | 34320/624 | 55 | 0 | 39,062%
strndup_user+3f | 14496/17 | 8917/10 | 829 | 0 | 38,486%
alloc_trial_cpuset+19 | 14336/1024 | 8848/632 | 14 | 0 | 38,281%
cpuset_css_alloc+29 | 5120/1024 | 3160/632 | 5 | 0 | 38,281%
proc_reg_open+33 | 48768/64 | 30480/40 | 762 | 0 | 37,500%
get_mountpoint+73 | 26432/64 | 16520/40 | 413 | 0 | 37,500%
alloc_pipe_info+aa | 219136/1024 | 136960/640 | 214 | 12 | 37,500%
alloc_fair_sched_group+f0 | 38400/512 | 24000/320 | 75 | 0 | 37,500%
__alloc_workqueue_key+77 | 15360/512 | 9600/320 | 30 | 0 | 37,500%
newary+69 | 15360/512 | 9600/320 | 30 | 0 | 37,500%
disk_expand_part_tbl+74 | 960/64 | 600/40 | 15 | 0 | 37,500%
alloc_dax+29 | 120/8 | 75/5 | 15 | 0 | 37,500%
kernfs_mount_ns+3c | 320/64 | 200/40 | 5 | 0 | 37,500%
bucket_table_alloc+be | 16640/978 | 10496/617 | 17 | 12 | 36,923%
__alloc_workqueue_key+250 | 7680/512 | 4920/328 | 15 | 0 | 35,938%
journal_init_common+1b9 | 61440/4096 | 40920/2728 | 15 | 0 | 33,398%
kernfs_fop_write+b3 | 2248/11 | 1507/7 | 191 | 0 | 32,963%
__alloc_skb+72 | 3698176/876 | 2578048/611 | 4217 | 115 | 30,289%
alloc_pid+33 | 80896/128 | 56944/90 | 632 | 43 | 29,608%
alloc_pipe_info+3d | 41088/192 | 29104/136 | 214 | 12 | 29,167%
device_create_groups_vargs+59 | 15360/1024 | 10920/728 | 15 | 0 | 28,906%
sget_userns+ee | 112640/2048 | 80960/1472 | 55 | 8 | 28,125%
key_alloc+13e | 480/96 | 350/70 | 5 | 0 | 27,083%
load_elf_phdrs+49 | 153600/602 | 113176/443 | 255 | 0 | 26,318%
alloc_vfsmnt+aa | 11752/22 | 8765/17 | 513 | 130 | 25,417%
__memcg_init_list_lru_node+6b | 35200/32 | 26400/24 | 1100 | 160 | 25,000%
proc_self_get_link+96 | 12352/16 | 9264/12 | 772 | 0 | 25,000%
memcg_kmem_get_cache+9e | 46336/64 | 34752/48 | 724 | 0 | 25,000%
kernfs_fop_open+286 | 45056/64 | 33792/48 | 704 | 0 | 25,000%
insert_shadow+27 | 16544/32 | 12408/24 | 517 | 3 | 25,000%
allocate_cgrp_cset_links+70 | 28800/64 | 21600/48 | 450 | 0 | 25,000%
ext4_ext_remove_space+8db | 12352/64 | 9264/48 | 193 | 0 | 25,000%
dev_exception_add+25 | 5760/64 | 4320/48 | 90 | 0 | 25,000%
mempool_create_node+4e | 8160/96 | 6120/72 | 85 | 0 | 25,000%
alloc_rt_sched_group+11d | 7200/96 | 5400/72 | 75 | 0 | 25,000%
copy_semundo+60 | 2400/32 | 1800/24 | 75 | 7 | 25,000%
ext4_readdir+825 | 3264/64 | 2448/48 | 51 | 0 | 25,000%
alloc_worker+1d | 8640/192 | 6480/144 | 45 | 0 | 25,000%
alloc_workqueue_attrs+27 | 1440/32 | 1080/24 | 45 | 0 | 25,000%
ext4_fill_super+57 | 30720/2048 | 23040/1536 | 15 | 0 | 25,000%
apply_wqattrs_prepare+32 | 960/64 | 720/48 | 15 | 0 | 25,000%
inotify_handle_event+68 | 960/64 | 720/48 | 15 | 1 | 25,000%
blk_alloc_queue_stats+1b | 480/32 | 360/24 | 15 | 0 | 25,000%
proc_self_get_link+57 | 160/16 | 120/12 | 10 | 0 | 25,000%
disk_seqf_start+25 | 256/32 | 192/24 | 8 | 0 | 25,000%
memcg_write_event_control+8a | 960/192 | 720/144 | 5 | 0 | 25,000%
eventfd_file_create.part.3+28 | 320/64 | 240/48 | 5 | 0 | 25,000%
do_seccomp+249 | 160/32 | 120/24 | 5 | 0 | 25,000%
mem_cgroup_oom_register_event+29 | 160/32 | 120/24 | 5 | 0 | 25,000%
bucket_table_alloc+32 | 512/512 | 384/384 | 1 | 0 | 25,000%
__kernfs_new_node+25 | 42424/33 | 32046/24 | 1284 | 2 | 24,463%
single_open_size+2f | 45056/4096 | 35024/3184 | 11 | 0 | 22,266%
alloc_fdtable+ae | 544/90 | 424/70 | 6 | 0 | 22,059%
__register_sysctl_paths+10f | 2304/256 | 1800/200 | 9 | 0 | 21,875%
pskb_expand_head+71 | 10240/2048 | 8000/1600 | 5 | 0 | 21,875%
cpuacct_css_alloc+28 | 1280/256 | 1000/200 | 5 | 0 | 21,875%
shmem_symlink+a5 | 1440/13 | 1135/10 | 105 | 1 | 21,181%
kernfs_fop_open+d5 | 135168/192 | 107008/152 | 704 | 0 | 20,833%
mb_cache_create+2c | 2880/192 | 2280/152 | 15 | 0 | 20,833%
crypto_create_tfm+32 | 1440/96 | 1140/76 | 15 | 0 | 20,833%
bpf_prog_alloc+9d | 960/192 | 760/152 | 5 | 0 | 20,833%
pidlist_array_load+172 | 768/192 | 608/152 | 4 | 0 | 20,833%
cgroup_mkdir+ca | 46080/1024 | 36540/812 | 45 | 2 | 20,703%
__proc_create+a1 | 17280/192 | 13740/152 | 90 | 0 | 20,486%
__nf_conntrack_alloc+4e | 20800/320 | 16640/256 | 65 | 2 | 20,000%
devcgroup_css_alloc+1b | 1280/256 | 1040/208 | 5 | 0 | 18,750%
ext4_htree_store_dirent+35 | 27584/77 | 22770/64 | 354 | 0 | 17,452%
copy_ipcs+63 | 5120/1024 | 4240/848 | 5 | 4 | 17,188%
__list_lru_init+225 | 10560/96 | 8800/80 | 110 | 16 | 16,667%
device_private_init+1f | 5760/192 | 4800/160 | 30 | 0 | 16,667%
alloc_rt_sched_group+ef | 153600/2048 | 129600/1728 | 75 | 0 | 15,625%
ext4_fill_super+2907 | 1920/128 | 1620/108 | 15 | 0 | 15,625%
__d_alloc+169 | 107648/115 | 91360/97 | 934 | 0 | 15,131%
copy_utsname+85 | 2560/512 | 2200/440 | 5 | 4 | 14,062%
kobject_set_name_vargs+1e | 11904/66 | 10261/57 | 179 | 33 | 13,802%
kasprintf+3a | 11744/91 | 10196/79 | 129 | 33 | 13,181%
prepare_creds+21 | 191808/192 | 167832/168 | 999 | 31 | 12,500%
__seq_open_private+1c | 16896/64 | 14784/56 | 264 | 0 | 12,500%
start_this_handle+2da | 29440/256 | 25760/224 | 115 | 90 | 12,500%
load_elf_binary+1e8 | 3520/32 | 3080/28 | 110 | 0 | 12,500%
alloc_fair_sched_group+11d | 38400/512 | 33600/448 | 75 | 0 | 12,500%
__kthread_create_on_node+5e | 3840/64 | 3360/56 | 60 | 0 | 12,500%
wb_congested_get_create+86 | 2560/64 | 2240/56 | 40 | 0 | 12,500%
bioset_create+2e | 3840/128 | 3360/112 | 30 | 0 | 12,500%
kobj_map+83 | 960/64 | 840/56 | 15 | 0 | 12,500%
ext4_mb_init+54 | 960/64 | 840/56 | 15 | 0 | 12,500%
ext4_mb_init+2c | 480/32 | 420/28 | 15 | 0 | 12,500%
alloc_fdtable+4b | 384/64 | 336/56 | 6 | 0 | 12,500%
unix_bind+1a2 | 640/128 | 560/112 | 5 | 0 | 12,500%
kobject_get_path+56 | 30016/100 | 26522/88 | 299 | 0 | 11,640%
__register_sysctl_table+51 | 20448/151 | 18288/135 | 135 | 0 | 10,563%
create_cache+3e | 45696/384 | 40936/344 | 119 | 33 | 10,417%
kvmalloc_node+3e | 2113536/25161 | 1897896/22594 | 84 | 0 | 10,203%
dev_create+ab | 1440/96 | 1300/86 | 15 | 0 | 9,722%
__anon_vma_prepare+d2 | 290576/88 | 264160/80 | 3302 | 81 | 9,091%
anon_vma_fork+5e | 166672/88 | 151520/80 | 1894 | 187 | 9,091%
kthread+3f | 5760/96 | 5280/88 | 60 | 0 | 8,333%
thin_ctr+6f | 2880/192 | 2640/176 | 15 | 0 | 8,333%
sock_alloc_inode+18 | 492096/704 | 452952/648 | 699 | 11 | 7,955%
jbd2_journal_add_journal_head+67 | 48480/120 | 45248/112 | 404 | 1 | 6,667%
do_execveat_common.isra.31+c0 | 37120/256 | 34800/240 | 145 | 6 | 6,250%
kernfs_iattrs.isra.4+59 | 10112/128 | 9480/120 | 79 | 0 | 6,250%
shmem_fill_super+25 | 3200/128 | 3000/120 | 25 | 0 | 6,250%
bdi_alloc_node+2a | 15360/1024 | 14400/960 | 15 | 0 | 6,250%
alloc_mnt_ns+54 | 1152/128 | 1080/120 | 9 | 4 | 6,250%
alloc_fair_sched_group+29 | 640/128 | 600/120 | 5 | 0 | 6,250%
alloc_fair_sched_group+4e | 640/128 | 600/120 | 5 | 0 | 6,250%
alloc_rt_sched_group+29 | 640/128 | 600/120 | 5 | 0 | 6,250%
alloc_rt_sched_group+4d | 640/128 | 600/120 | 5 | 0 | 6,250%
__register_sysctl_table+434 | 43776/270 | 41157/254 | 162 | 0 | 5,983%
__kernfs_new_node+42 | 839528/136 | 790144/128 | 6173 | 1025 | 5,882%
mqueue_alloc_inode+16 | 4800/960 | 4520/904 | 5 | 4 | 5,833%
bpf_prepare_filter+24b | 40960/8192 | 38920/7784 | 5 | 0 | 4,980%
bpf_convert_filter+57 | 20480/4096 | 19460/3892 | 5 | 0 | 4,980%
bpf_prepare_filter+111 | 10240/2048 | 9730/1946 | 5 | 0 | 4,980%
ep_alloc+3d | 24576/192 | 23552/184 | 128 | 11 | 4,167%
inet_twsk_alloc+3a | 1736/248 | 1680/240 | 7 | 0 | 3,226%
mm_alloc+16 | 296960/2048 | 290000/2000 | 145 | 10 | 2,344%
copy_process.part.40+9e6 | 202752/2048 | 198000/2000 | 99 | 11 | 2,344%
mempool_create_node+f3 | 235920/425 | 230520/415 | 555 | 0 | 2,289%
dax_alloc_inode+16 | 11520/768 | 11280/752 | 15 | 0 | 2,083%
bdev_alloc_inode+16 | 12480/832 | 12240/816 | 15 | 0 | 1,923%
ext4_find_extent+290 | 423552/99 | 415440/97 | 4243 | 0 | 1,915%
radix_tree_node_alloc.constprop.19+78 | 5331336/584 | 5258304/576 | 9129 | 8 | 1,370%
alloc_inode+66 | 528352/608 | 521400/600 | 869 | 20 | 1,316%
proc_alloc_inode+16 | 928200/680 | 917280/672 | 1365 | 45 | 1,176%
copy_process.part.40+95d | 483648/2112 | 478152/2088 | 229 | 8 | 1,136%
shmem_alloc_inode+16 | 326096/712 | 322432/704 | 458 | 6 | 1,124%
copy_process.part.40+10fe | 234496/1024 | 232664/1016 | 229 | 12 | 0,781%
ext4_alloc_inode+17 | 647360/1088 | 642600/1080 | 595 | 1 | 0,735%
__vmalloc_node_range+d3 | 13192/30 | 13112/30 | 426 | 36 | 0,606%
sk_prot_alloc+2f | 544768/1127 | 542080/1122 | 483 | 6 | 0,493%
...
SUMMARY (SLAB allocator)
========================
Total bytes requested: 818.739.691
Total bytes allocated: 821.951.696
Total bytes freed: 763.705.848
Net total bytes allocated: 58.245.848
Total bytes wasted on internal fragmentation: 3.212.005
Internal fragmentation: 0,390778%
Cross CPU allocations: 28.844/10.157.339