Я оцениваю производительность специализированной программы для обработки большого количества событий, связанных с задержкой, с рыночными данными.Одним из ограничений проекта является ограничение задержки реакции на событие до 1 секунды.Поскольку проект находится на C #, нашей самой большой проблемой является случайная задержка сборки мусора.Профиль памяти этого процесса состоит из множества маленьких объектов, и мы запускаем сборку мусора каждые 30 секунд, чтобы минимизировать любые выбросы gc.Это, кажется, работает хорошо за исключением нескольких более длинных коллекций, которые происходят в течение дня.У меня есть несколько вопросов о том, как настроить систему и GC для обработки этой нагрузки.
Что является причиной этих периодических более длительных сборок мусора?
Ниже приведены некоторые журналы мониторинга производства.Используя фоновый поток часов (должен запускаться каждые 10 мс или около того), эта первая таблица записывает максимальный временной интервал для каждого 1-минутного интервала.
[13:39:00.004] updates=5,453, gaps>35ms=2, gaps>100ms=0, gaps>1sec=0, max=91.1ms
[13:40:00.005] updates=5,456, gaps>35ms=2, gaps>100ms=0, gaps>1sec=0, max=94.1ms
[13:41:00.010] updates=5,459, gaps>35ms=2, gaps>100ms=0, gaps>1sec=0, max=86.1ms
[13:42:00.007] updates=5,454, gaps>35ms=2, gaps>100ms=2, gaps>1sec=0, max=121.1ms
[13:43:00.000] updates=5,456, gaps>35ms=2, gaps>100ms=0, gaps>1sec=0, max=92.1ms
[13:44:00.000] updates=5,449, gaps>35ms=2, gaps>100ms=2, gaps>1sec=0, max=150.1ms
[13:45:00.002] updates=5,253, gaps>35ms=2, gaps>100ms=2, gaps>1sec=1, max=2347.3ms
[13:46:00.008] updates=5,452, gaps>35ms=2, gaps>100ms=1, gaps>1sec=0, max=102.1ms
[13:47:00.000] updates=5,454, gaps>35ms=2, gaps>100ms=1, gaps>1sec=0, max=109.1ms
[13:48:00.009] updates=5,455, gaps>35ms=2, gaps>100ms=1, gaps>1sec=0, max=111.1ms
[13:49:00.010] updates=5,454, gaps>35ms=2, gaps>100ms=0, gaps>1sec=0, max=94.1ms
[13:50:00.001] updates=5,455, gaps>35ms=2, gaps>100ms=0, gaps>1sec=0, max=97.1ms
[13:51:00.007] updates=5,461, gaps>35ms=2, gaps>100ms=0, gaps>1sec=0, max=84.1ms
Запись в [13: 45: 00.002] показывает max =2347.3ms.В производстве мы видим несколько таких в день.
В следующей таблице мы записываем прошедшее время от принудительного сбора.
GC.Collect(1, GCCollectionMode.Forced, true);
Этот журнал предоставляет счетчики производительности памяти gc и должен запускатькаждые 30 секунд.
[13:42:01.272] elapsed= 85.76 ms, gcMem= 11,724.8 MB, gcCol= 833.5 MB, gen0=613, gen1=607, gen2=47
[13:42:31.248] elapsed= 60.79 ms, gcMem= 11,736.2 MB, gcCol= 825.0 MB, gen0=614, gen1=608, gen2=47
[13:43:01.312] elapsed= 110.64 ms, gcMem= 11,791.9 MB, gcCol= 1,125.7 MB, gen0=615, gen1=609, gen2=47
[13:43:31.345] elapsed= 140.41 ms, gcMem= 11,836.7 MB, gcCol= 1,158.9 MB, gen0=616, gen1=610, gen2=47
[13:44:01.315] elapsed= 108.77 ms, gcMem= 11,864.8 MB, gcCol= 1,004.8 MB, gen0=617, gen1=611, gen2=47
[13:44:33.565] elapsed=2,345.37 ms, gcMem= 11,895.0 MB, gcCol= 911.7 MB, gen0=618, gen1=612, gen2=47
[13:45:01.315] elapsed= 92.02 ms, gcMem= 11,927.0 MB, gcCol= 938.8 MB, gen0=619, gen1=613, gen2=47
[13:45:31.331] elapsed= 96.98 ms, gcMem= 11,950.3 MB, gcCol= 941.9 MB, gen0=620, gen1=614, gen2=47
[13:46:01.316] elapsed= 82.55 ms, gcMem= 11,965.5 MB, gcCol= 855.4 MB, gen0=621, gen1=615, gen2=47
[13:46:31.339] elapsed= 104.35 ms, gcMem= 12,012.3 MB, gcCol= 1,041.5 MB, gen0=622, gen1=616, gen2=47
[13:47:01.348] elapsed= 109.23 ms, gcMem= 12,040.4 MB, gcCol= 914.6 MB, gen0=623, gen1=617, gen2=47
Здесь мы также регистрируем всплеск в [13: 44: 33.565] истекший = 2 345,37 мс.Обратите внимание, что во время длинного сбора gc сбор gen2 не выполнялся.
Согласно счетчикам производительности, этот всплеск задержки происходит без каких-либо других существенных отличий от объема памяти, поколений gc и / или собранной памяти по сравнению снормальные коллекции с задержкой> 100 мс.
Вот наш журнал производительности системы во время пика задержки.
+----------------+-----------+---------+---------+---------+---------------+---------------+----------------+-----------------+----------------+----------------+----------------+----------------+-----------------+----------------+---------+---------+---------+----------------+----------------+---------------+------------+----------------+-------------+----------------+----------------+-------------+-------------+--------------+---------------+--------+
| Time | TimeDelta | SysFrac | UsrFrac | TotFrac | MinWorkingSet | MaxWorkingSet | WorkingSet | VirtualMemory | PrivateMemory | PagedMemory | NonPagedMemory | PagedSysMemory | PeakVirtualMem | PeakWorkingSet | gcCnt.0 | gcCnt.1 | gcCnt.2 | gcTotalBytes | TotalClrMem | Gen0Mem | Gen1Mem | Gen2Mem | LargeObjMem | ReservedMem | CommittedMem | ContendRate | CurQueueLen | LogicalThrds | PhysicalThrds | GC%CPU |
+----------------+-----------+---------+---------+---------+---------------+---------------+----------------+-----------------+----------------+----------------+----------------+----------------+-----------------+----------------+---------+---------+---------+----------------+----------------+---------------+------------+----------------+-------------+----------------+----------------+-------------+-------------+--------------+---------------+--------+
| [13:31:00.029] | 60 | 1.120 | 14.080 | 15.200 | 204,800 | 1,413,120 | 40,034,070,528 | 122,791,522,304 | 20,763,750,400 | 20,763,750,400 | 218,324 | 44,421,720 | 122,801,758,208 | 40,150,093,824 | 590 | 584 | 47 | 11,756,722,000 | 13,896,680,000 | 8,206,550,000 | 24,448,480 | 13,257,890,000 | 614,338,900 | 97,173,050,000 | 19,206,200,000 | 4 | 20 | 235 | 234 | 0.294 |
| [13:32:00.035] | 60 | 1.119 | 14.129 | 15.248 | 204,800 | 1,413,120 | 40,115,458,048 | 122,792,742,912 | 20,808,089,600 | 20,808,089,600 | 218,452 | 44,421,720 | 122,801,758,208 | 40,150,093,824 | 592 | 586 | 47 | 11,939,798,072 | 13,956,040,000 | 8,203,999,000 | 74,592,030 | 13,267,000,000 | 614,441,500 | 97,173,050,000 | 19,228,650,000 | 3 | 19 | 235 | 234 | 0.480 |
| [13:33:00.025] | 60 | 1.075 | 14.088 | 15.164 | 204,800 | 1,413,120 | 40,222,773,248 | 122,792,742,912 | 20,856,360,960 | 20,856,360,960 | 218,452 | 44,421,720 | 122,801,758,208 | 40,222,773,248 | 594 | 588 | 47 | 12,073,257,352 | 14,016,110,000 | 8,235,154,000 | 65,803,180 | 13,335,750,000 | 614,557,600 | 97,173,050,000 | 19,274,330,000 | 4 | 20 | 235 | 234 | 0.541 |
| [13:34:00.026] | 60 | 1.073 | 14.080 | 15.152 | 204,800 | 1,413,120 | 40,301,883,392 | 122,792,742,912 | 20,919,599,104 | 20,919,599,104 | 218,452 | 44,421,720 | 122,801,758,208 | 40,301,883,392 | 596 | 590 | 47 | 12,078,994,416 | 14,002,090,000 | 8,212,341,000 | 19,899,990 | 13,367,420,000 | 614,764,000 | 97,173,050,000 | 19,298,840,000 | 3 | 18 | 235 | 234 | 0.248 |
| [13:35:00.026] | 60 | 1.092 | 14.116 | 15.208 | 204,800 | 1,413,120 | 40,339,972,096 | 122,792,742,912 | 20,947,439,616 | 20,947,439,616 | 218,452 | 44,421,720 | 122,801,758,208 | 40,339,972,096 | 598 | 592 | 47 | 12,244,802,512 | 14,088,220,000 | 8,224,077,000 | 21,259,580 | 13,412,930,000 | 654,036,800 | 97,173,050,000 | 19,368,640,000 | 4 | 19 | 235 | 234 | 0.278 |
| [13:36:00.034] | 60 | 1.073 | 14.042 | 15.115 | 204,800 | 1,413,120 | 40,386,605,056 | 122,792,742,912 | 20,961,361,920 | 20,961,361,920 | 218,452 | 44,421,720 | 122,801,758,208 | 40,386,605,056 | 600 | 594 | 47 | 12,266,651,104 | 14,121,420,000 | 8,193,102,000 | 11,004,300 | 13,456,180,000 | 654,243,200 | 97,173,050,000 | 19,399,840,000 | 2 | 19 | 235 | 234 | 0.302 |
| [13:37:00.026] | 60 | 1.098 | 14.009 | 15.107 | 204,800 | 1,413,120 | 40,409,452,544 | 122,792,742,912 | 20,981,190,656 | 20,981,190,656 | 218,452 | 44,421,720 | 122,801,758,208 | 40,409,452,544 | 602 | 596 | 47 | 12,278,596,600 | 14,172,080,000 | 8,225,842,000 | 52,736,690 | 13,464,890,000 | 654,450,800 | 97,173,050,000 | 19,402,140,000 | 2 | 20 | 235 | 234 | 0.243 |
| [13:38:00.035] | 60 | 1.117 | 14.048 | 15.165 | 204,800 | 1,413,120 | 40,470,384,640 | 122,791,686,144 | 21,025,886,208 | 21,025,886,208 | 218,196 | 44,421,720 | 122,801,758,208 | 40,470,384,640 | 604 | 598 | 47 | 12,352,609,656 | 14,190,920,000 | 8,190,084,000 | 19,076,740 | 13,517,180,000 | 654,656,000 | 97,173,050,000 | 19,448,470,000 | 3 | 20 | 235 | 234 | 0.248 |
| [13:39:00.014] | 60 | 1.096 | 14.040 | 15.136 | 204,800 | 1,413,120 | 40,504,029,184 | 122,791,686,144 | 21,050,589,184 | 21,050,589,184 | 218,196 | 44,421,720 | 122,801,758,208 | 40,504,029,184 | 606 | 600 | 47 | 12,400,151,400 | 14,227,780,000 | 8,159,343,000 | 25,802,310 | 13,547,000,000 | 654,980,700 | 97,173,050,000 | 19,469,140,000 | 3 | 20 | 235 | 234 | 0.301 |
| [13:40:00.022] | 60 | 1.069 | 14.092 | 15.162 | 204,800 | 1,413,120 | 40,515,948,544 | 122,791,686,144 | 21,050,916,864 | 21,050,916,864 | 218,196 | 44,421,720 | 122,801,758,208 | 40,515,948,544 | 608 | 602 | 47 | 12,398,850,960 | 14,271,310,000 | 8,157,288,000 | 39,735,550 | 13,576,390,000 | 655,186,000 | 97,173,050,000 | 19,489,010,000 | 3 | 19 | 235 | 234 | 0.275 |
| [13:41:00.034] | 60 | 1.098 | 14.104 | 15.201 | 204,800 | 1,413,120 | 40,547,528,704 | 122,791,686,144 | 21,070,147,584 | 21,070,147,584 | 218,196 | 44,421,720 | 122,801,758,208 | 40,547,528,704 | 610 | 604 | 47 | 12,638,109,400 | 14,285,210,000 | 8,121,626,000 | 24,978,900 | 13,604,720,000 | 655,509,600 | 97,173,050,000 | 19,493,130,000 | 3 | 20 | 235 | 234 | 0.224 |
| [13:42:00.022] | 60 | 1.092 | 14.056 | 15.148 | 204,800 | 1,413,120 | 40,620,322,816 | 122,791,686,144 | 21,124,751,360 | 21,124,751,360 | 218,196 | 44,421,720 | 122,801,758,208 | 40,620,322,816 | 612 | 606 | 47 | 12,522,555,576 | 14,344,950,000 | 8,121,881,000 | 28,175,860 | 13,661,060,000 | 655,716,000 | 97,173,050,000 | 19,557,770,000 | 3 | 20 | 235 | 234 | 0.369 |
| [13:43:00.042] | 60 | 1.120 | 14.079 | 15.199 | 204,800 | 1,413,120 | 40,714,616,832 | 122,791,686,144 | 21,218,025,472 | 21,218,025,472 | 218,196 | 44,421,720 | 122,801,758,208 | 40,714,616,832 | 614 | 608 | 47 | 12,868,348,064 | 14,372,630,000 | 8,089,421,000 | 13,960,660 | 13,702,750,000 | 655,920,300 | 97,173,050,000 | 19,574,740,000 | 3 | 19 | 235 | 234 | 0.202 |
| [13:44:00.023] | 60 | 1.085 | 14.163 | 15.249 | 204,800 | 1,413,120 | 40,793,210,880 | 122,791,686,144 | 21,293,572,096 | 21,293,572,096 | 218,196 | 44,421,720 | 122,801,758,208 | 40,793,210,880 | 616 | 610 | 47 | 12,807,955,696 | 14,458,860,000 | 8,179,772,000 | 51,066,620 | 13,751,670,000 | 656,126,700 | 97,173,050,000 | 19,726,070,000 | 5 | 18 | 235 | 234 | 0.466 |
| [13:45:00.020] | 60 | 1.029 | 13.608 | 14.637 | 204,800 | 1,413,120 | 40,834,953,216 | 122,791,686,144 | 21,324,902,400 | 21,324,902,400 | 218,196 | 44,421,720 | 122,801,758,208 | 40,834,953,216 | 618 | 612 | 47 | 12,827,588,224 | 14,497,620,000 | 8,160,795,000 | 25,559,010 | 13,815,730,000 | 656,331,900 | 97,173,050,000 | 19,756,560,000 | 3 | 19 | 235 | 234 | 7.271 |
| [13:46:00.027] | 60 | 1.082 | 14.065 | 15.147 | 204,800 | 1,413,120 | 40,874,991,616 | 122,792,898,560 | 21,356,466,176 | 21,356,466,176 | 218,324 | 44,421,720 | 122,801,758,208 | 40,874,991,616 | 620 | 614 | 47 | 12,773,554,288 | 14,533,490,000 | 8,180,897,000 | 25,771,470 | 13,851,180,000 | 656,538,400 | 97,173,050,000 | 19,785,200,000 | 2 | 19 | 235 | 234 | 0.322 |
| [13:47:00.016] | 60 | 1.114 | 14.072 | 15.186 | 204,800 | 1,413,120 | 40,956,710,912 | 122,792,898,560 | 21,433,704,448 | 21,433,704,448 | 218,324 | 44,421,720 | 122,801,758,208 | 40,956,710,912 | 622 | 616 | 47 | 12,921,358,936 | 14,588,320,000 | 8,204,076,000 | 50,835,140 | 13,880,650,000 | 656,830,000 | 97,173,050,000 | 19,854,300,000 | 2 | 20 | 235 | 234 | 0.346 |
| [13:48:00.019] | 60 | 1.107 | 14.030 | 15.137 | 204,800 | 1,413,120 | 41,003,507,712 | 122,792,898,560 | 21,460,283,392 | 21,460,283,392 | 218,324 | 44,421,720 | 122,801,758,208 | 41,003,507,712 | 624 | 618 | 47 | 13,012,250,176 | 14,597,210,000 | 8,169,155,000 | 18,384,390 | 13,921,790,000 | 657,036,400 | 97,173,050,000 | 19,884,190,000 | 2 | 20 | 235 | 234 | 0.246 |
| [13:49:00.017] | 60 | 1.111 | 14.023 | 15.134 | 204,800 | 1,413,120 | 41,020,157,952 | 122,792,898,560 | 21,467,410,432 | 21,467,410,432 | 218,324 | 44,421,720 | 122,801,758,208 | 41,020,157,952 | 626 | 620 | 47 | 12,983,589,400 | 14,638,320,000 | 8,155,601,000 | 28,285,010 | 13,952,900,000 | 657,137,900 | 97,173,050,000 | 19,893,340,000 | 2 | 19 | 235 | 234 | 0.294 |
| [13:50:00.022] | 60 | 1.120 | 14.033 | 15.152 | 204,800 | 1,413,120 | 41,065,877,504 | 122,792,898,560 | 21,505,044,480 | 21,505,044,480 | 218,324 | 44,421,720 | 122,801,758,208 | 41,065,877,504 | 628 | 622 | 47 | 12,966,301,480 | 14,664,660,000 | 8,160,342,000 | 35,044,860 | 13,972,360,000 | 657,254,000 | 97,173,050,000 | 19,929,060,000 | 2 | 20 | 235 | 234 | 0.316 |
| [13:51:00.037] | 60 | 1.093 | 14.070 | 15.162 | 204,800 | 1,413,120 | 41,109,229,568 | 122,792,898,560 | 21,544,714,240 | 21,544,714,240 | 218,324 | 44,421,720 | 122,801,758,208 | 41,109,229,568 | 630 | 624 | 47 | 13,046,612,984 | 14,703,170,000 | 8,154,384,000 | 37,274,800 | 14,008,440,000 | 657,457,000 | 97,173,050,000 | 19,967,270,000 | 2 | 20 | 235 | 234 | 0.274 |
| [13:52:00.030] | 60 | 1.085 | 14.026 | 15.111 | 204,800 | 1,413,120 | 41,142,714,368 | 122,792,898,560 | 21,560,086,528 | 21,560,086,528 | 218,324 | 44,421,720 | 122,801,758,208 | 41,142,714,368 | 632 | 626 | 47 | 13,085,486,720 | 14,722,970,000 | 8,135,368,000 | 24,030,770 | 14,041,380,000 | 657,560,200 | 97,173,050,000 | 19,987,170,000 | 1 | 19 | 235 | 234 | 0.297 |
| [13:53:00.029] | 60 | 1.077 | 14.041 | 15.118 | 204,800 | 1,413,120 | 41,165,729,792 | 122,792,898,560 | 21,572,124,672 | 21,572,124,672 | 218,324 | 44,421,720 | 122,801,758,208 | 41,165,729,792 | 634 | 628 | 47 | 13,182,848,552 | 14,748,780,000 | 8,116,007,000 | 23,681,860 | 14,067,340,000 | 657,764,400 | 97,173,050,000 | 19,999,100,000 | 1 | 20 | 235 | 234 | 0.227 |
| [13:54:00.030] | 60 | 1.087 | 14.097 | 15.183 | 204,800 | 1,413,120 | 41,212,456,960 | 122,792,898,560 | 21,617,262,592 | 21,617,262,592 | 218,324 | 44,421,720 | 122,801,758,208 | 41,212,456,960 | 636 | 630 | 47 | 13,357,287,176 | 14,781,100,000 | 8,099,477,000 | 20,468,650 | 14,102,660,000 | 657,969,700 | 97,173,050,000 | 20,043,010,000 | 2 | 20 | 235 | 234 | 0.269 |
| [13:55:00.033] | 60 | 1.090 | 14.024 | 15.114 | 204,800 | 1,413,120 | 41,235,304,448 | 122,792,898,560 | 21,632,643,072 | 21,632,643,072 | 218,324 | 44,421,720 | 122,801,758,208 | 41,235,304,448 | 638 | 632 | 47 | 13,274,559,640 | 14,850,320,000 | 8,144,056,000 | 26,831,810 | 14,165,200,000 | 658,292,200 | 97,173,050,000 | 20,059,910,000 | 2 | 20 | 235 | 234 | 0.336 |
| [13:56:00.025] | 60 | 1.090 | 14.003 | 15.093 | 204,800 | 1,413,120 | 41,277,251,584 | 122,792,898,560 | 21,665,263,616 | 21,665,263,616 | 218,324 | 44,421,720 | 122,801,758,208 | 41,277,251,584 | 640 | 634 | 47 | 13,343,955,968 | 14,872,170,000 | 8,106,031,000 | 20,415,720 | 14,193,260,000 | 658,496,300 | 97,173,050,000 | 20,063,000,000 | 1 | 19 | 235 | 234 | 0.225 |
| [13:57:00.023] | 60 | 1.127 | 14.558 | 15.685 | 204,800 | 1,413,120 | 41,018,687,488 | 122,792,898,560 | 21,709,570,048 | 21,709,570,048 | 218,324 | 44,421,720 | 122,801,758,208 | 41,322,450,944 | 642 | 636 | 48 | 11,972,380,824 | 14,922,300,000 | 8,184,630,000 | 95,170,150 | 14,193,260,000 | 633,870,800 | 97,173,050,000 | 20,096,390,000 | 6 | 20 | 232 | 231 | 0.592 |
| [13:58:00.032] | 60 | 1.074 | 14.149 | 15.223 | 204,800 | 1,413,120 | 41,062,780,928 | 122,809,479,168 | 21,752,737,792 | 21,752,737,792 | 218,452 | 44,421,720 | 122,809,479,168 | 41,322,450,944 | 644 | 638 | 48 | 12,016,674,992 | 14,881,730,000 | 8,209,373,000 | 39,635,030 | 14,208,120,000 | 633,974,100 | 97,173,050,000 | 20,177,100,000 | 3 | 19 | 232 | 231 | 0.358 |
| [13:59:00.031] | 60 | 1.019 | 13.970 | 14.990 | 204,800 | 1,413,120 | 41,064,423,424 | 122,809,479,168 | 21,752,930,304 | 21,752,930,304 | 218,452 | 44,421,720 | 122,817,601,536 | 41,322,450,944 | 646 | 640 | 48 | 11,809,759,384 | 14,862,300,000 | 8,142,180,000 | 19,593,650 | 14,208,530,000 | 634,177,200 | 97,173,050,000 | 20,179,600,000 | 2 | 20 | 232 | 231 | 0.196 |
| [14:00:00.036] | 60 | 1.056 | 13.984 | 15.041 | 204,800 | 1,413,120 | 41,067,610,112 | 122,809,544,704 | 21,755,564,032 | 21,755,564,032 | 218,580 | 44,421,720 | 122,817,601,536 | 41,322,450,944 | 648 | 642 | 48 | 11,906,742,832 | 14,855,660,000 | 8,086,772,000 | 12,748,120 | 14,208,530,000 | 634,381,300 | 97,173,050,000 | 20,182,000,000 | 2 | 20 | 232 | 231 | 0.179 |
+----------------+-----------+---------+---------+---------+---------------+---------------+----------------+-----------------+----------------+----------------+----------------+----------------+-----------------+----------------+---------+---------+---------+----------------+----------------+---------------+------------+----------------+-------------+----------------+----------------+-------------+-------------+--------------+---------------+--------+
Может ли кто-нибудь объяснить дополнительную нагрузку, которая вызывает эти случайные более длинные коллекции?
Также:
1) Существуют ли какие-либо дополнительные показатели, которые мы можем отслеживать, чтобы понять эти пики задержки?
2) Существуют ли другие стратегии, которые мы должны использовать дляуменьшить продолжительность этих пиков задержки?
Заранее спасибо.