Мусорщик: сбой выделения - JavaScript куча из памяти - PullRequest
Новая
       95

Мусорщик: сбой выделения - JavaScript куча из памяти

1 голос
/ 19 июня 2020

Вот сообщение об ошибке: 

<--- Last few GCs --->
[2383:0x7efe08001450]     6100 ms: Scavenge 30.3 (39.5) -> 30.5 (42.7) MB, 73.5 / 0.0 ms  (average mu = 1.000, curr
ent mu = 1.000) allocation failure 
[2383:0x7efe08001450]     8464 ms: Scavenge 35.1 (44.5) -> 35.3 (44.8) MB, 2336.2 / 0.0 ms  (average mu = 1.000, cu
rrent mu = 1.000) allocation failure 

[2383:0x7efe08001450]    32349 ms: Scavenge 36.1 (44.8) -> 36.0 (45.8) MB, 23879.5 / 0.2 ms  (average mu = 1.000, c
urrent mu = 1.000) allocation failure 
<--- JS stacktrace --->
FATAL ERROR: Scavenger: semi-space copy Allocation failed - JavaScript heap out of memory
 1: 0xa2b020 node::Abort() [node]
 2: 0x97a467 node::FatalError(char const*, char const*) [node]
 3: 0xb9e0ee v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb9e467 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd3e875  [node]
 6: 0xdab25e  [node]
 7: 0xdae060 v8::internal::Scavenger::Process(v8::internal::OneshotBarrier*) [node]
 8: 0xdaeb29 v8::internal::ScavengingTask::ProcessItems() [node]
 9: 0xdaecd1 v8::internal::ScavengingTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [node]
10: 0xd590f9 v8::internal::ItemParallelJob::Run() [node]
11: 0xdaf9c5 v8::internal::ScavengerCollector::CollectGarbage() [node]
12: 0xd37aba v8::internal::Heap::Scavenge() [node]
13: 0xd4d37b v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [nod
e]
14: 0xd4de65 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReaso
n, v8::GCCallbackFlags) [node]
15: 0xd5082c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal
::AllocationOrigin, v8::internal::AllocationAlignment) [node]
16: 0xd20e82 v8::internal::Factory::AllocateRawWithAllocationSite(v8::internal::Handle<v8::internal::Map>, v8::inte
rnal::AllocationType, v8::internal::Handle<v8::internal::AllocationSite>) [node]
17: 0xd20f38 v8::internal::Factory::NewJSObjectFromMap(v8::internal::Handle<v8::internal::Map>, v8::internal::Alloc
ationType, v8::internal::Handle<v8::internal::AllocationSite>) [node]
18: 0xd2130e v8::internal::Factory::NewJSArrayWithUnverifiedElements(v8::internal::Handle<v8::internal::FixedArrayB
ase>, v8::internal::ElementsKind, int, v8::internal::AllocationType) [node]
19: 0xd24a32 v8::internal::Factory::NewJSArray(v8::internal::ElementsKind, int, int, v8::internal::ArrayStorageAllo
cationMode, v8::internal::AllocationType) [node]
20: 0xf6edbb v8::internal::ValueDeserializer::ReadDenseJSArray() [node]
21: 0xf6f1cd v8::internal::ValueDeserializer::ReadObjectInternal() [node]
22: 0xf6f338 v8::internal::ValueDeserializer::ReadObject() [node]
23: 0xf6d80d v8::internal::ValueDeserializer::ReadJSObjectProperties(v8::internal::Handle<v8::internal::JSObject>, 
v8::internal::SerializationTag, bool) [node]
24: 0xf6ec06 v8::internal::ValueDeserializer::ReadSparseJSArray() [node]
25: 0xf6f118 v8::internal::ValueDeserializer::ReadObjectInternal() [node]
26: 0xf6f338 v8::internal::ValueDeserializer::ReadObject() [node]
27: 0xf6e186 v8::internal::ValueDeserializer::ReadJSObjectProperties(v8::internal::Handle<v8::internal::JSObject>, 
v8::internal::SerializationTag, bool) [node]
28: 0xf6e959 v8::internal::ValueDeserializer::ReadJSObject() [node]
29: 0xf6f140 v8::internal::ValueDeserializer::ReadObjectInternal() [node]
30: 0xf6f338 v8::internal::ValueDeserializer::ReadObject() [node]
31: 0xf6d80d v8::internal::ValueDeserializer::ReadJSObjectProperties(v8::internal::Handle<v8::internal::JSObject>, 
v8::internal::SerializationTag, bool) [node]
32: 0xf6e959 v8::internal::ValueDeserializer::ReadJSObject() [node]
33: 0xf6f140 v8::internal::ValueDeserializer::ReadObjectInternal() [node]
34: 0xf6f338 v8::internal::ValueDeserializer::ReadObject() [node]
35: 0xba6e15 v8::ValueDeserializer::ReadValue(v8::Local<v8::Context>) [node]
36: 0xa720ff node::worker::Message::Deserialize(node::Environment*, v8::Local<v8::Context>) [node]
37: 0xa72d86 node::worker::MessagePort::ReceiveMessage(v8::Local<v8::Context>, bool) [node]
38: 0xa72ee4 node::worker::MessagePort::OnMessage() [node]
39: 0x13247e9  [node]
40: 0x1336c50  [node]
41: 0x13250f8 uv_run [node]
42: 0xae3533 node::worker::Worker::Run() [node]
43: 0xae3fc8  [node]
44: 0x7eff53bdafa3  [/lib/x86_64-linux-gnu/libpthread.so.0]
45: 0x7eff53b0b4cf clone [/lib/x86_64-linux-gnu/libc.so.6]

сервер узла на GCP - Linux x64, 96 ядер, 135 ГБ ОЗУ. Я обрабатываю массив из 100 000 000 элементов, разделенных на 95 ядер с созданием рабочих.

Это только один из типов ошибок, которые я получаю, например, я получил NewArray Allocation failed. Я играл с количеством рабочих, --max-old-space-size, --max-semi-space-size, но никто не работает. Самое интересное, что моя ma c с 6 ядрами, 16 ГБ оперативной памяти (только 10 отдано узлу) работает хорошо, просто медленно, с увеличением аппаратных ресурсов я ожидаю более быстрых результатов.

Это Что я делаю не так, когда впервые пытаюсь обработать такой объем данных и многопоточность с помощью узла?

...