У меня есть кластер mongodb, который содержит наши базы данных development и production.По-видимому, прошлой ночью, когда обрабатывались данные, один из серверов набора репликации сегментов получил от службы mongod сообщение об ошибке.После перезапуска службы я неоднократно получаю ошибки, связанные с ошибками пакетных операций из-за ошибок DuplicateKey: E11000.
Предыстория:
Я создал механизм ETL, который импортирует данные из нескольких источников и сохраняет их в реализации базы данных графа, используя mongodb.Интеграторы переводят данные в известные узлы и сохраняют их с помощью массовых операций.
Я также иногда замечаю, что с некоторыми правилами для узла что-то не так.Поэтому мне нужно запустить инструмент для обработки данных после того, как они существуют в базе данных.
Я выполнял эти две операции на development и production.
Кластер mongodbдовольно просто прямо сейчас.Два сервера запросов общаются с 1 сервером конфигурации (набор из 4 виртуальных машин), общаются с 1 сервером сегмента (набор из 4 виртуальных машин).
Когда я перезагружаю одну службу mongod, служба возвращается и довольно скоро после этого выдает другую трассировку стека.
Что мне действительно нужно, так это просто, чтобы одна виртуальная машина прекратила генерировать трассировку стека и продолжила работу, даже если операции резервного копирования явно не выполняются.
Можно ли указать серверу набора реплик, чтобы он прекратил свои операции в очереди в журнале операций?
2019-03-02T08:37:47.373-0500 F REPL [repl writer worker 2] writer worker caught exception: DuplicateKey: E11000 duplicate key error collection: securityportal-development.core.nodes index: _id_ dup key: { : ObjectId('5c7a14795fe5e41db50036e3') } on: { op: "u", ns: "securityportal-development.core.nodes", ui: UUID("e2d48024-2c6d-4e46-a162-670959b8ae1e"), o: { $v: 1, $set: {...} }, ts: Timestamp(1551505421, 10), t: 8, h: -4266186036746248747, v: 2, wall: new Date(1551505421050) }
2019-03-02T08:37:47.472-0500 F REPL [repl writer worker 14] writer worker caught exception: DuplicateKey: E11000 duplicate key error collection: securityportal-development.core.nodes index: _id_ dup key: { : ObjectId('5c7a147a5fe5e41db50036f3') } on: { op: "u", ns: "securityportal-development.core.nodes", ui: UUID("e2d48024-2c6d-4e46-a162-670959b8ae1e"), o: { $v: 1, $set: {... }, o2: { _id: ObjectId('5c7a147a5fe5e41db50036f3') }, ts: Timestamp(1551505421, 78), t: 8, h: -6279039002132381396, v: 2, wall: new Date(1551505421585) }
2019-03-02T08:37:47.796-0500 F REPL [initandlisten] Failed to apply batch of operations. Number of operations in batch: 3422. First operation: { op: "u", ns: "config.system.sessions", ui: UUID("31fff722-6a5c-4947-b6f0-a83a98ef5a32"), o: { $v: 1, $set: { lastUse: new Date(1551505383010) } }, o2: { _id: { id: UUID("459b61e6-6744-41d8-a3ff-c2dfa3d8b71e"), uid: BinData(0, 56C1974FE3C40DBCDBFF60F66420E9AA79BA0B37F1138022B88B41E9BDF29678) } }, ts: Timestamp(1551505383, 24), t: 8, h: -8280419510924350842, v: 2, wall: new Date(1551505383004) }. Last operation: { op: "u", ns: "securityportal-development.core.nodes", ui: UUID("e2d48024-2c6d-4e46-a162-670959b8ae1e"), o: { $v: 1, $set: { ... }, o2: { _id: ObjectId('5c66c9dafa24cd52d2276e6d') }, ts: Timestamp(1551505423, 13), t: 8, h: 2759588341675508350, v: 2, wall: new Date(1551505423101) }. Oplog application failed in writer thread 1: DuplicateKey: E11000 duplicate key error collection: securityportal-development.core.nodes index: _id_ dup key: { : ObjectId('5c7a14795fe5e41db50036ca') }
2019-03-02T08:37:47.801-0500 F REPL [initandlisten] Caught exception during replication recovery: DuplicateKey: E11000 duplicate key error collection: securityportal-development.core.nodes index: _id_ dup key: { : ObjectId('5c7a14795fe5e41db50036ca') }
2019-03-02T08:37:47.801-0500 F - [initandlisten] terminate() called. An exception is active; attempting to gather more information
2019-03-02T08:37:47.811-0500 F - [initandlisten] DBException::toString(): DuplicateKey: E11000 duplicate key error collection: securityportal-development.core.nodes index: _id_ dup key: { : ObjectId('5c7a14795fe5e41db50036ca') }
Actual exception type: mongo::error_details::ExceptionForImpl<(mongo::ErrorCodes::Error)11000, mongo::AssertionException>
0x563bd0976761 0x563bd0976145 0x563bd0a6ac66 0x563bd0a6acb1 0x563bcf279e3f 0x563bcf2b401c 0x563bcf2b48e7 0x563bcef15ef8 0x563bceffd76b 0x563bcef92bc9 0x7f129f0b7b97 0x563bceffb8da
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"563BCE57A000","o":"23FC761","s":"_ZN5mongo15printStackTraceERSo"},{"b":"563BCE57A000","o":"23FC145"},{"b":"563BCE57A000","o":"24F0C66","s":"_ZN10__cxxabiv111__terminateEPFvvE"},{"b":"563BCE57A000","o":"24F0CB1"},{"b":"563BCE57A000","o":"CFFE3F","s":"_ZN5mongo4repl23ReplicationRecoveryImpl16recoverFromOplogEPNS_16OperationContextEN5boost8optionalINS_9TimestampEEE"},{"b":"563BCE57A000","o":"D3A01C","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl21_startLoadLocalConfigEPNS_16OperationContextE"},{"b":"563BCE57A000","o":"D3A8E7","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl7startupEPNS_16OperationContextE"},{"b":"563BCE57A000","o":"99BEF8"},{"b":"563BCE57A000","o":"A8376B","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"563BCE57A000","o":"A18BC9","s":"main"},{"b":"7F129F096000","o":"21B97","s":"__libc_start_main"},{"b":"563BCE57A000","o":"A818DA","s":"_start"}],"processInfo":{ "mongodbVersion" : "4.0.6", "gitVersion" : "caa42a1f75a56c7643d0b68d3880444375ec42e3", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-45-generic", "version" : "#48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "563BCE57A000", "elfType" : 3, "buildId" : "6C174EAF077366438A078C99465007456D76C8E4" }, { "b" : "7FFF29307000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "025AD7BDF9DF8C7251192246537C451A988B98AE" }, { "b" : "7F12A0965000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "CFC1F864E5F85EDA8E37B3BB238EC438D6D524A5" }, { "b" : "7F12A074A000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "390E9CC4C215314B6D8ADE6D6E28F8518418039C" }, { "b" : "7F12A02D2000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "A409BCD92DCDD3298E92C359876AC800FDD499AC" }, { "b" : "7F12A0068000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "9787FF1FFFE8D82268D89F6BFA3AF07B3E9D140F" }, { "b" : "7F129FE64000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25AD56E902E23B490A9CCDB08A9744D89CB95BCC" }, { "b" : "7F129FC5C000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "9826FBDF57ED7D6965131074CB3C08B1009C1CD8" }, { "b" : "7F129F8BE000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "A33761AB8FB485311B3C85BF4253099D7CABE653" }, { "b" : "7F129F6A6000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "92E0BE1929D28508CF9C6D5754C7EB48C12255B3" }, { "b" : "7F129F487000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "28C6AADE70B2D40D1F0F3D0A1A0CAD1AB816448F" }, { "b" : "7F129F096000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B417C0BA7CC5CF06D1D1BED6652CEDB9253C60D0" }, { "b" : "7F12A0BE4000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "64DF1B961228382FE18684249ED800AB1DCEAAD4" }, { "b" : "7F129EE71000", "path" : "/usr/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "4F00E5207693FDC249DA42EC6472ACA6A7B929AE" }, { "b" : "7F129EC54000", "path" : "/usr/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "BA5BF9A5C44F48C647E9D8270A5421AE81CCAD61" }, { "b" : "7F129EA38000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "69465D8AA6B19086ABF2455A703F9168BF82A69F" }, { "b" : "7F129E82A000", "path" : "/usr/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "CDAF1F1946846941F9D06414EC8C812D131A168E" }, { "b" : "7F129E5DF000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "00F419F64B0E70D8C5EEF7050369AA40B2A6E090" }, { "b" : "7F129E38D000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "9AFD2677E6CD2A67823EA74FDD8448F683028484" }, { "b" : "7F129E17F000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "445EBBF9B2F0FC928017DF247244256F56D431E1" }, { "b" : "7F129DF62000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "EF3E006DFE3132A41D4D4DC0E407D6EA658E11C4" }, { "b" : "7F129DBE4000", "path" : "/usr/lib/x86_64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "62A44D226F3A74CF720305A52D3E876E214C248A" }, { "b" : "7F129D87F000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "F67AAC89381F6ACFE9979B24D4002668C4F70ABE" }, { "b" : "7F129D64B000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "842BDF0B0EAAB82E19F1EABFC38769F4040FBE31" }, { "b" : "7F129D415000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "C20D4B3BA13FCDCC3BF6857689BA9FC70BE3F6A5" }, { "b" : "7F129D194000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "D40EA9B5EC5BC46799E4A412319617BD38BE9341" }, { "b" : "7F129CEBE000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "69FBCF425EE6DF03DE93B82FBC2FC33790E68A96" }, { "b" : "7F129CC8C000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "F400D5D643A7F9696DF0E6148FA99BEE6C1BDDF7" }, { "b" : "7F129CA88000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "C0CB7E35A4566A443F99DFBC1A54D3A0677C8A10" }, { "b" : "7F129C87D000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "D78D71E8E016A534281B25B97CD7E5E9DB5FE00A" }, { "b" : "7F129C662000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "ABB7E3F40302E6509DAD1F91DFB1F04B6A5FD072" }, { "b" : "7F129C421000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "A1A98DB481968073636BBAECB561A3EA8ED198AE" }, { "b" : "7F129C0F2000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "8DBD451EA5651283905E16FA7DFA9908688893A3" }, { "b" : "7F129BEDF000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "6036B89A3BB671B32E01464C0C82BFA016186352" }, { "b" : "7F129BCDB000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "F463E107B099910463BC32E837C73D341A52C27B" }, { "b" : "7F129BAD2000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "C2376C5B831991591F1A67B976758185F86896D8" }, { "b" : "7F129B845000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "69BDEE5FA0FEEDF317308BE850F78761861D520A" }, { "b" : "7F129B5A3000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "315D74995AAA32DE4D15BA25F335066988B1B230" }, { "b" : "7F129B36D000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "6673972A1C24A89EBAFBAE696188A4CB26C6DDEB" }, { "b" : "7F129B157000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "430827C33259C12248CF44B91A9A9821114376F5" }, { "b" : "7F129AF4F000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "3555B5F599C9787DFDDBF9E8DF6F706B9044D985" }, { "b" : "7F129AD26000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "93A0931B1C2818F0EA224CE6FE5E31E84A9B55BB" }, { "b" : "7F129AB17000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "669D4CCE42FA4382796EFFCF0C16F459F4382C4C" }, { "b" : "7F129A8CD000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "4B80C543356EE0AF9039EFE7C9EA1CC1F74C426A" }, { "b" : "7F129A5C4000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "9FD7B69263297314C83BB205234E182BD4D8B86A" }, { "b" : "7F129A38C000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "810686AF0D5FD350A4FB1CC4B5AFF44A05C102CB" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x563bd0976761]
mongod(+0x23FC145) [0x563bd0976145]
mongod(_ZN10__cxxabiv111__terminateEPFvvE+0x6) [0x563bd0a6ac66]
mongod(+0x24F0CB1) [0x563bd0a6acb1]
mongod(_ZN5mongo4repl23ReplicationRecoveryImpl16recoverFromOplogEPNS_16OperationContextEN5boost8optionalINS_9TimestampEEE+0xC7F) [0x563bcf279e3f]
mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl21_startLoadLocalConfigEPNS_16OperationContextE+0x54C) [0x563bcf2b401c]
mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl7startupEPNS_16OperationContextE+0x237) [0x563bcf2b48e7]
mongod(+0x99BEF8) [0x563bcef15ef8]
mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x10DB) [0x563bceffd76b]
mongod(main+0x9) [0x563bcef92bc9]
libc.so.6(__libc_start_main+0xE7) [0x7f129f0b7b97]
mongod(_start+0x2A) [0x563bceffb8da]
----- END BACKTRACE -----