Мы используем Greenplum 4.3.14.0 в одном из наших проектов, и недавно произошел сбой.Я нашел информацию, похожую на наш случай, но не совсем:
https://discuss.pivotal.io/hc/en-us/articles/225773448-Greenplum-Postmaster-Resets-with-Error-PID-terminated-by-signal-6-Aborted-
Может быть, вы расскажете нам что-нибудь о причине этого сбоя.
Данные из журнала (эточасть журнала повторяется много раз):
2018-05-26 18:15:01.607440 MSK,,,p2561,th1921165184,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
addr2line: /usr/local/greenplum-db/./lib/libz.so.1: no version information available (required by /lib64/libbfd-2.25.1-32.base.el7_4.1.so)",,,,,,,,"SysLoggerMain","syslogger.c",550,
2018-05-26 18:15:01.673235 MSK,"gpadmin","prod-ofd",p6611,th1921165184,"10.8.196.68","53568",2018-05-24 10:20:03 MSK,0,con1520839,cmd380300,seg-1,,,,sx1,"FATAL","XX000","reached limit of 4294967295 global transactions per start (cdbtm.c:3023)",,,,,,,0,,"cdbtm.c",3023,"Stack trace:
1 0xb05a4e postgres errstart (elog.c:502)
2 0xcc64a2 postgres createDtx (cdbtm.c:3022)
3 0x5433df postgres StartTransaction (xact.c:3043)
4 0x543a75 postgres StartTransactionCommand (xact.c:4354)
5 0x99978e postgres PostgresMain (postgres.c:3155)
6 0x8f422e postgres <symbol not found> (postmaster.c:6673)
7 0x8f6ed0 postgres PostmasterMain (postmaster.c:7604)
8 0x7f81bf postgres main (main.c:206)
9 0x7fd26e111c05 libc.so.6 __libc_start_main + 0xf5
10 0x4bdf59 postgres <symbol not found> + 0x4bdf59
"
2018-05-26 18:15:01.673664 MSK,,,p2561,th1921165184,,,,0,,,seg-1,,,,,"LOG","00000","Retrying emails now...",,,,,,,,"send_alert_via_email","sendalert.c",556,
2018-05-26 18:15:01.674872 MSK,,,p2561,th1921165184,,,,0,,,seg-1,,,,,"LOG","00000","could not connect to SMTP server","couldn't connect to host",,,,,,,"send_alert_via_email","sendalert.c",720,
2018-05-26 18:15:01.678701 MSK,,,p2561,th1921165184,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
addr2line: /usr/local/greenplum-db/./lib/libz.so.1: no version information available (required by /lib64/libbfd-2.25.1-32.base.el7_4.1.so)",,,,,,,,"SysLoggerMain","syslogger.c",550,
2018-05-26 18:15:01.749374 MSK,"gpadmin","prod-ofd",p6611,th1921165184,"10.8.196.68","53568",2018-05-24 10:20:03 MSK,0,con1520839,cmd380300,seg-1,,,,sx1,"PANIC","XX000","Waiting on lock already held! (lwlock.c:557)",,,,,,,0,,"lwlock.c",557,"Stack trace:
1 0xb01852 postgres <symbol not found> (elog.c:502)
2 0xb03868 postgres elog_finish (elog.c:1446)
3 0x980cd6 postgres LWLockAcquire (lwlock.c:557)
4 0x95f967 postgres ProcArrayRemove (procarray.c:181)
5 0x95b9cd postgres proc_exit (ipc.c:254)
6 0xb01cff postgres errfinish (elog.c:674)
7 0xcc656a postgres createDtx (cdbtm.c:3022)
8 0x5433df postgres StartTransaction (xact.c:3043)
9 0x543a75 postgres StartTransactionCommand (xact.c:4354)
10 0x99978e postgres PostgresMain (postgres.c:3155)
11 0x8f422e postgres <symbol not found> (postmaster.c:6673)
12 0x8f6ed0 postgres PostmasterMain (postmaster.c:7604)
13 0x7f81bf postgres main (main.c:206)
14 0x7fd26e111c05 libc.so.6 __libc_start_main + 0xf5
15 0x4bdf59 postgres <symbol not found> + 0x4bdf59
"
2018-05-26 18:15:02.752408 MSK,,,p2561,th1921165184,,,,0,,,seg-1,,,,,"LOG","00000","could not connect to SMTP server","couldn't connect to host",,,,,,,"send_alert_via_email","sendalert.c",720,
2018-05-26 18:15:02.753379 MSK,,,p2560,th1921165184,,,,0,,,seg-1,,,,,"LOG","00000","server process (PID 6611) was terminated by signal 6: Aborted",,,,,,,0,,"postmaster.c",5894,
2018-05-26 18:15:02.753414 MSK,,,p2560,th1921165184,,,,0,,,seg-1,,,,,"LOG","00000","terminating any other active server processes",,,,,,,0,,"postmaster.c",5581,
2018-05-26 18:15:02.753429 MSK,,,p2560,th1921165184,,,,0,,,seg-1,,,,,"LOG","00000","seqserver process (PID 2572) exited with exit code 2",,,,,,,0,,"postmaster.c",5872,
2018-05-26 18:15:02.753459 MSK,"gpadmin","prod-ofd",p485,th1921165184,"10.8.196.68","51043",2018-05-26 18:15:01 MSK,0,,,seg-1,,,,,"FATAL","57P03","the database system is in recovery mode",,,,,,,0,,"postmaster.c",2966,
Then many various messages that db try to recover.
Then
2018-05-26 18:15:04.017036 MSK,,,p2560,th1921165184,,,,0,,,seg-1,,,,,"LOG","00000","PostgreSQL 8.2.15 (Greenplum Database 4.3.14.0 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 25 2017 07:17:30",,,,,,,0,,"postmaster.c",4618,
2018-05-26 18:15:04.017070 MSK,,,p2560,th1921165184,,,,0,,,seg-1,,,,,"LOG","00000","database system is ready to accept connections","PostgreSQL 8.2.15 (Greenplum Database 4.3.14.0 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 25 2017 07:17:30",,,,,,0,,"postmaster.c",4624,
And again
2018-05-26 18:15:05.078777 MSK,,,p2561,th1921165184,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
addr2line: /usr/local/greenplum-db/./lib/libz.so.1: no version information available (required by /lib64/libbfd-2.25.1-32.base.el7_4.1.so)",,,,,,,,"SysLoggerMain","syslogger.c",550,
2018-05-26 18:15:05.147311 MSK,"gpadmin","prod-ofd",p1914,th1921165184,"10.8.196.92","54613",2018-05-26 18:15:04 MSK,0,con1564021,,seg-1,,,,sx1,"FATAL","XX000","reached limit of 4294967295 global transactions per start (cdbtm.c:3023)",,,,,,"SHOW TRANSACTION ISOLATION LEVEL",0,,"cdbtm.c",3023,"Stack trace:
1 0xb05a4e postgres errstart (elog.c:502)
2 0xcc64a2 postgres createDtx (cdbtm.c:3022)
3 0x5433df postgres StartTransaction (xact.c:3043)
4 0x543a75 postgres StartTransactionCommand (xact.c:4354)
5 0x998d15 postgres PostgresMain (postgres.c:3155)
6 0x8f422e postgres <symbol not found> (postmaster.c:6673)
7 0x8f6ed0 postgres PostmasterMain (postmaster.c:7604)
8 0x7f81bf postgres main (main.c:206)
9 0x7fd26e111c05 libc.so.6 __libc_start_main + 0xf5
10 0x4bdf59 postgres <symbol not found> + 0x4bdf59
"
Бесконечный цикл.
После уничтожения всех процессов БД командой
gpadmin @ gp-master $ gpssh -f / usr/ local / greenplum-db / hostfile_allhosts -e "pkill -9 postgres"
и перезапуск - db снова начал работать правильно.
Заранее спасибо.