Я пытаюсь создать безопасный кворум между кластером Zookeeper 3.5.7 из 3 узлов. Я следовал инструкциям из официального руководства здесь .
Однако я все еще не могу установить sh кворум по TLS. Вот журнал при запуске сервера
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/conf/zoo.cfg
2020-04-27 14:22:55,973 [myid:] - INFO [main:QuorumPeerConfig@135] - Reading configuration from: /opt/zookeeper/conf/zoo.cfg
2020-04-27 14:22:55,979 [myid:] - INFO [main:QuorumPeerConfig@377] - clientPort is not set
2020-04-27 14:22:55,988 [myid:] - INFO [main:QuorumPeerConfig@398] - secureClientPortAddress is 192.168.152.61:2182
2020-04-27 14:22:55,994 [myid:] - INFO [main:X509Util@79] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2020-04-27 14:22:56,009 [myid:1] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2020-04-27 14:22:56,009 [myid:1] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2020-04-27 14:22:56,009 [myid:1] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2020-04-27 14:22:56,010 [myid:1] - INFO [main:ManagedUtil@46] - Log4j found with jmx enabled.
2020-04-27 14:22:56,025 [myid:1] - INFO [main:QuorumPeerMain@141] - Starting quorum peer
2020-04-27 14:22:56,114 [myid:1] - INFO [main:NettyServerCnxnFactory@387] - zookeeper.client.portUnification=false
2020-04-27 14:22:56,179 [myid:1] - INFO [main:ServerCnxnFactory@135] - Using org.apache.zookeeper.server.NettyServerCnxnFactory as server connection factory
2020-04-27 14:22:56,194 [myid:1] - INFO [main:FileTxnSnapLog@115] - zookeeper.snapshot.trust.empty : false
2020-04-27 14:22:56,200 [myid:1] - INFO [main:QuorumPeer@1470] - Local sessions disabled
2020-04-27 14:22:56,200 [myid:1] - INFO [main:QuorumPeer@1481] - Local session upgrading disabled
2020-04-27 14:22:56,200 [myid:1] - INFO [main:QuorumPeer@1448] - tickTime set to 2000
2020-04-27 14:22:56,200 [myid:1] - INFO [main:QuorumPeer@1492] - minSessionTimeout set to 4000
2020-04-27 14:22:56,202 [myid:1] - INFO [main:QuorumPeer@1503] - maxSessionTimeout set to 40000
2020-04-27 14:22:56,202 [myid:1] - INFO [main:QuorumPeer@1518] - initLimit set to 10
2020-04-27 14:22:56,224 [myid:1] - INFO [main:ZKDatabase@117] - zookeeper.snapshotSizeFactor = 0.33
2020-04-27 14:22:56,227 [myid:1] - INFO [main:QuorumPeer@1761] - Using TLS encrypted quorum communication
2020-04-27 14:22:56,227 [myid:1] - INFO [main:QuorumPeer@1769] - Port unification disabled
2020-04-27 14:22:56,228 [myid:1] - INFO [main:QuorumPeer@2136] - QuorumPeer communication is not secured! (SASL auth disabled)
2020-04-27 14:22:56,228 [myid:1] - INFO [main:QuorumPeer@2165] - quorum.cnxn.threads.size set to 20
2020-04-27 14:22:56,233 [myid:1] - INFO [main:FileSnap@83] - Reading snapshot /etc/zookeeper/version-2/snapshot.1a00000000
2020-04-27 14:22:56,329 [myid:1] - INFO [main:NettyServerCnxnFactory@590] - binding to port test-server-01/192.168.152.61:2182
2020-04-27 14:22:56,430 [myid:1] - INFO [main:NettyServerCnxnFactory@595] - bound to port 2182
2020-04-27 14:22:56,436 [myid:1] - INFO [main:QuorumCnxManager$Listener@867] - Election port bind maximum retries is 3
2020-04-27 14:22:56,438 [myid:1] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@900] - Creating TLS-only quorum server socket
2020-04-27 14:22:56,439 [myid:1] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@917] - My election bind port: test-server-01/192.168.152.61:3888
2020-04-27 14:22:56,449 [myid:1] - INFO [QuorumPeer[myid=1](plain=disabled)(secure=192.168.152.61:2182):QuorumPeer@1175] - LOOKING
2020-04-27 14:22:56,450 [myid:1] - INFO [QuorumPeer[myid=1](plain=disabled)(secure=192.168.152.61:2182):FastLeaderElection@885] - New election. My id = 1, proposed zxid=0x23000003d7
2020-04-27 14:22:56,594 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@685] - Cannot open channel to 2 at election address test-server-02/192.168.152.62:3888
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2020)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1127)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:713)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:626)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456)
at java.lang.Thread.run(Thread.java:748)
2020-04-27 14:22:58,062 [myid:1] - INFO [QuorumPeer[myid=1](plain=disabled)(secure=192.168.152.61:2182):FastLeaderElection@919] - Notification time out: 1600
2020-04-27 14:22:59,168 [myid:1] - INFO [test-server-01/192.168.152.61:3888:QuorumCnxManager$Listener@924] - Received connection request 192.168.152.62:40548
2020-04-27 14:22:59,226 [myid:1] - INFO [test-server-01/192.168.152.61:3888:UnifiedServerSocket$UnifiedSocket@273] - Accepted TLS connection from /192.168.152.62:40548 - NONE - SSL_NULL_WITH_NULL_NULL
2020-04-27 14:22:59,227 [myid:1] - WARN [test-server-01/192.168.152.61:3888:QuorumCnxManager@548] - Exception reading or writing challenge: {}
javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: no cipher suites in common
at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1533)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:95)
at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.read(UnifiedServerSocket.java:694)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:524)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:478)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:934)
Caused by: javax.net.ssl.SSLHandshakeException: no cipher suites in common
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:306)
at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:1127)
at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:814)
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:221)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.security.ssl.SSLSocketImpl.getSession(SSLSocketImpl.java:2288)
at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.detectMode(UnifiedServerSocket.java:273)
at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.getSocket(UnifiedServerSocket.java:301)
at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.access$400(UnifiedServerSocket.java:180)
at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.getRealInputStream(UnifiedServerSocket.java:700)
... 9 more
Вот мой zoo.cfg, который одинаков для всех серверов
secureClientPort=2182
secureClientPortAddress=test-server-01
maxClientCnxns=1000
admin.enableServer=false
tickTime=2000
initLimit=10
syncLimit=5
server.1=test-server-01:2888:3888
server.2=test-server-02:2888:3888
server.3=test-server-03:2888:3888
sslQuorum=true
ssl.quorum.keyStore.location=/home/zookeeper/.ssl/keyStore.jks
ssl.quorum.keyStore.password=privateKeyStorePass
ssl.quorum.trustStore.location=/home/zookeeper/.ssl/trustStore.jks
ssl.quorum.trustStore.password=privateTrustStorePass
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.keyStore.location=/home/zookeeper/.ssl/keyStore.jks
ssl.keyStore.password=privateKeyStorePass
ssl.trustStore.location=/home/zookeeper/.ssl/trustStore.jks
ssl.trustStore.password=privateTrustStorePass
ssl.clientAuth=none
authProvider.1=org.apache.zookeeper.server.auth.X509AuthenticationProvider
При попытке подключения через s_client выдается следующее сообщение
$ openssl s_client -debug -connect test-server-01:2182
CONNECTED(00000003)
write to 0x174f490 [0x1761730] (289 bytes => 289 (0x121))
0000 - 16 03 01 01 1c 01 00 01-18 03 03 d4 b7 bb d6 03 ................
0010 - 1e 79 43 39 1c 3b c0 f0-52 94 0c c5 9c c7 78 40 .yC9.;..R.....x@
0020 - cd 66 10 15 34 33 1a 47-1f 48 b1 00 00 ac c0 30 .f..43.G.H.....0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1 .,.(.$..........
0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37 ...k.j.i.h.9.8.7
0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a .6.........2...*
0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f .&.......=.5.../
0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0 .+.'.#..........
0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31 ...g.@.?.>.3.2.1
0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43 .0.........E.D.C
00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c .B.1.-.).%......
00b0 - 00 3c 00 2f 00 96 00 41-c0 12 c0 08 00 16 00 13 .<./...A........
00c0 - 00 10 00 0d c0 0d c0 03-00 0a 00 07 c0 11 c0 07 ................
00d0 - c0 0c c0 02 00 05 00 04-00 ff 01 00 00 43 00 0b .............C..
00e0 - 00 04 03 00 01 02 00 0a-00 0a 00 08 00 17 00 19 ................
00f0 - 00 18 00 16 00 23 00 00-00 0d 00 20 00 1e 06 01 .....#..... ....
0100 - 06 02 06 03 05 01 05 02-05 03 04 01 04 02 04 03 ................
0110 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01 ................
0120 - 01 .
read from 0x174f490 [0x1766c90] (7 bytes => 7 (0x7))
0000 - 15 03 03 00 02 02 28 ......(
140328963696528:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1587990980
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
Кто-нибудь может дать мне несколько советов? Попытался как добавить сертификаты в доверенные хранилища всех серверов, так и добавить подписанный CSR от CA в доверенное хранилище / w сертификат CA без какой-либо помощи.
РЕДАКТИРОВАТЬ:
По неизвестной причине после После многих часов отладки кажется, что эта проблема решается, когда пароль хранилища ключей совпадает с паролем ключа при генерации пары ключей на сервере.
Может кто-нибудь объяснить, почему это работает, или это ошибка?
Вот скрипт, как я генерировал ключи и хранилища. Обратите внимание, что он немного отличается от описанного в посте в ссылке, но он работает / не работает с обоими подходами одинаково при смене паролей.
#!/bin/bash
TRUSTSTORE_PASS=Passw1rd
KEYSTORE_PASS=Passw2rd
KEY_PASS=Passw3rd
CA_PASS=caPass
DOMAINS_ZOOKEEPER=( domain-01 domain-02 domain-03 )
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365 -subj "/C=CZ/ST=Czechia/L=Prague/O=company/OU=unit/CN=domain.com" -passout pass:$CA_PASS
for i in "${DOMAINS[@]}"
do
keytool -genkeypair -alias $i -keyalg RSA -keysize 2048 -dname "cn=$i" -keypass $KEY_PASS -keystore ./$i/keyStore.jks -storepass $KEYSTORE_PASS
keytool -keystore ./$i/trustStore.jks -alias CARoot -import -file ca-cert -trustcacerts -storepass $TRUSTSTORE_PASS -noprompt
keytool -keystore ./$i/keyStore.jks -alias $i -certreq -file ./$i/$i.req -ext san=dns:$i -storepass $KEYSTORE_PASS -keypass $KEY_PASS
openssl x509 -req -CA ca-cert -CAkey ca-key -in ./$i/$i.req -out ./$i/$i-signed.cer -days 3650 -CAcreateserial -passin pass:$CA_PASS
keytool -keystore ./$i/keyStore.jks -alias CARoot -import -file ca-cert -noprompt -storepass $KEYSTORE_PASS
keytool -keystore ./$i/keyStore.jks -alias $i -import -file ./$i/$i-signed.cer -storepass $KEYSTORE_PASS -noprompt -keypass $KEY_PASS
done