Искробезопасное приложение передается в отдельном кластере с 2-мя рабочими. В моем приложении чтение данных из темы kafka, обработка некоторых вычислений и отправка в тему результатов. Дублирующие записи видны в теме результатов. В консоли видны следующие исключения. Проверка установлен на s3, и даже auto .offset.reset установлен как последний "kafkaParams.put (" auto.offset.reset "," latest ");" тогда почему я получаю дубликат записи в теме результатов.
2018-08-28 16:32:36 INFO JobScheduler:54 - Finished job streaming job 1535454155000 ms.0 from job set of time 1535454155000 ms
2018-08-28 16:32:36 INFO JobScheduler:54 - Total delay: 1.870 s for time 1535454155000 ms (execution: 0.201 s)
2018-08-28 16:32:42 INFO CheckpointWriter:54 - Checkpoint for time 1535454142000 ms saved to file 's3://cffcs5/cffccheck/cffc2/checkpoint-1535454143000', took 5590 bytes and 8507 ms
2018-08-28 16:32:42 INFO CheckpointWriter:54 - Saving checkpoint for time 1535454143000 ms to file 's3://cffcs5/cffccheck/cffc2/checkpoint-1535454143000'
2018-08-28 16:32:43 INFO NativeS3FileSystem:237 - OutputStream for key 'cffccheck/cffc2/temp' writing to tempfile '\tmp\hadoop-DELL\s3\output-7698969883279329562.tmp'
2018-08-28 16:32:43 INFO NativeS3FileSystem:271 - OutputStream for key 'cffccheck/cffc2/temp' closed. Now beginning upload
2018-08-28 16:32:44 INFO NativeS3FileSystem:283 - OutputStream for key 'cffccheck/cffc2/temp' upload complete
2018-08-28 16:32:45 INFO CheckpointWriter:54 - CheckpointWriter executor terminated? false, waited for 10008 ms.
2018-08-28 16:32:45 INFO JobGenerator:54 - Stopped JobGenerator
2018-08-28 16:32:45 INFO JobScheduler:54 - Stopped JobScheduler
2018-08-28 16:32:45 INFO ContextHandler:910 - Stopped o.s.j.s.ServletContextHandler@120d3fd{/streaming,null,UNAVAILABLE,@Spark}
2018-08-28 16:32:45 INFO ContextHandler:910 - Stopped o.s.j.s.ServletContextHandler@410ae5ac{/streaming/batch,null,UNAVAILABLE,@Spark}
2018-08-28 16:32:45 INFO ContextHandler:910 - Stopped o.s.j.s.ServletContextHandler@4fd05028{/static/streaming,null,UNAVAILABLE,@Spark}
2018-08-28 16:32:45 INFO StreamingContext:54 - StreamingContext stopped successfully
2018-08-28 16:32:45 INFO AbstractConnector:318 - Stopped Spark@562c877a{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-08-28 16:32:45 INFO SparkUI:54 - Stopped Spark web UI at http://DELL-PC:4040
2018-08-28 16:32:45 INFO StandaloneSchedulerBackend:54 - Shutting down all executors
2018-08-28 16:32:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Asking each executor to shut down
2018-08-28 16:32:45 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2018-08-28 16:32:45 WARN TransportChannelHandler:78 - Exception in connection from /192.168.1.179:58235
java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
2018-08-28 16:32:45 WARN TransportChannelHandler:78 - Exception in connection from /192.168.1.179:58251
java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
2018-08-28 16:32:45 INFO MemoryStore:54 - MemoryStore cleared
2018-08-28 16:32:45 INFO BlockManager:54 - BlockManager stopped
2018-08-28 16:32:45 INFO BlockManagerMaster:54 - BlockManagerMaster stopped
2018-08-28 16:32:45 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2018-08-28 16:32:45 INFO SparkContext:54 - Successfully stopped SparkContext
2018-08-28 16:32:50 INFO CheckpointWriter:54 - Checkpoint for time 1535454143000 ms saved to file 's3://cffcs5/cffccheck/cffc2/checkpoint-1535454143000', took 5590 bytes and 8615 ms
2018-08-28 16:32:50 INFO ShutdownHookManager:54 - Shutdown hook called
2018-08-28 16:32:50 INFO ShutdownHookManager:54 - Deleting directory E:\temp8094\localdir\spark-aa95d0d5-6b7a-4d5b-ae51-a832fd2b4074
2018-08-28 16:32:50 INFO ShutdownHookManager:54 - Deleting directory C:\Users\DELL\AppData\Local\Temp\spark-28b3a422-0a53-4049-9fb0-2425af002354
2018-08-28 16:32:53 INFO SecurityManager:54 - Changing view acls to: DELL
2018-08-28 16:32:53 INFO SecurityManager:54 - Changing modify acls to: DELL
2018-08-28 16:32:53 INFO SecurityManager:54 - Changing view acls groups to:
2018-08-28 16:32:53 INFO SecurityManager:54 - Changing modify acls groups to:
2018-08-28 16:32:53 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(DELL); groups with view permissions: Set(); users with modify permissions: Set(DELL); groups with modify permissions: Set()
2018-08-28 16:32:53 INFO Utils:54 - Successfully started service 'Driver' on port 58363.
2018-08-28 16:32:53 INFO DriverWrapper:54 - Driver address: 192.168.1.179:58363
2018-08-28 16:32:53 INFO WorkerWatcher:54 - Connecting to worker spark://Worker@192.168.1.179:53448
2018-08-28 16:32:53 INFO SecurityManager:54 - Changing view acls to: DELL
2018-08-28 16:32:53 INFO SecurityManager:54 - Changing modify acls to: DELL
2018-08-28 16:32:53 INFO SecurityManager:54 - Changing view acls groups to:
2018-08-28 16:32:53 INFO SecurityManager:54 - Changing modify acls groups to:
2018-08-28 16:32:53 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(DELL); groups with view permissions: Set(); users with modify permissions: Set(DELL); groups with modify permissions: Set()
2018-08-28 16:32:53 INFO WorkerWatcher:54 - Successfully connected to spark://Worker@192.168.1.179:53448
2018-08-28 16:32:53 INFO TransportClientFactory:267 - Successfully created connection to /192.168.1.179:53448 after 129 ms (0 ms spent in bootstraps)
2018-08-28 16:32:56 INFO CheckpointReader:54 - Checkpoint files found: s3://cffcs5/cffccheck/cffc2/checkpoint-1535454143000,s3://cffcs5/cffccheck/cffc2/checkpoint-1535454143000.bk,s3://cffcs5/cffccheck/cffc2/checkpoint-1535454142000
2018-08-28 16:32:56 INFO CheckpointReader:54 - Attempting to load checkpoint from file s3://cffcs5/cffccheck/cffc2/checkpoint-1535454143000
2018-08-28 16:32:56 INFO NativeS3FileSystem:615 - Opening 's3://cffcs5/cffccheck/cffc2/checkpoint-1535454143000' for reading
2018-08-28 16:32:57 INFO Checkpoint:54 - Checkpoint for time 1535454143000 ms validated
2018-08-28 16:32:57 INFO CheckpointReader:54 - Checkpoint successfully loaded from file s3://cffcs5/cffccheck/cffc2/checkpoint-1535454143000
2018-08-28 16:32:57 INFO CheckpointReader:54 - Checkpoint was generated at time 1535454143000 ms
2018-08-28 16:32:57 INFO SparkContext:54 - Running Spark version 2.3.0