У нас есть сервер DWH, который получает данные из разных источников с помощью задания jenkins.
Недавно у нас возникла проблема, когда служба jerkins была остановлена автоматически, и после этого никакие задания не запускаются, если мы не перезапустим jenkins вручную. service.
Итак, у меня есть два вопроса:
1.Можно написать сценарий, чтобы проверить, остановлена ли служба jenkins, и перезапустить ее, если она остановлена.
2. Ниже приведена информация журнала, которую я получил от модуля мониторинга jenkins. Я не эксперт в jenkins. Если какие-либо настройки необходимо изменить, чтобы избежать сбоев в JVM?
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# JVM is running with Zero Based Compressed Oops mode in which the Java heap is
# placed in the first 32GB address space. The Java Heap base address is the
# maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
# to set the Java Heap base and to place the Java Heap above 32GB virtual address.
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2757), pid=28669, tid=0x00007f396ffff700
#
# JRE version: OpenJDK Runtime Environment (8.0_232-b09) (build 1.8.0_232-8u232-b09-1~deb9u1-b09)
# Java VM: OpenJDK 64-Bit Server VM (25.232-b09 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
--------------- T H R E A D ---------------
Current thread (0x00007f3958048000): JavaThread "FilePath.localPool [#5]" daemon [_thread_new, id=13423, stack(0x00007f396feff000,0x00007f3970000000)]
Stack: [0x00007f396feff000,0x00007f3970000000], sp=0x00007f396fffeb50, free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xa7dc70]
V [libjvm.so+0x4cd0bc]
V [libjvm.so+0x8cc91c]
V [libjvm.so+0x8c3eae]
V [libjvm.so+0x8cdea7]
V [libjvm.so+0xa1f488]
V [libjvm.so+0xa235bc]
V [libjvm.so+0x8ca132]
C [libpthread.so.0+0x74a4] start_thread+0xc4
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x00007f39d404c000 JavaThread "qtp1642360923-5099" [_thread_new, id=13450, stack(0x00007f3a45f19000,0x00007f3a4601a000)]
0x00007f3960019800 JavaThread "qtp1642360923-5101" [_thread_new, id=13449, stack(0x00007f39353fa000,0x00007f39354fb000)]
0x00007f39a801f800 JavaThread "qtp1642360923-5102" [_thread_new, id=13448, stack(0x00007f39355fc000,0x00007f39356fd000)]
0x00007f397c03f800 JavaThread "qtp1642360923-5100" [_thread_new, id=13447, stack(0x00007f39356fd000,0x00007f39357fe000)]
0x00007f397c03e800 JavaThread "qtp1642360923-5098" [_thread_new, id=13446, stack(0x00007f396fdfe000,0x00007f396feff000)]
0x00007f397c018800 JavaThread "qtp1642360923-5097" [_thread_new, id=13445, stack(0x00007f39357fe000,0x00007f39358ff000)]
0x00007f3960019000 JavaThread "qtp1642360923-5096" [_thread_new, id=13444, stack(0x00007f39354fb000,0x00007f39355fc000)]
0x00007f39a801f000 JavaThread "qtp1642360923-5093" [_thread_new, id=13443, stack(0x00007f3935d03000,0x00007f3935e04000)]
0x00007f397c018000 JavaThread "qtp1642360923-5095" [_thread_new, id=13442, stack(0x00007f3935e04000,0x00007f3935f05000)]
0x00007f39d404b000 JavaThread "qtp1642360923-5094" [_thread_new, id=13441, stack(0x00007f3a18279000,0x00007f3a1837a000)]
0x00007f3960017000 JavaThread "qtp1642360923-5092" [_thread_new, id=13440, stack(0x00007f39f07f7000,0x00007f39f08f8000)]
0x00007f397c050800 JavaThread "qtp1642360923-5091" [_thread_new, id=13439, stack(0x00007f39358ff000,0x00007f3935a00000)]
0x00007f39a8015000 JavaThread "qtp1642360923-5089" [_thread_new, id=13438, stack(0x00007f3935c02000,0x00007f3935d03000)]
0x00007f39d4044000 JavaThread "qtp1642360923-5090" [_thread_new, id=13437, stack(0x00007f3935f05000,0x00007f3936006000)]
0x00007f3960056800 JavaThread "qtp1642360923-5088" [_thread_new, id=13436, stack(0x00007f3935a00000,0x00007f3935b01000)]
0x00007f39a805d800 JavaThread "qtp1642360923-5087" [_thread_new, id=13435, stack(0x00007f39374e2000,0x00007f39375e3000)]
0x00007f398c29f000 JavaThread "qtp1642360923-5085" [_thread_new, id=13434, stack(0x00007f3935b01000,0x00007f3935c02000)]
0x00007f39d816a800 JavaThread "qtp1642360923-5084" [_thread_new, id=13433, stack(0x00007f396c0b3000,0x00007f396c1b4000)]
0x00007f399c015800 JavaThread "qtp1642360923-5086" [_thread_new, id=13432, stack(0x00007f3a45d17000,0x00007f3a45e18000)]
0x00007f39d4025000 JavaThread "qtp1642360923-5083" [_thread_new, id=13431, stack(0x00007f3a45a14000,0x00007f3a45b15000)]
0x00007f397c050000 JavaThread "qtp1642360923-5080" [_thread_new, id=13430, stack(0x00007f39f09f9000,0x00007f39f0afa000)]
0x00007f39a802c800 JavaThread "qtp1642360923-5081" [_thread_new, id=13429, stack(0x00007f39f0afa000,0x00007f39f0bfb000)]
0x00007f399c011800 JavaThread "qtp1642360923-5082" [_thread_new, id=13428, stack(0x00007f39f04f4000,0x00007f39f05f5000)]
0x00007f39d8169800 JavaThread "qtp1642360923-5077" [_thread_new, id=13427, stack(0x00007f3936206000,0x00007f3936307000)]
0x00007f398c29e000 JavaThread "qtp1642360923-5078" [_thread_new, id=13426, stack(0x00007f39f05f5000,0x00007f39f06f6000)]
0x00007f39b0008000 JavaThread "ThinBackup Worker Thread thread" daemon [_thread_new, id=13425, stack(0x00007f3a18077000,0x00007f3a18178000)]
0x00007f399c027000 JavaThread "qtp1642360923-5076" [_thread_new, id=13424, stack(0x00007f3a45913000,0x00007f3a45a14000)]
=>0x00007f3958048000 JavaThread "FilePath.localPool [#5]" daemon [_thread_new, id=13423, stack(0x00007f396feff000,0x00007f3970000000)]
0x00007f39d8072800 JavaThread "qtp1642360923-5074" [_thread_in_vm, id=13379, stack(0x00007f3a45e18000,0x00007f3a45f19000)]
0x00007f399000c800 JavaThread "qtp1642360923-5062" [_thread_blocked, id=13087, stack(0x00007f3a18178000,0x00007f3a18279000)]
0x00007f399001d800 JavaThread "qtp1642360923-5060" [_thread_in_vm, id=12973, stack(0x00007f39f01f3000,0x00007f39f02f4000)]
0x00007f39a0064800 JavaThread "qtp1642360923-5047" [_thread_in_vm, id=12643, stack(0x00007f39f0dfd000,0x00007f39f0efe000)]
0x00007f39a802c000 JavaThread "qtp1642360923-5030" [_thread_blocked, id=11724, stack(0x00007f39f08f8000,0x00007f39f09f9000)]
0x00007f39d4046000 JavaThread "qtp1642360923-5025" [_thread_in_vm, id=11439, stack(0x00007f3a4601a000,0x00007f3a4611b000)]
0x00007f39a005c800 JavaThread "qtp1642360923-5019" [_thread_blocked, id=11074, stack(0x00007f3a18b7e000,0x00007f3a18c7f000)]
0x00007f399c042000 JavaThread "qtp1642360923-4936" [_thread_blocked, id=8691, stack(0x00007f3936307000,0x00007f3936408000)]
0x00007f3994137000 JavaThread "qtp1642360923-4931" [_thread_blocked, id=8378, stack(0x00007f39f06f6000,0x00007f39f07f7000)]
0x00007f397803f800 JavaThread "/bin/sh -xe /tmp/jenkins4350180492712963730.sh: stdout copier" daemon [_thread_in_native, id=26999, stack(0x00007f39f00f2000,0x00007f39f01f3000)]
0x00007f39c409e000 JavaThread "Executor #1 for master : executing dl.magento_data_layer_cleanup #135" daemon [_thread_blocked, id=26997, stack(0x00007f3a45b15000,0x00007f3a45c16000)]
0x00007f39ac448000 JavaThread "process reaper" daemon [_thread_in_native, id=24642, stack(0x00007f396f247000,0x00007f396f280000)]
0x00007f39ac41e000 JavaThread "Windows IOHub Watcher for IOHub#1: Selector[keys:0, gen:0] / Computer.threadPoolForRemoting [#3]" daemon [_thread_blocked, id=28820, stack(0x00007f39f0cfc000,0x00007f39f0dfd000)]
0x00007f39ac41d000 JavaThread "IOHub#1: Selector[keys:0, gen:0] / Computer.threadPoolForRemoting [#2]" daemon [_thread_in_native, id=28819, stack(0x00007f396c1b4000,0x00007f396c2b5000)]
0x00007f39a8011000 JavaThread "Session-Scheduler-51b279c9" [_thread_blocked, id=28796, stack(0x00007f396f925000,0x00007f396fa26000)]
0x00007f39c42a4000 JavaThread "NioChannelHub keys=0 gen=0: Computer.threadPoolForRemoting [#1]" daemon [_thread_in_native, id=28773, stack(0x00007f39f0bfb000,0x00007f39f0cfc000)]
0x00007f39c42a1000 JavaThread "Jenkins UDP 33848 monitoring thread" [_thread_in_native, id=28771, stack(0x00007f39f0efe000,0x00007f39f0fff000)]
0x00007f3988046000 JavaThread "jenkins.util.Timer [#10]" daemon [_thread_blocked, id=28766, stack(0x00007f396c2b5000,0x00007f396c3b6000)]
0x00007f39b0003000 JavaThread "jenkins.util.Timer [#9]" daemon [_thread_blocked, id=28765, stack(0x00007f396c5b6000,0x00007f396c6b7000)]
0x00007f398c10f000 JavaThread "jenkins.util.Timer [#8]" daemon [_thread_blocked, id=28757, stack(0x00007f396ceb7000,0x00007f396cfb8000)]
0x00007f398c10c800 JavaThread "jenkins.util.Timer [#7]" daemon [_thread_blocked, id=28756, stack(0x00007f396cfb8000,0x00007f396d0b9000)]
0x00007f398c10a000 JavaThread "jenkins.util.Timer [#6]" daemon [_thread_blocked, id=28755, stack(0x00007f396d0b9000,0x00007f396d1ba000)]
0x00007f398c109000 JavaThread "jenkins.util.Timer [#5]" daemon [_thread_blocked, id=28754, stack(0x00007f396d1ba000,0x00007f396d2bb000)]
0x00007f398c02b800 JavaThread "jenkins.util.Timer [#4]" daemon [_thread_blocked, id=28753, stack(0x00007f396d2bb000,0x00007f396d3bc000)]
0x00007f398c02b000 JavaThread "jenkins.util.Timer [#3]" daemon [_thread_blocked, id=28752, stack(0x00007f396ea1a000,0x00007f396eb1b000)]
0x00007f39b0001800 JavaThread "jenkins.util.Timer [#2]" daemon [_thread_blocked, id=28748, stack(0x00007f396f3df000,0x00007f396f4e0000)]
0x00007f3998094000 JavaThread "javamelody" daemon [_thread_blocked, id=28742, stack(0x00007f396f813000,0x00007f396f914000)]
0x00007f39d0004800 JavaThread "Connector-Scheduler-6b143ee9" [_thread_blocked, id=28721, stack(0x00007f3a1837a000,0x00007f3a1847b000)]
0x00007f39c4099800 JavaThread "jenkins.util.Timer [#1]" daemon [_thread_blocked, id=28718, stack(0x00007f3a1867b000,0x00007f3a1877c000)]
0x00007f39c4078000 JavaThread "Jenkins cron thread" [_thread_blocked, id=28717, stack(0x00007f3a1877c000,0x00007f3a1887d000)]
0x00007f3a7000b000 JavaThread "DestroyJavaVM" [_thread_blocked, id=28671, stack(0x00007f3a7a101000,0x00007f3a7a201000)]
0x00007f3a70e1b800 JavaThread "LauncherControlThread[ControlPort=-1]" [_thread_blocked, id=28716, stack(0x00007f3a1887d000,0x00007f3a1897e000)]
0x00007f3a70d76800 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=28714, stack(0x00007f3a18c7f000,0x00007f3a18d80000)]
0x00007f3a70b0a800 JavaThread "Session-HouseKeeper-27fe3806" [_thread_blocked, id=28712, stack(0x00007f3a45282000,0x00007f3a45383000)]
0x00007f3a704bd800 JavaThread "qtp1642360923-15-acceptor-0@12a46961-ServerConnector@6b143ee9{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" [_thread_in_native, id=28696, stack(0x00007f3a45c16000,0x00007f3a45d17000)]
0x00007f3a70211000 JavaThread "Service Thread" daemon [_thread_blocked, id=28688, stack(0x00007f3a46a51000,0x00007f3a46b52000)]
0x00007f3a7020e000 JavaThread "C1 CompilerThread3" daemon [_thread_blocked, id=28687, stack(0x00007f3a46b53000,0x00007f3a46c53000)]
0x00007f3a7020c000 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=28686, stack(0x00007f3a46c54000,0x00007f3a46d54000)]
0x00007f3a7020a800 JavaThread "C2 CompilerThread1" daemon [_thread_in_native, id=28685, stack(0x00007f3a46d55000,0x00007f3a46e55000)]
0x00007f3a70207800 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=28684, stack(0x00007f3a46e56000,0x00007f3a46f56000)]
0x00007f3a70206000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=28683, stack(0x00007f3a46f56000,0x00007f3a47057000)]
0x00007f3a701d3800 JavaThread "Finalizer" daemon [_thread_blocked, id=28682, stack(0x00007f3a471f2000,0x00007f3a472f3000)]
0x00007f3a701d1000 JavaThread "Reference Handler" daemon [_thread_blocked, id=28681, stack(0x00007f3a472f3000,0x00007f3a473f4000)]
Other Threads:
0x00007f3a701c7800 VMThread [stack: 0x00007f3a473f5000,0x00007f3a474f5000] [id=28680]
0x00007f3a70213800 WatcherThread [stack: 0x00007f3a46951000,0x00007f3a46a51000] [id=28689]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
heap address: 0x00000003db000000, size: 15952 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
Compressed class space size: 1073741824 Address: 0x00000007c0000000
Heap:
PSYoungGen total 4711424K, used 1209684K [0x0000000673b00000, 0x0000000798080000, 0x00000007c0000000)
eden space 4658176K, 24% used [0x0000000673b00000,0x00000006ba459038,0x0000000790000000)
from space 53248K, 99% used [0x0000000793d00000,0x00000007970fc288,0x0000000797100000)
to space 62464K, 0% used [0x0000000790000000,0x0000000790000000,0x0000000793d00000)
ParOldGen total 1458688K, used 97397K [0x00000003db000000, 0x0000000434080000, 0x0000000673b00000)
object space 1458688K, 6% used [0x00000003db000000,0x00000003e0f1d608,0x0000000434080000)
Metaspace used 92553K, capacity 101142K, committed 101248K, reserved 1138688K
class space used 10216K, capacity 12402K, committed 12416K, reserved 1048576K
Card table byte_map: [0x00007f3a7489a000,0x00007f3a767c3000] byte_map_base: 0x00007f3a729c2000
Marking Bits: (ParMarkBitMap*) 0x00007f3a793e6560
Begin Bits: [0x00007f3a18d80000, 0x00007f3a286c0000)
End Bits: [0x00007f3a286c0000, 0x00007f3a38000000)
Polling page: 0x00007f3a7a20e000
CodeCache: size=245760Kb used=55049Kb max_used=56198Kb free=190710Kb
bounds [0x00007f3a61000000, 0x00007f3a64750000, 0x00007f3a70000000]
total_blobs=14549 nmethods=13505 adapters=956
compilation: enabled
Compilation events (10 events):
Event: 40945.189 Thread 0x00007f3a7020a800 18156 4 javax.servlet.ServletResponseWrapper::getOutputStream (10 bytes)
Event: 40945.190 Thread 0x00007f3a7020a800 nmethod 18156 0x00007f3a63ab5e10 code [0x00007f3a63ab5f60, 0x00007f3a63ab5ff8]
Event: 40945.192 Thread 0x00007f3a7020c000 18157 4 sun.nio.cs.US_ASCII$Encoder::isLegalReplacement (26 bytes)
Event: 40945.192 Thread 0x00007f3a7020c000 nmethod 18157 0x00007f3a62d19750 code [0x00007f3a62d198a0, 0x00007f3a62d19978]
Event: 40970.054 Thread 0x00007f3a7020a800 18158 4 net.bull.javamelody.MonitoringFilter::getCompleteRequestName (242 bytes)
Event: 40970.085 Thread 0x00007f3a7020a800 nmethod 18158 0x00007f3a632019d0 code [0x00007f3a63201d00, 0x00007f3a63203908]
Event: 41015.066 Thread 0x00007f3a7020a800 18159 4 org.kohsuke.stapler.compression.FilterServletOutputStream::close (8 bytes)
Event: 41015.071 Thread 0x00007f3a7020a800 nmethod 18159 0x00007f3a6180f190 code [0x00007f3a6180f340, 0x00007f3a6180f728]
Event: 41065.070 Thread 0x00007f3a7020a800 18160 ! 4 org.apache.commons.jelly.tags.core.ArgTag::doTag (382 bytes)
Event: 41065.092 Thread 0x00007f3a7020a800 nmethod 18160 0x00007f3a61f13950 code [0x00007f3a61f13be0, 0x00007f3a61f154e8]
Ниже приведена информация о системе:
OS: OS Linux, 4.9.0-11-amd64 , amd64/64 (8 cores)
Java: OpenJDK Runtime Environment, 1.8.0_232-8u232-b09-1~deb9u1-b09
JVM: OpenJDK 64-Bit Server VM, 25.232-b09, mixed mode
PID of process: 14969
Nb of opened files 502 / 1,048,576 ++++++++++++
Server: Server jetty/9.4.z-SNAPSHOT
Webapp context:
Start: 09/02/20 19:37
JVM arguments: -Djava.awt.headless=true
-Dhudson.DNSMultiCast.disabled=true
Mean age of http sessions (min): 12
Memory: Non heap memory = 149 Mb (Perm Gen, Code Cache),
Buffered memory = 0 Mb,
Loaded classes = 14,929,
Garbage collection time = 1,227 ms,
Process cpu time = 260,390 ms,
Committed virtual memory = 22,164 Mb,
Free physical memory = 10,194 Mb,
Total physical memory = 63,805 Mb
Free disk space: 49,460 Mb
Usable disk space: 48,215 Mb
