У меня есть набор jenkins maste (1 ЦПУ 8 ГБ ОЗУ) и агент jenkins (4 ЦП 16 ГБ ОЗУ), имеющие Centos7 и Java1.8 - работающие на виртуальных машинах GCP. Все мои сборки (CI и CD работают на агенте). Он обрабатывает ~ 30 репозиториев, все настроены как многоотраслевой проект jenkins. Все задания CI - это проекты gradle с использованием различных версий gradle. Время от времени я вижу сбой сборок с журналами ниже, которые исчезают при повторном запуске:
./gradlew clean
.
.
.
.
FAILURE: Build failed with an exception.
* What went wrong:
A problem occurred configuring root project 'xyz123'.
> Could not resolve all artifacts for configuration ':classpath'.
> Could not resolve com.eriwen:gradle-js-plugin:2.14.1.
Required by:
project : > com.eriwen.gradle.js:com.eriwen.gradle.js.gradle.plugin:2.14.1
> Could not resolve com.eriwen:gradle-js-plugin:2.14.1.
> Could not get resource 'https://plugins.gradle.org/m2/com/eriwen/gradle-js-plugin/2.14.1/gradle-js-plugin-2.14.1.pom'.
> Could not GET 'https://plugins.gradle.org/m2/com/eriwen/gradle-js-plugin/2.14.1/gradle-js-plugin-2.14.1.pom'.
> Connect to plugins-artifacts.gradle.org:443 [plugins-artifacts.gradle.org/104.18.191.9, plugins-artifacts.gradle.org/104.18.190.9, plugins-artifacts.gradle.org/2606:4700:0:0:0:0:6812:be09, plugins-artifacts.gradle.org/2606:4700:0:0:0:0:6812:bf09] failed: Network is unreachable (connect failed)
+ ./gradlew clean
Downloading https://services.gradle.org/distributions/gradle-4.10.2-all.zip
...............................................................................................................
Starting a Gradle Daemon (subsequent builds will be faster)
FAILURE: Build failed with an exception.
* Where:
Build file '/var/lib/jenkins/workspace/Auth_CI_master/build.gradle' line: 8
* What went wrong:
Could not apply requested plugin [id: 'nu.studer.credentials', version: '1.0.4'] as it does not provide a plugin with id 'nu.studer.credentials'. This is caused by an incorrect plugin implementation. Please contact the plugin author(s).
> Plugin with id 'nu.studer.credentials' not found.
+ ./gradlew clean
The message received from the daemon indicates that the daemon has disappeared.
Build request sent: Build{id=3313d464-e64f-4eeb-87df-9b373d6c47fa, currentDir=/var/lib/jenkins/workspace/Model_CI_PR-299}
Attempting to read last messages from the daemon log...
Daemon pid: 28652
log file: /var/lib/jenkins/.gradle/daemon/4.8.1/daemon-28652.out.log
----- Last 20 lines from daemon log file - daemon-28652.out.log -----
13:12:39.097 [DEBUG] [org.gradle.launcher.daemon.server.DefaultDaemonConnection] thread 49: Received non-IO message from client: Build{id=3313d464-e64f-4eeb-87df-9b373d6c47fa, currentDir=/var/lib/jenkins/workspace/Model_CI_PR-299}
13:12:39.098 [INFO] [org.gradle.launcher.daemon.server.DefaultIncomingConnectionHandler] Received command: Build{id=3313d464-e64f-4eeb-87df-9b373d6c47fa, currentDir=/var/lib/jenkins/workspace/Model_CI_PR-299}.
13:12:39.098 [DEBUG] [org.gradle.launcher.daemon.server.DefaultIncomingConnectionHandler] Starting executing command: Build{id=3313d464-e64f-4eeb-87df-9b373d6c47fa, currentDir=/var/lib/jenkins/workspace/Model_CI_PR-299} with connection: socket connection from /0:0:0:0:0:0:0:1:41856 to /0:0:0:0:0:0:0:1:40654.
13:12:39.098 [ERROR] [org.gradle.launcher.daemon.server.DaemonStateCoordinator] Command execution: started DaemonCommandExecution[command = Build{id=3313d464-e64f-4eeb-87df-9b373d6c47fa, currentDir=/var/lib/jenkins/workspace/Model_CI_PR-299}, connection = DefaultDaemonConnection: socket connection from /0:0:0:0:0:0:0:1:41856 to /0:0:0:0:0:0:0:1:40654] after 0.0 minutes of idle
13:12:39.098 [INFO] [org.gradle.launcher.daemon.server.DaemonRegistryUpdater] Marking the daemon as busy, address: [453d948b-1298-4102-b309-f6b1490fd31e port:41856, addresses:[/0:0:0:0:0:0:0:1%lo, /127.0.0.1]]
13:12:39.098 [DEBUG] [org.gradle.launcher.daemon.registry.PersistentDaemonRegistry] Marking busy by address: [453d948b-1298-4102-b309-f6b1490fd31e port:41856, addresses:[/0:0:0:0:0:0:0:1%lo, /127.0.0.1]]
13:12:39.099 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire exclusive lock on daemon addresses registry.
13:12:39.100 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
13:12:39.101 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
13:12:39.101 [DEBUG] [org.gradle.launcher.daemon.server.DaemonStateCoordinator] resetting idle timer
13:12:39.101 [DEBUG] [org.gradle.launcher.daemon.server.DaemonStateCoordinator] daemon is running. Sleeping until state changes.
13:12:39.101 [INFO] [org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy] Daemon is about to start building Build{id=3313d464-e64f-4eeb-87df-9b373d6c47fa, currentDir=/var/lib/jenkins/workspace/Model_CI_PR-299}. Dispatching build started information...
13:12:39.102 [DEBUG] [org.gradle.launcher.daemon.server.SynchronizedDispatchConnection] thread 16: dispatching class org.gradle.launcher.daemon.protocol.BuildStarted
13:12:39.102 [DEBUG] [org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment] Configuring env variables: {PATH=/opt/gradle/gradle-5.1/bin:/usr/local/bin:/usr/bin, CHANGE_BRANCH=f_update_order_status, RUN_DISPLAY_URL=http://10.30.0.133:8080/job/Model_CI/job/PR-299/6/display/redirect, HUDSON_HOME=/var/lib/jenkins, RUN_CHANGES_DISPLAY_URL=http://10.30.0.133:8080/job/Model_CI/job/PR-299/6/display/redirect?page=changes, JOB_URL=http://10.30.0.133:8080/job/Model_CI/job/PR-299/, HUDSON_COOKIE=ff8d2b01-1ecf-4578-81de-41c2e6728da0, JENKINS_NODE_COOKIE=5916860b-7925-4de5-9a5a-bd1e698b141f, MAIL=/var/mail/jenkins, JENKINS_SERVER_COOKIE=durable-8d6a1400e0b69566c97ac3c70ccbb340, LOGNAME=jenkins, PWD=/var/lib/jenkins/workspace/Model_CI_PR-299, JENKINS_URL=http://10.30.0.133:8080/, LESSOPEN=||/usr/bin/lesspipe.sh %s, BUILD_TAG=jenkins-Model_CI-PR-299-6, SHELL=/bin/bash, STAGE_NAME=clean, OLDPWD=/var/lib/jenkins/workspace/Model_CI_PR-299, CHANGE_URL=https://github.com/PawsForLife/model/pull/299, JENKINS_HOME=/var/lib/jenkins, NODE_NAME=Agent-007, BUILD_DISPLAY_NAME=#6, JOB_DISPLAY_URL=http://10.30.0.133:8080/job/Model_CI/job/PR-299/display/redirect, GIT_BRANCH=PR-299, SHLVL=3, GIT_PREVIOUS_COMMIT=df3be799640564caf268caa461cf6bff04f29601, BUILD_ID=6, LANG=en_US.UTF-8, XDG_SESSION_ID=500, JOB_NAME=Model_CI/PR-299, GRADLE_HOME=/opt/gradle/gradle-5.1, GIT_PREVIOUS_SUCCESSFUL_COMMIT=eb4923c9ad8da298e99cea563c7dea068d51c2a0, NODE_LABELS=Agent-007 Agent007, HUDSON_URL=http://10.30.0.133:8080/, CHANGE_TITLE=ERP-2159:interface for the endpoint of updating the order status, WORKSPACE=/var/lib/jenkins/workspace/Model_CI_PR-299, GIT_LOCAL_BRANCH=PR-299, CHANGE_ID=299, CHANGE_TARGET=master, library.instanceGroups.version=master, GIT_COMMIT=4f9236381d724a7ef57410bcc4834042587361f0, EXECUTOR_NUMBER=3, SSH_CLIENT=10.30.0.133 56618 22, HUDSON_SERVER_COOKIE=0cf2baaadf0780f3, JOB_BASE_NAME=PR-299, USER=jenkins, BUILD_NUMBER=6, SSH_CONNECTION=10.30.0.133 56618 10.30.0.134 22, BUILD_URL=http://10.30.0.133:8080/job/Model_CI/job/PR-299/6/, CHANGE_AUTHOR=rafael-tugaon, GIT_URL=https://github.com/=====/model.git, XDG_RUNTIME_DIR=/run/user/1008, BRANCH_NAME=PR-299, HOME=/var/lib/jenkins}
13:12:39.104 [DEBUG] [org.gradle.launcher.daemon.server.exec.LogToClient] About to start relaying all logs to the client via the connection.
13:12:39.104 [INFO] [org.gradle.launcher.daemon.server.exec.LogToClient] The client will now receive all logging from the daemon (pid: 28652). The daemon log file: /var/lib/jenkins/.gradle/daemon/4.8.1/daemon-28652.out.log
13:12:39.104 [INFO] [org.gradle.launcher.daemon.server.exec.LogAndCheckHealth] Starting 3rd build in daemon [uptime: 14.887 secs, performance: 97%]
13:12:39.105 [DEBUG] [org.gradle.launcher.daemon.server.exec.ExecuteBuild] The daemon has started executing the build.
13:12:39.105 [DEBUG] [org.gradle.launcher.daemon.server.exec.ExecuteBuild] Executing build with daemon context: DefaultDaemonContext[uid=e4e323b7-ed33-4a56-9984-a31b8de7d600,javaHome=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64,daemonRegistryDir=/var/lib/jenkins/.gradle/daemon,pid=28652,idleTimeout=10800000,daemonOpts=-XX:+HeapDumpOnOutOfMemoryError,-Xmx1024m,-Dfile.encoding=UTF-8,-Duser.country=US,-Duser.language=en,-Duser.variant]
Daemon vm is shutting down... The daemon has exited normally or was terminated in response to a user interrupt.
----- End of the daemon log -----
Хосты jenkins находятся за брандмауэром и в отношении первого сбоя, который я проверил, NAT облака Google журнал для любого сброса пакета - нет ни одного.
Вопрос: Как бы я расследовал эти ошибки? Любое предложение будет высоко оценено.