Я вижу некоторые проблемы с непредсказуемым выполнением запросов к базе данных.
Это приложение, размещенное на Payara 4, подключающееся к базе данных Oracle с помощью EclipseLink.Мы видели, как это происходит в нашей производственной среде, в двух случаях.Похоже, что это происходит при более высоких временах использования, хотя я не уверен, что это потому, что это является причиной или потому, что больше запросов, так что это более вероятно.
Ниже приведен график, который показывает количествонеудавшиеся запросы (красным цветом) и успешные запросы (серым цветом): 
Я подтвердил, что на стороне базы данных нет проблем, например, разорванные соединения.
Ниже приведена ошибка, которая возникает при сбое (хотя это происходит во многих запросах):
javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.4.qualifier): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: This pool is not registered with the runtime environment : <redacted>
Error Code: 0
Call: <redacted>
bind => [1 parameter bound]
Query: DoesExistQuery(<redacted>)
at org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:879)
at org.eclipse.persistence.internal.jpa.QueryImpl.performPreQueryFlush(QueryImpl.java:967)
at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:207)
at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:473)
at <redacted>
at <redacted>
at <redacted>
at sun.reflect.GeneratedMethodAccessor6005.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4836)
at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:836)
at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
at sun.reflect.GeneratedMethodAccessor3907.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
at sun.reflect.GeneratedMethodAccessor3908.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)
at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:374)
at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4808)
at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4796)
at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:205)
at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:79)
at com.sun.proxy.$Proxy132454.<redacted>(Unknown Source)
at sun.reflect.GeneratedMethodAccessor6004.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:239)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:150)
at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:226)
at <redacted>
at <redacted>
at <redacted>
at <redacted>
at org.apache.struts.chain.commands.servlet.ExecuteAction.execute(ExecuteAction.java:58)
at org.apache.struts.chain.commands.AbstractExecuteAction.execute(AbstractExecuteAction.java:67)
at org.apache.struts.chain.commands.ActionCommandBase.execute(ActionCommandBase.java:51)
at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:191)
at org.apache.commons.chain.generic.LookupCommand.execute(LookupCommand.java:305)
at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:191)
at org.apache.struts.chain.ComposableRequestProcessor.process(ComposableRequestProcessor.java:283)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1692)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:339)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:209)
at <redacted>
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:251)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:209)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:654)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:593)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:159)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:654)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:593)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:368)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:238)
at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:483)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:180)
at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:206)
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:180)
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:539)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573)
at java.lang.Thread.run(Thread.java:748)
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.4.qualifier): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: This pool is not registered with the runtime environment : <redacted>
Error Code: 0
Call: <redacted>
bind => [1 parameter bound]
Query: DoesExistQuery(<redacted>)
at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:316)
at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:147)
at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)
at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:346)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:309)
at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:581)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1634)
at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:321)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:615)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:560)
at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2056)
at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:570)
at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:258)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:242)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:228)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectRowForDoesExist(DatasourceCallQueryMechanism.java:736)
at org.eclipse.persistence.queries.DoesExistQuery.executeDatabaseQuery(DoesExistQuery.java:241)
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:904)
at org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:803)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1857)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1790)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.checkForUnregisteredExistingObject(UnitOfWorkImpl.java:785)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.discoverAndPersistUnregisteredNewObjects(UnitOfWorkImpl.java:4192)
at org.eclipse.persistence.mappings.ObjectReferenceMapping.cascadeDiscoverAndPersistUnregisteredNewObjects(ObjectReferenceMapping.java:951)
at org.eclipse.persistence.mappings.ObjectReferenceMapping.cascadeDiscoverAndPersistUnregisteredNewObjects(ObjectReferenceMapping.java:929)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.cascadeDiscoverAndPersistUnregisteredNewObjects(ObjectBuilder.java:2515)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.discoverAndPersistUnregisteredNewObjects(UnitOfWorkImpl.java:4205)
at org.eclipse.persistence.mappings.CollectionMapping.cascadeDiscoverAndPersistUnregisteredNewObjects(CollectionMapping.java:483)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.cascadeDiscoverAndPersistUnregisteredNewObjects(ObjectBuilder.java:2515)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.discoverAndPersistUnregisteredNewObjects(UnitOfWorkImpl.java:4205)
at org.eclipse.persistence.mappings.CollectionMapping.cascadeDiscoverAndPersistUnregisteredNewObjects(CollectionMapping.java:483)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.cascadeDiscoverAndPersistUnregisteredNewObjects(ObjectBuilder.java:2515)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.discoverAndPersistUnregisteredNewObjects(UnitOfWorkImpl.java:4205)
at org.eclipse.persistence.mappings.CollectionMapping.cascadeDiscoverAndPersistUnregisteredNewObjects(CollectionMapping.java:483)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.cascadeDiscoverAndPersistUnregisteredNewObjects(ObjectBuilder.java:2515)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.discoverAndPersistUnregisteredNewObjects(UnitOfWorkImpl.java:4205)
at org.eclipse.persistence.mappings.ObjectReferenceMapping.cascadeDiscoverAndPersistUnregisteredNewObjects(ObjectReferenceMapping.java:951)
at org.eclipse.persistence.mappings.ObjectReferenceMapping.cascadeDiscoverAndPersistUnregisteredNewObjects(ObjectReferenceMapping.java:929)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.cascadeDiscoverAndPersistUnregisteredNewObjects(ObjectBuilder.java:2515)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.discoverAndPersistUnregisteredNewObjects(UnitOfWorkImpl.java:4205)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.discoverUnregisteredNewObjects(RepeatableWriteUnitOfWork.java:308)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.calculateChanges(UnitOfWorkImpl.java:723)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.writeChanges(RepeatableWriteUnitOfWork.java:441)
at org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:874)
... 92 more
Caused by: java.sql.SQLException: This pool is not registered with the runtime environment : <redacted>
at com.sun.enterprise.connectors.service.ConnectorConnectionPoolAdminServiceImpl.getConnection(ConnectorConnectionPoolAdminServiceImpl.java:1558)
at com.sun.enterprise.connectors.ConnectorRuntime.getConnection(ConnectorRuntime.java:639)
at org.glassfish.jdbcruntime.service.JdbcDataSource.getConnection(JdbcDataSource.java:88)
at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:135)
... 136 more
Caused by: javax.resource.ResourceException: This pool is not registered with the runtime environment : <redacted>
at com.sun.enterprise.connectors.service.ConnectorConnectionPoolAdminServiceImpl.getUnpooledConnection(ConnectorConnectionPoolAdminServiceImpl.java:583)
at com.sun.enterprise.connectors.service.ConnectorConnectionPoolAdminServiceImpl.getConnection(ConnectorConnectionPoolAdminServiceImpl.java:1550)
... 139 more
Caused by: com.sun.appserv.connectors.internal.api.ConnectorRuntimeException: This pool is not bound in JNDI : <redacted>
at com.sun.enterprise.connectors.service.ConnectorConnectionPoolAdminServiceImpl.obtainManagedConnectionFactory(ConnectorConnectionPoolAdminServiceImpl.java:1022)
at com.sun.enterprise.connectors.service.ConnectorConnectionPoolAdminServiceImpl.obtainManagedConnectionFactory(ConnectorConnectionPoolAdminServiceImpl.java:911)
at com.sun.enterprise.connectors.service.ConnectorConnectionPoolAdminServiceImpl.getUnpooledConnection(ConnectorConnectionPoolAdminServiceImpl.java:576)
... 140 more
Caused by: javax.naming.NamingException: Lookup failed for '__SYSTEM/pools/<redacted>' in SerialContext[myEnv={java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory, java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl, java.naming.factory.url.pkgs=com.sun.enterprise.naming} [Root exception is javax.naming.NameNotFoundException: <redacted> not found]
at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:496)
at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:442)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at org.glassfish.resourcebase.resources.naming.ResourceNamingService.lookup(ResourceNamingService.java:238)
at com.sun.enterprise.connectors.service.ConnectorConnectionPoolAdminServiceImpl.getConnectorConnectionPool(ConnectorConnectionPoolAdminServiceImpl.java:799)
at com.sun.enterprise.connectors.service.ConnectorConnectionPoolAdminServiceImpl.obtainManagedConnectionFactory(ConnectorConnectionPoolAdminServiceImpl.java:938)
... 142 more
Caused by: javax.naming.NameNotFoundException: <redacted> not found
at com.sun.enterprise.naming.impl.TransientContext.doLookup(TransientContext.java:237)
at com.sun.enterprise.naming.impl.TransientContext.lookup(TransientContext.java:204)
at com.sun.enterprise.naming.impl.TransientContext.lookup(TransientContext.java:208)
at com.sun.enterprise.naming.impl.TransientContext.lookup(TransientContext.java:208)
at com.sun.enterprise.naming.impl.SerialContextProviderImpl.lookup(SerialContextProviderImpl.java:70)
at com.sun.enterprise.naming.impl.LocalSerialContextProviderImpl.lookup(LocalSerialContextProviderImpl.java:114)
at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:483)
... 148 more
Кроме того, в server.log есть следующие сообщения RAR.Кажется, что это происходит только тогда, когда использование превышает определенную сумму:
RAR6021: Failed to remove the MCF : <connection_pool>RAR5114: Error allocating connection : [Task already scheduled or cancelled]RAR5114 : Error allocating connection : [null]RAR5114 : Error allocating connection : [This pool is not registered with the runtime environment : <connection_pool>]RAR5114 : Error allocating connection : [This pool is not bound in JNDI : <connection_pool><connection_pool>]RAR7010: Pool not reachable.
Существует также один экземпляр RAR6021 :Failed to create MCF : <connection_pool> в один день;однако я не уверен, что это связано.
Что может быть причиной этого?Существуют ли методы, которые можно использовать для диагностики проблемы?