Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 366102

Summary: Connection leak during reconnect
Product: z_Archived Reporter: Nobody - feel free to take it <nobody>
Component: EclipselinkAssignee: Project Inbox <eclipselink.foundation-inbox>
Status: NEW --- QA Contact:
Severity: normal    
Priority: P2 CC: marcin, pascalm.deville, rgoncalves, ruebe356, tom.ware
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: All   
Whiteboard:
Attachments:
Description Flags
Stacktrace from connection leak
none
Set of OSGi bundles that run automatic test inside GlassFish none

Description Nobody - feel free to take it CLA 2011-12-08 15:06:49 EST
Build Identifier: 2.2.1.v20110722-r9776

I've just run some stress-tests with our application inside GlassFish 3.1.

The application uses completely container-managed JPA EntityManagers inside stateless EJBs. Usually we don't see this, but under a real stress test we saw this:

A potential connection leak detected for connection pool pgXXXPool.

From the documentation for GlassFish, the stacktrace (attached) points to the source of the connection leak.

Looking into the EclipseLink code, the DatasourceAccessor.connectInternal method looks suspicious to me. Also within the attached stacktrace I couldn't find any code that returns the connection back to the pool of the container.

The initial reason for this was, that the underlying physical connection has been invalidated by the container (option for connection reuse limit), which resulted in this message:

"Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query."

The retry than works and the application is not affected (apart from the delay).



Reproducible: Always
Comment 1 Nobody - feel free to take it CLA 2011-12-08 15:12:58 EST
Created attachment 208117 [details]
Stacktrace from connection leak

The attached stacktrace is one example of the many we received during our application stress testing.

All stacktraces (coming from different EJBs and executing different things) share the following lines:

org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:126)
org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:94)
org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)
org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:330)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:291)
org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:565)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1434)
org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:305)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:570)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:526)
org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:1729)
org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:566)
Comment 2 Nobody - feel free to take it CLA 2011-12-08 20:41:15 EST
Created attachment 208132 [details]
Set of OSGi bundles that run automatic test inside GlassFish

I've also got it reproducible with the current GlassFish 3.1.2-b12 and a little test setup:

1.) Download GlassFish from here and unzip it:
http://dlc.sun.com.edgesuite.net/glassfish/3.1.2/promoted/latest-glassfish.zip

2.) Modify glassfish/domains/domain1/config/domain.xml and add the following attributes to the "DerbyPool" <jdbc-connection-pool> tag:

steady-pool-size="1"
idle-timeout-in-seconds="12"
connection-leak-reclaim="true"
statement-leak-timeout-in-seconds="10"
max-pool-size="10"
max-connection-usage-count="10"
connection-leak-timeout-in-seconds="15"
statement-leak-reclaim="true"
fail-all-connections="true"

3.) Extract the set of bundles into:
.../glassfish/domains/domain1/autodeploy/bundles/

4.) Start the derby server:
.../glassfish/$ ./bin/asadmin start-database

5.) Start GlassFish:
.../glassfish/$ ./bin/asadmin start-domain domain1

6.) tail -f glassfish/domains/domain1/logs/server.log, wait some time and have a fun time with a lot of connection leaks.
Comment 3 Tom Ware CLA 2011-12-22 10:35:49 EST
Setting target and priority.  See the following page for the meanings of these fields:

http://wiki.eclipse.org/EclipseLink/Development/Bugs/Guidelines

Community: Please vote for this bug if it is important to you.  Votes are one of the main criteria we use to determine which bugs to fix next.
Comment 4 Marcin Cinik CLA 2013-01-30 11:54:10 EST
I have just stumbled over that issue on my production installation. This bug prevents me from operating using eclipse-link. Is there any way to fix it ?

 Definitely Importance for this issue should be higher in my opinion.
I am using version 2.3.2
Comment 5 Rui Goncalves CLA 2018-01-22 06:19:06 EST
We are also hitting this problem in production.  Here's the stack trace:

org.apache.commons.dbcp2.managed.ManagedConnection.updateTransactionStatus(ManagedConnection.java:125)
  at org.apache.commons.dbcp2.managed.ManagedConnection.<init>(ManagedConnection.java:59)
  at org.apache.commons.dbcp2.managed.ManagedDataSource.getConnection(ManagedDataSource.java:81)
  at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413)
  at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:123)
  at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)
  at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:346)
  at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:307)
  at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:581)
  at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1625)
  at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:321)
  at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:613)
  at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:558)
  at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2002)
  at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:570)


We're using the latest eclipselink version (2.7.1). I think this is a critical defect, provided compromise the normal functioning of an entire system.

Does anyone managed to work around this bug?

Thanks.
Comment 6 Rui Goncalves CLA 2018-01-23 15:44:54 EST
Dear developers,

could you please take action in an analysis of this issue and provide feedback?

Thanks in advance for your support.

Rui
Comment 7 Eclipse Webmaster CLA 2022-06-09 10:09:58 EDT
The Eclipselink project has moved to Github: https://github.com/eclipse-ee4j/eclipselink