Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 366102 - Connection leak during reconnect
Summary: Connection leak during reconnect
Status: NEW
Alias: None
Product: z_Archived
Classification: Eclipse Foundation
Component: Eclipselink (show other bugs)
Version: unspecified   Edit
Hardware: PC All
: P2 normal with 7 votes (vote)
Target Milestone: ---   Edit
Assignee: Project Inbox CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-12-08 15:06 EST by Nobody - feel free to take it CLA
Modified: 2022-06-09 10:09 EDT (History)
5 users (show)

See Also:


Attachments
Stacktrace from connection leak (4.36 KB, text/plain)
2011-12-08 15:12 EST, Nobody - feel free to take it CLA
no flags Details
Set of OSGi bundles that run automatic test inside GlassFish (43.58 KB, application/octet-stream)
2011-12-08 20:41 EST, Nobody - feel free to take it CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Nobody - feel free to take it CLA 2011-12-08 15:06:49 EST
Build Identifier: 2.2.1.v20110722-r9776

I've just run some stress-tests with our application inside GlassFish 3.1.

The application uses completely container-managed JPA EntityManagers inside stateless EJBs. Usually we don't see this, but under a real stress test we saw this:

A potential connection leak detected for connection pool pgXXXPool.

From the documentation for GlassFish, the stacktrace (attached) points to the source of the connection leak.

Looking into the EclipseLink code, the DatasourceAccessor.connectInternal method looks suspicious to me. Also within the attached stacktrace I couldn't find any code that returns the connection back to the pool of the container.

The initial reason for this was, that the underlying physical connection has been invalidated by the container (option for connection reuse limit), which resulted in this message:

"Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query."

The retry than works and the application is not affected (apart from the delay).



Reproducible: Always
Comment 1 Nobody - feel free to take it CLA 2011-12-08 15:12:58 EST
Created attachment 208117 [details]
Stacktrace from connection leak

The attached stacktrace is one example of the many we received during our application stress testing.

All stacktraces (coming from different EJBs and executing different things) share the following lines:

org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:126)
org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:94)
org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)
org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:330)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:291)
org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:565)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1434)
org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:305)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:570)
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:526)
org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:1729)
org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:566)
Comment 2 Nobody - feel free to take it CLA 2011-12-08 20:41:15 EST
Created attachment 208132 [details]
Set of OSGi bundles that run automatic test inside GlassFish

I've also got it reproducible with the current GlassFish 3.1.2-b12 and a little test setup:

1.) Download GlassFish from here and unzip it:
http://dlc.sun.com.edgesuite.net/glassfish/3.1.2/promoted/latest-glassfish.zip

2.) Modify glassfish/domains/domain1/config/domain.xml and add the following attributes to the "DerbyPool" <jdbc-connection-pool> tag:

steady-pool-size="1"
idle-timeout-in-seconds="12"
connection-leak-reclaim="true"
statement-leak-timeout-in-seconds="10"
max-pool-size="10"
max-connection-usage-count="10"
connection-leak-timeout-in-seconds="15"
statement-leak-reclaim="true"
fail-all-connections="true"

3.) Extract the set of bundles into:
.../glassfish/domains/domain1/autodeploy/bundles/

4.) Start the derby server:
.../glassfish/$ ./bin/asadmin start-database

5.) Start GlassFish:
.../glassfish/$ ./bin/asadmin start-domain domain1

6.) tail -f glassfish/domains/domain1/logs/server.log, wait some time and have a fun time with a lot of connection leaks.
Comment 3 Tom Ware CLA 2011-12-22 10:35:49 EST
Setting target and priority.  See the following page for the meanings of these fields:

http://wiki.eclipse.org/EclipseLink/Development/Bugs/Guidelines

Community: Please vote for this bug if it is important to you.  Votes are one of the main criteria we use to determine which bugs to fix next.
Comment 4 Marcin Cinik CLA 2013-01-30 11:54:10 EST
I have just stumbled over that issue on my production installation. This bug prevents me from operating using eclipse-link. Is there any way to fix it ?

 Definitely Importance for this issue should be higher in my opinion.
I am using version 2.3.2
Comment 5 Rui Goncalves CLA 2018-01-22 06:19:06 EST
We are also hitting this problem in production.  Here's the stack trace:

org.apache.commons.dbcp2.managed.ManagedConnection.updateTransactionStatus(ManagedConnection.java:125)
  at org.apache.commons.dbcp2.managed.ManagedConnection.<init>(ManagedConnection.java:59)
  at org.apache.commons.dbcp2.managed.ManagedDataSource.getConnection(ManagedDataSource.java:81)
  at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413)
  at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:123)
  at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)
  at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:346)
  at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:307)
  at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:581)
  at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1625)
  at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:321)
  at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:613)
  at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:558)
  at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2002)
  at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:570)


We're using the latest eclipselink version (2.7.1). I think this is a critical defect, provided compromise the normal functioning of an entire system.

Does anyone managed to work around this bug?

Thanks.
Comment 6 Rui Goncalves CLA 2018-01-23 15:44:54 EST
Dear developers,

could you please take action in an analysis of this issue and provide feedback?

Thanks in advance for your support.

Rui
Comment 7 Eclipse Webmaster CLA 2022-06-09 10:09:58 EDT
The Eclipselink project has moved to Github: https://github.com/eclipse-ee4j/eclipselink