Description David Minsky 2011-10-18 17:32:47 EDT

Multiple problems were observed using the EclipseLink Cache Coordination feature with an RMI-JNDI configuration on IBM WebSphere Websphere 8 (8.0.0.1) server. When attempting to look up an EclipseLink RMIRemoteCommandConnectionImpl within JNDI, the code fails with a com.ibm.websphere.naming.CannotInstantiateObjectException : ClassNotFoundException occurred while attempting to deserialize retrieved object.

This occurs regardless of where the EclipseLink library is placed:

- <WAR>/lib/
- <WAR>/classes/
- <EAR>/lib/
- Websphere /ext/ folder
- Websphere shared library

This occurs in all configurations with different WebSphere classloader configurations (application, and global classloader settings) set to both ("Classes loaded with parent class loader first") and ("Classes loaded with local class loader first (parent last)").

To diagnose and investigate this issue, a standalone application (without any EclipseLink code) was created, using a standard RMI "HelloServer" example (attached to this bug). Websphere allowed the binding of the Remote object into JNDI, but the lookup of that object failed with the same exception observed with the lookup failure for EclipseLink's RMIRemoteCommandConnectionImpl:

JNDI Bind:

Properties env = new Properties();
env.put(InitialContext.SECURITY_PRINCIPAL, "wasuser");
env.put(InitialContext.SECURITY_CREDENTIALS, "waspassword");

InitialContext context = new InitialContext(env);
HelloServer helloServer = new HelloServerImpl();
String name = "helloserver";
context.rebind(name, helloServer);

out.println("Bound" + helloServer + " into JNDI with name: " + name + " name in namespace: " + context.getNameInNamespace());

Produced: (expected result)

BoundHelloServerImpl[UnicastServerRef [liveRef: [endpoint:[10.156.52.231:2090](local),objID:[-25669d5f:133182aef88:-7fc9, -989722282943293654]]]] into JNDI with name: helloserver name in namespace: DMINSKY-LAPNode01Cell/nodes/DMINSKY-LAPNode01/servers/server1


JNDI Lookup:

HelloServer helloServerLookedUp = (HelloServer)context.lookup(name);

Produced: (unexpected result)

Exception occurred: com.ibm.websphere.naming.CannotInstantiateObjectException: ClassNotFoundException occurred while attempting to deserialize retrieved object. [Root exception is java.lang.ClassNotFoundException: interface sustaining.test.HelloServer is not visible from class loader]
com.ibm.websphere.naming.CannotInstantiateObjectException: ClassNotFoundException occurred while attempting to deserialize retrieved object. 
[Root exception is java.lang.ClassNotFoundException: interface sustaining.test.HelloServer is not visible from class loader] 
at com.ibm.ws.naming.util.Helpers.processSerializedObjectForLookupExt(Helpers.java:902) 
at com.ibm.ws.naming.util.Helpers.processSerializedObjectForLookup(Helpers.java:796) 
at com.ibm.ws.naming.jndicos.CNContextImpl.processBoundObjectForLookup(CNContextImpl.java:2876) 
at com.ibm.ws.naming.jndicos.CNContextImpl.processResolveResults(CNContextImpl.java:3973) 
at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1875) 
at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1776) 
at com.ibm.ws.naming.jndicos.CNContextImpl.lookupExt(CNContextImpl.java:1433) 
at com.ibm.ws.naming.jndicos.CNContextImpl.lookup(CNContextImpl.java:615) 
at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:165) 
at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:179) 
at org.apache.aries.jndi.DelegateContext.lookup(DelegateContext.java:161) 
at javax.naming.InitialContext.lookup(InitialContext.java:431) 
at sustaining.servlet.JNDIBindServlet.doGet(JNDIBindServlet.java:40) 
at javax.servlet.http.HttpServlet.service(HttpServlet.java:575) 
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) 
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1147) 
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:722) 
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:449) 
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178) 
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1020) 
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3703) 
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304) 
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:953) 
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1655) 
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195) 
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452) 
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511) 
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305) 
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:276) 
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214) 
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113) 
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165) 
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217) 
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161) 
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138) 
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204) 
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775) 
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905) 
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1650) 
Caused by: java.lang.ClassNotFoundException: interface sustaining.test.HelloServer is not visible from class loader 
at com.ibm.ws.util.WsObjectInputStream.resolveProxyClass(WsObjectInputStream.java:281) 
at java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1548) 
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1510) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1749) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1346) 
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:365) 
at com.ibm.ws.naming.util.Serialization$1.run(Serialization.java:127) 
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118) 
at com.ibm.ws.naming.util.Serialization.deserializeObject(Serialization.java:123) 
at com.ibm.ws.naming.util.Helpers.processSerializedObjectForLookupExt(Helpers.java:887) ... 38 more 
Caused by: java.lang.IllegalArgumentException: interface sustaining.test.HelloServer is not visible from class loader 
at java.lang.reflect.Proxy.getProxyClass(Proxy.java:365) 
at com.ibm.ws.util.WsObjectInputStream.resolveProxyClass(WsObjectInputStream.java:277) ... 47 more 


This result was obtained regardless of the classloader configuration, and the location of the HelloServer/HelloServerImpl code.

When an attempt is made to bind in a non Remote object (not subclassing UnicastRemoteObject) WebSphere reports an error. However, binding a non-EJB UnicastRemoteObject subclass is successful (see above).

An additional issue is remote lookup with WebSphere's namespaced JNDI tree. Currently, by default EclipseLink generates a unique String identifier within a ServiceId, and binds a RMIRemoteCommandConnection into JNDI with that id. To remotely look up an object in WebSphere JNDI, the full namespace must also be specified, and so the current code cannot lookup an object bound in a remote WebSphere JNDI tree. To obtain the context namespace in WAS, the following Context/InitialContext code can potentially be used:

context.getNameInNamespace();

ServiceId does not currently have a mechanism to allow the user to specify a namespace name, and a subclass cannot easily be substituted, because the ServiceAnnouncement class is used to serialize and unserialize the contents of the ServiceId for the purposes of announcing a session's availability. (Note also that with a larger ServiceAnnouncement implementation, DiscoveryManager.startListening() needs to have its byte[] buffer size increased to account for the added number of bytes to send across the wire).

Because of these issues, the current Cache Coordination RMI-JNDI configuration cannot be integrated with WebSphere. A way to get RMI cache coordination working within WAS is to not utilize JNDI and instead use an RMI-RMIRegistry configuration:

e.g. (SessionCustomizer customize(Session session) code)

import java.rmi.RemoteException;
import java.rmi.registry.LocateRegistry;
import java.rmi.registry.Registry;

...

DatabaseSessionImpl dbSession = (DatabaseSessionImpl)session;
RemoteCommandManager rcm = new RemoteCommandManager(dbSession);

int port = 1099;
try {
	System.out.println("Creating RMI Registry on port " + port);
	Registry registry = LocateRegistry.createRegistry(port); // RMI registry
	System.out.println("Created: " + registry);
} catch (RemoteException r) {
	System.out.println("RMI registry was not created on port " + port);
	System.out.println(r.getMessage());
}

RMITransportManager transport = new RMITransportManager(rcm);
rcm.setTransportManager(transport);

rcm.setUrl("rmi://localhost:" + port);
rcm.setChannel("AChannelNameOfYourChoosing");
rcm.setShouldPropagateAsynchronously(true);

transport.setNamingServiceType(TransportManager.REGISTRY_NAMING_SERVICE);
transport.setShouldRemoveConnectionOnError(true);

dbSession.setCommandManager(rcm);
dbSession.setShouldPropagateChanges(true);