Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 327623 - OpenMRS trunk fails to load - org.apache.jasper.JasperException: /index.jsp(1,1) Unable to initialize TldLocationsCache: Illegal class loader binding
Summary: OpenMRS trunk fails to load - org.apache.jasper.JasperException: /index.jsp(1...
Status: CLOSED WONTFIX
Alias: None
Product: Virgo
Classification: RT
Component: unknown (show other bugs)
Version: unspecified   Edit
Hardware: PC Linux
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: Glyn Normington CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-10-12 22:58 EDT by Misha Koshelev CLA
Modified: 2010-10-22 03:42 EDT (History)
1 user (show)

See Also:


Attachments
Error report from going to http://localhost:8080/openmrs for the first time (3.90 KB, text/html)
2010-10-12 22:59 EDT, Misha Koshelev CLA
no flags Details
virgo-web-server-2.1.0.M06-incubation/serviceability/logs/openmrs-0.0.0/log.log (33.24 KB, text/x-log)
2010-10-12 23:00 EDT, Misha Koshelev CLA
no flags Details
openmrs page returned when Glyn tried to reproduce the problem (143.29 KB, image/png)
2010-10-13 03:30 EDT, Glyn Normington CLA
no flags Details
result of tar cvf openmrs.tar ~/.OpenMRS (10.00 KB, application/x-tar)
2010-10-14 00:31 EDT, Misha Koshelev CLA
no flags Details
result of mysqldump --user=root --password=openmrs --all-databases > mysqldump.alldatabases (740.60 KB, application/octet-stream)
2010-10-14 00:32 EDT, Misha Koshelev CLA
no flags Details
result of mysqldump --user=root --password=openmrs openmrs > mysqldump.alldatabases (274.92 KB, application/octet-stream)
2010-10-14 00:32 EDT, Misha Koshelev CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Misha Koshelev CLA 2010-10-12 22:58:50 EDT
Build Identifier: 2.1.0.M06-incubation

OpenMRS trunk functions well on standalone Tomcat.

However, when deployed through the Web interface on a clean install of Virgo Web Server, it fails to deploy with indicated error.

Thank you
Misha

Reproducible: Always
Comment 1 Misha Koshelev CLA 2010-10-12 22:59:22 EDT
Created attachment 180718 [details]
Error report from going to http://localhost:8080/openmrs for the first time
Comment 2 Misha Koshelev CLA 2010-10-12 23:00:07 EDT
Created attachment 180719 [details]
virgo-web-server-2.1.0.M06-incubation/serviceability/logs/openmrs-0.0.0/log.log

Relevant serviceability log
Comment 3 Misha Koshelev CLA 2010-10-12 23:05:59 EDT
Here is a nightly build that can reproduce the problem. Thank you.

Misha

http://sourceforge.net/projects/openmrsdev/files/nightly/2010-10-12/openmrs.war/download
Comment 4 Glyn Normington CLA 2010-10-13 03:29:12 EDT
Well I tried to reproduce and it seemed to work ok. I'll attach a screenshot of the page that was presented. So perhaps there is an environmental difference??

My environment:

Virgo web server 2.1.0.M06-incubation

java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-10M3065)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode)

Mac OS X 10.6.4
Comment 5 Glyn Normington CLA 2010-10-13 03:30:08 EDT
Created attachment 180727 [details]
openmrs page returned when Glyn tried to reproduce the problem
Comment 6 Misha Koshelev CLA 2010-10-13 15:43:45 EDT
I have to admit I'm a little stumped at the moment.

In a 32-bit Ubuntu 10.04 Server VM all works well either with open or sun java 6 JDK.

On my machine (64-bit Ubuntu 10.04 _Desktop_ - technically _lubuntu_ desktop), I still get the _same_ error with open or sun java 6 JDK. I've pretty much purged my entire set-up so I don't think it's that but just in case I'm downloading an Ubuntu 64-bit 10.04 iso).

Thank you
Misha
Comment 7 Misha Koshelev CLA 2010-10-13 17:43:35 EDT
Ok quite frankly I am stumped.

I did _not_ encounter the problem in a VMWare Ubuntu 10.04 64 bit desktop image using Sun JDK.

Furthermore, there is one additional issue that might be a hint.

Specifically, on _my_ machine, I actually get:
java.lang.OutOfMemoryError: PermGen space
when I try to deploy the OpenMRS artifact. I work around this per the instructions here:
http://www.eclipse.org/virgo/documentation/virgo-documentation-2.1.0.M06-incubation/docs/virgo-user-guide/htmlsingle/virgo-user-guide.html#known-issues-perm-gen-sun-vm
export JAVA_OPTS="-XX:MaxPermSize=128M"

_then_ I get the Jasper error.

This is completely consistent on my machine :(

I don't get any PermGen issues in any of the VMs. I'm not really sure how to diagnose the problem.

I've purged and reinstalled all Java packages (or anything possibly related to Java) but no change :(

Thank you
Misha
Comment 8 Misha Koshelev CLA 2010-10-13 17:48:55 EDT
Ok sorry for fast second post, but I figured it out. I have an OpenMRS setup already on _my_ machine. If I simply rename the ~/.OpenMRS folder (which is where OpenMRS stores its setup), I get to the initial setup wizard as well.

I suspect you need to install mysql and go through the setup first. I will investigate later.

Thank you
Misha
Comment 9 Misha Koshelev CLA 2010-10-14 00:30:52 EDT
To reproduce:
1. Install mysql if not installed already
2. Download openmrs.war:
http://sourceforge.net/projects/openmrsdev/files/nightly/2010-10-12/openmrs.war/download
3. Install artifact
4. Go to http://localhost:8080/openmrs
5. Proceed through step by step setup setup
http://wiki.openmrs.org/display/docs/Step+6+-+Configuring+OpenMRS
6. Wait for database to create (sorry)
7. Works fine first time... now:
8. Uninstall openmrs.war from GUI

9. Re-install artifact
10. Go to http://localhost:8080/openmrs.

Voila!

If you would like to _just_ do steps 9 and 10, I will attached:
openmrs.tar - this needs to be extracted using:
tar xvf openmrs.tar
in user's home directory and creates .OpenMRS folder.
.OpenMRS/openmrs-runtime.properties points to database user and password.

mysqldump.alldatabases is a dump of all mysql databases. You can re-create by doing:
mysql --user=root --password=yourinitialmysqlpassword < mysqldump.alldatabases 
echo flush privileges | mysql --user=root --password=yourinitialmysqlpassword

new root mysql password will be openmrs

mysqldump.openmrs is a dump _only_ of the openmrs database.

If you do this, you will have to create a mysql user:
openmrs_user
with password (below is a password, not a hash)
6D7a4&0XjZc|

and grant them access to the openmrs database.

Thank you
Misha
Comment 10 Misha Koshelev CLA 2010-10-14 00:31:22 EDT
Created attachment 180841 [details]
result of tar cvf openmrs.tar ~/.OpenMRS
Comment 11 Misha Koshelev CLA 2010-10-14 00:32:08 EDT
Created attachment 180842 [details]
result of mysqldump --user=root --password=openmrs --all-databases > mysqldump.alldatabases
Comment 12 Misha Koshelev CLA 2010-10-14 00:32:38 EDT
Created attachment 180843 [details]
result of mysqldump --user=root --password=openmrs openmrs > mysqldump.alldatabases
Comment 13 Glyn Normington CLA 2010-10-14 05:59:59 EDT
Well done for reproducing the problem. Unfortunately I'm too busy to install myqsl and take the other steps to reproduce, so I wonder if you could do some debugging yourself?

I would start Virgo with the -debug option, attach the Eclipse debugger, add a breakpoint for JasperException, re-create the problem, and then focus on the stack frame:

org.apache.jasper.compiler.Parser.processIncludeDirective(Parser.java:300)

for starters as that seems to initiate the diagnostics. The code there is:

err.jspError(start, ex.getMessage());

so I would find out the type of ex and then add a breakpoint for that type and either reproduce from scratch or, if you're feeling lucky, drop to frame on the above stack frame and let the thread run to re-issue the same exception. Follow your nose and hopefully you'll get to the point where "Illegal class loader binding" is detected. You may then need some help debugging class loading, which I can probably help with.

Hope that helps and good luck!
Comment 14 Misha Koshelev CLA 2010-10-14 13:08:15 EDT
(In reply to comment #13)
Thanks. Will do shortly.

Perhaps I am missing something simple wrt the Eclipse debugger, but do I understand correctly that I need to have the source project in order to actually attach the debugger?

I have lots of history with Java programming but am fairly new to Eclipse (I am an Emacs user traditionally).

Thank you
Misha
Comment 15 Misha Koshelev CLA 2010-10-14 14:43:55 EDT
(In reply to comment #14)
> (In reply to comment #13)
> Thanks. Will do shortly.
> 
> Perhaps I am missing something simple wrt the Eclipse debugger, but do I
> understand correctly that I need to have the source project in order to
> actually attach the debugger?
> 
> I have lots of history with Java programming but am fairly new to Eclipse (I am
> an Emacs user traditionally).
> 
> Thank you
> Misha

Nevermind. Got it. Thank you.
Comment 16 Glyn Normington CLA 2010-10-14 15:41:36 EDT
It took years before I would use Eclipse instead of emacs, so I can sympathise. ;-)
Comment 17 Misha Koshelev CLA 2010-10-14 18:53:25 EDT
For what it's worth, the same exact error occurs in springsource-dm-server-2.0.3.RELEASE, so it's clearly not a regression from that release.

Misha
Comment 18 Misha Koshelev CLA 2010-10-14 19:04:23 EDT
Ok I must admit I'm a little stumped. I'm able to debug Virgo Web Server in Eclipse either with or without the SpringSource dm Server Tools but I still can't add an exception breakpoint for JasperException.

There seem to be no Exceptions available at all in "Add Java Breakpoint Exception." I suspect this is as I have no relevant source files.

For what it's worth the SpringSource Enterprise Bundle repo only has the binary jar:
http://www.springsource.com/repository/app/bundle/version/detail?name=com.springsource.org.apache.jasper.springsource&version=6.0.29.S2-r1559&searchType=bundlesByName&searchQuery=jasper

and I'm not sure where to get the sources so Eclipse will allow me to add the exception.

I can get to the Exception via jdb and catch it (and do "where" command to dump stack), but then I'm in the boat where I have no sources to look at the corresponding lines :(

Any hints much appreciated. Clearly I need some Virgo Web Server debugging hints, etc.

Misha

p.s. I've actually tried other ways to deploy openmrs.war within an OSGi web environment, including Pax Web, but they don't quite seem to work quite as advertised for even mvc-basic example :(
Comment 19 Glyn Normington CLA 2010-10-15 00:25:03 EDT
I've put the Tomcat 6.0.29 (+ SpringSource mods) source tree here for you to download:

http://dl.dropbox.com/u/2487064/6029_08%20Sep.zip

This came from the Eclipse Contribution Questionnaire for Virgo's dependency on that code, but since CQs are only visible to Eclipse Committers, I downloaded it.
Comment 20 Misha Koshelev CLA 2010-10-15 12:55:35 EDT
(In reply to comment #19)
Thank you so much! Now I can even see JasperException in Eclipse's "Add Java Exception Breakpoint"
Comment 21 Glyn Normington CLA 2010-10-19 05:44:47 EDT
Hi Misha. How are you getting on in the debugger?
Comment 22 Misha Koshelev CLA 2010-10-19 14:11:37 EDT
My apologies I have become a little side tracked trying to actually modify OpenMRS to be deployable as an OSGi module. I will try to do some of the debugging as well but, based on a careful review and testing of OSGi options available for deploying unmodified WAR files, as well as an understanding that the ultimate goal of this research is actually to make OpenMRS a fully modular OSGi framework, it makes the most sense for the OpenMRS project to deploy OpenMRS within Virgo Web Server as a true OSGi module... at least that is the theory.

Misha
Comment 23 Glyn Normington CLA 2010-10-20 04:08:19 EDT
Fair enough. I'm always a little nervous if I can't keep a running syste, and make incremental changes, but it's your call and maybe modularising OpenMRS right from the start will reap dividends, albeit with a higher initial risk. Good luck!
Comment 24 Misha Koshelev CLA 2010-10-20 22:45:17 EDT
(In reply to comment #23)
> Fair enough. I'm always a little nervous if I can't keep a running syste, and
> make incremental changes, but it's your call and maybe modularising OpenMRS
> right from the start will reap dividends, albeit with a higher initial risk.
> Good luck!

Thanks. I am so far having some major problems with the modularity approach:
http://tickets.openmrs.org/browse/TRUNK-1596?focusedCommentId=162728&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_162728
Specifically, since we have separate web and api jars that contain all classes, and our webapp contains the webapp (web.xml and spring beans), we seem to have some classloader issues that I can't figure out, and maven bundle plugin doesn't seem to be doing a great job with this setup either (as it can't see any of our imports :( ).

If I have time, I may try to debug the issue within Virgo.

The main problem is that I know the core devs _do not_ want to be dependent on any one server, and it seems (to my naive understanding) that to incrementally go through the Virgo Web Server we will have to necessarily go through the PAR stage.

Perhaps, though, I am wrong.

In any case, thank you for all your helpful suggestions.

Misha
Comment 25 Glyn Normington CLA 2010-10-21 03:58:39 EDT
You're welcome.

One hack is to put each JAR in its own bundle - probably what you are doing already - but make all of the bundles except the web app bundle fragments of the web app bundle. This should side-step class loading issues in a standard way. If that works, you'll be in a position to start breaking out the bundles by changing them back into normal (non-fragment) bundles, which may be more incremental than your current approach.

Meanwhile, what do you want to do with this bug as there's not much we can do with it and it seems a shame to leave it on the books unless someone is going to actively work on it.
Comment 26 Glyn Normington CLA 2010-10-22 03:42:08 EDT
Closing for now. Please reopen if you come back to this approach in the future.