Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 178392 - Tomcat on mylar.eclipse.org has been down repeatedly
Summary: Tomcat on mylar.eclipse.org has been down repeatedly
Status: RESOLVED FIXED
Alias: None
Product: z_Archived
Classification: Eclipse Foundation
Component: Mylyn (show other bugs)
Version: unspecified   Edit
Hardware: PC Windows XP
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: Steffen Pingel CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-03-20 15:38 EDT by meghan CLA
Modified: 2007-05-17 10:51 EDT (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description meghan CLA 2007-03-20 15:38:53 EDT
The Tomcat server installed on mylar.eclipse.org has been down at least 2 times recently.  Based on our conversation on the dev call today, neither Steffen nor I stopped it.  Steffen said that they rebooted the machine once, so it could just be that the machine has been rebooted a couple of times.  We need to figure out why Tomcat is stopping (or crashing) and probably set it to start running when the machine is rebooted.

I wasn't sure what component to put this under, so I just picked monitor.  We're using the Tomcat server for the monitor usage stuff and for some Jira stuff.
Comment 1 Mik Kersten CLA 2007-04-03 11:55:34 EDT
It appears to be down again?  I'm getting a "Bad Gateway" message.
Comment 2 Steffen Pingel CLA 2007-04-03 12:05:48 EDT
Yes, it is not running. From catalina.err: 

30/03/2007 19:41:17 18447 jsvc.exec error: Service did not exit cleanly

Let's try an upgrade to the latest JDK and Tomcat version.
Comment 3 meghan CLA 2007-04-04 13:25:16 EDT
Hi Steffen,

Did you update the server?  or should I? thanks.
Comment 4 Steffen Pingel CLA 2007-04-04 13:48:24 EDT
I'll do it on Friday unless you want to do it before that.
Comment 5 Steffen Pingel CLA 2007-04-06 20:29:50 EDT
Updated as per comment #4. Please reopen if Tomcat fails again.
Comment 6 Steffen Pingel CLA 2007-04-06 20:30:05 EDT
Marking as resolved.
Comment 7 meghan CLA 2007-05-02 16:48:36 EDT
Tomcat was just down again... I checked the uptime of mylar.eclipse.org in case Tomcat is just going down because the server is being rebooted, but the server has been up for 58 days.  I restarted it, but we should figure out why it keeps crashing.
Comment 8 Steffen Pingel CLA 2007-05-05 12:20:54 EDT
Tomcat is hit by the OOM killer. 

# grep Memory /var/log/messages
[...]
May  5 12:09:56 mylar kernel: Out of Memory: Kill process 4493 (java) score 36291 and children.

I have limited the Java heap memory to 192 MB for now but getting more memory (or at least some swap) for the machine would be prefereable. 
Comment 9 Robert Elves CLA 2007-05-14 15:28:20 EDT
Now mysql is suffering form this.

	May 14 15:05:27 mylar kernel: Out of Memory: Kill process 31779 (mysqld) score 31265 and children.

I see what appears to be two instances of tomcat running on mylar.eclipse.org?
Comment 10 Steffen Pingel CLA 2007-05-14 15:43:19 EDT
Sorry, I think I started the second instance, not realizing you were working on the server at the same time. I think we should  configure JIRA to use mysql as well instead of the in memory database. 
Comment 11 Steffen Pingel CLA 2007-05-16 23:48:21 EDT
I made a few changes to the server setup:

 * Downgraded Tomcat to 5.5.23 since I could not get JIRA to run on 6.0.10 without getting crazy amounts of logged errors
 * Upgraded JIRA to 3.9
 * Switched JIRA database from hsql to mysql
 
 Meghan, I copied the monitor webapp to the new Tomcat directory. Please make sure that everything is in place (a test upload of my usage statistics was successful).
 
In addition I have contacted the Eclipse webmaster and asked for additional memory for the server.
Comment 12 Steffen Pingel CLA 2007-05-17 10:51:00 EDT
I have reduced the number of pre-forked apache processes which should help save some memory as well. Current stats:

[root@mylar ~]# free
             total       used       free     shared    buffers     cached
Mem:        484612     315864     168748          0       6152      52088
-/+ buffers/cache:     257624     226988
Swap:            0          0          0

Please reopen the bug if processes are killed again.