Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 446479

Summary:	[server] Massive number of LockObtainFailedException: Lock obtain timed out exceptions
Product:	[ECD] Orion	Reporter:	Anthony Hunter <ahunter.eclipse>
Component:	Server	Assignee:	Project Inbox <orion.server-inbox>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	P3	CC:	pwebster
Version:	unspecified
Target Milestone:	8.0
Hardware:	PC
OS:	Linux
Whiteboard:

Description Anthony Hunter

2014-10-09 10:47:05 EDT

Ken noticed the log.txt file has exploded in size on http://orionhub.org. There is an error in the log over 1.5 million times:
% egrep 'Lock obtain timed out' log.txt | wc
3060101 28305946 801746568

This is the error, not sure if this is an orion issue or a filesystem issue.

2014-10-09 10:35:29.207 [Worker-1544] ERROR o.e.o.internal.server.search.Indexer - Error during searching indexing on file: /home/data/nfs/serverworkspace/st/studioeight/OrionContent/org.eclipse.orion.client/bundles/org.eclipse.orion.client.ui/web/orion/compare/compareHighlighter.js
org.apache.solr.client.solrj.SolrServerException: org.apache.solr.client.solrj.SolrServerException: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/home/data/nfs/serverworkspace/.metadata/.plugins/org.eclipse.orion.server.core.search/v16/data/index/write.lock
	at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:153) ~[org.apache.solr.core_3.5.0.v20120530-0900.jar:na]
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105) ~[na:na]
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:121) ~[org.apache.solr.client.solrj_3.5.0.v20120530-0900.jar:na]
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:106) ~[org.apache.solr.client.solrj_3.5.0.v20120530-0900.jar:na]
	at org.eclipse.orion.internal.server.search.Indexer.indexProject(Indexer.java:277) [org.eclipse.orion.server.search_1.0.0.v20140813-1354.jar:na]
	at org.eclipse.orion.internal.server.search.Indexer.indexWorkspace(Indexer.java:188) [org.eclipse.orion.server.search_1.0.0.v20140813-1354.jar:na]
	at org.eclipse.orion.internal.server.search.Indexer.indexUser(Indexer.java:167) [org.eclipse.orion.server.search_1.0.0.v20140813-1354.jar:na]
	at org.eclipse.orion.internal.server.search.Indexer.run(Indexer.java:371) [org.eclipse.orion.server.search_1.0.0.v20140813-1354.jar:na]
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54) ~[org.eclipse.core.jobs_3.6.0.v20140424-0053.jar:na]
Caused by: org.apache.solr.client.solrj.SolrServerException: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/home/data/nfs/serverworkspace/.metadata/.plugins/org.eclipse.orion.server.core.search/v16/data/index/write.lock
	at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:142) ~[org.apache.solr.core_3.5.0.v20120530-0900.jar:na]
	... 8 common frames omitted
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/home/data/nfs/serverworkspace/.metadata/.plugins/org.eclipse.orion.server.core.search/v16/data/index/write.lock
	at org.apache.lucene.store.Lock.obtain(Lock.java:84) ~[na:na]
	at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1108) ~[na:na]
	at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:83) ~[na:na]
	at org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:101) ~[na:na]
	at org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:171) ~[na:na]
	at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:219) ~[na:na]
	at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61) ~[na:na]
	at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:158) ~[na:na]
	at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:79) ~[na:na]
	at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:58) ~[na:na]
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) ~[na:na]
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:1372) ~[org.apache.solr.core_3.5.0.v20120530-0900.jar:na]
	at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:140) ~[org.apache.solr.core_3.5.0.v20120530-0900.jar:na]
	... 8 common frames omitted

Comment 1 Anthony Hunter

2014-10-09 10:52:31 EDT

(In reply to Anthony Hunter from comment #0)
> This is the error, not sure if this is an orion issue or a filesystem issue.
There is no way this could be a file system issue, the error is limited to the Indexer

Comment 2 Anthony Hunter

2014-10-31 11:23:15 EDT

The problem occurred again on Orionhub, the first error in the chain is actually some kind of Lucine out of memory failure?

I have restarted orionhub.

Exception in thread "Lucene Merge Thread #0" org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: Map failed
        at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:517)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
Caused by: java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:849)
        at org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:265)
        at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
        at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:89)
        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:115)
        at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:705)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4270)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3940)
        at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method)
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:846)
        ... 9 more

Comment 3 Anthony Hunter

2015-01-19 15:20:48 EST

We are no longer using Apache Solr on the Orion server for search.