Community
Participate
Working Groups
I'd like to bring Lucene 5.x to Orbit for Neon M1. This has been announced on cross-project-issues-dev to make other Eclipse projects aware that there will be two versions of Lucene around starting from Neon M1. This bug is for tracking progress and discussing potential issues or complains.
I would rather see single version used and do what we can to get platform.ua using 5.x for Neon M1 which should be much simpler if you take care of the the Orbit/cq stuff :).
Is Lucene 5.x backwards compatible or are there breaking changes that clients have to adopt?
(In reply to Dani Megert from comment #2) > Is Lucene 5.x backwards compatible or are there breaking changes that > clients have to adopt? It contains API breaking changes. The concept around documents changed, several constants moved etc. But it's easy to convert existing code.
(In reply to Alexander Kurtakov from comment #1) > I would rather see single version used and do what we can to get platform.ua > using 5.x for Neon M1 which should be much simpler if you take care of the > the Orbit/cq stuff :). So you take care of platform / help if I do the CQs? Deal. :-)
(In reply to Marcel Bruch from comment #3) > (In reply to Dani Megert from comment #2) > > Is Lucene 5.x backwards compatible or are there breaking changes that > > clients have to adopt? > > It contains API breaking changes. The concept around documents changed, > several constants moved etc. But it's easy to convert existing code. This makes it very unlikely that the Platform will adopt it for SR1.
(In reply to Dani Megert from comment #5) > (In reply to Marcel Bruch from comment #3) > > (In reply to Dani Megert from comment #2) > > > Is Lucene 5.x backwards compatible or are there breaking changes that > > > clients have to adopt? > > > > It contains API breaking changes. The concept around documents changed, > > several constants moved etc. But it's easy to convert existing code. > > This makes it very unlikely that the Platform will adopt it for SR1. To clarify for Platform work I speak for Neon release train - updates for SR1 are not something I even thought of. My initial analysis shows that such an update would be internal to o.e.help.base and should be possible to do all the changes with no API churn to the bundle.
What are the expected benefits of Lucene 5 ? I only know Lucene being used in the Eclipse Help search, and many of our customers complain about the quality of search results not being good -- it does find the search strings that one types, but there seems to be no page ranking at all related to relevance of the documents found. Not even whether the search string is in the headline / title of a document or somewhere in the middle or how often it appears in the document. I don't have data at hand right now to prove this, but it's what I've heard from customers ... would love to see this situation improved. Is this something one could get from the Lucene Update or would Lucene extensions need to be implemented to make it better aware of the help xhtml's semantics of relevance ?
(In reply to Martin Oberhuber from comment #7) > What are the expected benefits of Lucene 5 ? > > I only know Lucene being used in the Eclipse Help search, and many of our > customers complain about the quality of search results not being good -- it > does find the search strings that one types, but there seems to be no page > ranking at all related to relevance of the documents found. Not even whether > the search string is in the headline / title of a document or somewhere in > the middle or how often it appears in the document. > > I don't have data at hand right now to prove this, but it's what I've heard > from customers ... would love to see this situation improved. Is this > something one could get from the Lucene Update or would Lucene extensions > need to be implemented to make it better aware of the help xhtml's semantics > of relevance ? AFAICT, platform.ua use a really simple html stripper that is copied from really old lucene version(1.x?) and it was shipped as demo in lucene. Improving the search semantics would need changes in this code and this lucene update would not magically improve it but is a prereq for any such work in my eyes.
I am investigating the platform migration to Lucene jars and it would be nice to start with the latest Lucene that we have right now which is 5.1.0. We might need a couple of additional jars because of the migration (e.g. chinese analyzer which has moved to separate bundle and a lucene-provided parser). I'll make a list shortly about what jars we need.
(In reply to Sopot Cela from comment #9) > I am investigating the platform migration to Lucene jars and it would be > nice to start with the latest Lucene that we have right now which is 5.1.0. > We might need a couple of additional jars because of the migration (e.g. > chinese analyzer which has moved to separate bundle and a lucene-provided > parser). I'll make a list shortly about what jars we need. Sopot, would you please create platform.ua bug for the update which depends on this one? This would make it easier to further discuss the platform.ua specifics.
So what's the status of this Marcel? I have some input on a list of bundles that we would need in order to do migration properly. https://bugs.eclipse.org/bugs/show_bug.cgi?id=466829#c1 has some info on what is needed in broad terms. The list of bundles is (all version 5.1): lucene-core lucene-analyzers-common lucene-analyzers-smartcn : Needed to support a decent Chinese analyzer lucene-queryparser : Lucene now has a nice multi-field query parser which is really useful. It's better to have this than do parsing ourselves in o.e.h.base code. lucene-highlighter : There are some issues presently with the highlighting mechanism when you use wildcards. It tokenizes 'around' wildcards and highlights those tokens, not the actual match. So for te?t it would highlight 'te' and 't' rather than "test" or "text". Using this bundle (and it's dependencies, described below) I managed to have a full-featured highlighter which does the highlighting right (previously it was custom-coded). lucene-join: Needed by highlighter. lucene-memory: Same lucene-queries: Same The reason on these dependencies is because Lucene 'rewrites' a wildcard query in base queries and for that it needs these bundles to do the work. I think these bundles are definitely needed if we want to do the migration properly as there is a lot of code on o.e.h.base which is provided out-of-the box by Lucene (and probably better-suited). Let me know if you need help with the CQs.
I have an update on the needed bundles. I decided for now to drop the highlighting enhancements as it turned out that in order to do proper highlighting of wildcard matches Lucene needs the fields to be also stored, not just indexed (see http://lucene.472066.n3.nabble.com/Highlighting-for-non-stored-fields-td1773015.html and https://wiki.apache.org/solr/FieldOptionsByUseCase). This means that for the first pass of the migration there should be no need for the highlighting related bundles and that brings the list to : lucene-core lucene-analyzers-common lucene-analyzers-smartcn lucene-queryparser I've put a temporary copy (to test https://git.eclipse.org/r/#/c/48360/ as the build is expected to fail) here if you want to try it out : https://sopotc.fedorapeople.org/Lucene5MigrationJars/
(In reply to Sopot Cela from comment #12) > I have an update on the needed bundles. I decided for now to drop the > highlighting enhancements as it turned out that in order to do proper > highlighting of wildcard matches Lucene needs the fields to be also stored, > not just indexed (see > http://lucene.472066.n3.nabble.com/Highlighting-for-non-stored-fields- > td1773015.html and https://wiki.apache.org/solr/FieldOptionsByUseCase). > > This means that for the first pass of the migration there should be no need > for the highlighting related bundles and that brings the list to : > > lucene-core > lucene-analyzers-common > lucene-analyzers-smartcn > lucene-queryparser > > I've put a temporary copy (to test https://git.eclipse.org/r/#/c/48360/ as > the build is expected to fail) here if you want to try it out : > https://sopotc.fedorapeople.org/Lucene5MigrationJars/ Marcel, any plan on taking a look at this?
I (just) filed these IP requests for tools.orbit: > > lucene-core https://dev.eclipse.org/ipzilla/show_bug.cgi?id=9967 > > lucene-analyzers-common https://dev.eclipse.org/ipzilla/show_bug.cgi?id=9968 > > lucene-analyzers-smartcn https://dev.eclipse.org/ipzilla/show_bug.cgi?id=9969 > > lucene-queryparser https://dev.eclipse.org/ipzilla/show_bug.cgi?id=9970
David, Gunnar, I remember that there was a discussion going on to use EBR for Orbit. Should I do these Orbit contributions via EBR (preferred) or rather "classical" via CVS (really not preferred) ? If the former, is there some documentation in place how to get started with EBR for Orbit?
(In reply to Marcel Bruch from comment #15) > I remember that there was a discussion going on to use EBR for Orbit. Should > I do these Orbit contributions via EBR (preferred) or rather "classical" via > CVS (really not preferred) ? CVS is still possible. https://wiki.eclipse.org/Orbit/Transition_to_Git_Plan > If the former, is there some documentation in place how to get started with > EBR for Orbit? https://git.eclipse.org/c/orbit/orbit-recipes.git/tree/README.md
We had to rewrite the CQs to eclipse.platform first before creating ATOs. Here they go: http://dev.eclipse.org/ipzilla/show_bug.cgi?id=10145 http://dev.eclipse.org/ipzilla/show_bug.cgi?id=10146 http://dev.eclipse.org/ipzilla/show_bug.cgi?id=10147 http://dev.eclipse.org/ipzilla/show_bug.cgi?id=10148 As soon as we got those approved I can start working on bringing them to Orbit.
New Gerrit change created: https://git.eclipse.org/r/56670 WARNING: this patchset contains 1347 new lines of code and may require a Contribution Questionnaire (CQ) if the author is not a committer on the project. Please see:https://wiki.eclipse.org/Project_Management_Infrastructure/Creating_A_Contribution_Questionnaire
Gerrit change https://git.eclipse.org/r/56670 was merged to [master]. Commit: http://git.eclipse.org/c/orbit/orbit-recipes.git/commit/?id=6ca2ee1f0e99d8592badc14e76ef7498647b3d09
(In reply to Marcel Bruch from comment #17) > We had to rewrite the CQs to eclipse.platform first before creating ATOs. > > Here they go: > > http://dev.eclipse.org/ipzilla/show_bug.cgi?id=10145 > http://dev.eclipse.org/ipzilla/show_bug.cgi?id=10146 > http://dev.eclipse.org/ipzilla/show_bug.cgi?id=10147 > http://dev.eclipse.org/ipzilla/show_bug.cgi?id=10148 FYI, we (Code Recommenders) would also like to use Lucene Queries 5.2.1 and hence bring it to Orbit. I have just filed a corresponding CQ: <https://dev.eclipse.org/ipzilla/show_bug.cgi?id=10240>
Just FYI. The DLTK project has been approved to reuse Lucene Core 5.2.1 [1] and Lucene Analyzers Common 5.2.1 [2]. DLTK will have a new code indexer based on Apache Lucene. For now, these libs are embedded in the requiring DLTK bundle. Once they are available in Orbit we will require them as bundles. [1] https://dev.eclipse.org/ipzilla/show_bug.cgi?id=11192 [2] https://dev.eclipse.org/ipzilla/show_bug.cgi?id=11193
(In reply to Kaloyan Raev from comment #21) > Just FYI. The DLTK project has been approved to reuse Lucene Core 5.2.1 [1] > and Lucene Analyzers Common 5.2.1 [2]. DLTK will have a new code indexer > based on Apache Lucene. > > For now, these libs are embedded in the requiring DLTK bundle. Once they are > available in Orbit we will require them as bundles. > > [1] https://dev.eclipse.org/ipzilla/show_bug.cgi?id=11192 > [2] https://dev.eclipse.org/ipzilla/show_bug.cgi?id=11193 Thanks for letting us know. I checked the platform, and we seem to correctly specify upper and lower bounds (e.g. "[3.5.0,4.0.0)")) so don't think it would impact "us" such as when part of EPP. Plus, the few I had loaded were not singleton's. Would be good if you double checked? Plus, might send a message to the cross-project list if you think many other depend on Lucene. (Or, I suppose, directly check content.jar in Neon staging, if you had the time and energy). Thanks again for letting us know.
This went worse than I hoped. I mean adding Lucene to Orbit using the new recipes way, cvs supposed to be locked and probably many things misunderstood from my side. I wish we went the old cvs way and delivered platform update via bug 466829. Let's hope that Orbit aggregation with the new recipes way works soon enough so we can deliver it for Oxygen M1.
Alexander, can you help me understanding what the pain is with recipes? Is it recipes and Git in general or is the pain more in the slowness for adoption/merge?
(In reply to Gunnar Wagenknecht from comment #24) > Alexander, can you help me understanding what the pain is with recipes? Is > it recipes and Git in general or is the pain more in the slowness for > adoption/merge? There was no pain with recipes(themself). The pain is the unclear state which way to go in order to get smth in orbit blessed repo (in whatever meaning Orbit project decides this) in reliable time. Aka I'm definetely not looking into having platform.ua shipping lucene jars (not even considered this as an option). But at the time all CQs were approved, there was the announcement that cvs is going to be frozen so we went the new recipes way (Roland did it) but months later there is still no repo where we can consume them from and etc. Overall I have the feeling that whatever the benefits of recipes are (Roland convinces me of that) the whole move to it was premature(aka not planned who do what and etc.) effectively leading to blocking adding new stuff to Orbit (or at least we haven't figure what's the right way is).
(In reply to Alexander Kurtakov from comment #25) > Overall I have the feeling that whatever the benefits of recipes are (Roland > convinces me of that) the whole move to it was premature(aka not planned who > do what and etc.) effectively leading to blocking adding new stuff to Orbit > (or at least we haven't figure what's the right way is). I see, I think the issue is staffing/capacity. None of us is able to work full time on the plan we have. This makes the transition much slower then I hoped it would be. Sorry.
(In reply to Gunnar Wagenknecht from comment #26) > (In reply to Alexander Kurtakov from comment #25) > > Overall I have the feeling that whatever the benefits of recipes are (Roland > > convinces me of that) the whole move to it was premature(aka not planned who > > do what and etc.) effectively leading to blocking adding new stuff to Orbit > > (or at least we haven't figure what's the right way is). > > I see, I think the issue is staffing/capacity. None of us is able to work > full time on the plan we have. This makes the transition much slower then I > hoped it would be. Sorry. That's perfectly fine and understandable. My complain is that old(CVS) way should haven't been declared obsolete/frozen and should have stayed as viable option for people needing stuff in orbit fast until the new (recipes) way has been proven to be fully working.
(In reply to Alexander Kurtakov from comment #27) > > ... My complaint is that old(CVS) way > should haven't been declared obsolete/frozen and should have stayed as > viable option for people needing stuff in orbit fast until the new (recipes) > way has been proven to be fully working. Alex, what do you mean "declared obsolete/frozen". I do not think anything came from Orbit saying that. So I am wondering if we have miscommunicated, or if you are going by some of the notes/bugs from webmasters?
(In reply to David Williams from comment #28) > (In reply to Alexander Kurtakov from comment #27) > > > > > ... My complaint is that old(CVS) way > > should haven't been declared obsolete/frozen and should have stayed as > > viable option for people needing stuff in orbit fast until the new (recipes) > > way has been proven to be fully working. > > Alex, what do you mean "declared obsolete/frozen". I do not think anything > came from Orbit saying that. So I am wondering if we have miscommunicated, > or if you are going by some of the notes/bugs from webmasters? From https://wiki.eclipse.org/Orbit/Transition_to_Git_Plan: "Migrate from hand crafted bundles in CVS to templated approach implemented by EBR" and "(October, after Mars SR1) Freeze CVS, stop building bundles out of CVS" . Maybe I read a bit too much but for me these ment that no old style builds from git, cvs frozen in autumn.
(In reply to Alexander Kurtakov from comment #29) > (In reply to David Williams from comment #28) > > (In reply to Alexander Kurtakov from comment #27) > > > > > > > > ... My complaint is that old(CVS) way > > > should haven't been declared obsolete/frozen and should have stayed as > > > viable option for people needing stuff in orbit fast until the new (recipes) > > > way has been proven to be fully working. > > > > Alex, what do you mean "declared obsolete/frozen". I do not think anything > > came from Orbit saying that. So I am wondering if we have miscommunicated, > > or if you are going by some of the notes/bugs from webmasters? > > From https://wiki.eclipse.org/Orbit/Transition_to_Git_Plan: > "Migrate from hand crafted bundles in CVS to templated approach implemented > by EBR" and > "(October, after Mars SR1) Freeze CVS, stop building bundles out of CVS" . > Maybe I read a bit too much but for me these ment that no old style builds > from git, cvs frozen in autumn. Ok, that wiki plan is out of date if not completely invalid. Gunnar, since that "draft plan" was written by you, do you want to decide if it should be "removed"? Or, updated? Since so much of it is wrong, (not only dates but no "migration" of old bundles in cvs to the new system will occur) I would recommend removing it.
Lucene 5.x is in the Orbit contribution for Oxygen M2 ( http://download.eclipse.org/tools/orbit/downloads/drops/S20160916162009/repository/ ). Can this bug be closed now ? Also, having spoken to Sopot, I assume a request for Lucene 6.x will probably be filed soon :)
(In reply to Roland Grunberg from comment #31) > Lucene 5.x is in the Orbit contribution for Oxygen M2 ( > http://download.eclipse.org/tools/orbit/downloads/drops/S20160916162009/ > repository/ ). Can this bug be closed now ? Also, having spoken to Sopot, I > assume a request for Lucene 6.x will probably be filed soon :) Hi Roland. As I said in comment 20, Code Recommenders would like to see org.apache.lucene.queries in Orbit alongside org.apache.lucene.core and org.apache.lucene.queryparser. The CQ 10161 for lucene-queries 5.2.1 has been approved, but I didn't see it in the repository you pointed to yet: > org.apache.lucene.analyzers-common=5.2.1.v20160301-1110 > org.apache.lucene.analyzers-common.source=5.2.1.v20160301-1110 > org.apache.lucene.analyzers-smartcn=5.2.1.v20160301-1110 > org.apache.lucene.analyzers-smartcn.source=5.2.1.v20160301-1110 > org.apache.lucene.core=5.2.1.v20160301-1110 > org.apache.lucene.core.source=5.2.1.v20160301-1110 > org.apache.lucene.queryparser=5.2.1.v20160301-1110 > org.apache.lucene.queryparser.source=5.2.1.v20160301-1110 What needs to be done (and by whom) to make this a reality, after the legal paperwork has apparently been done a while ago already?
(In reply to Andreas Sewe from comment #32) > What needs to be done (and by whom) to make this a reality, after the legal > paperwork has apparently been done a while ago already? There needs to be an Add-To-Orbit CQ and the bundle recipe should be added here: http://git.eclipse.org/c/orbit/orbit-recipes.git/tree/apache/lucene Are you an Orbit committer? Can you create the CQ?
(In reply to Gunnar Wagenknecht from comment #33) > (In reply to Andreas Sewe from comment #32) > > What needs to be done (and by whom) to make this a reality, after the legal > > paperwork has apparently been done a while ago already? > > There needs to be an Add-To-Orbit CQ and the bundle recipe should be added > here: > http://git.eclipse.org/c/orbit/orbit-recipes.git/tree/apache/lucene > > Are you an Orbit committer? No, I am not. But AFAIK I don't have to be, now that there is Gerrit. So I can prepare the recipe. > Can you create the CQ? Why are CQs 10161 and 10240 not sufficient for this?
(In reply to Andreas Sewe from comment #34) > > Are you an Orbit committer? > > No, I am not. But AFAIK I don't have to be, now that there is Gerrit. So I > can prepare the recipe. Cool. That's how it should work. However, I'm afraid to have to be an committer for filing CQ. :) > > Can you create the CQ? > > Why are CQs 10161 and 10240 not sufficient for this? Those are the ones for projects. We need one for "adding CQ 10161" to Orbit. They are called ATO CQs. Then everyone else using the library can PB on this one. Filed it as CQ12009. Please reference that one in the recipe's ip log.
(In reply to Gunnar Wagenknecht from comment #35) > (In reply to Andreas Sewe from comment #34) > > > Are you an Orbit committer? > > > > No, I am not. But AFAIK I don't have to be, now that there is Gerrit. So I > > can prepare the recipe. > > Cool. That's how it should work. However, I'm afraid to have to be an > committer for filing CQ. :) Pushed to Gerrit [1]. Let's discuss the change further over there. [1] <https://git.eclipse.org/r/81765>
(In reply to Andreas Sewe from comment #36) > Pushed to Gerrit [1]. Let's discuss the change further over there. > > [1] <https://git.eclipse.org/r/81765> Anyone has time for this? Gunnar? Roland?
Lucene 5.x has been in Orbit now for some time so I wil probably close this soon. Bug 466829 is focusing on bringing Lucene 6.1.0 into Orbit. The bundles to be included are all from 5.2.1 except lucene-queries (it seems this isn't necessary for platform.ua) If Code Recommenders has no interest in upgrading, that's fine as long as you're explicit about which version you're taking from Orbit in your builds.
(In reply to Roland Grunberg from comment #38) > If Code Recommenders has no interest in upgrading, that's fine as long as > you're explicit about which version you're taking from Orbit in your builds. Yes, we use Import-Package with [5.2,6) version ranges, so we should be fine.
Marking as CLOSED (FIXED). I'll say it's in M3 since lucene-queries made it in for that particular milestone.
