Community
Participate
Working Groups
We are trying to adapt Orion to run on Virgo, the first obstacle we face is the PDE build which is hard to reproduce on a local developer machine. Hence we think it would simplify contributing to Orion if its build would be implemented with CBI (Maven/Tycho). This should simplify running reproducible builds both on developer machines and Hudson for continuous integration and release builds.
I pushed a draft of the maven build to GitHub orion.server: branch tycho: https://github.com/msohn/orion.server/tree/tycho commit https://github.com/msohn/orion.server/commit/5d384dd082716a7ca003c208c482b5fa5022860b orion.client: branch tycho: https://github.com/msohn/orion.client/tree/tycho commit https://github.com/msohn/orion.client/commit/86ca86095e7d1b9a77554101601467bae51f6c76 Instructions how to run maven build https://github.com/msohn/orion.server/blob/tycho/README.md
Very cool! How are things going on this?
Not well on the Orion side. However, we will start looking at this now. We can't continue to use our existing build infrastructure, it's just not usable by others.
I rebased Matthias' patch for orion.server against the latest master branch and was able to successfully build after making some modifications in my clone at: https://github.com/zxiiro/orion.server/commits/tycho Specifically I had to add a new requirement org.eclipse.orion.server.npm: https://github.com/zxiiro/orion.server/commit/1d0aafb994482ff8a8b36a5e7bf413388d5c2ca1 I'm happy to help where I can to move this bug forward.
I am going to rebase my proposal onto current master soon.
I rebased onto current master, resolved conflicts and added pom.xmls for the new server bundles, also fixed the license headers for orion.client pom.xml files. Find the new version here: https://github.com/msohn/orion.client/tree/tycho https://github.com/msohn/orion.server/tree/tycho
(In reply to comment #6) I managed to build Orion. When start Orion I get this: rion\orion.conf 2013-04-15 17:23:02.041 [Start Level Event Dispatcher] INFO org.eclipse.orion.server.config - No server configuration file found at: C:\home\orions\o rion\serverworkspace\orion.conf !SESSION 2013-04-15 17:23:00.646 ----------------------------------------------- eclipse.buildId=@qualifier@ java.fullversion=JRE 1.6.0 IBM J9 2.4 Windows 7 amd64-64 jvmwa6460sr13-20130114_134867 (JIT enabled, AOT enabled) J9VM - 20130114_134867 JIT - r9_20130108_31100 GC - 20121212_AA BootLoader constants: OS=win32, ARCH=x86_64, WS=win32, NL=pl_PL Command-line arguments: -os win32 -ws win32 -arch x86_64 -consoleLog -console -data serverworkspace !ENTRY org.eclipse.equinox.ds 4 0 2013-04-15 17:23:02.821 !MESSAGE Could not bind a reference of component org.eclipse.orion.server.authentication.form. The reference is: Reference[name = GuestUserAdmin, inte rface = org.eclipse.orion.server.useradmin.IOrionGuestCredentialsService, policy = dynamic, cardinality = 0..1, target = null, bind = setGuestUserAdmi n, unbind = unsetGuestUserAdmin] !ENTRY org.eclipse.equinox.ds 4 0 2013-04-15 17:23:02.826 !MESSAGE Could not bind a reference of component org.eclipse.orion.server.authentication.form. The reference is: Reference[name = GuestUserAdmin, inte rface = org.eclipse.orion.server.useradmin.IOrionGuestCredentialsService, policy = dynamic, cardinality = 0..1, target = null, bind = setGuestUserAdmi n, unbind = unsetGuestUserAdmin] !ENTRY org.eclipse.equinox.ds 4 0 2013-04-15 17:23:02.837 !MESSAGE Could not bind a reference of component org.eclipse.orion.server.core.useradmin. The reference is: Reference[name = GuestUserAdmin, interface = org.eclipse.orion.server.useradmin.IOrionGuestCredentialsService, policy = dynamic, cardinality = 0..1, target = null, bind = setGuestUserAdmin, un bind = unsetGuestUserAdmin] !ENTRY org.eclipse.equinox.ds 4 0 2013-04-15 17:23:02.842 !MESSAGE Could not bind a reference of component org.eclipse.orion.server.core.useradmin. The reference is: Reference[name = GuestUserAdmin, interface = org.eclipse.orion.server.useradmin.IOrionGuestCredentialsService, policy = dynamic, cardinality = 0..1, target = null, bind = setGuestUserAdmin, un bind = unsetGuestUserAdmin] osgi> Probably related to latest Mark's changes. When this is fixed, I would like to merge poms to master right after M1.
I also get this error when I try to open http://localhost:8080 !ENTRY org.eclipse.equinox.ds 4 0 2013-04-15 17:23:02.842 !MESSAGE Could not bind a reference of component org.eclipse.orion.server.core.useradmin. The reference is: Reference[name = GuestUserAdmin, interface = org.eclipse.orion.server.useradmin.IOrionGuestCredentialsService, policy = dynamic, cardinality = 0..1, target = null, bind = setGuestUserAdmin, un bind = unsetGuestUserAdmin] osgi> 2013-04-15 17:28:52.372 [qtp438966826-41] WARN o.e.jetty.servlet.ServletHandler - Error for /index.html java.lang.AbstractMethodError: org/eclipse/jetty/http/gzip/CompressedResponseWrapper.newCompressedStream(Ljavax/servlet/http/HttpServletRequest;Ljavax /servlet/http/HttpServletResponse;)Lorg/eclipse/jetty/http/gzip/AbstractCompressedStream; at org.eclipse.jetty.http.gzip.CompressedResponseWrapper.getOutputStream(CompressedResponseWrapper.java:393) ~[org.eclipse.jetty.http_8.1.9.v2 0130131.jar:8.1.9.v20130131] at org.eclipse.equinox.http.servlet.internal.ResourceServlet$1.run(ResourceServlet.java:111) ~[na:na] at java.security.AccessController.doPrivileged(AccessController.java:310) ~[na:na] at org.eclipse.equinox.http.servlet.internal.ResourceServlet.writeResource(ResourceServlet.java:60) ~[na:na] at org.eclipse.equinox.http.servlet.internal.ResourceServlet.service(ResourceServlet.java:50) ~[na:na] at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) ~[javax.servlet_3.0.0.v201112011016.jar:na] at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61) ~[na:na] at org.eclipse.equinox.http.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:38) ~[na:na] at org.eclipse.orion.internal.server.hosting.HostedSiteRequestFilter.doFilter(HostedSiteRequestFilter.java:50) ~[na:na] at org.eclipse.equinox.http.registry.internal.FilterManager$FilterWrapper.doFilter(FilterManager.java:173) ~[na:na] at org.eclipse.equinox.http.servlet.internal.FilterRegistration.doFilter(FilterRegistration.java:81) ~[na:na] at org.eclipse.equinox.http.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:35) ~[na:na] at org.eclipse.orion.server.configurator.servlet.WelcomeFileFilter.doFilter(WelcomeFileFilter.java:99) ~[na:na] at org.eclipse.equinox.http.registry.internal.FilterManager$FilterWrapper.doFilter(FilterManager.java:173) ~[na:na] at org.eclipse.equinox.http.servlet.internal.FilterRegistration.doFilter(FilterRegistration.java:81) ~[na:na] at org.eclipse.equinox.http.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:35) ~[na:na] at org.eclipse.orion.server.configurator.servlet.CacheFilter.doFilter(CacheFilter.java:73) ~[na:na] at org.eclipse.equinox.http.registry.internal.FilterManager$FilterWrapper.doFilter(FilterManager.java:173) ~[na:na] at org.eclipse.equinox.http.servlet.internal.FilterRegistration.doFilter(FilterRegistration.java:81) ~[na:na] at org.eclipse.equinox.http.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:35) ~[na:na] at org.eclipse.orion.server.configurator.servlet.CacheFilter.doFilter(CacheFilter.java:73) ~[na:na] at org.eclipse.equinox.http.registry.internal.FilterManager$FilterWrapper.doFilter(FilterManager.java:173) ~[na:na] at org.eclipse.equinox.http.servlet.internal.FilterRegistration.doFilter(FilterRegistration.java:81) ~[na:na] at org.eclipse.equinox.http.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:35) ~[na:na] at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:77) ~[na:na] at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:226) ~[na:na] at org.eclipse.orion.server.configurator.servlet.ExcludedExtensionGzipFilter.doFilter(ExcludedExtensionGzipFilter.java:42) ~[na:na] at org.eclipse.equinox.http.registry.internal.FilterManager$FilterWrapper.doFilter(FilterManager.java:173) ~[na:na] at org.eclipse.equinox.http.servlet.internal.FilterRegistration.doFilter(FilterRegistration.java:81) ~[na:na] at org.eclipse.equinox.http.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:35) ~[na:na] at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:132) ~[na:na] at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:76) ~[na:na] at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) ~[javax.servlet_3.0.0.v201112011016.jar:na] at org.eclipse.equinox.http.jetty.internal.HttpServerManager$InternalHttpServiceServlet.service(HttpServerManager.java:386) ~[org.eclipse.equi nox.http.jetty_3.0.100.v20130327-1442.jar:na] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669) ~[org.eclipse.jetty.servlet_8.1.9.v20130131.jar:8.1.9.v20130131]
(In reply to comment #8) This is about win32.x86_64 package.
I get the service warnings in Comment 7 when I launch Orion normally from the Eclipse IDE or the current builds using orion.exe (see bug 405740). It's unrelated to the Tycho build.
java.lang.AbstractMethodError: org/eclipse/jetty/http/gzip/CompressedResponseWrapper.newCompressedStream(Ljavax/servlet/http/HttpServletRequest;Ljavax /servlet/http/HttpServletResponse;)Lorg/eclipse/jetty/http/gzip This one likely *is* a problem with the tycho build. If you check I suspect you have a mix of jetty bundles from different jetty releases? I have seen this error in my workspace. Orion really needs to move up to the latest Jetty as consumed by the platform in Kepler, but this may point to some inconsistency in where the Tycho build is pulling pre-requisites from.
(In reply to comment #11) > This one likely *is* a problem with the tycho build. If you check I suspect > you have a mix of jetty bundles from different jetty releases? You are right. I changed the target definition and now it works.
I'll look into updating the target platform. Also feedback on the client would be helpful as I am not a Javascript expert so I am not sure what might be missing. Any hints on how you want to run client side tests (which test framework to use) in the client build ? Should we run some static code checks like jslint in the client build ? Another problem I recognized right now is the current split of server and client build. With this version of the patch both client and server need to be built in one build run which is convenient when running the build locally but isn't so easy on Hudson since it assumes a 1-1 mapping of repository to build jobs i.e. it's not so simple to track and fetch 2 repositories in the same Hudson job. Currently the common parent pom.xml is located in the server repository but also is needed by the client build. I see 2 solutions here: - 1. solution: define 3 jobs - orion.parent job which just builds the parent pom in the server repository and deploys it to Nexus - orion.client job which builds the client and deploys the results to Nexus. For that we need to add another client reactor pom.xml in order to be able to run a maven build for all client bundles. - orion.server job which builds the server, runs the server tests, builds all the features, the p2 repository and the product - 2. solution: introduce git submodules and 1 build job - the super project (requiring another git repository) will just contain the parent pom and other global stuff if there is any - the build job tracks and fetches the super project's git repository and updates all submodules accordingly and the runs the build for everything - disadvantage of this solution is that at least those who are involved in changing the build have to learn how to work with submodules I'd propose we first go for solution 1 and then talk to some other projects using submodules to explore if this approach would be better in the long run.
Matthias, please wait till I merge your commit to master. I already updated the target and fixed licenses. When the code is in master, we can add fixes incrementally.
for the IP records: - I am the author 100% of the content of this patch, some of the ideas came from Michael Ochmann (also SAP employee) hence some files contain also his name in the license header - SAP owns the copyright of the contributed content - We have the right to contribute the content to Eclipse.
Thanks Matthias. I am waiting when 3.0M1 is announces and then I will merge the commit. It will be likely tomorrow morning (in EU).
(In reply to comment #13) > Any hints on how you want to run client side tests (which test framework to > use) in the client build ? We currently run javascript tests using a framework call jstestdriver: http://code.google.com/p/js-test-driver/ We already have a separate hudson job for performing these tests that we could probably continue using in some form (orion-jstest-windows). Currently we just have a job for running tests on windows but we could run Linux tests in the same way. We have had a lock of difficulty with stability of the hudson.eclipse.org windows slave so these tests have been failing recently. This test is kicked off remotely by our build so it can't be run independently. > Should we run some static code checks like jslint in the client build ? That would be nice to have. We have jslint integration in Orion editor but still errors get through. This isn't the highest priority but it would be cool to have it. > Another problem I recognized right now is the current split of server and > client build... Splitting apart client and server builds sounds reasonable. In reality they are pretty tightly related - if you were building the client to be used separately from the Orion server you likely wouldn't bundle it up as OSGi bundles. However multiple jobs could enable us to do things like run the client or server tests automatically on gerrit checkin independently from each other.
I merged client and server changes + made some changes e.g. added license feature.
I pushed another 2 patches to github in order to enable separate build of client and server on Hudson: client change: https://github.com/msohn/orion.client/commit/efcc76f700b0a8fc4439977023b12ca5101d09bd server change: https://github.com/msohn/orion.server/commit/6c49d19d0d5cd3f63a3ec6ae1d367524d434b278 as soon as these changes are in I could setup the corresponding Hudson jobs. I propose to name these jobs "orion.client" and "orion.server"
(In reply to comment #19) In comment for the server commit you say that there is no easy way to fetch from many git repos in one Hudson job. I don't know how Hudson and Jenkins are similar, but in Jenkins there is Multiple Scm plug-in that lets you do it.
When I run mvn clean install on pom in the server bundle, I get this: [ERROR] Internal error: java.lang.RuntimeException: "No solution found because the problem is unsatisfiable.": ["Unable to satisfy dependency from org.eclipse.j git 3.0.0.201304220225 to package javaewah 0.5.6.", "Unable to satisfy dependency from org.eclipse.orion.server.tests.feature.feature.group 1.0.0.v20130419-0910 to org.eclipse.orion.server.tests [1.0.0.v20130419-0910].", "No solution found because the problem is unsatisfiable."] -> [Help 1] org.apache.maven.InternalErrorException: Internal error: java.lang.RuntimeException: "No solution found because the problem is unsatisfiable.": ["Unable to sati sfy dependency from org.eclipse.jgit 3.0.0.201304220225 to package javaewah 0.5.6.", "Unable to satisfy dependency from org.eclipse.orion.server.tests.feature.f eature.group 1.0.0.v20130419-0910 to org.eclipse.orion.server.tests [1.0.0.v20130419-0910].", "No solution found because the problem is unsatisfiable."]
(In reply to comment #21) > When I run mvn clean install on pom in the server bundle, I get this: > This is because the test features are commented out for some reason. https://git.eclipse.org/c/orion/org.eclipse.orion.server.git/tree/pom.xml#n114 lines 114 and 128 - 129, Uncommenting them allowed me to successfully build.
(In reply to comment #22) You are right. I uncommented them because build was failing when subsequent Jenkins tasks were triggered. The problem was Orion instance started each time and using a port. The next time I got an error that the port is already bound. Any idea how to workaround that or fix?
Created attachment 229968 [details] Remove tests patch
(In reply to comment #23) > (In reply to comment #22) > > You are right. I uncommented them because build was failing when subsequent > Jenkins tasks were triggered. The problem was Orion instance started each > time and using a port. The next time I got an error that the port is already > bound. Any idea how to workaround that or fix? I attached a patch in comment #24 which removes the tests requirement from the repository module's category.xml file. This should allow the build to continue without tests that were commented out.
(In reply to comment #20) > (In reply to comment #19) > In comment for the server commit you say that there is no easy way to fetch > from many git repos in one Hudson job. I don't know how Hudson and Jenkins > are similar, but in Jenkins there is Multiple Scm plug-in that lets you do > it. I didn't know multiple Scm plugin. I couldn't find a Hudson variant for that, also the Jenkins home page for this plugin says: "This plugin is more of a proof-of-concept than a robust and fully functional component." and I checked hudson.eclipse.org and found that it's not installed there. Being able to build the Orion client bundles separately also has other advantages, e.g. as soon as the client bundles have been built js tests could be executed and first feedback could be sent back to Gerrit (as soon as we use Gerrit for Orion ;-).
(In reply to comment #23) > (In reply to comment #22) > > You are right. I uncommented them because build was failing when subsequent > Jenkins tasks were triggered. The problem was Orion instance started each > time and using a port. The next time I got an error that the port is already > bound. Any idea how to workaround that or fix? I'll look into that soon, seems we have to wait until the Orion instance is properly shutdown before starting the next test.
(In reply to comment #27) > I'll look into that soon, seems we have to wait until the Orion instance is > properly shutdown before starting the next test. This is what we do in current Orion js tests on hudson: for /F "tokens=2" %%I in ('TASKLIST /NH /FI "Imagename eq orion.exe"') do set ORION_PID=%%I taskkill /f /t /pid %ORION_PID% taskkill /f /im chrome.exe tasklist /v
(In reply to comment #19) > I pushed another 2 patches to github in order to enable separate build of > client and server on Hudson: > > client change: > https://github.com/msohn/orion.client/commit/ > efcc76f700b0a8fc4439977023b12ca5101d09bd > > server change: > https://github.com/msohn/orion.server/commit/ > 6c49d19d0d5cd3f63a3ec6ae1d367524d434b278 > > as soon as these changes are in I could setup the corresponding Hudson jobs. > I propose to name these jobs "orion.client" and "orion.server" I pushed a new version of these 2 commits separating the build of client and server bundles so that they can be built using separate Hudson jobs: orion.client change: https://github.com/msohn/orion.client/commit/4ff96053e6de03692363db63568898e6e6ae5c60 orion.server change: https://github.com/msohn/orion.server/commit/b1d0a0fccd0d3bbeadbd5e6b2bdd2fef1d29533e In addition I pushed another server change enabling the Orion server tests by using the build-helper-maven-plugin to reserve a free port number for running the tests: https://github.com/msohn/orion.server/commit/f2885c712c72c9ca638a3d2ded3664d7a879ba1b
I released all 3 commits + one minor fix in the server parent pom. I think the next step would be to run minification.
(In reply to comment #30) > I released all 3 commits + one minor fix in the server parent pom. I'll setup some Hudson jobs to get a regular Maven CI build up and running. Any preference for naming these jobs ? > I think the next step would be to run minification. Which minifier do you want to use ? It seems the following maven plugin is popular to be used to minify Javscript, it supports YUI or Google Closure Compiler http://samaxes.github.io/minify-maven-plugin/ Do you want a build option to skip minification to simplify debugging ? Regarding static checks for javascript sources: do you use jslint or jshint ?
(In reply to comment #31) > Which minifier do you want to use ? It seems the following maven plugin is > popular > to be used to minify Javscript, it supports YUI or Google Closure Compiler > http://samaxes.github.io/minify-maven-plugin/ > > Do you want a build option to skip minification to simplify debugging ? We currently use r.js [1], which is a minification tool that is aware of the AMD module syntax we use in Orion. Under the covers it actually uses google closure for the minification. It looks like there is a Maven plugin for it [2] that has had some positive reviews [3]. If you could get an initial hudson job setup, either myself or someone here should be able to help with getting the minification going. [1] https://github.com/jrburke/r.js/ [2] https://github.com/mcheely/requirejs-maven-plugin [3] http://pseudobry.com/maven-and-r-js-optimization/
I'm adding our current student to this bug. He can help out after you get the build added.
When we have the job ready, will it publish the artifacts in some public repo?
(In reply to comment #34) > When we have the job ready, will it publish the artifacts in some public > repo? I'll configure it to deploy artifacts to Nexus available at http://repo.eclipse.org
Bug 408169: requested Nexus repositories for Orion
Bug 408171: requested Hudson jobs for Orion
I pushed 2 more changes to enable using Nexus: - orion client: https://github.com/msohn/orion.client/commit/42fd6601609c4b30c241c7dee0a62001875ea81f - orion server: https://github.com/msohn/orion.server/commit/048efe3a705fa33a02a93a88f3c0d8e32ee0045f
(In reply to comment #38) > I pushed 2 more changes to enable using Nexus: Both commits merged to master.
we've made some progress: - Hudson jobs created [1] - client job works and deploys artifacts to Nexus [2] still some open issues: The orion-server job can't directly consume the client bundles from Nexus, Tycho can only consume the client bundles from a p2 repository hence the client job needs to creates a p2 repository. I propose the following solution: - we move the client feature and the doc plugins (which are packed into the client feature) to orion.client. If you don't want to move the doc bundles we could put them into a separate feature in orion.server. - and create a client p2 repo which the server build can then consume via Nexus. Let me know if this approach is ok. [1] https://hudson.eclipse.org/hudson/job/orion-client/ https://hudson.eclipse.org/hudson/job/orion-server/ [2] https://repo.eclipse.org/content/repositories/orion-snapshots/org/eclipse/orion/
(In reply to comment #40) > we've made some progress: > - Hudson jobs created [1] > - client job works and deploys artifacts to Nexus [2] > > still some open issues: > > The orion-server job can't directly consume the client bundles from Nexus, > Tycho can only consume the client bundles from a p2 repository hence the > client job needs to creates a p2 repository. > > I propose the following solution: > - we move the client feature and the doc plugins (which are packed into the > client feature) to orion.client. If you don't want to move the doc bundles > we could put them into a separate feature in orion.server. > - and create a client p2 repo which the server build can then consume via > Nexus. This doesn't work as it would break the old PDE build, so I'll create a client p2 repository without moving any parts from orion.server to orion.client. We can clean this up later when PDE build isn't needed anymore.
(In reply to comment #41) > This doesn't work as it would break the old PDE build, so I'll create a > client p2 repository without moving any parts from orion.server to > orion.client. I pushed a solution for this to github: "Bug 399023 - Add a p2 repository for assembling client bundles" https://github.com/msohn/orion.client/commit/ffa96dc4491f847c232a91bf9b3ff07868dcef27 Two more improvements for orion.server: "Add test features to p2 repository" https://github.com/msohn/orion.server/commit/05f4eb60cf14c8f857103a1085b10300fbda7324 "Update Maven build to use Tycho 0.17" https://github.com/msohn/orion.server/commit/5dcba4a40cd936122f2a0c490c85f3c9969fff7a
I merged all three commits to master.
We've reached a working hudson build, find the resulting output on Nexus: --------------------------------------------------- client p2 repository built by orion-client job https://hudson.eclipse.org/hudson/job/orion-client/ --------------------------------------------------- - zipped: https://repo.eclipse.org/content/groups/orion/org/eclipse/orion/org.eclipse.orion.client.repository/1.0.0-SNAPSHOT/ - unzipped: https://repo.eclipse.org/content/unzip/snapshots.unzip/org/eclipse/orion/org.eclipse.orion.client.repository/1.0.0-SNAPSHOT/org.eclipse.orion.client.repository-1.0.0-SNAPSHOT.zip-unzip/ --------------------------------------------------- server p2 repository built by orion-server job https://hudson.eclipse.org/hudson/job/orion-server/ --------------------------------------------------- - zipped: https://repo.eclipse.org/content/groups/orion/org/eclipse/orion/org.eclipse.orion.server.repository/1.0.0-SNAPSHOT/ - unzipped: https://repo.eclipse.org/content/unzip/snapshots.unzip/org/eclipse/orion/org.eclipse.orion.server.repository/1.0.0-SNAPSHOT/ next steps I'll look into now: - configure Maven/Hudson to sign build results - minify js bundles
The latest artifacts for orion-server build at Hudson should be good to deploy to orion.eclipse.org.
