Bugzilla – Full Text Bug Listing
| Summary: | PBS: Runtime jar and build dependencies | ||
|---|---|---|---|
| Product: | [Tools] PTP | Reporter: | Roland Schulz <roland> |
| Component: | RM.PBS | Assignee: | Roland Schulz <roland> |
| Status: | CLOSED WONTFIX | QA Contact: | Greg Watson <g.watson> |
| Severity: | normal | ||
| Priority: | P3 | CC: | dieter.krachtus |
| Version: | 4.0.1 | ||
| Target Milestone: | --- | ||
| Hardware: | PC | ||
| OS: | All | ||
| Whiteboard: | |||
| Attachments: | |||
Created attachment 173729 [details]
New utils package with classes which don't require Eclipse
I think the name org.eclipse.ptp.server.utils.core would be confusing. I suggest the way to do this is to separate the current packages out into separate plugins. So the plugins would become: org.eclipse.ptp.utils.core org.eclipse.ptp.utils.extensionpoints.core org.eclipse.ptp.utils.file.core However, this is not going to prevent someone from adding a dependency to the utils.core plugin in the future. I'm not sure if there is any way to guard against that. Created attachment 174206 [details]
New projects org.eclipse.ptp.utils.extensionpoints.core, org.eclipse.ptp.utils.file.core
Created attachment 174212 [details]
Patch to split org.eclipse.ptp.utils.core
Greg, I changed it to your naming. Do you think this way it is OK?
You are right, that we can't block someone adding dependencies.
Can we change the build that it works like the "Export->Runnable Jar"? This function automatically adds all needed dependencies to the Jar. Then we are guaranteed to always have all dependencies. And hopefully someone would notice if the jar is suddenly gets much bigger, if someone add large dependencies.
I don't think it's possible to use the export wizard because the jar must be built with a custom builder. Maybe it's possible to add dependency checking to the build step? I committed it to HEAD. I hope I have updated everything which is required for building it. Still looking at how to best build the jproxy plugin. If one uses the Export->RunnableJar option and saves it as ANT script, it places the jar-in-jar-loader.zip in the same folder as the ant script. I don't think this is an ideal solution to include the jar-in-jar-loader.zip in the CVS but I think it is a better solution than our current one, requiring the user to modify the script to point to the location of the eclipse installation. Thus I suggest: Add jar-in-jar-loader.zip to CVS Use ANT script instead of build.sh Add ANT script to Builders to create the JAR automatically This doesn't solve the dependency problem (yet) but it does solve the problem that when using the CVS version the user has to: Modify the build.sh Run the build.sh (requiring cygwin on Windows) Remember to rerun it when files change And I think the same JAR could be used for the automatic build, removing differences between manual and automatic build. I see errors on the extensionpoints plugin that org.eclipse.core can't be resolved. Why do you add dependencies to "Imported Packages" rather than "Required Plug-ins"? The new plugins are also not externalized and they should be marked as singletons. The should also be activated when a class is loaded. You also need to add an about.html to each. As this is is adding new plugins, you'll also need to announce this to ptp-dev (once the issues have been resolved) and let people know what they need to do to fix their workspace. Thanks. (In reply to comment #7) > If one uses the Export->RunnableJar option and saves it as ANT script, it places > the jar-in-jar-loader.zip in the same folder as the ant script. > I don't think this is an ideal solution to include the jar-in-jar-loader.zip in > the CVS but I think it is a better solution than our current one, requiring the > user to modify the script to point to the location of the eclipse installation. Presumably it doesn't change much, so this is probably ok. (In reply to comment #8) > I see errors on the extensionpoints plugin that org.eclipse.core can't be > resolved. Why do you add dependencies to "Imported Packages" rather than > "Required Plug-ins"? I used the Quick-Fix to fix the dependencies and that's where Eclipse added them. I must admit I don't know the difference. I changed it to "Required" now. I hope this also solves the resolve error. Because I didn't get that error. > The new plugins are also not externalized and they should be marked as > singletons. The should also be activated when a class is loaded. You also need > to add an about.html to each. done > As this is is adding new plugins, you'll also need to announce this to ptp-dev > (once the issues have been resolved) and let people know what they need to do > to fix their workspace. I'll wait until he here from you whether the resolve error is fixed. (In reply to comment #9) > (In reply to comment #7) > > If one uses the Export->RunnableJar option and saves it as ANT script, it places > > the jar-in-jar-loader.zip in the same folder as the ant script. > > I don't think this is an ideal solution to include the jar-in-jar-loader.zip in > > the CVS but I think it is a better solution than our current one, requiring the > > user to modify the script to point to the location of the eclipse installation. > > Presumably it doesn't change much, so this is probably ok. I committed it. I didn't remove or modify the customBuild.xml because I don't know how to make the automatic build use the same build.xml. It looks fine now. For the dependencies I had to move PBSProxyServerRunner out of .rm.pbs.jproxy and I moved it into .rm.pbs.core. Because of that the org.eclipse.ptp.rm.pbs.PBSProxyServer extension point also had to be moved because otherwise the class wasn't found. Without any other change the payload isn't found. There are two options: - Reference the payload from the jar file in the other plugin. Is this possible? - Move the payload also to .rm.pbs.core. I thought the 2nd option is better and updated the ant build script for the workspace (build.xml) to store the jar in .rm.pbs.core. Greg, is this OK? If it is OK, then we also need to update the location for the automatic build. Can we use the build.xml also for the automatic build? If not, could you update the location? I don't want to change the automatic build scripts without being able to test it. Thanks. I don't understand why PBSProxyServerRunner needs to be moved. Please explain. I would like to keep this and the payload in their own plugin. I just added a new build.xml that build pbs_proxy.jar locally. For that I wrote a custom JAR ANT task which determines all external Eclipse project dependencies. New dependencies are added only once to the Eclipse project and not hard-coded in the build.xml (no redundancies). The ANT task figures out what is needed to build pbs_proxy.jar at 'build-time'. (In reply to comment #14) > I don't understand why PBSProxyServerRunner needs to be moved. Please explain. > I would like to keep this and the payload in their own plugin. PBSProxyServerRunner depends on org.eclipse.core.runtime.IProgressMonitor and org.eclipse.ptp.remote.launch.core.AbstractRemoteServerRunner and they in-turn depend on many libraries. Including all this would make the JAR several MB. Of course this class (and thus the dependency) is not needed at run-time on the server but you can't add the build-time dependency only for this one file. Thus you would need to add these build dependencies to the whole package and thus wouldn't notice if the run and build dependencies don't match. Of course if you don't like it to be in rm.ptp.core we could create a new rm.ptp.server package. (In reply to comment #15) > I just added a new build.xml that build pbs_proxy.jar locally. For that I wrote > a custom JAR ANT task which determines all external Eclipse project > dependencies. > > New dependencies are added only once to the Eclipse project and not hard-coded > in the build.xml (no redundancies). The ANT task figures out what is needed to > build pbs_proxy.jar at 'build-time'. I doesn't work for me I get: [jar-eclipse] java.io.FileNotFoundException: C:\Users\rschulz\Desktop\eclipse\.classpath (The system cannot find the file specified) [jar-eclipse] at java.io.FileInputStream.open(Native Method) Also I don't see any copyright information for the binary only JAR file you added. Please add this. Also either adding the source or a pointer to the source is important. The backtrace: [jar-eclipse] at java.io.FileInputStream.open(Native Method) [jar-eclipse] at java.io.FileInputStream.<init>(Unknown Source) [jar-eclipse] at java.io.FileInputStream.<init>(Unknown Source) [jar-eclipse] at sun.net.www.protocol.file.FileURLConnection.connect(Unknown Source) [jar-eclipse] at sun.net.www.protocol.file.FileURLConnection.getInputStream(Unknown Source) [jar-eclipse] at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(Unknown Source) [jar-eclipse] at com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(Unknown Source) [jar-eclipse] at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source) [jar-eclipse] at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source) [jar-eclipse] at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source) [jar-eclipse] at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source) [jar-eclipse] at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source) [jar-eclipse] at javax.xml.parsers.DocumentBuilder.parse(Unknown Source) [jar-eclipse] at com.chartle.ant.JarEclipseProject.getFiles(JarEclipseProject.java:74) Strange. Before commiting the changes I tested it under Windows and Linux both from Eclipse-ANT view and the command line executing ANT directly. Still I cannot reproduce your stacktrace. Can you give a little more information: What OS did you use? Eclipse ANT view or command line? Please give the location of your workspace and the path of the org.eclipse.rm.pbs.jproxy project (In reply to comment #17) > The backtrace: > [jar-eclipse] at java.io.FileInputStream.open(Native Method) > [jar-eclipse] at java.io.FileInputStream.<init>(Unknown Source) > [jar-eclipse] at java.io.FileInputStream.<init>(Unknown Source) > [jar-eclipse] at (In reply to comment #18) > Strange. Before commiting the changes I tested it under Windows and Linux both > from Eclipse-ANT view and the command line executing ANT directly. Still I > cannot reproduce your stacktrace. If I run in a separate JVM it works. I had set it to run in the same JVM as Eclipse. Also the separate JVM has to be >=1.6 for it to work. (for more details see PM) One more problem. The resulting jar doesn't contain a MANIFEST and I trying to execute it, I get: $java -jar pbs_proxy.jar Invalid or corrupt jarfile pbs_proxy.jar I think I fixed this problem and commited. Does the previous problem still appear? What happens when you run the ant script from the console? (In reply to comment #19) > (In reply to comment #18) > > Strange. Before commiting the changes I tested it under Windows and Linux both > > from Eclipse-ANT view and the command line executing ANT directly. Still I > > cannot reproduce your stacktrace. > > If I run in a separate JVM it works. I had set it to run in the same JVM as > Eclipse. Also the separate JVM has to be >=1.6 for it to work. (for more > details see PM) I fixed this problem. (In reply to comment #20) > One more problem. The resulting jar doesn't contain a MANIFEST and I trying to > execute it, I get: > $java -jar pbs_proxy.jar > Invalid or corrupt jarfile pbs_proxy.jar MANIFEST and jdk>=1.6 is fixed. Great. Thanks! Running with same JVM as Eclipse still gives the error. To reproduce: - Right Click build.xml - Run As->Ant Build... - JRE Tab - Run in the same JRE as the workspace In my opinion this is a problem with ANT by setting the wrong working directory. Still I know how to fix it at our end (ASAP). Until then run in forked jvm. (In reply to comment #23) > MANIFEST and jdk>=1.6 is fixed. Great. Thanks! > > Running with same JVM as Eclipse still gives the error. > To reproduce: > - Right Click build.xml > - Run As->Ant Build... > - JRE Tab > - Run in the same JRE as the workspace If there is a jdk 1.6 dependency then it needs to be removed. (In reply to comment #16) IMHO you're creating a whole load of complexity just in order to build the server manually. Previously, apart from having to edit the build.sh file, there were no build dependency issues. Now you're moving things around just to accommodate a new build process. I think it would be better to stay with the current approach, live with a larger jar during development, or work out how to solve the issue in the build.xml. For the latter, you may end up with a more complex build file, but that is better than making the whole proxy more complicated. (In reply to comment #25) > If there is a jdk 1.6 dependency then it needs to be removed. There was no 1.6 dependency. It was merely a problem with compiler settings that has been resolved. (In reply to comment #26) > (In reply to comment #16) > IMHO you're creating a whole load of complexity just in order to build the > server manually. Previously, apart from having to edit the build.sh file, there > were no build dependency issues. Now you're moving things around just to > accommodate a new build process. I think it would be better to stay with the > current approach, live with a larger jar during development, or work out how to > solve the issue in the build.xml. For the latter, you may end up with a more > complex build file, but that is better than making the whole proxy more > complicated. I agree, sacrificing consistency to optimize the server payload is a bad idea. Simplicity should be paramount. In similar cases I used a JAR build post-processing by removing unused dependencies though bytecode analysis/modification and/or specialized bytecode compression. Both may be realized in a project-based build.xml that works with the server-JAR the old build process gives us. How well that approach works I could only tell after a few tests. (In reply to comment #26) > (In reply to comment #16) > IMHO you're creating a whole load of complexity just in order to build the > server manually. Previously, apart from having to edit the build.sh file, there > were no build dependency issues. Now you're moving things around just to > accommodate a new build process. I think it would be better to stay with the > current approach, live with a larger jar during development, or work out how to > solve the issue in the build.xml. For the latter, you may end up with a more > complex build file, but that is better than making the whole proxy more > complicated. This is a misunderstanding. My aim has nothing to do with the build process. I totally agree that the build process should adapt to the code and not the other way around. I wouldn't have moved away from the build.sh if it would have meant code change and I would be happy to move back to it if this makes things easier. Instead the aim of everything is that I would like it to be guaranteed that all dependencies are fulfilled at runtime. So far the dependencies included in the JAR (both with the automatic build, the build.sh and my build.xml) are hard-coded. Thus if you used a class which is part of the compile dependencies but not of the run-time dependencies (e.g. a eclipse.core class for e.g. externalized strings) you didn't get an error message at compile time but the proxy crashed with a unresolved dependency at run-time. Thus this is a very easy error to make. If this is in code which is only executed seldom (e.g. error handling) than this is an error which can be very easily be overlooked and make the proxy less stable. Thus the aim of all this is to make the proxy more robust and any change which may be required to reach this aim would be only a side product. If there is any other way to guarantee that all dependencies used at compile time are available at run-time or detect a mismatch reliable, than we can of course also go with such an alternative. But I feel that the changes and added complexity are worth it if it can prevent such easy to make runtime errors. (In reply to comment #28) > (In reply to comment #26) > > (In reply to comment #16) > > IMHO you're creating a whole load of complexity just in order to build the > > server manually. Previously, apart from having to edit the build.sh file, there > > were no build dependency issues. Now you're moving things around just to > > accommodate a new build process. I think it would be better to stay with the > > current approach, live with a larger jar during development, or work out how to > > solve the issue in the build.xml. For the latter, you may end up with a more > > complex build file, but that is better than making the whole proxy more > > complicated. > > I agree, sacrificing consistency to optimize the server payload is a bad idea. > Simplicity should be paramount. I agree too. Match of build- and run-time dependencies is my aim. (see last comment) > In similar cases I used a JAR build post-processing by removing unused > dependencies though bytecode analysis/modification and/or specialized bytecode > compression. Both may be realized in a project-based build.xml that works with > the server-JAR the old build process gives us. > How well that approach works I could only tell after a few tests. As said in my last comment I'm happy with alternative solutions which guarantee that all dependencies are fulfilled and which make the JAR not ridiculous big. Thus if the removing of dependencies wouldn't be hard-coded but would automatically remove only those classes/jars which are not required than this would be possibility. But I'm not sure whether it would help the simplicity. First it would require this additional automatic removal and second it would be less obvious for the programmer in which parts of the code he/she should try to limit the used dependencies (e.g. not use eclipse.core) and in which parts all codependencies can be used. After my change this is IMO rather clear because some packages are used also by the server and others are only used local. Of course the programmer can add dependencies which would increase the JAR size (good thing - big JAR is MUCH better than failed run-time dependencies) but he would first need to add these dependencies to the project as a warning. With the new setup you've broken the headless build. Currently the customBuild.xml gets run during the build process to create the pbs_proxy.jar. This has to run prior to the plugin being jarred. Now you're going to have to either work out how to put the jar file into the pbs.core plugin (so you'll have to ensure that jproxy gets built before pbs.core) or move customeBuild.xml to pbs.core and work out how to build it from there. Also, pbs.jproxy should no longer be included in the plugins to be installed as it is not needed during runtime, so that needs to be fixed also. I agree with you that my solution requires some difficult changes. Do you have an idea how to avoid the possibility of unresolved dependency at run-time some easier way? (In reply to comment #31) > With the new setup you've broken the headless build. Currently the > customBuild.xml gets run during the build process to create the pbs_proxy.jar. > This has to run prior to the plugin being jarred. Now you're going to have to > either work out how to put the jar file into the pbs.core plugin (so you'll > have to ensure that jproxy gets built before pbs.core) or move customeBuild.xml > to pbs.core and work out how to build it from there. It seems to be difficult to test the build on different machine, because the build.xml uses many /home/data paths. I assume you run the builds on build.eclipse.org. I don't have a shell account on that machine thus I can't test it there unless you want to request a shell account for me. Without testing and documentation I don't think I can figure out how the head-less build works. How was originally guaranteed that the required plugins (.remote.core,.rm.proxy.core,.proxy.protocol) are build before .pbs.jproxy? Does the build system uses the Required-bundle line in the Manifest.mf to determine the build order? > Also, pbs.jproxy should no > longer be included in the plugins to be installed as it is not needed during > runtime, so that needs to be fixed also. Is it enough to remove it from the feature.xml? I finally extracted the sources for the Ant task that dynamically and recursively determines the dependencies to build a functional pbs_proxy.jar at runtime. I added the proper copyright notice to the Ant task source file. I decided to put the source file into the task-jar instead of the jproxy 'src' directory. Putting it there would have caused build errors since it depends on ant.jar (Apache Ant) which is not part of the jproxy project (yet). I also fixed a previous problem where execution of the Ant task using the Eclipse view broke if Ant was run in the same JRE as the workspace. Tested under Windows/Linux, using the console and the Eclipse Ant view, receptively. Has been made obsolete by new RM framework. |
Created attachment 173728 [details] Patch to fix dependencies for jproxy Currently the build dependencies don't match the jar files provided at runtime. For build-time jproxy is configured to have the "Plug-in Dependencies" as Library. The jar which is deployed (either with the binary built or with the build.sh) doesn't include any of the eclipse core packages. This currently doesn't produce any error because no class which uses any of the eclipse.core packages is used by the proxy. But it can easily lead to errors, because adding a dependency on a eclipse.core class doesn't produce any compile time error but causes run-time errors. And this could be easily overlooked. Their are two solutions to improve this solution: 1) Include the eclipse core classes into the JAR file. 2) Remove the "Plug-in Dependencies" from the Library build configuration Solution 1 has the problem that the JAR file would increase from 220kb to 6.2MB. Thus I would strongly prefer solution 2. For this to work the "Plug-In Dependencies" have to be removed from org.eclipse.ptp.rm.pbs.jproxy, org.eclipse.ptp.proxy.protocol, org.eclipse.ptp.utils.core and org.eclipse.ptp.rm.proxy.core. For all but org.eclipse.ptp.util this is no problem at all since they don't use any Eclipse classes. To be able to use most of org.eclipse.ptp.utils from packages which shouldn't have a dependency on eclipse classes, I split the package into two. org.eclipse.ptp.utils.core only contains classes using Eclipse classes and org.eclipse.ptp.server.utils.core contains classes don't requiring Eclipse. Is this a good solution? Attached is a patch for all packages affected and a zip for the new package.