Community
Participate
Working Groups
Some feel it's a good idea to merge all 5 source-editing repos into 1. This can simplify the future releng changes that are to be made, to re-enable tests, and to ensure that each consolidated component can stand on its own and be responsible for its own tests, rather than be a monolith. The question here is what structure sourceediting might like to take if it were combined into one repo. There are 51 plugin bundles, not including docs, tests, etc. With everything included, the number of individual buildables jumps to 126. This seems a bit much for one repository with a standard structure of bundles / tests / docs / features. But, it might be suitable for a repository structured like dali... example: sourceediting: |- common |---|-- bundles |---|-- docs |---|-- features |---|-- tests |- xpath |---|-- bundles |---|-- docs |---|-- features |---|-- tests |- xsl |---|-- bundles |---|-- docs |---|-- features |---|-- tests
Hi Rob, I like the idea, especially since we've always had the tests in separate repos. Instead of just common, xpath, and xsl, maybe we could break it down even more sourceediting |- json (All the JSON bits) |- jst (All the Java Server bits) |- wst (The core Web technologies) |- xpath |- xsl
Hi Keith, Do you have any thoughts on this in regards to XSL?
Let me know when quorum/agreement is achieved and I can start working on the reorg/making the tests work again. :D
(In reply to Nick Sandonato from comment #1) To look at the relative distribution for just existing sourceediting: - json: 11 bundles, 4 features, 1 test bundle - jst : 3 bundles, 3 test bundles - wst : 16 bundles, 10 features, 21 test bundles "jst" (which is really just JSP for us) would be dwarfed by the other two, but I don't think it's an outright bad idea.
Created attachment 269795 [details] Splitting the bundles Here's a plugin heirarchy and possible methods of splitting the repo. I think it's clear that the splitting it into core/common, json, xpath, and xsl, is the best. If we wanted to break out the jst stuff, we'd need to separate jst.jsp.core and jst.jsp.ui, which would also necessitate breaking out wst.jsdt.web.support.jsp since it depends on a jst plugin. This, of course, makes me wonder why a wst plugin is depending on a jst plugin at all. Seems backwards and broken. To prevent questions like this, I propose we just leave the jst stuff in a core/common component, if you all agree. But I'm definitely +1 on breaking out the json stuff.
(In reply to Rob Stryker from comment #5) > But I'm definitely +1 on breaking out the json stuff. Rob, your image places the XML Schema bundles in the same "pool" with XPath, while I think they really belong in "core" (the name I'd advocate for over "common"). https://wiki.eclipse.org/WTP_Git_Migration_Checklist details why it was split/spread among 6 repos in the first place, and why XSL is an open question now vs. 5 years ago.
For reference, nitind means two bundles colored green (xsd.*) should be moved out of the xpath group into the 'core' group.
(In reply to Rob Stryker from comment #7) > For reference, nitind means two bundles colored green (xsd.*) should be > moved out of the xpath group into the 'core' group I agree that the *xsd* plugins should not belong in the xpath group. But, the XSD plugins are not really 'core' to JSON. Can we organize them by 'language', regardless of size? Is there a big impact on releng if more 'folders' are created in the hierarchy?: source editing |- core |- json (All the JSON bits) |- jst (All the Java Server bits) |- wst (The core Web technologies) |- xpath |- xml |- xsl |- xsd
There's no real impact, but it's annoying to developers if you're dividing things up when they don't need to be. Remember, each top-level item (core, json, etc) will have its own plugins / features / tests folders. If we only have 2 plugins in some of them, and no features, and no tests specific to that folder, then it kinda becomes useless and more annoying than not.
https://gist.github.com/robstryker/054b9f2315a9bddd0ca76acebbd44610 So, this is the script that merges the 5 repos into 1. This issue does not yet address the sourceediting issue. This puts the repo in a merged state that can be built when using the -Pintegration profile. However, this is only required for one of the subcomponents. This script creates a repo with the following dependency tree: core <- (json | xml <- xpath <- xsl <- web) Basically, 'web' sits at the top. Web is the only component that won't build if you don't use the -Pintegration profile. It has a feature dependency on jee (which needs to be solved separately). It also is the component containing the troublesome jsdt integrations. I believe strongly in this patch, and think we should move to have the webmaster open up the restrictions so it can be merged ASAP. This will allow us to move on to the next steps.
After running the script, this is part of the layout: core/bundles: org.eclipse.wst.sse.core org.eclipse.wst.sse.ui pom.xml json/bundles: org.eclipse.json org.eclipse.wst.json.jshint.ui org.eclipse.wst.json.bower.core org.eclipse.wst.json.npm.core org.eclipse.wst.json.bower.ui org.eclipse.wst.json.npm.ui org.eclipse.wst.json.core org.eclipse.wst.json.schemaprocessor org.eclipse.wst.json.eslint.core org.eclipse.wst.json.ui org.eclipse.wst.json.eslint.ui pom.xml org.eclipse.wst.json.jshint.core xml/bundles: org.eclipse.wst.dtd.core org.eclipse.wst.xsd.core org.eclipse.wst.dtd.ui org.eclipse.wst.xsd.ui org.eclipse.wst.xml.core org.eclipse.wtp.xml.capabilities org.eclipse.wst.xml.ui pom.xml xpath/bundles: org.eclipse.wst.xml.xpath2 org.eclipse.wst.xml.xpath.core org.eclipse.wst.xml.xpath2.processor org.eclipse.wst.xml.xpath.ui org.eclipse.wst.xml.xpath2.wtptypes pom.xml xsl/bundles: org.eclipse.wst.xsl org.eclipse.wst.xsl.jaxp.launching org.eclipse.wst.xsl.core org.eclipse.wst.xsl.launching org.eclipse.wst.xsl.debug.ui org.eclipse.wst.xsl.saxon org.eclipse.wst.xsl.exslt.core org.eclipse.wst.xsl.ui org.eclipse.wst.xsl.exslt.ui org.eclipse.wst.xsl.xalan org.eclipse.wst.xsl.jaxp.debug pom.xml org.eclipse.wst.xsl.jaxp.debug.ui web/bundles: org.eclipse.jst.jsp.core org.eclipse.wst.jsdt.web.core org.eclipse.jst.jsp.ui org.eclipse.wst.jsdt.web.support.jsp org.eclipse.jst.standard.schemas org.eclipse.wst.jsdt.web.ui org.eclipse.wst.css.core org.eclipse.wst.standard.schemas org.eclipse.wst.css.ui org.eclipse.wtp.javascript.capabilities org.eclipse.wst.html.core org.eclipse.wtp.web.capabilities org.eclipse.wst.html.ui pom.xml I realize some might not like the label "xml" for a component that includes dtd. Unfortunately I couldn't think of a better name for it, and I was really trying to avoid breakout out the schema stuff directly. A lot of this structure was dictated by what features depend on what. I think this is a really good structure in general, though. I also think it will be much better once the jsdt stuff is moved back into the jsdt repo as Victor wants. Regaring jsdt, I know there's a lot of opinions there, but Victor insists that removing jsdt's dependency on sourceediting would be extremely difficult for him to accomplish, and moving the 3 jsdt plugins into jsdt repo and having jsdt depend on sse instead is the better solution. I've proceeded with my faith in his opinion. Remember, the goal here is to avoid stagnation, get reproducable builds, and improve and streamline the process. Let's try not to get too bogged down with smaller details if it will stagnate the reorganization :( I'd like to see some +1's here though. It'd be great of some people could verify this structure works for them.
Note that when building, the warnings about invalid relativePath for the parent pom will be solved in a future fix, once we sort out how to publish the wtp parent pom by itself to a nexus repo from which these builds can then resolve it without the need for a complicated relativePath requirement to a local file. (This is how we do things in JBoss Tools and it's proven very effective for the last several years.)
Correction to my previous. Seems this is already a thing: https://hudson.eclipse.org/webtools/job/deploy-webtools-parent-pom-3.6/ --> https://repo.eclipse.org/#nexus-search;quick~wtp-parent
(In reply to Rob Stryker from comment #11) > Regaring jsdt, I know there's a lot of opinions there, but Victor insists > that removing jsdt's dependency on sourceediting would be extremely > difficult for him to accomplish, and moving the 3 jsdt plugins into jsdt > repo and having jsdt depend on sse instead is the better solution. I've > proceeded with my faith in his opinion. I put a patch on bug 519774 to remove the dependency, although it would require the review of someone more recently familiar with JSDT.
Since the JSDT / SSE dependency has been fixed, I think now's the time to address this issue specifically. If anyone has a problem with the proposed layout (core, json, xml, xpath, xsl, web), please speak now. Otherwise, I think the sse merge should go ahead ASAP.
(In reply to Rob Stryker from comment #11) > After running the script, this is part of the layout: +1
I sent nsand.dev@gmail.com an email on September 26th, and again today. I think it is likely he will not respond.
@webmaster: With @Carl approving this change (and getting @Nitin to also review and +1 it), and with the project lead [1] Nick S. seemingly unresponsive, what's the next step to get this change done? Do we need to vote in another (acting) project lead? Or can we safely assume that Nick S's lack or response == no objections, and proceed with this merge, which we've talked about doing since at least July 2017 on various lists [2] and in wtp-dev meetings [3]? [1] https://projects.eclipse.org/projects/webtools.sourceediting/who [2] https://dev.eclipse.org/mhonarc/lists/wtp-dev/msg10406.html [3] https://wiki.eclipse.org/WTP_2017-08-31#Main_Agenda_Items
+1. The new layout looks good to me.
Overall this sounds fine to me, but it's really EMO(and EMO legal) that needs to sign-off on this plan. Wayne, are you ok with this? -M.
(In reply to Eclipse Webmaster from comment #20) > Wayne, are you ok with this? Will the commit history (authors and dates) be preserved in the merge?
Yes, authors and dates are preserved. Very similar scripts have now been run against servertools (to merge 3 repos, last week) and webtools-common (to merge several repos, 5 months ago). The scripts all preserve author, history, time of commit, etc.
(In reply to Rob Stryker from comment #22) > Yes, authors and dates are preserved. > > Very similar scripts have now been run against servertools (to merge 3 > repos, last week) and webtools-common (to merge several repos, 5 months > ago). > > The scripts all preserve author, history, time of commit, etc. +1 Have at it.
OK. So... here's the script https://gist.github.com/robstryker/054b9f2315a9bddd0ca76acebbd44610 ** WARNING ** This script is intended to be used on a linux environment. It may fail in a large number of ways if run via OSX. Differences between what was expected and what this script does: 1) org.eclipse.wst.standard.schemas has been moved to the 'xml' subdirectory rather than web. The reason is because features in the 'xml' repository depend on it. While in theory things built in the same reactor can live anywhere, that'd just be an excuse for being messy. So let's group things properly. 2) Two plugins from javaee repo will be moved into sse. These plugins are org.eclipse.wst.web and org.eclipse.wst.web.ui. This is ALL of the features in the 'web' folder basically require this bundle. The features have names like org.eclipse.wst.web_core.feature, so it makes sense the wst.web features need the wst.web bundles. It further makes sense because the jeetools repo is almost entirely org.eclipse.jst plugins and these two wst plugins are WAAAY out of place there. This repository [1] is a snapshot of what it all looks like AFTER the script is run. Now, we can handle this two ways. We can have someone else run the script, verify everything works, etc, OR, we can just have someone add my github repo as a remote, check out the branch, build it, verify everything LOOKS good, verify no plugins are missing, history is maintained, etc etc, and then push that directly once webmaster opens it up. The script can take up to 45 minutes to run. A lot of time is spent building hte pre-reqs (common, server, jsdt, releng, etc) and a good deal of time is spent rewriting javaeetools history to isolate the two plugins that are moved. (This is done to make sure we only bring commits which touched those two plugins into the new repo.) [1] https://github.com/robstryker/webtools.sourceediting/commits/519774 As to the question of history and authorship: History and authorship **ARE** maintained, BUT, the person doing a git log must do 'git log --follow' in order to see it all clearly. Here's an example on github: https://github.com/robstryker/webtools.sourceediting/blame/e65ac79d0d72c3f72099691d5014eb5c963133a5/web/tests/org.eclipse.wst.css.tests.encoding/pom.xml You can see that 'git blame' still accurately shows the author of each line. But if you switch to the 'history' tab, you will only see commits from the time I ran the script, **AFTER** I moved the files. Unfortunately this is the best we can do without filter-branching and rewriting history all over the place (HUGE NO-NO). The fact is, though, authorship and commit dates **are** preserved in the repo. They're just a bit harder to see. This script builds everything successfully. The only thing that fails is [INFO] org.eclipse.wst.sse.unittests ...................... FAILURE [02:32 min] Tests run: 1752, Failures: 244, Errors: 5, Skipped: 0 Now, I can do a lot of work, but I can't fix 244 test failures in a repo I barely understand ;) So, I need the following: 1) A +1 from an SSE committer / lead. @nitin / @victor? 2) A +1 from a javaee committer / lead. As a committer I can +1 but I'd prefer a second opinion. 3) Webmaster to open up the repo. I will only do this once 1) and 2) are complete. 4) Someone from sse to verify the repo looks good, builds, and is willing to volunteer to push this in. @victor?
The script takes 1458 seconds. Just for the record.
I ran the script locally and it works! After the merge, the repo build takes 15 mins on my Fedora 24 vm. So, verified that it's ready to be run by the SSE lead or a committer.
+1. With the only note that I cannot run the JUnit tests (my Gnome crashes during the tests), but this problem doesn't seem to be related to the script and resulting merged repository. The merged repository looks consistent.
I've rebased my merged repo. https://github.com/robstryker/webtools.sourceediting/commits/519774 @victor @nitin It'd be good to get this in before any further changes. Who can promise to commit this on a certain date, so we can alert webmaster in advance to open up the repository?
Merge was done. http://git.eclipse.org/c/webtools/webtools.releng.aggregator.git/commit/?id=aaef7c476defee7d0a37aa1f65a0d30a8f0c8b2b Now updating releng.aggregator to source from 1 repo instead of 5: http://git.eclipse.org/c/webtools/webtools.releng.aggregator.git/commit/?id=ec6b129d35710c5beaf29201d9e5436dd7bcb575
Seems we are missing a wst.web infopop plugin. gist script: https://gist.github.com/robstryker/cbf087cb100134551d0a95a8dcac8684 output: https://github.com/robstryker/webtools.sourceediting/tree/520044_part2 Seems pretty safe to me :| @nboldt can you test it? Or at least test the result?
New Gerrit change created: https://git.eclipse.org/r/112139
Nearly done here... but building locally, I get this error: $ git clone ssh://nickb@git.eclipse.org:29418/sourceediting/webtools.sourceediting.git $ cd webtools.sourceediting $ mvn clean install ... [ERROR] You requested to install 'bundle org.eclipse.jst.j2ee.core 0.0.0' but it could not be found [ERROR] Cannot resolve dependencies of MavenProject: org.eclipse.webtools.sourceediting:org.eclipse.jst.jsp.core.tests:1.0.902-SNAPSHOT @ /home/nboldt/4-eclipse.org/WTP/webtools.sourceediting/web/tests/org.eclipse.jst.jsp.core.tests/pom.xml: See log for details -> [Help 1] So we have a 4th plugin (after the wst.web.* stuff) to move from webtools.javaee over to webtools.sourceediting? Or else we need to have sse depend on javaee :( webtools.javaee/plugins/org.eclipse.jst.j2ee.core -> webtools.sourceediting/web/plugins/org.eclipse.jst.j2ee.core
Solution to the missing deps noted before appears to be to move two test plugins into the integration profile so they're not run for the default build. https://git.eclipse.org/r/112354
Actually, it's three plugins that have to move: https://git.eclipse.org/r/#/c/112519/
SSE now only depends on upstream, having moved / disabled all the plugins that were causing circular deps. https://hudson.eclipse.org/webtools/job/webtools-sourceediting-gerrit_master/ and https://hudson.eclipse.org/webtools/job/webtools-sourceediting_R3_10/
