Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 471835

Summary: Build succeeds, even when "old" parent pom is referred to
Product: [Eclipse Project] Platform Reporter: David Williams <david_williams>
Component: RelengAssignee: Platform-Releng-Inbox <platform-releng-inbox>
Status: VERIFIED FIXED QA Contact:
Severity: major    
Priority: P3 CC: akurtakov, arunkumar.thondapu, daniel_megert, jarthana, Lars.Vogel, markus.kell.r, mikael.barbero, mistria, pwebster, sptaszkiewicz, thanh.ha, tjwatson
Version: 4.5   
Target Milestone: 4.5.1   
Hardware: PC   
OS: Linux   
See Also: https://git.eclipse.org/r/51454
https://git.eclipse.org/r/54112
https://git.eclipse.org/r/54114
https://git.eclipse.org/r/54118
https://git.eclipse.org/r/54121
https://git.eclipse.org/r/54130
https://git.eclipse.org/r/54144
https://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/?id=aa4615a817bb63c9b8c03fbedb7c603694c5ecd2
https://git.eclipse.org/c/jdt/eclipse.jdt.ui.git/commit/?id=b0c6c38e9793009d7b18ca08df919788fb9473c9
https://git.eclipse.org/r/54151
https://git.eclipse.org/r/54152
https://git.eclipse.org/r/54157
https://git.eclipse.org/c/platform/eclipse.platform.swt.git/commit/?id=6eef1897321cfe779aa47a489d4f424d7e92a291
https://git.eclipse.org/c/platform/eclipse.platform.runtime.git/commit/?id=11fb475be8bf7d7bdfa6d726acdaaf1fa913cd5c
https://git.eclipse.org/c/platform/eclipse.platform.ui.git/commit/?id=dad291c2000b1113a115d69c3aecabae446de086
https://git.eclipse.org/c/platform/eclipse.platform.ua.git/commit/?id=14360cb54932c6d56d2b63596a546e8a072c5068
https://git.eclipse.org/c/platform/eclipse.platform.ui.tools.git/commit/?id=847ed216d3452582296bafd75812788bfc7e3559
https://git.eclipse.org/r/54172
https://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/?id=c9583e8d9dd9bea9a6a778744fa40b3586c0ae82
https://git.eclipse.org/r/54173
https://git.eclipse.org/r/54174
https://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/?id=46014c5bd11d5274722cbaedc32c86f646e0c18a
https://git.eclipse.org/r/54175
https://git.eclipse.org/c/platform/eclipse.platform.resources.git/commit/?id=0c2a0575118e9a58f095360846cc62e28ed9fff2
https://git.eclipse.org/c/platform/eclipse.platform.team.git/commit/?id=7d572cec4c7ccc179d0c13ef44998a8d746148cb
https://git.eclipse.org/r/54217
https://git.eclipse.org/r/54218
https://git.eclipse.org/r/54219
https://git.eclipse.org/r/54222
https://git.eclipse.org/c/equinox/rt.equinox.framework.git/commit/?id=70eb4f06a03655928597e7f6c9399ce7c370faf9
https://git.eclipse.org/c/equinox/rt.equinox.p2.git/commit/?id=803fb88bedb715be74bcaf9f1b3b963194a5521b
https://git.eclipse.org/c/pde/eclipse.pde.build.git/commit/?id=cff38cc0fcf06caa3b8c0c8a601a92d0382891b6
https://git.eclipse.org/c/pde/eclipse.pde.git/commit/?id=13a28a9a449ca1ac32cdc5889d4a808d995c1973
https://git.eclipse.org/r/54588
https://git.eclipse.org/r/54589
https://git.eclipse.org/c/pde/eclipse.pde.build.git/commit/?id=85748f1437f895a5f6292519eb122b493b49c965
https://git.eclipse.org/c/pde/eclipse.pde.git/commit/?id=b09a2f2ce1b1c3fba3ab25e89d3a6f30f543a8b7
Whiteboard: backport
Bug Depends on: 469765    
Bug Blocks:    

Description David Williams CLA 2015-07-03 14:59:45 EDT
This bug was opened due to observation made elsewhere, bug 469765 comment 30. 

As mentioned there, the build should fail if we have changed parent pom to "4.6", but some sub-projects still refer to parent pom "4.5".
Comment 1 David Williams CLA 2015-07-03 15:19:18 EDT
My best guess at the moment is that we somehow "pull in" the old version of 4.5, from some repository we specify, so it can be found by maven/tycho even though  not intended. 

My first guess was 
https://repo.eclipse.org/content/repositories/releases/

This is where we get released versions of the shared license, and CBI. 

But, I do not see how it could lead to the parent pom (which should just be "staged" somewhere, on a daily basis?) [Not sure if someone releases it somewhere, that's semi-hidden (such as maven central)? If so, we should probably "exclude" maven central. 

Mikael, added you to CC to see if you have an insight into where "old" parent pom might be being pulled in? 

I see in the deploy-eclipse-sdk-target-pom-4.5 job, it "deploys" the parent pom daily (ordinarily, though I've disabled it, for now). 
But I think it deploys to "eclipse.repo.org"? Is that the same as "repo.eclipse.org"? 

No, I think not. By looking in the log ... I see it deploys to 
https://repo.eclipse.org/content/repositories/eclipse-snapshots

I bet it might have more in the debug logs we produce. I'll check there too.
Comment 2 David Williams CLA 2015-07-03 15:49:37 EDT
(In reply to David Williams from comment #1)

> I bet it might have more in the debug logs we produce. I'll check there too.

So far, in log, I see these repositories, added to (most?) "BUILD PLAN"s: 

[DEBUG] Repositories (dependencies): [eclipse-hosted (https://repo.eclipse.org/content/repositories/eclipse/, releases+snapshots), central (http://repo.maven.apache.org/maven2, releases)]

[DEBUG] Repositories (plugins)     : [tycho (https://oss.sonatype.org/content/repositories/public/, releases+snapshots), cbi-jdt (https://repo.eclipse.org/content/repositories/eclipse-staging/, releases+snapshots), eclipse (https://repo.eclipse.org/content/repositories/releases/, releases+snapshots), central (http://repo.maven.apache.org/maven2, releases)]
Comment 3 David Williams CLA 2015-07-03 16:28:06 EDT
Found better "documentation" in "pom-updater" log, as that's presumably the first time it is needed, and added to local maven repo, at that time. 


[DEBUG] Could not find metadata org.eclipse:eclipse-platform-parent:4.5.0-SNAPSHOT/maven-metadata.xml in local (/shared/eclipse/builds/4N/localMavenRepo)
[DEBUG] Using connector WagonRepositoryConnector with priority 0.0 for https://repo.eclipse.org/content/repositories/eclipse/
Downloading: https://repo.eclipse.org/content/repositories/eclipse/org/eclipse/eclipse-platform-parent/4.5.0-SNAPSHOT/maven-metadata.xml
615/615 B   ^M            ^MDownloaded: https://repo.eclipse.org/content/repositories/eclipse/org/eclipse/eclipse-platform-parent/4.5.0-SNAPSHOT/maven-metadata.xml (615 B at 1.2 KB/sec)
[DEBUG] Writing tracking file /shared/eclipse/builds/4N/localMavenRepo/org/eclipse/eclipse-platform-parent/4.5.0-SNAPSHOT/resolver-status.properties
[DEBUG] Using connector WagonRepositoryConnector with priority 0.0 for https://repo.eclipse.org/content/repositories/eclipse/
Downloading: https://repo.eclipse.org/content/repositories/eclipse/org/eclipse/eclipse-platform-parent/4.5.0-SNAPSHOT/eclipse-platform-parent-4.5.0-20150629.040448-367.pom
4/33 KB     ^M8/33 KB   ^M12/33 KB   ^M16/33 KB   ^M20/33 KB   ^M24/33 KB   ^M28/33 KB   ^M31/33 KB   ^M33/33 KB   ^M           ^MDownloaded: https://repo.eclipse.org/content/repositories/eclipse/org/eclipse/eclipse-platform-parent/4.5.0-SNAPSHOT/eclipse-platform-parent-4.5.0-20150629.040448-367.pom (33 KB at 682.4 KB/sec)
[DEBUG] Writing tracking file /shared/eclipse/builds/4N/localMavenRepo/org/eclipse/eclipse-platform-parent/4.5.0-SNAPSHOT/eclipse-platform-parent-4.5.0-20150629.040448-367.pom.lastUpdated
Comment 4 David Williams CLA 2015-07-03 16:45:56 EDT
It seems we (in parent pom nor production scripts), mention 
https://repo.eclipse.org/content/repositories/eclipse/

But, it is mentioned in 6 pom files: 

pom.xml - eclipse.platform.releng
pom.xml - eclipse.platform.releng.aggregator/eclipse.jdt
pom.xml - eclipse.platform.swt
pom.xml - eclipse.platform.swt.binaries
pom.xml - eclipse.platform.ui
pom.xml - org.eclipse.equinox.p2.releng/org.eclipse.equinox.p2-parent


All are the classic "repositories" form: 

  <repositories>
    <repository>
      <releases>
        <enabled>true</enabled>
      </releases>
      <snapshots>
        <enabled>true</enabled>
      </snapshots>
      <id>eclipse-hosted</id>
      <url>https://repo.eclipse.org/content/repositories/eclipse/</url>
    </repository>
  </repositories>

I'm guessing those got put in, at some point, to help build "isolated pieces" in Gerrit, or similar. 

But 1) not sure why that would be needed, and 
2) not sure there might be a better one to use?

I am not even sure these 6 pom's are directly related to all 7 projects that had the old parent listed, but ... once it is in "local maven repo" then suspect anyone can find it, even if they do not have the "repositories" element in their pom.
Comment 5 David Williams CLA 2015-07-03 17:29:19 EDT
From what I can tell, the one in eclipse.platform.releng was made long ago, very near the beginning of our "git efforts", see bug 403963 but the bug was about updating "parent version" so does not shed any insight into why it was added. 

For eclipse.platform.releng I plan to just remove that element, and see what happens. 

Paul, it was so long ago, I doubt if you remember, but any recollection on they the "eclipse-hosted" repo was added at such low levels?
Comment 6 Mickael Istria CLA 2015-07-04 04:07:54 EDT
I guess those repositories are specified in order to actually resolve the parent pom, which is not part of the reactor for individual builds (local or Gerrit).
Even if there is a newer parent pom published, the previous one is still in Maven repository and will still be resolved. It's the responsibility of components to update to newer parent pom when available.
If you build all at once, it believe it would fail because the reference to ../eclipse-platform-parent wouldn't resolve to the version specified in pom.xml.
Comment 7 Thanh Ha CLA 2015-07-04 09:42:43 EDT
(In reply to Mickael Istria from comment #6)
This is correct, I wouldn't remove the repo sections from the sub-repos as they will fail to build in their verify jobs.

When the aggregator repo is built though if any parents have a mismatch version with the platform-parent Maven should fail the build by default as the GAV much match. If it's not doing this, then my suspicion would be that the <relativePath> for that pom is not pointing correctly to platform-parent's relative directory so Maven's not doing the comparison.
Comment 8 David Williams CLA 2015-07-04 13:39:10 EDT
(In reply to Thanh Ha from comment #7)
> (In reply to Mickael Istria from comment #6)
> This is correct, I wouldn't remove the repo sections from the sub-repos as
> they will fail to build in their verify jobs.
> 
> When the aggregator repo is built though if any parents have a mismatch
> version with the platform-parent Maven should fail the build by default as
> the GAV much match. If it's not doing this, then my suspicion would be that
> the <relativePath> for that pom is not pointing correctly to
> platform-parent's relative directory so Maven's not doing the comparison.

Perhaps it is the relative directory, then, since as stated in
bug 469765 comment 30
the build does still succeed. 

But, no matter how fixed, this does need to be fixed. Otherwise, we could be doing a production build, and getting something different that we anticipate. 

Any easy way to tell who doesn't have there right relative directory? 

Plus, I'd prefer to see the "individual build" case handle this by specifying a variable. They already have to specify a profile. But, if done with a variable, then we'd know for sure not interfering with production builds. 

I am also not sure why "https://repo.eclipse.org/content/repositories/eclipse/" is used. 

When we deploy the daily version of parent pom, and eclipse pre-reqs, we deploy it to 

https://repo.eclipse.org/content/repositories/eclipse-snapshots/org/eclipse/

Wouldn't that be the better choice for building individual pieces? 
There may be some "magic" about the URLs ... I don't know ... but seems the more specific you could make it the better? 

Please review, but I believe to use a command line property, those sections of repo-poms would change to something similar to 

<properties>
  <property>
    <fallbackrepo>${fallbackrepo}</fallbackrepo>
  </property>
</properties>


  <repositories>
    <repository>
      <releases>
        <enabled>true</enabled>
      </releases>
      <snapshots>
        <enabled>true</enabled>
      </snapshots>
      <id>eclipse-hosted</id>
      <url>${fallbackrepo}</url>
    </repository>
  </repositories>

So that, if fallbackrepo not specified, on command line, or anywhere else, the build would fail if parent pom was wrong, or relative directory was wrong? 

But, on individual builds, the fall back repo would be specified, so would succeed. Right?
Comment 9 Mickael Istria CLA 2015-07-06 06:42:53 EDT
Using a property would most likely not work: this repo is used in order to resolve parent pom, and if parent pom cannot be resolved, neither the property can...
Wrapping the repository declaration under the build-individual-bundles profile in the root pom of each repository seems to work and to be a good trade-off. The profile first allows to resolve parent pom, and later, it allows to resolve other bundles.
Comment 10 Thanh Ha CLA 2015-07-06 09:58:17 EDT
(In reply to Mickael Istria from comment #9)
> Using a property would most likely not work: this repo is used in order to
> resolve parent pom, and if parent pom cannot be resolved, neither the
> property can...
> Wrapping the repository declaration under the build-individual-bundles
> profile in the root pom of each repository seems to work and to be a good
> trade-off. The profile first allows to resolve parent pom, and later, it
> allows to resolve other bundles.

Agreed, the repository unfortunately cannot be a property that can be overrided as it would need to be able to resolve the parent pom to get that value which is the purpose of having this repository defined in the first place.

Unfortunately there is no easy way to detect that these mismatches are happening as Maven magic makes it try to resolve it via Maven Central and other repositories when it's relativePath does not link back to the correct artifact. I think putting <repositories> in a profile might help but if the aggregator build lists a repo that contains the artifact it is looking for it will still be able to download it so that's something to keep in mind with this approach.
Comment 11 David Williams CLA 2015-07-06 11:32:53 EDT
(In reply to Mickael Istria from comment #9)
> Using a property would most likely not work: this repo is used in order to
> resolve parent pom, and if parent pom cannot be resolved, neither the
> property can...
> Wrapping the repository declaration under the build-individual-bundles
> profile in the root pom of each repository seems to work and to be a good
> trade-off. The profile first allows to resolve parent pom, and later, it
> allows to resolve other bundles.

That's why I put the property in the same pom (not parent) are you saying that would not work? I wonder why not?
Comment 12 Mickael Istria CLA 2015-07-06 13:00:12 EDT
(In reply to David Williams from comment #11)
> That's why I put the property in the same pom (not parent) are you saying
> that would not work? I wonder why not?

Ok, I didn't get that part. So yeah, it seems like your proposal would work.
However, I still believe the approach of re-using the build-individual-bundles profile is simpler, since it actually shows what this repository definition is for.
Comment 13 David Williams CLA 2015-07-06 16:11:05 EDT
(In reply to Mickael Istria from comment #12)
> (In reply to David Williams from comment #11)
> > That's why I put the property in the same pom (not parent) are you saying
> > that would not work? I wonder why not?
> 
> Ok, I didn't get that part. So yeah, it seems like your proposal would work.
> However, I still believe the approach of re-using the
> build-individual-bundles profile is simpler, since it actually shows what
> this repository definition is for.

Do you mean to "copy down" the profile that is currently in the parent pom, to each of these "repo poms"? Thats fine with me too, if it'd work. 

My only concern .. in this bug ... is that I want the production build to fail, if someone is referring to the wrong parent (either via version, or relative path ... when, of course, we are doing a "complete production build"). And the way things are now, that's not happening, so gives be pause that we may at time not build exactly what we think we are. One solution is to simply remove those "repositories" sections. You have an alternative solution (to allow same "small build" function to work) and that's fine with me, if you want to try it, and advocate that solution. 

But otherwise, I'm going to advocate their removal, until the "small build" solution can be found that does not taint the production build.
Comment 14 Mickael Istria CLA 2015-07-06 16:44:57 EDT
(In reply to David Williams from comment #13)
> Do you mean to "copy down" the profile that is currently in the parent pom,
> to each of these "repo poms"? Thats fine with me too, if it'd work.

Not copy the whole profile, but just adding the repo definition to the profile (Maven will merge different pieces of the profile together smartly enough to handle that case according to my local tests)

> My only concern .. in this bug ... is that I want the production build to
> fail, if someone is referring to the wrong parent

If you build with the suggestion above (wrapping repo def in the build-individual-bundles) but without profile enabled, without a repo in settings.xml providing the older parent pom, and without the older parent pom in local Maven repo, it will fail as you'd like.

> One solution is to simply remove those "repositories" sections. You have an alternative
> But otherwise, I'm going to advocate their removal, until the "small build"
> solution can be found that does not taint the production build.

This is as bad as introducing risk in production builds. The individual build and Gerrit triggers have proven much value in term of contributors and reviewers ease of work and productivity

Anyway, it seems like my suggestion would please all of us, so let's give it a try.
Comment 15 David Williams CLA 2015-07-06 17:14:23 EDT
(In reply to Mickael Istria from comment #14)
> (In reply to David Williams from comment #13)
> > Do you mean to "copy down" the profile that is currently in the parent pom,
> > to each of these "repo poms"? Thats fine with me too, if it'd work.
> 
> Not copy the whole profile, but just adding the repo definition to the
> profile (Maven will merge different pieces of the profile together smartly
> enough to handle that case according to my local tests)

Does that assume a correct "local maven repo"? I'm just thinking, it seems that the reason it was put in there, is that sometimes the parent pom can not be "found", so if that happens (i.e. no complete local maven repo) then it still could not find the parent to merge with, right? 

> > One solution is to simply remove those "repositories" sections. You have an alternative
> > But otherwise, I'm going to advocate their removal, until the "small build"
> > solution can be found that does not taint the production build.
> 
> This is as bad as introducing risk in production builds. The individual
> build and Gerrit triggers have proven much value in term of contributors and
> reviewers ease of work and productivity

Yes, the ... you can gain one thing, by breaking another. Especially the fidelity the production builds. I mostly meant, I think you have other ways to solve your "build individual bundles" case: such as, I think you can even specify "extra repositories" on the command line? If not, perhaps you'd just have to learn to build the whole thing, but much faster, caching more, cleaning less, skipping some steps such as docs or api generation? 

> Anyway, it seems like my suggestion would please all of us, so let's give it
> a try.

I'm game. I can test locally, once changes are in place, by changing some random repositories parent version, and make sure they fail. Or, guess we have a list of them, somewhere near bug 469765 comment 30. 

Plus, should they fail if relative path to parent is wrong? 

Not sure if they will, if relative path is wrong. Seems that they should, but not sure due to comment 10. As far as I know, we do not have any general repo listed that has the parent pom in it. I hope no one is putting it in maven central.
Comment 16 David Williams CLA 2015-07-06 21:16:42 EDT
For reference, if needed, here's a link to the "repositories on command line" approach I mentioned. 

http://stackoverflow.com/questions/71030/can-i-add-maven-repositories-in-the-command-line

As it says, would not be recommended for a production build (should be in POM, for that) for the "partial build" case, it sounds like it might be one approach.
Comment 17 David Williams CLA 2015-07-07 00:39:06 EDT
(In reply to David Williams from comment #16)
> For reference, if needed, here's a link to the "repositories on command
> line" approach I mentioned. 
> 
> http://stackoverflow.com/questions/71030/can-i-add-maven-repositories-in-the-
> command-line
> 
> As it says, would not be recommended for a production build (should be in
> POM, for that) for the "partial build" case, it sounds like it might be one
> approach.

Upon re-reading ... there's suggestions near the end of that thread that suggest "organizing poms" (instead of command line, which may no longer work?) but the organizing poms that may be similar to what you are talking about? If it helps, I know p2 has their own pom for ?bulding p2? that is not even read, during production build. I know for patches, I "start off" with a differ pom, that can build just a patch, using current parent, etc., but is not touched during production build. Can't wait to see your solution.
Comment 18 Eclipse Genie CLA 2015-07-07 03:30:05 EDT
New Gerrit change created: https://git.eclipse.org/r/51454
Comment 19 Mickael Istria CLA 2015-07-07 04:17:53 EDT
See proposed patch for platform.ui: https://git.eclipse.org/r/51454
As you can see on Gerrit, it doesn't prevent individual builds to succeed, however, it should fail in the case of production build if parent pom version is wrong (saying that it can't resolve parent).
Comment 20 David Williams CLA 2015-07-08 14:11:57 EDT
(In reply to Mickael Istria from comment #19)
> See proposed patch for platform.ui: https://git.eclipse.org/r/51454
> As you can see on Gerrit, it doesn't prevent individual builds to succeed,
> however, it should fail in the case of production build if parent pom
> version is wrong (saying that it can't resolve parent).

I can't seem to connect to Gerrit at the moment, to leave a comment there, but the change looks great to me! Assuming it works, of course. Seems a perfect case to have a "split profile". Very nice. 

Wouldn't hurt to add a comment, similar to 

<!-- To build individual bundles, we specify a repository where to find parent pom, in case it is not in local maven cache already, and that parent pom also has fuller individual-bundle profile defined that is combined with this one. --> 

Or, something like that.
Comment 21 David Williams CLA 2015-07-08 15:21:43 EDT
(In reply to David Williams from comment #20)
> (In reply to Mickael Istria from comment #19)
> > See proposed patch for platform.ui: https://git.eclipse.org/r/51454
> > As you can see on Gerrit, it doesn't prevent individual builds to succeed,
> > however, it should fail in the case of production build if parent pom
> > version is wrong (saying that it can't resolve parent).
> 
> I can't seem to connect to Gerrit at the moment, to leave a comment there,
> but the change looks great to me! Assuming it works, of course. Seems a
> perfect case to have a "split profile". Very nice. 
> 

I did update eclipse.platform.releng "repository pom" in both master, and R4_5_maintenance. CGit isn't updating quickly, so will just paste here to show I used the 2 space indent, and I think current comment is fine (I didn't remember it being there, when I suggested to leave a comment). 


  <profiles>
    <profile>
      <id>build-individual-bundles</id>
      <repositories>
        <repository>
          <!-- to resolve parent pom, only useful for individual builds -->
          <releases>
            <enabled>true</enabled>
          </releases>
          <snapshots>
            <enabled>true</enabled>
          </snapshots>
          <id>eclipse-hosted</id>
          <url>https://repo.eclipse.org/content/repositories/eclipse/</url>
        </repository>
      </repositories>
    </profile>
  </profiles>
Comment 22 David Williams CLA 2015-07-08 15:28:27 EDT
So, there's a Gerrit patch for eclipse.ui, I just did releng, so think that leaves 3 or 4 other repositories to add this to (and remove the "always available" one). 

eclipse.jdt
eclipse.platform.swt
eclipse.platform.swt.binaries

Technically, I doubt this one is absolutely required (do not think it interferes with production build), but would be a good idea, to be consistent. 

org.eclipse.equinox.p2.releng/org.eclipse.equinox.p2-parent

Please comment once fixed, in both branches, and I'll try some test "production builds" in my local machine -- both, "as things are" then will also try to change parent, and make sure it fails.
Comment 23 David Williams CLA 2015-07-09 02:10:28 EDT
(In reply to Thanh Ha from comment #10)
> .... I think putting <repositories> in a profile might help but if the
> aggregator build lists a repo that contains the artifact it is looking for
> it will still be able to download it so that's something to keep in mind
> with this approach.

There is a place, in parent pom, where we specify 

<eclipse-repo.url>https://repo.eclipse.org/content/repositories/releases/</eclipse-repo.url>

and later, 

    <pluginRepository>
      <id>eclipse</id>
      <url>${eclipse-repo.url}</url>
      <releases>
        <enabled>true</enabled>
      </releases>
      <snapshots>
        <enabled>true</enabled>
      </snapshots>
    </pluginRepository>

So, seems to me that is sort of "wide open". 

From what I can tell, from the comments, this is where we get released CBI plugins from. But, I do not think we intend to get anything else from there. 


Is there a way to make that more specific? Just for CBI? 
Anyone know if they can be specified exactly, in .target file, and left out of the "wide open repositories"? 

Or, perhaps specify the repo as 
https://repo.eclipse.org/content/repositories/cbi/ ? 

I'll try some test builds to see if cbi is all we get from there (during production build). (And, this would not change the need for the other change, exemplified in Gerrit patch.)
Comment 24 Mickael Istria CLA 2015-07-09 04:07:26 EDT
(In reply to David Williams from comment #23)
> Or, perhaps specify the repo as 
> https://repo.eclipse.org/content/repositories/cbi/ ? 

It would indeed make sense to use "finer-grained" repositories. However, don't we need the Eclipse Maven repo to resolve Tycho snapshots?
Comment 25 David Williams CLA 2015-07-09 09:19:13 EDT
(In reply to Mickael Istria from comment #24)
> (In reply to David Williams from comment #23)
> > Or, perhaps specify the repo as 
> > https://repo.eclipse.org/content/repositories/cbi/ ? 
> 
> It would indeed make sense to use "finer-grained" repositories. However,
> don't we need the Eclipse Maven repo to resolve Tycho snapshots?

At present, our parent pom has these comments and URLs for tycho (staging and releases). 
(But, you might be right, that they also go into repo.eclipse.org at some point? 



    <!-- using snapshot repo for Tycho 0.23.0-SNAPSHOT, such as for bug 461207 -->
    <!-- Does it hurt to always leave snapshot repo here? Even if not used? 
    <tycho-snapshot-repo.url>https://repo.eclipse.org/content/repositories/tycho-snapshots/</tycho-snapshot-repo.url>
    -->
    <tycho-repo.url>https://oss.sonatype.org/content/repositories/public/</tycho-repo.url>
Comment 26 Thanh Ha CLA 2015-07-09 10:55:57 EDT
(In reply to Mickael Istria from comment #24)
> (In reply to David Williams from comment #23)
> > Or, perhaps specify the repo as 
> > https://repo.eclipse.org/content/repositories/cbi/ ? 
> 
> It would indeed make sense to use "finer-grained" repositories. However,
> don't we need the Eclipse Maven repo to resolve Tycho snapshots?

That's correct, we need Tycho and CBI repos defined. In theory this should work but should test to make sure we aren't pulling in anything else from "releases" unexpectedly.
Comment 27 David Williams CLA 2015-07-10 01:41:59 EDT
(In reply to Thanh Ha from comment #26)
> (In reply to Mickael Istria from comment #24)
> > (In reply to David Williams from comment #23)
> > > Or, perhaps specify the repo as 
> > > https://repo.eclipse.org/content/repositories/cbi/ ? 
> > 
> > It would indeed make sense to use "finer-grained" repositories. However,
> > don't we need the Eclipse Maven repo to resolve Tycho snapshots?
> 
> That's correct, we need Tycho and CBI repos defined. In theory this should
> work but should test to make sure we aren't pulling in anything else from
> "releases" unexpectedly.

On my local machine, I've tested using 
https://repo.eclipse.org/content/repositories/cbi/
and the build still worked. I'll make that change in master 
and maintenance, so in case there is more subtle problem, is will show up sooner rather than later. Note: this *might* effect some of the build-individual-bundle jobs ... I hope note, but, if so, suspect they too will need the "extra profile" added (somewhere).
Comment 28 David Williams CLA 2015-08-19 10:08:15 EDT
So .. to summarize, in both R4_5_maintenance and master,
to prevent "too much" being pulled in during production builds every pom.xml that has 


 <repositories>
    <repository>
      <releases>
        <enabled>true</enabled>
      </releases>
      <snapshots>
        <enabled>true</enabled>
      </snapshots>
      <id>eclipse-hosted</id>
      <url>https://repo.eclipse.org/content/repositories/eclipse/</url>
    </repository>
  </repositories>

needs to change that to 

  <!-- 
    To build individual bundles, we specify a repository where to find parent pom, 
    in case it is not in local maven cache already
    and that parent pom also has fuller individual-bundle profile 
    defined that is combined with this one. --> 
  <profiles>
    <profile>
      <id>build-individual-bundles</id>
      <repositories>
        <repository>
          <releases>
            <enabled>true</enabled>
          </releases>
          <snapshots>
            <enabled>true</enabled>
          </snapshots>
          <id>eclipse-hosted</id>
          <url>https://repo.eclipse.org/content/repositories/eclipse/</url>
        </repository>
      </repositories>
    </profile>
  </profiles>


There are about 29 such places (times 2 branches) that need the fix. 

This "wide open" mechanism was interfering in my ability to produce a BETA_JAVA9 patch, 
so am hoping these fixes can get in soon, to make it easier to spot where the "real" problems 
are in producing the patch. Without this change, while trying to produce the patch, I could see that
both the 4.5 parent and the 4.6 parent were being "pulled in" ... eventually resulting in a conflict 
of one sort or another.
Comment 29 Eclipse Genie CLA 2015-08-19 10:34:03 EDT
New Gerrit change created: https://git.eclipse.org/r/54112
Comment 30 Eclipse Genie CLA 2015-08-19 10:37:37 EDT
New Gerrit change created: https://git.eclipse.org/r/54114
Comment 31 Lars Vogel CLA 2015-08-19 10:42:17 EDT
(In reply to Eclipse Genie from comment #29)
> New Gerrit change created: https://git.eclipse.org/r/54112

David, can you have a short look, if that change is correct? If yes, I change also runtime, ua and ui.tools.
Comment 32 Eclipse Genie CLA 2015-08-19 10:45:06 EDT
New Gerrit change created: https://git.eclipse.org/r/54118
Comment 33 Lars Vogel CLA 2015-08-19 10:47:47 EDT
(In reply to Eclipse Genie from comment #32)
> New Gerrit change created: https://git.eclipse.org/r/54118

eclipse.platform.ua still reference to 4.5.0-SNAPSHOT. Should that also be updated?
Comment 34 Eclipse Genie CLA 2015-08-19 10:48:35 EDT
New Gerrit change created: https://git.eclipse.org/r/54121
Comment 35 David Williams CLA 2015-08-19 11:40:47 EDT
(In reply to Lars Vogel from comment #33)
> (In reply to Eclipse Genie from comment #32)
> > New Gerrit change created: https://git.eclipse.org/r/54118
> 
> eclipse.platform.ua still reference to 4.5.0-SNAPSHOT. Should that also be
> updated?

Yes. I assume you mean R4_5_maintenance, since master looked correct in referencing 4.6.0-SNAPSHOT. 

But, good catch ... and, just the sort of problem what will be caught automatically in production builds ... once this fix is complete. :)
Comment 36 Eclipse Genie CLA 2015-08-19 11:54:38 EDT
New Gerrit change created: https://git.eclipse.org/r/54130
Comment 37 Eclipse Genie CLA 2015-08-19 13:52:32 EDT
New Gerrit change created: https://git.eclipse.org/r/54144
Comment 40 Markus Keller CLA 2015-08-19 14:23:12 EDT
Here's a bash command that can be executed in the parent directory that holds all your Git repos:

$ find . -maxdepth 2 -name pom.xml -print0 | xargs -0 sed -i -s -z -r 's#  <repositories>\n    <repository>\n      <releases>\n        <enabled>true</enabled>\n      </releases>\n      <snapshots>\n        <enabled>true</enabled>\n      </snapshots>\n      <id>eclipse-hosted</id>\n      <url>https://repo.eclipse.org/content/repositories/eclipse/</url>\n    </repository>\n  </repositories>\n#  <!-- \n    To build individual bundles, we specify a repository where to find parent pom, \n    in case it is not in local maven cache already\n    and that parent pom also has fuller individual-bundle profile \n    defined that is combined with this one. --> \n  <profiles>\n    <profile>\n      <id>build-individual-bundles</id>\n      <repositories>\n        <repository>\n          <releases>\n            <enabled>true</enabled>\n          </releases>\n          <snapshots>\n            <enabled>true</enabled>\n          </snapshots>\n          <id>eclipse-hosted</id>\n          <url>https://repo.eclipse.org/content/repositories/eclipse/</url>\n        </repository>\n      </repositories>\n    </profile>\n  </profiles>\n#'

This avoids all copy/paste bugs and ensures we can easily do similar batch changes in the future. To verify/commit the changes in EGit, you can click Refresh in the Git Staging view.

I've pushed/fixed all eclipse.jdt* repos in master.
Comment 41 Eclipse Genie CLA 2015-08-19 14:44:55 EDT
New Gerrit change created: https://git.eclipse.org/r/54151
Comment 42 Eclipse Genie CLA 2015-08-19 14:48:18 EDT
New Gerrit change created: https://git.eclipse.org/r/54152
Comment 43 Markus Keller CLA 2015-08-19 15:44:13 EDT
> New Gerrit change created: https://git.eclipse.org/r/54151
> New Gerrit change created: https://git.eclipse.org/r/54152

These are for resources.git and team.git.

I've done the rest of platform in master (except for UI, where Lars is working).

TODO: 
- Equinox/p2 (I've pushed rt.equinox.bundles)
- PDE
Comment 44 Eclipse Genie CLA 2015-08-19 16:25:16 EDT
New Gerrit change created: https://git.eclipse.org/r/54157
Comment 50 Eclipse Genie CLA 2015-08-19 18:16:56 EDT
New Gerrit change created: https://git.eclipse.org/r/54172
Comment 52 Eclipse Genie CLA 2015-08-19 18:20:22 EDT
New Gerrit change created: https://git.eclipse.org/r/54173
Comment 53 Eclipse Genie CLA 2015-08-19 18:21:55 EDT
New Gerrit change created: https://git.eclipse.org/r/54174
Comment 54 Eclipse Genie CLA 2015-08-19 18:22:27 EDT
Gerrit change https://git.eclipse.org/r/54174 was merged to [R4_5_maintenance].
Commit: http://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/?id=46014c5bd11d5274722cbaedc32c86f646e0c18a
Comment 55 Lars Vogel CLA 2015-08-19 18:23:13 EDT
(In reply to Markus Keller from comment #43)
> TODO: 
> - Equinox/p2 (I've pushed rt.equinox.bundles)
> - PDE

I did PDE for master and R4_5_maintenance.
Comment 56 Eclipse Genie CLA 2015-08-19 18:26:20 EDT
New Gerrit change created: https://git.eclipse.org/r/54175
Comment 57 Lars Vogel CLA 2015-08-19 18:35:01 EDT
eclipse.platform.ui.tools
eclipse.platform.ui
eclipse.platform.runtime
eclipse.platform.ua
pde.ui

Should be updated in master and in R4_5_maintenance.
Comment 58 Jay Arthanareeswaran CLA 2015-08-20 01:44:45 EDT
JDT Core changes have been picked into R4_5_maintenance:

http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/?h=R4_5_maintenance&id=5528f8c07e309d70250a52f7d60d06f5fe77ac0f
Comment 61 Dani Megert CLA 2015-08-20 06:21:42 EDT
(In reply to Markus Keller from comment #43)
> > New Gerrit change created: https://git.eclipse.org/r/54151
> > New Gerrit change created: https://git.eclipse.org/r/54152
> 
> These are for resources.git and team.git.

Done, see above two comments.
Comment 62 Eclipse Genie CLA 2015-08-20 07:52:22 EDT
New Gerrit change created: https://git.eclipse.org/r/54217
Comment 63 Eclipse Genie CLA 2015-08-20 07:52:34 EDT
New Gerrit change created: https://git.eclipse.org/r/54218
Comment 64 Eclipse Genie CLA 2015-08-20 07:52:57 EDT
New Gerrit change created: https://git.eclipse.org/r/54219
Comment 65 Eclipse Genie CLA 2015-08-20 08:08:53 EDT
New Gerrit change created: https://git.eclipse.org/r/54222
Comment 68 Markus Keller CLA 2015-08-20 10:27:46 EDT
Remaining Gerrit reviews for master:

eclipse.pde: https://git.eclipse.org/r/#/c/54217/
eclipse.pde.build: https://git.eclipse.org/r/#/c/54218/

Once these are done and David can confirm that master works fine, we can put the changes into R4_5_maintenance as well.
Comment 71 David Williams CLA 2015-08-20 16:08:18 EDT
(In reply to Markus Keller from comment #68)

> Once these are done and David can confirm that master works fine, we can put
> the changes into R4_5_maintenance as well.

I have done a local, test I-build and it succeeded. 

Ideally, someone interested in the Gerrit jobs that use "build individual bundle" might want to test it too. (though, I know Mickael had done some of that. 

Otherwise, thanks for fixing master ... on to maintenance?
Comment 72 Markus Keller CLA 2015-08-21 11:13:33 EDT
Pushed to R4_5_maintenance:
- JDT (including eclipse.jdt.core.binaries)
- Platform, except for Resources & Team
  - also fixed unnecessary whitespace changes in UI repos that only make life
    harder for existing and new contributors

For the remaining repos, the respective commits in master were fine on the first attempt, so they can just be cherry-picked to R4_5_maintenance.
Comment 74 Thomas Watson CLA 2015-08-21 14:33:45 EDT
R4_5_maintenance done for:

rt.equinox.bundles
rt.equinox.framework
rt.equinox.p2
Comment 75 Eclipse Genie CLA 2015-08-26 09:45:11 EDT
New Gerrit change created: https://git.eclipse.org/r/54588
Comment 76 Eclipse Genie CLA 2015-08-26 09:45:29 EDT
New Gerrit change created: https://git.eclipse.org/r/54589
Comment 77 Eclipse Genie CLA 2015-08-26 09:51:02 EDT
Gerrit change https://git.eclipse.org/r/54589 was merged to [R4_5_maintenance].
Commit: http://git.eclipse.org/c/pde/eclipse.pde.build.git/commit/?id=85748f1437f895a5f6292519eb122b493b49c965
Comment 78 Eclipse Genie CLA 2015-08-26 09:51:15 EDT
Gerrit change https://git.eclipse.org/r/54588 was merged to [R4_5_maintenance].
Commit: http://git.eclipse.org/c/pde/eclipse.pde.git/commit/?id=b09a2f2ce1b1c3fba3ab25e89d3a6f30f543a8b7
Comment 79 Markus Keller CLA 2015-08-26 09:54:25 EDT
Thanks Dani for finishing up with PDE. R4_5_maintenance is now done as well.
Comment 80 David Williams CLA 2015-09-27 00:15:51 EDT
As we prepare to move to "4.5.2", I thought a good opportunity to test this fix. To make sure the build failed, if parent pom's not correctly referred to. 

On my local test system, I changed parent pom, to 4.5.2, observed "build failed", then one or three at a time, changed each of the main repo pom's to point to 4.5.2 parent, and indeed the build continued to fail, until *all* were changed. 

I also, then, had the idea I've entered into bug 478483. During production builds if someone is accidentally still relying on previous build, then the production build will likely fail if we pass in a bogus "previous build" URL. (If does not fail, it will at least be accurate). 

But the fix that was applied here, besides being very cool, did work.