Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 349300 - [releng] Automate the creation of p2.mirrorsURL and p2.statsURI properties in the generated properties
Summary: [releng] Automate the creation of p2.mirrorsURL and p2.statsURI properties in...
Status: NEW
Alias: None
Product: OCL
Classification: Modeling
Component: Core (show other bugs)
Version: 3.1.0   Edit
Hardware: PC Windows 7
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: OCL Inbox CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on: 373667
Blocks:
  Show dependency tree
 
Reported: 2011-06-14 06:31 EDT by Adolfo Sanchez-Barbudo Herrera CLA
Modified: 2014-05-27 12:51 EDT (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Adolfo Sanchez-Barbudo Herrera CLA 2011-06-14 06:31:29 EDT
During the quite week, I've been working on the (manual) introduction of some interesting p2 properties in our Indigo release repository[1]

- p2.mirrorsURL which allows to download our P2 repository artifacts from eclipse mirrors.
- p2.statsURI which does some kind of downloads count of our artifacts. I've decided to only register stats for the following "main" features:
    org.eclipse.ocl.examples
    org.eclipse.ocl.all.sdk

More information about these properties may be found here [2] [3].

It could be interesting to automate the introduction of such p2 properties during the build process. It needs investigation.

[1] http://download.eclipse.org/modeling/mdt/ocl/updates/releases/3.1.0 
[2] http://wiki.eclipse.org/Equinox/p2/p2.mirrorsURL
[3] http://wiki.eclipse.org/Equinox_p2_download_stats
Comment 1 Laurent Goubet CLA 2011-06-14 06:55:15 EDT
Adolfo,

I don't know how you add the p2.statsURI to the update site, but we do this (semi-)automatically for Acceleo and EMF Compare through the evaluation of an xslt stylesheet when we promote new versions. We promote manually by logging in on dev.eclipse.org and using a command such as "ant -f promoter.ant"... and this ant file calls the xslt engine on our update site before publishing it.

I don't know if such a method can be applied for OCL, but for the record, you can find how we do this in :pserver:dev.eclipse.org:/cvsroot/modeling/org.eclipse.m2t/org.eclipse.acceleo/releng/org.eclipse.acceleo.releng.

The files of interest in there are addDownloadStats.xsl and promoter.ant.

If you wish to use this method and need more information, you have my Skype id :).
Comment 2 Adolfo Sanchez-Barbudo Herrera CLA 2011-07-27 13:41:45 EDT
My last tests looks like to produce the expected artifacts.xml in the p2.repository version (including the zipped one which will be available in downloads). 

However, there is a weird behavior:

- In my local Eclipse all works as expected.
- The artifacts.xml file for the the I-buid I did two days ago actually has those properties.(http://download.eclipse.org/modeling/mdt/ocl/updates/interim/3.2.0)
- The artifacts.xml files for the the last N-build (from master) and M-build (from maintenance branch) don't have the properties.

Looking at our last master's successful build, the tuning scripts were properly executed, because in our ocl-build feature we have a modified artifacts.xml which properly contains the p2.statsURI and p2.mirrorsURL properties [1]. However the produced p2 repository doesn't contain them [2]

On the other hand, where are the archived artifacts from said I-build [3]? ... It's not the first time it happens >.<

P.S: Laurent, thanks for the information. I finally took as reference the Xtext stuff, since it also had a similar stuff to do that and I already had it in my workspace ;). If it helps to you we are now also including the mirrorsURL in the generated p2 repository

P.S2: UPDATED: The last build[4] (run automatically) contains the properties... 

[1] https://hudson.eclipse.org/hudson/job/buckminster-mdt-ocl-core-3.2-master/ws/org.eclipse.ocl.git/releng/org.eclipse.ocl.build-feature/artifacts.xml/*view*/
[2] https://hudson.eclipse.org/hudson/job/buckminster-mdt-ocl-core-3.2-master/401/artifact/MDT-OCL.p2.repository/

[3] https://hudson.eclipse.org/hudson/job/buckminster-mdt-ocl-core-3.2-master/397/

[4] https://hudson.eclipse.org/hudson/job/buckminster-mdt-ocl-core-3.2-master/402/artifact/MDT-OCL.p2.repository/

Best Regards,
Adolfo.
Comment 3 Adolfo Sanchez-Barbudo Herrera CLA 2011-07-28 09:24:57 EDT
Update with our last automatically run builds:

Maintenance (M-builds) [1]: Does  include the modified artifact.xml
Master (N-build) [2]: Doest NOT include the modified artifact.xml

Random Powah !! ... :(

Laurent, have you had any similar issue to this one ?. Does your generated p2 repository artifact.xml ALWAYS include the p2.statsURI property ?

[1] https://hudson.eclipse.org/hudson/job/buckminster-mdt-ocl-3.1-maintenance/621/artifact/MDT-OCL.p2.repository/
[2] https://hudson.eclipse.org/hudson/job/buckminster-mdt-ocl-core-3.2-master/403/artifact/MDT-OCL.p2.repository/
Comment 4 Laurent Goubet CLA 2011-07-28 09:33:30 EDT
(In reply to comment #3)
> Update with our last automatically run builds:
> 
> Maintenance (M-builds) [1]: Does  include the modified artifact.xml
> Master (N-build) [2]: Doest NOT include the modified artifact.xml
> 
> Random Powah !! ... :(
> 
> Laurent, have you had any similar issue to this one ?. Does your generated p2
> repository artifact.xml ALWAYS include the p2.statsURI property ?
> 
> [1]
> https://hudson.eclipse.org/hudson/job/buckminster-mdt-ocl-3.1-maintenance/621/artifact/MDT-OCL.p2.repository/
> [2]
> https://hudson.eclipse.org/hudson/job/buckminster-mdt-ocl-core-3.2-master/403/artifact/MDT-OCL.p2.repository/

We add these properties only when we promote our builds... So I can't really help here. I am confident that our promoted builds have them since we promote through an ANT file and it is that very ant file which is in charge of adding that stuff.

For something automated with every build ... I don't know :s. Maybe your script does add the two necessary properties, but does/can not re-zip the artifacts file?
Comment 5 Adolfo Sanchez-Barbudo Herrera CLA 2011-12-22 13:52:18 EST
Remembering to change the scripts to discern stats for core and tools downloads
Comment 6 Adolfo Sanchez-Barbudo Herrera CLA 2012-03-08 11:48:24 EST
(In reply to comment #5)
> Remembering to change the scripts to discern stats for core and tools downloads

Before doing that, I should do the merge between core and tools build, so that I only have one place on which work
Comment 7 Adolfo Sanchez-Barbudo Herrera CLA 2013-06-10 13:04:55 EDT
Today, I've been trying to track what's wrong with this issue.

My experience on randomness implies some hudson node related issues:

I've tried the following nodes, with no stats/mirrorsURL after all:
- hudson-slave2 [1]
- master [2]

If I'm not able to produce the desired P2 repository, I'll promote it anyway. If no luck tomorrow with Tools one, I'll add them manually during the quiet week.

[1] https://hudson.eclipse.org/hudson/job/buckminster-mdt-ocl-core-kepler-master/804/
[2] https://hudson.eclipse.org/hudson/job/buckminster-mdt-ocl-core-kepler-master/806/
Comment 8 Adolfo Sanchez-Barbudo Herrera CLA 2013-06-11 07:59:23 EDT
The third try worked:
- hudson-slave1 [1]

After having lunch, I'll use the Tools RC4 build to verify if the hudson node is related to the randomness of the success of the p2 repostiory tuning activities.

[1] https://hudson.eclipse.org/hudson/job/buckminster-mdt-ocl-core-kepler-master/808/
Comment 9 Adolfo Sanchez-Barbudo Herrera CLA 2013-06-11 14:19:55 EDT
Tools report. No stats after kicking out the build in the current configured node:
- hudson-slave2 [1]

I'm trying hudson-slave1 to see if now succeeds as it happened yesterday with the Core build.

Curiously, every time I changed the node the next build fails with a "NFS stale" error. The next one works (as my experience Yesterday). Running the job again to see the final result, what respects the P2 repository tuning activities.
Comment 10 Adolfo Sanchez-Barbudo Herrera CLA 2013-06-11 15:09:34 EDT
(In reply to comment #9)
> Running the job again to see the final result, what respects the P2 repository tuning activities.

No way :(. The randomness is not related to specific nodes. In hudson-slave1 p2 repo tuning activities worked yesterday with OCL Core build, however it didn't work today with OCL Tools one.
Comment 11 Ed Willink CLA 2014-05-27 12:51:10 EDT
(In reply to Adolfo Sanchez-Barbudo Herrera from comment #9)
> "NFS stale" error

can be caused by Buckminsters allocation of temp directories from a shared pool. Some other job may re-use and evn lock out with wrong ownership.

Now that we have a HIP for OCL+QVTd such conflicts should not arise.

Now that we don't have distinct Core/Tools builds it may be easier.

May be worth another try.