Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 318090 - [releng] Introduce distinct Core and Tools builds
Summary: [releng] Introduce distinct Core and Tools builds
Status: CLOSED FIXED
Alias: None
Product: OCL
Classification: Modeling
Component: Core (show other bugs)
Version: 3.0.0   Edit
Hardware: PC Windows Vista
: P3 normal (vote)
Target Milestone: 3.2.0   Edit
Assignee: Adolfo Sanchez-Barbudo Herrera CLA
QA Contact:
URL:
Whiteboard: Usability
Keywords:
Depends on:
Blocks: 318358
  Show dependency tree
 
Reported: 2010-06-26 14:17 EDT by Ed Willink CLA
Modified: 2014-05-27 09:52 EDT (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ed Willink CLA 2010-06-26 14:17:56 EDT
The OCL core functionality (o.e.ocl, o.e.ocl.ecore, o.e.ocl.uml and associated .edit plugins) should be built at Platform+1 as at present.

The OCL tools functionality (editors, console, examples) should be built at +3 to ensure sensible dependency on Xtext.
Comment 1 Adolfo Sanchez-Barbudo Herrera CLA 2010-06-28 06:06:30 EDT
Picking up a Kenn's comment from the newsgroup:

"Be careful regarding the "Core" vs. "Tools" issue, so that the content you are providing in the project still fits within the project's scope. If it does not, you can either ask the PMC to allow you to change/broaden the scope or create a separate project."
Comment 2 Ed Willink CLA 2010-06-28 12:24:59 EDT
I'm waiting for a reply to my mdt-ocl-dev posting asking where the scope of MDT/OCL is defined. If no response within a week we'd better ping him.

The only project for which there might be a conflict is sphinx with whom we should be working. I'm waiting for feedback from Papyrus on use of the one line expression editor, which we could practice integration on in the OCL Console.
Comment 3 Adolfo Sanchez-Barbudo Herrera CLA 2011-08-03 10:33:05 EDT
Ed,

I'm trying to a elaborate a sensible layout for what a Core and a Tool job should produce. In principle, my idea is the following:

Core job:
- It will produce a p2 repository with the org.eclipse.ocl.all.sdk feature (SDK) and all the transitively required ocl features/plugins (including lpg bundles).
- It will produce the following zips
		a zipped update site with the SDK.
		a zip with the SDK plugins/features.
		a zip with the Core SDK plugins/features.
		a zip with the runtime plugins/features.
		
Tools job:
- It will produce a p2 repository with the org.eclipse.ocl.examples feature (Examples feature) and all the transitively required ocl examples plugins.
- It will produce the following zips
       a zipped update site with the Examples.
       a zip with the Examples.
       
The idea is that when a core-build successfully runs, it triggers a run of the tools one.

When publishing, each p2 repository will be published in a distinct location. However our public repositories will show both SDK and Examples features (by the means of the composite repository fancies)

On the other hand, I'm not sure what to do with the org.eclipse.ocl.master feature (All in one SDK). IMO, this feature has not been very useful so far. In the simultaneous release train builds, we contribute the SDK and the Examples feature. Besides, when I try to include said master feature in the Tools job, due to transitive features inclusion all OCL features (and plugins) are included in the generated p2 repository.

Do we want to expose the CoreSDK and Runtime features (category.xml file) in the Eclipse Core p2 repositories ?

Another worrying point is the downloads web page (which is currently shared along most MDT projects). Since different produce different artifacts, we will have different builds which provides different artifacts. For example, milestones:

3.2.0M1 would produce some artifacts (core artifacts)
3.2.0M1a would produce the remaining ones (tools/examples one).

This could be more confusing for what concerns N-builds. Another approach I could try to do is also preparing some zips with the core stuff when publishing a tools build.

The last point to sort out is that I'll have to some kind of distinction concerning tests. I was thinking about having two tests features, one for core tests and a new one for tools (examples) one.

Finally, I would also to bring up the idea (for the future) about having a similar distinction for examples plugins. Do we want to have some examples in the core stuff and other examples in the tools stuff?.

BTW, I'll have to take some vacations next week (before I start a new project at work in middle August). I'm not sure if this stuff will be ready for JunoM1... In any case, I've already created a branch for it, and it will be pushed to Eclipse repository today and/or tomorrow.

Best Regards,
Adolfo.
Comment 4 Ed Willink CLA 2011-08-03 12:02:55 EDT
M1, M1a. I don't understand this. Each of Core and Tools should produce M1, jusr Tools is two days later. Look at MWE's download page that actually has 3 build contributions.

Examples is certainly a confusion. We may want the option of distinct Core/Tools examples one day, but for the most part examples are for big installations with tools, so for now just examples in Tools. All examples use some tool anyway.

The master feature combines both core and examples. It can become a bigger Tools feature for all of Core and all of Tools. I'm inclined to try to leave features largely unchanged, because any change always upsets someone.
Comment 5 Adolfo Sanchez-Barbudo Herrera CLA 2011-08-03 13:16:43 EDT
(In reply to comment #4)
> M1, M1a. I don't understand this. Each of Core and Tools should produce M1, jusr
> Tools is two days later. Look at MWE's download page that actually has 3 build
> contributions.

Have you looked at Nightly builds in the page you mentioned?. As a similar issue/example, look at the following N-builds:

N201108020306 (2011/08/02)
N201107141823 (2011/07/14) 

N-builds section looks like purely automated section. R-builds undoubtedly have its manual counterpart (confirmed by Dennis), to make the "same" build area be fed by artifacts which come from different builds.... 

> 
> Examples is certainly a confusion. We may want the option of distinct Core/Tools
> examples one day, but for the most part examples are for big installations with
> tools, so for now just examples in Tools. All examples use some tool anyway.
> 

I agree... Anyway, once the releng infrastructure is working to make this Core/Tools distinction, we shouldn't find too many problems to make such a distinction at an examples level. In any case, it's a future concern. By the moment, all examples will be moved 

> The master feature combines both core and examples. It can become a bigger Tools
> feature for all of Core and all of Tools. I'm inclined to try to leave features
> largely unchanged, because any change always upsets someone.

I'm not sure I've received a convincing reply. It looks like clear the contents for the Eclipse OCL Core repository. Concerning the Tools repository one. I see two Options:

1. Excluding the master feature from the Eclipse Tools Repository, so that we'd have two different repositories (Core and Tools) with different content (SDK in one repo, Examples in the other one).

2. Including master feature in the Tools repository, so that we'd have the Core features/plugins "replicated" in the Tools repository. In that case, we could only expose the Master and Examples features /via Cactegory.xml).

A good point for the second option is that installing/updating the master feature, should automatically install/update Examples and Core features/plugins (taking them from the Tools repository, rather than the Core one). It may be possible in both solutions that somebody updates the Examples feature without updating the SDK one.

As I understand from your comments, you prefer the second one, right ?

Best Regards,
Adolfo.
Comment 6 Ed Willink CLA 2011-08-04 01:11:40 EDT
I'm aware that the MWE builds are not perfect; 2 out of 3 are actually the same; it just demonstrates that more than one build is possible.

I don't really understand repositories, so my thinking may not be appropriate.

I see a repository as like a directory, some files can come from a +1 build, others from a +3 build, so I was not expecting multiple repositories.

If multiple leaf repositories are necessary, then I'm not keen on replicated content. Surely a distinct Tools repository can aggregate/delegate to the Core repository? The cost of the master feature should just be an extra 10kB in the Tools repository.

If a master feature is too hard, leave it out at least for now. The P2 Installation should automatically install the Core when it install the tools, just the same as it automatically install Xtext and UML.
Comment 7 Adolfo Sanchez-Barbudo Herrera CLA 2011-08-04 06:36:15 EDT
(In reply to comment #6)
> I'm aware that the MWE builds are not perfect; 2 out of 3 are actually the same;
> it just demonstrates that more than one build is possible.

Of course, I'm just trying to agree how level of automation we want for our builds.... I really don't mind having to do manual tasks (copy/move/delete files in the ssh console to include the TOOLS artifacts inside the downloads created for a CORE build) in our stable and releases builds to have a similar MWE download webpage, providing the frequency is every 6 weeks... I prefer a pure automated mechanism, though.

> 
> I don't really understand repositories, so my thinking may not be appropriate.
> 
> I see a repository as like a directory, some files can come from a +1 build,
> others from a +3 build, so I was not expecting multiple repositories.

It was not easy, but I finally found the following threads:

http://dev.eclipse.org/mhonarc/lists/mdt-ocl.dev/msg00718.html
http://dev.eclipse.org/mhonarc/lists/mdt-ocl.dev/msg00719.html

The agreed solution I had in mind from this months-ago discussion, is simply what I've tried to do... Of course, I'm opened to change the solution (minds change ;) ), as long as the solution doesn't make me spend too much time.

By the way, the idea you pointed out was using 3.2.0M1-Core and 3.2.0M1-Tools (instead using the 'a' suffix). Again, concerning the download web page, I'm opened to MANUALLY include the examples artifacts from the second build into the downloads area of the first build (to have a similar MWE releases download area). Then I'd have to delete the downloads area created by the promotion script of the second build.... However, I'm in favor of the opinion that the lesser manual tasks we have to do the better.

> 
> If multiple leaf repositories are necessary, then I'm not keen on replicated
> content. Surely a distinct Tools repository can aggregate/delegate to the Core
> repository? The cost of the master feature should just be an extra 10kB in the
> Tools repository.
> 

The point is that buckminster (well, probably the pde/p2 tasks on which it relies) automatically generates the repository based on the features organization. The transitive inclusion of the OCL features (starting from org.eclipse.ocl.master feature) will make include every ocl feature/and plugin in the generated repository. I don't know if there is a way to change this... 

So it's not a problem of making the Tools repository composes different repositories, the point is that an automatically generated repository in which we want the master feature, will always have ALL ocl the stuff (unless a more experienced user points us another solution)

I've just checked the EMF 2.7 releases repository, they have a composite repository which composes a BASE repository (to be early used by the e4 project) and a CORE repository (all the EMF stuff). If I have a look to the features in each leaf repository, I can check that they provide replicated content:

BASE contains the org.eclipse.emf.ecore_2.7.0.v20110605-0747.jar
CORE contains the org.eclipse.emf.ecore_2.7.0.v20110605-0747.jar 

So it looks like that EMF has opted for the replicated content solution.

> If a master feature is too hard, leave it out at least for now. The P2
> Installation should automatically install the Core when it install the tools,
> just the same as it automatically install Xtext and UML.

Well, this assertion needs some accuracy. Installing tools (current Examples feature) will install/update whatever is specified at the Examples feature's dependencies/requirements. This doesn't necessarily install the SDK feature (and therefore, documentation, for example), actually it won't be installed. For example, if you install OCL Examples, you will have installed some xtext stuff, but you will still be allowed to install the Xtext SDK in the Install New Software wizard.

It looks like we still need some thoughts about how our features and/repositories should be organized ;P
Comment 8 Ed Willink CLA 2011-08-05 09:10:20 EDT
I don't really understand the problems so I'm finding it difficult to make helpful comments. I think you know roughly what we're trying to achieve, so perhaps it's best for you to produce a strawman taking the easier options, then we can see how it works.

I think that the usages we must support at the feature level are:

Provision of a run-time feature for Ecore only.
Provision of a run-time feature for Ecore + UML.
[Provision of a run-time feature for Pivot]
Provision of a full SDK for Ecore + UML.
[Provision of a full SDK for Pivot]
[Provision of a full SDK for Ecore + UML + Pivot]

Provision of a run-time feature for Editors and Console
Provision of a full SDK for everything

[once examples promoted to core]

At the update level

Provision of a full SDK for Ecore + UML
[Provision of a full SDK for Pivot]
[Provision of a full SDK for Ecore + UML + Pivot]
Provision of a full SDK for everything

How the repositories are organised is an implementation detail; just so long as they're useable, unsurprising, and have as few name changes as possible for traditional Ecore and UML consumers.
Comment 9 Adolfo Sanchez-Barbudo Herrera CLA 2011-11-08 13:41:37 EST
Well,

A first tools - job has been published at:

http://download.eclipse.org/modeling/mdt/ocl/updates/nightly/4.0.0/tools

Things start to work, but there are still pending issues.

1. Tools tests are unstable and they don't include Axel's test cases yet.
2. Fix the generated p2 repository.

Tomorrow I want to:
- Fix 1 and 2.
- Modify our core-job so that it uses our new org.eclipse.ocl.releng.core.build feature (working in  branch-tests, although I've not published the artifacts)
- Run a verify the correct publishing process for the core job.
- Prepare our public p2 repositories to adopt the new core/tools layout.
- Modify the web page to be able to download the new zips (distinct between Update-Tools and Update-Core).
- Create a list of pending enhancements for this bugzilla.

Best Regards,
Adolfo.
Comment 10 Adolfo Sanchez-Barbudo Herrera CLA 2011-11-09 08:26:13 EST
Great !!

I have our public nightly repository showing the proper features which belong to different p2 repositories generated by different builds (Core and Tools).

http://download.eclipse.org/modeling/mdt/ocl/updates/nightly/4.0.0

I'll be fixing around other bits.

Cheers,
Adolfo.
Comment 11 Adolfo Sanchez-Barbudo Herrera CLA 2011-11-09 09:32:55 EST
Good,

1. Tools tests pass (axel's tests are inexplicably failing)
2. Generated repository from tools job has been fixed.
3. Generated repository from Core job looks good.
4. Nightly repository has been configured to expose core and tools repositories.
5. Web page has been accordingly modified (see nightly downloads).

Current inconveniences:
1. Examples zip packaging fails, with no clues about the error.
2. Axel's tests are not executing, with no clues about how to solve the error.

I'm worried about the inconvenience 2, I'm not sure if I should push and configure the core and tools jobs to use the new releng infrastructure.

I'm going to have lunch. Please, Ed could you provide some thought about what to do .... I only have this afternoon to finish this bug. Tomorrow, I have to take my companu project up again.

BTW, I'll not change the milestones repository layout until Juno M3 is completed.
Comment 12 Ed Willink CLA 2011-11-09 13:36:41 EST
Good. This is looking better than I expected; the Tools ZIP includes the Core ZIP so users only really need the Tools ZIP. I've not looked at the p2 repos; preumably these should be non-overlapping if we're making two contributions.

I don't think we need two Docs; just leave the Docs out of the Core ZIP.

Comparing Tools ZIP with the M3 ZIP, the Tools ZIP is 5 files and 5 MB smaller. Differences seem to be in feature and doc sources.

The download page isn't very pretty; probably work in progress. With Tools ZIP including Core ZIP, I'm not sure that it is worth offering the Core Build ZIPs at all. This may make the download page easier to maintain and use. The only consumers I can see for a Core ZIP are Acceleo debugging at +1.5, and I'm sure they can manage to download from the Hudson page.

codegen is not yet in the Tools build.

We need to check that p2 can resolve an All-In-One install from the Tools ZIP.
Comment 13 Adolfo Sanchez-Barbudo Herrera CLA 2011-11-10 06:18:20 EST
(In reply to comment #12)
> Good. This is looking better than I expected; the Tools ZIP includes the Core
> ZIP so users only really need the Tools ZIP. I've not looked at the p2 repos;
> preumably these should be non-overlapping if we're making two contributions.

As stated in the old comments above, the tools p2 repository will include Core (duplicated) content because of the inclusion of the Master feature, which transitevely INCLUDES the remaining features.

The Tools and Core Update Zip, it's a simple zip of the p2 repository which will be published in http://download.eclipse.org/modeling/mdt/ocl/updates/...

We will need to discuss later if the tools build should include only examples or master + examples + sdk.

> 
> I don't think we need two Docs; just leave the Docs out of the Core ZIP.

I remember an old discussion about having documentation SDK. An SDK needs sources and documentation. The problem is that our documentation contains info about components which are not distributed in the SDK, that is, the examples.

> 
> Comparing Tools ZIP with the M3 ZIP, the Tools ZIP is 5 files and 5 MB smaller.
> Differences seem to be in feature and doc sources.

I'll have a look to it...

> 
> The download page isn't very pretty; probably work in progress. With Tools ZIP
> including Core ZIP, I'm not sure that it is worth offering the Core Build ZIPs
> at all. This may make the download page easier to maintain and use. The only
> consumers I can see for a Core ZIP are Acceleo debugging at +1.5, and I'm sure
> they can manage to download from the Hudson page.

Well remember Anthony hunter issues who was expecting some specific zips (SDK I think) to build EMF QTV.

I see the downloads web page as folows:

In regular (nightly, interim) builds:

- The Core build publishing process moves to downloads the 4 Core downloadable zips: SDK P2 repository, SDK, CoreSDK and Runtime.
- The Tools build publishing process moves to downloads the 2 Tools downloadable zips: All-In-One P2 repository, Examples.

In milestones/releases builds:

- When +1, The Core build publishing process move to downloada the 4 Core downloadable zips: SDK P2 repository, SDK, CoreSDK and Runtime.
- When +3, The Tools build publishing process move to downloads the 2 Tools downloadable zips: All-In-One P2 repository, Examples.
- The All-In-One P2 repository and the Examples zips would be moved to the folder in which The Core build resides, and the the Tools folder removed. Having a unique Milestone link with 6 zips.

For this last step, there is not automated script yet.

> 
> codegen is not yet in the Tools build.

Not sure how to do that, explain or do the changes in bug/318090.

> 
> We need to check that p2 can resolve an All-In-One install from the Tools ZIP.

I tried that from M2 installation, but I got some dependency problem with a platforom plugin version. I'll try it again when M3 is out.

I guess that all the stuff will be in stand by until december, or do we want to
adopt it before all the bits are sorted out ? As commented, my main concern (apart from the differences you mentioned above) is that Axel's test cases are not integrated.

Regards,
Adolfo.
Comment 14 Ed Willink CLA 2011-11-10 13:13:53 EST
(In reply to comment #13)
> As stated in the old comments above, the tools p2 repository will include Core
> (duplicated) content because of the inclusion of the Master feature, which
> transitevely INCLUDES the remaining features.

How do we ensure that a +3 contribution has the exactly the same plugins as the +1 contribution. With CVS we build tags, so builds were reproducible? Currently we build from master so if there is a commit to master at +2, +3 will differ from +1.

I was assuming no overlap, so no conflict. I though the Tools ZIP was just copying the Core build.

> 
> We will need to discuss later if the tools build should include only examples
> or master + examples + sdk.
> 
> > 
> > I don't think we need two Docs; just leave the Docs out of the Core ZIP.
> 
> I remember an old discussion about having documentation SDK. An SDK needs
> sources and documentation. The problem is that our documentation contains info
> about components which are not distributed in the SDK, that is, the examples.

I would like the Tools ZIPs to be everything and the Core builds minimal. It is just confusing to have any Docs or Examples in Core. With Docs in Tools only there is no omission. I see no real need for a Core SDK at all, apart from a bit of feature compatibility. 

> Well remember Anthony hunter issues who was expecting some specific zips (SDK I
> think) to build EMF QTV.

Yes these need to be available for builds, but need not clutter the download page, where I would like to see a Tools All-In-One and perhaps a few smaller ZIPs. Only one entry per milestone.


> I see the downloads web page as folows:
> 
> In regular (nightly, interim) builds:
> 
> - The Core build publishing process moves to downloads the 4 Core downloadable
> zips: SDK P2 repository, SDK, CoreSDK and Runtime.
> - The Tools build publishing process moves to downloads the 2 Tools
> downloadable zips: All-In-One P2 repository, Examples.
> 
> In milestones/releases builds:
> 
> - When +1, The Core build publishing process move to downloada the 4 Core
> downloadable zips: SDK P2 repository, SDK, CoreSDK and Runtime.

Ok for now. I'm inclined to try to trim to SDK P2, Runtime. (No docs or examples). No move to downloads. Just repos for +2 builds and aggrcon contribution.

> - When +3, The Tools build publishing process move to downloads the 2 Tools
> downloadable zips: All-In-One P2 repository, Examples.

I would do most of the options here, so that all the move to downloads is in the one build.

> - The All-In-One P2 repository and the Examples zips would be moved to the
> folder in which The Core build resides, and the the Tools folder removed.
> Having a unique Milestone link with 6 zips.
> 
> For this last step, there is not automated script yet.
> 
If only the Tools build moves to downloads, there is nothing to automate.
> > 
> > codegen is not yet in the Tools build.
> 
> Not sure how to do that, explain or do the changes in bug/318090.

It's in the GIT master history, since I added it and retracted it.

Add o.e.o.examples.codegen to examples feature
Add Acceleo to repo search path
Add Acceleo, UML codegen plugins to plugin dependencies
 
I'll doi it when you've merged to master if you like.
> > 
> > We need to check that p2 can resolve an All-In-One install from the Tools ZIP.
> 
> I tried that from M2 installation, but I got some dependency problem with a
> platforom plugin version. I'll try it again when M3 is out.

Yes. M2 was troubled.
> 
> I guess that all the stuff will be in stand by until december, or do we want to
> adopt it before all the bits are sorted out ? As commented, my main concern
> (apart from the differences you mentioned above) is that Axel's test cases are
> not integrated.

Since they used to work it must be a 'simple' dependency typo.

I'd like to have a complete Tools build soon so that I can check out the codegen and we are ready well before M4.
Comment 15 Adolfo Sanchez-Barbudo Herrera CLA 2011-11-10 14:57:58 EST
(In reply to comment #14)
> 
> How do we ensure that a +3 contribution has the exactly the same plugins as the
> +1 contribution. With CVS we build tags, so builds were reproducible? Currently
> we build from master so if there is a commit to master at +2, +3 will differ
> from +1.

Tools build doesn't use SCM for "core" plugins/features, but use a p2 repository generated from the Core build. In a similar way we use p2 repository for dependent plugins (f.e, EMF). You may verify this looking at the build workspace:

1. The target platfrom contains core ocl plugins/features: https://hudson.eclipse.org/hudson/job/buckminster-mdt-ocl-tools-3.2-master/ws/buildroot/target.platform/plugins/

2. There is no output for ocl bundles other than examples bundles: https://hudson.eclipse.org/hudson/job/buckminster-mdt-ocl-tools-3.2-master/ws/buildroot/buckminster.output/

In principle the "core" p2 repository is configured to use the last successful p2 repository of the Core build and my idea is parameterizing this so that we may set the proper p2 repository for milestones and releases builds. However, at the time of writing I've realized that we could rather use our public core repositories (nightly, milestones, etc) depending of the build type, in a similar way we do for EMF, UML, Xtext, etc

> 
> I was assuming no overlap, so no conflict. I though the Tools ZIP was just
> copying the Core build.

As commented, Core and Tools jobs are different builds which generate different artifacts

> 
> I would like the Tools ZIPs to be everything and the Core builds minimal. It is
> just confusing to have any Docs or Examples in Core. With Docs in Tools only
> there is no omission. I see no real need for a Core SDK at all, apart from a bit
> of feature compatibility.
> 

Again it's a matter of feature organization and selection of such features for each build job. Core build produces documentation because it's building the SDK which transitively includes documentation.

Another alternative is that the Core job built the CoreSDK (no doc, no edit) so that it would produce a proper CoreSDK for downstream projects. The SDK (with edit and doc) + examples would be produced by the tools job.

This looks good in terms usage (Core build produces CoreSDK). This doesn't look good from the point of view of not building the SDK at +1 as we used to, but assuming we will be producing more content at +3 (examples which we also use to produce at +1), this could be acceptable as soon as we don't break anything.

> 
> Yes these need to be available for builds, but need not clutter the download
> page, where I would like to see a Tools All-In-One and perhaps a few smaller
> ZIPs. Only one entry per milestone.
> 

A downloadable zips based build, will require having them in the downloads area and a web page to make the release engineer know the URL of the new zips. A user who likes installing components via zips (dropins ?) would like to see the different packaged zips we have always provided, wouldn't he ?

As commented the milestones and releases will have only one entry with 6 zips (Core Update Site, Tools Update Site, SDK, CoreSDK, Runtime and Examples).

> 
> Ok for now. I'm inclined to try to trim to SDK P2, Runtime. (No docs or
> examples). No move to downloads. Just repos for +2 builds and aggrcon
> contribution.
> 

I'm not sure what you are suggesting. Are you suggesting no creating downlodable zips for nightly builds ? Creating them but no moving them to the downloads area ? We could think about this, but I'd create a different enhacement to discuss and tackle it.

> 
> Since they used to work it must be a 'simple' dependency typo.
> 

It's not so trivial, core-master job is successfully running the test cases.

> I'd like to have a complete Tools build soon so that I can check out the codegen
> and we are ready well before M4.

I'm not sure how much time I saved yesterday for OCL..... but I'll try to have it the next week, after M3, let's see if I don't have to stay too many afternoons/evenings at work ;P... argg too many things to do too little for them >.<

Cheers,
Adolfo.
Comment 16 Adolfo Sanchez-Barbudo Herrera CLA 2011-11-17 08:20:39 EST
Ed,

Missing files come from some source features which belong to the Core build.  Sources are generally generated by buckminster, but the sources are usually located in the workspace and Core build features are in the target platform.

Moreover if we look at the Tools build target platform we have some features with sources and we have some features without them... The clue is that the Core build source features which are explicitly included in a feature, they are resolved in the target platform, for instance the following features:

org.eclipse.ocl
org.eclipse.ocl.uml
org.eclipse.ocl.edit

Source features which are not explicitly included: 

org.eclipse.ocl.all.sdk
org.eclipse.ocl.all
org.eclipse.ocl.core.sdk
org.eclipse.ocl.doc
org.eclipse.ocl.sdk

As commented in the Tools build, the org.eclipse.ocl.master feature has source counterpart because it's a feature of the workspace and buckminter generate source for them (if configured to do that, I need to add).

A similar issue occurs with the missing org.eclipse.ocl.doc.source plugin

So the question now is :

Should we include the source features for every feature, including the coordinating one ? I guess so...
Should we configure our features to explicitly include the source features ? If we shouldn't, why some source features are explicitly included and others are not ?.

BTW, EMF-Core repositories suffer a similar issue: There is no emf.common.source neither emf.ecore.source features...

While we decide about this issue, I'll try other means to fix this issue via buckminster tricks.

** Concerning the installation of the nightly p2 repo:

I had some problems installing the nightly repo due to signing issues. I forgot to include the SITE_SIGNING parameter in the build configuration. After fixing that, the current p2 nightly repo content may be successfully installed in a Eclipse Modeling Juno M3 :)
Comment 17 Adolfo Sanchez-Barbudo Herrera CLA 2011-11-17 08:44:46 EST
(In reply to comment #16)
> Ed,
> 
> Missing files come from some source features which belong to the Core build.
> Sources are generally generated by buckminster, but the sources are usually
> located in the workspace and Core build features are in the target platform.
> 
> 
> Source features which are not explicitly included:
> 
> org.eclipse.ocl.all.sdk
> org.eclipse.ocl.all
> org.eclipse.ocl.core.sdk
> org.eclipse.ocl.doc
> org.eclipse.ocl.sdk

I'm sorry, the last one doesn't exist... simply ignore it
Comment 18 Adolfo Sanchez-Barbudo Herrera CLA 2011-11-17 10:31:54 EST
Ed,

Doc plugin source has been fixed via including the source feature into the SDK feature, in a similar way ocl.edit sources are included.

Concerning the coordinating features, I've only got to include the source features into the workspace, however the buckminster action of generating the P2 repository doesn't consider it.

I don't want to delay the deployment for this issue .... if we wanted source features of the coordinating features (which doesn't provide any different from the original on) we could always configure them by explicitly including them.

I'll start the deployment of this stuff.

Regards,
Adolfo.
Comment 19 Adolfo Sanchez-Barbudo Herrera CLA 2011-11-17 12:06:15 EST
Good,

Core and Tools jobs are set and they have produced a proper nightly repository which allows a successful installation of every Eclipse OCL feature and plugin.

In the end I've configured the Tools rmap so that Eclipse Core artifacts are fetched from different repositories depending on the build-type, i.e:

N-Build: Uses buckminster-mdt-ocl-core-3.2-master last successful build
I-Build: Uses Eclipse OCL iterim repository
S-Build: Uses Eclipse OCL milestones repository
R-Build: Uses Eclipse OCL releases repository.

whenever the core job is run, the tools job will run after it.... a Core build is running to check that.

Regards,
Adolfo.
Comment 20 Ed Willink CLA 2011-11-17 14:34:44 EST
(In reply to comment #16)
> So the question now is :
> 
> Should we include the source features for every feature, including the
> coordinating one ? I guess so...

I've always found source features hard but eventually found the Athena templates. Buckminster is different, so simplistically it's all your code, if something's missing you've not got it quite right yet.

Naively, I expect that every plugin installed from an SDK should have corresponding sources, so that Import Plugin with source folders should give the users

a) full Java that auto-builds
b) full more abstract sources such as *.gi, *.xtext and launches etc so that they can regenerate the auto-generated Java.

[b) requires that the source build include o.e.o.examples.build.]

As such source features could be paired with SDK features, but whatever is easiest. I'm happy with a single Core and a single Tools source feature or even just a single Tools feature. As far as I'm concerned, we provide very small runtime features for indirect users such as GMF who are not aware that they're using OCL, but we really want to encourage modeling users to see the full range of the project, so a small SDK is undesirable.

I'm confused by the reference to 'the coordinating one'. Do features have source features? I don't think I've ever usefully imported features into a workspace.
Comment 21 Adolfo Sanchez-Barbudo Herrera CLA 2011-11-17 15:42:09 EST
> 
> I've always found source features hard but eventually found the Athena
> templates. Buckminster is different, so simplistically it's all your code, if
> something's missing you've not got it quite right yet.

Take into account that what we are trying to do was not done with Athena, it's not usual (including features from target platform into the generated repository). It's not a problem of the build system used, but the PDE task which is invoked by them.

The p2 repository generation come from PDE tasks (invoked buckminster) and the source features generation and inclusion into the generated repository, comes from features which are in the workspace not in the target platform ... so, if we don't have via features dependencies that a feature source must be included in the generated repository, we won't never have the desired feature.

That could be the reason why EMF is having the same issue.... basically it's not directly posible ... perhaps creating complicated scripts to include the source jar (which is really present in the target platform).

> 
> Naively, I expect that every plugin installed from an SDK should have
> corresponding sources, so that Import Plugin with source folders should give
> the users
> 
> a) full Java that auto-builds
> b) full more abstract sources such as *.gi, *.xtext and launches etc so that
> they can regenerate the auto-generated Java.
> 
> [b) requires that the source build include o.e.o.examples.build.]

We don't have problem with plugins...so it's ok... basically, every source plugin is included in the generated repository because the corresponding source features are included as well. And most of the source features are included because they are explicitly defined in the coordinating features (see the CoreSDK feature definition for example).


> 
> As such source features could be paired with SDK features, but whatever is
> easiest. I'm happy with a single Core and a single Tools source feature or even
> just a single Tools feature.

"The single Core feature" currently maps with the SDK (o.e.o.all.sdk) feature and the "The single Core feature" maps with the All-in-one (o.e.o.master) feature one.

As commented it could be interesting mapping the Core build with the CoreSDK feature, instead.

Have a look to our nightly repository, to check what users should be currently seeing.

> As far as I'm concerned, we provide very small
> runtime features for indirect users such as GMF who are not aware that they're
> using OCL, but we really want to encourage modeling users to see the full range
> of the project, so a small SDK is undesirable.
> 

Modeling users will probably want to install the All-in-one (o.e.o.master) feature generated by the Tools build.

> I'm confused by the reference to 'the coordinating one'.

When I say "Coordinating feature", I mean those features just used to group other features so that they don't provide any plugin. See 
http://wiki.eclipse.org/MDT/OCL/Dev/Releng/Features_Organization

> Do features have
> source features? I don't think I've ever usefully imported features into a
> workspace.

Yes, at least PDE (if desired) generates a source jar for both features and plugins, so if the build.properties file have something different in the "sources" counterpart to the "binary" one, will probably make sense that "source feature".... I think it's not our case.

Regards,
Adolfo.
Comment 22 Adolfo Sanchez-Barbudo Herrera CLA 2011-12-15 07:27:34 EST
After reviewing this thread, I can see a couple of pending issues:

1) zips packaging (the problem now is also extended to the core job > who knows why <).
2) Axel's test cases.

There is already a bugzilla for 1) -> Bug 363208

For 2) I've run the tools build again with the Axel's test cases activated. The build fails with a weird failure ( there has not been any change apart from the junit reactivation). Perhaps the non-stable servers may be causing the issue. If Axels tests fails I'll open a different bugzilla to track the issue.

Resolving this issue as fixed.
Comment 23 Ed Willink CLA 2013-05-17 15:15:05 EDT
Removing "iplog+". Shouldn't be on a bug; there is no attachment. This is all releneg by committers. Must have been a typo.
Comment 24 Ed Willink CLA 2014-05-27 09:43:33 EDT
CLOSED after more than a year in RESOLVED state.
Comment 25 Ed Willink CLA 2014-05-27 09:52:29 EDT
and CLOSE