Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 400733

Summary: [CBI] CBI builds missing source bundles for branding bundles
Product: [Eclipse Project] Platform Reporter: Curtis Windatt <curtis.windatt.public>
Component: RelengAssignee: Platform-Releng-Inbox <platform-releng-inbox>
Status: RESOLVED FIXED QA Contact:
Severity: major    
Priority: P2 CC: akurtakov, daniel_megert, david_williams, jan.sievers, john.arthorne, pwebster
Version: 4.3   
Target Milestone: 4.2.2+   
Hardware: PC   
OS: Windows 7   
Whiteboard:
Bug Depends on:    
Bug Blocks: 393922, 396445, 434596    
Attachments:
Description Flags
generate source bundle for org.eclipse.jdt branding bundle none

Description Curtis Windatt CLA 2013-02-13 15:09:18 EST
Noticed that the PDE UI import tests were failing in the CBI builds.  The tests import content from the target platform (running Eclipse) into the workspace.  Failures there often indicate something has changed in the Eclipse installation.

Downloaded I20130212-2011-win32_x86_64 to test.

One of the tests that fails is looking for the source bundle for our branding plug-in (org.eclipse.pde.source).  It exists in the PDE build, but not in the CBI build.  Also noticed that JDT is missing their bundle as well (org.eclipse.jdt.source).  org.eclipse.platform.source exists, but has different content than the PDE built version.

I don't believe the source bundle for the branding is required by anything, so the easy solution is for us to change the tests.  However, this could indicate something wrong with the pom files.
Comment 1 David Williams CLA 2013-02-13 15:30:06 EST
Yes, and these are often listed "in" source features too, so if the feature says it includes one, and its not there, that'll break something I'm sure. 

[Unless of course we decide to remove from features too ... I've not checked the features ... just assuming they are (still) listed there as they used to be. ]
Comment 2 Paul Webster CLA 2013-02-13 15:31:10 EST
AFAIK tycho doesn't produce source bundles for the branding ones as they have no source in them.  I thought we had a bug where we acknowledged this already.  Thanh?

PW
Comment 3 Thanh Ha CLA 2013-02-13 16:12:33 EST
(In reply to comment #2)
> AFAIK tycho doesn't produce source bundles for the branding ones as they
> have no source in them.  I thought we had a bug where we acknowledged this
> already.  Thanh?
> 

I recall this being the case as well. However I can't seem to find the bug for this where it was discussed. I'll post again later if I can find it.
Comment 4 Curtis Windatt CLA 2013-02-13 16:33:04 EST
I modified the tests to use pde.ui.source instead of pde.source.

http://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/?id=0d4293ba96d71403880c18fdb36624a8e971b117
Comment 5 Curtis Windatt CLA 2013-02-13 17:01:12 EST
I assume this is a separate issue, but the other test failures are because the source bundles have a different structure.  For example, org.eclipse.jdt.debug.source puts the source at the root of the jar whereas in PDE built eclipse they are in folders named after the library the source belongs to (jdimodelsrc, jdisrc).

This shouldn't cause problems for JDT because it is very good at tracking down where the package fragment roots are.  However, when PDE does a plug-in import we need to recognize the folder names to create a source folder for each library.
Comment 6 Paul Webster CLA 2013-02-13 17:05:11 EST
(In reply to comment #5)
> I assume this is a separate issue, but the other test failures are because
> the source bundles have a different structure.  For example,
> org.eclipse.jdt.debug.source puts the source at the root of the jar whereas
> in PDE built eclipse they are in folders named after the library the source
> belongs to (jdimodelsrc, jdisrc).
> 
> This shouldn't cause problems for JDT because it is very good at tracking
> down where the package fragment roots are.  However, when PDE does a plug-in
> import we need to recognize the folder names to create a source folder for
> each library.

If the source bundles tycho is producing cannot be imported by PDE, then please open another releng bug about that and then we'll open a tycho bug.

PW
Comment 7 Curtis Windatt CLA 2013-02-13 17:21:53 EST
(In reply to comment #6)
> If the source bundles tycho is producing cannot be imported by PDE, then
> please open another releng bug about that and then we'll open a tycho bug.
> 
> PW

See Bug 400740
Comment 8 Jan Sievers CLA 2013-03-20 16:43:53 EDT
(In reply to comment #2)
> AFAIK tycho doesn't produce source bundles for the branding ones as they
> have no source in them.  I thought we had a bug where we acknowledged this
> already.  

bug 401890
Comment 9 David Williams CLA 2013-04-21 14:39:40 EDT
If I understand this right, regardless of "previous decision", the desire is that branding bundles do have source. 

There are two uses of "source". One that the debugger can find and make use of it during debugging. 

The second, it that PDE can "import source" so that users can see what the source it. 

Correct?
Comment 10 Dani Megert CLA 2013-04-22 04:43:55 EDT
(In reply to comment #9)
> If I understand this right, regardless of "previous decision", the desire is
> that branding bundles do have source. 
> 
> There are two uses of "source". One that the debugger can find and make use
> of it during debugging. 
> 
> The second, it that PDE can "import source" so that users can see what the
> source it. 
> 
> Correct?

Yes, those bundles need to be present. I've targeted 4.2.2+, since this should also be fixed there.
Comment 11 John Arthorne CLA 2013-04-29 13:15:20 EDT
I don't understand the use case where importing the source of branding plugins has any value. If you want to do anything useful you will need to import the source from Git anyway. Is the main problem that it makes our metadata inconsistent because the SDK lists a plugin that is missing from the build?
Comment 12 David Williams CLA 2013-04-29 16:28:52 EDT
(In reply to comment #11)
> I don't understand the use case where importing the source of branding
> plugins has any value. If you want to do anything useful you will need to
> import the source from Git anyway. Is the main problem that it makes our
> metadata inconsistent because the SDK lists a plugin that is missing from
> the build?

I don't think there is any technical problem ... at least from my point of view. 

From my "casual user" point of view, I have in the past, just been exploring, and wanted to take a quick peek at something, not "do anything useful", and the branding bundles all sound so important and central, ... I guess because of their short names ... 
org.eclipse.cvs
org.eclipse.jdt

I'd often want to pick one and import the source just to see what it was. 
And, admittedly usually disappointed to learn it was just a branding bundle :) 

I personally do not have a strong preference one way or the other. My comment 9 was meant to help drive this issue to closure.
Comment 13 Jan Sievers CLA 2013-04-30 03:11:42 EDT
(In reply to comment #12)
in case you want source bundles generated for bundles with resources only, just define a src.includes in build.properties
Comment 14 Dani Megert CLA 2013-04-30 03:55:38 EDT
Looks like some clarification is needed here: the missing bundles are *not* source bundles for the branding bundles. Instead, they are the branding bundles for the source features.

Example:
feature org.eclipse.jdt has org.eclipse.jdt as branding plug-in
feature org.eclipse.jdt.source has org.eclipse.jdt.source as branding plug-in

The latter is currently missing from our builds.
Comment 15 John Arthorne CLA 2013-04-30 11:51:51 EDT
(In reply to comment #14)
> Looks like some clarification is needed here: the missing bundles are *not*
> source bundles for the branding bundles. Instead, they are the branding
> bundles for the source features.

Ah! That makes much more sense. Sorry for confusion.
Comment 16 Jan Sievers CLA 2013-05-03 09:38:03 EDT
(In reply to comment #14)
> Looks like some clarification is needed here: the missing bundles are *not*
> source bundles for the branding bundles. Instead, they are the branding
> bundles for the source features.
> 
> Example:
> feature org.eclipse.jdt has org.eclipse.jdt as branding plug-in
> feature org.eclipse.jdt.source has org.eclipse.jdt.source as branding plug-in
> 
> The latter is currently missing from our builds.

found related bug 379415 and I think I finally understood why plugins org.eclipse.jdt.source and org.eclipse.pde.source are missing:

These plugins used to be generated using the "sourceTemplatePlugin" folder mechanism [1,2] by PDE headless build (docs see [3]).

Tycho does not support sourceTemplatePlugin folders.

The same effect can mostly be achieved simply by specifying src.includes in the bundle.
However there is a problem if both binary and source bundle should include the same file paths, such as about.html, but with differing content.

Opened tycho bug 407172 for this.

[1] http://git.eclipse.org/c/pde/eclipse.pde.git/tree/org.eclipse.pde-feature
[2] http://git.eclipse.org/c/jdt/eclipse.jdt.git/tree/org.eclipse.jdt-feature
[3] http://help.eclipse.org/juno/index.jsp?topic=/org.eclipse.pde.doc.user/tasks/pde_individual_source.htm
Comment 17 John Arthorne CLA 2013-05-06 10:46:28 EDT
(In reply to comment #14)
> Looks like some clarification is needed here: the missing bundles are *not*
> source bundles for the branding bundles. Instead, they are the branding
> bundles for the source features.
> 
> Example:
> feature org.eclipse.jdt has org.eclipse.jdt as branding plug-in
> feature org.eclipse.jdt.source has org.eclipse.jdt.source as branding plug-in
> 
> The latter is currently missing from our builds.


I was investigating this today, and I think you are wrong about this. org.eclipse.jdt.source is not the branding plugin for the org.eclipse.jdt.source feature. It is (or was) the source bundle for the bundle org.eclipse.jdt. You can see this by looking at its MANIFEST.MF in Eclipse 4.2.2:

Eclipse-SourceBundle: org.eclipse.jdt;version="3.8.1.v201302041200"

That header means it is a source bundle, not a branding bundle. It is still missing, but I'm not sure it is critical. It contains all the same information as the org.eclipse.jdt bundle itself and doesn't add any value that I can see.
Comment 18 John Arthorne CLA 2013-05-06 13:55:10 EDT
Changing back to original title.
Comment 19 Dani Megert CLA 2013-05-07 05:26:36 EDT
(In reply to comment #17)

You're right, I got confused when looking at the other branding bug. And I agree that if we miss those bundles, it's not a big deal.
Comment 20 Jan Sievers CLA 2013-05-17 07:46:33 EDT
Created attachment 231147 [details]
generate source bundle for org.eclipse.jdt branding bundle

Here is how to generate the source bundle for the org.eclipse.jdt branding bundle.
Other branding bundles will be similar I guess.
Comment 21 David Williams CLA 2013-05-18 15:10:31 EDT
(In reply to comment #20)
> Created attachment 231147 [details]
> generate source bundle for org.eclipse.jdt branding bundle
> 
> Here is how to generate the source bundle for the org.eclipse.jdt branding
> bundle.
> Other branding bundles will be similar I guess.

Jan, is this fix dependent on some recent fix in 0.18.0-SNAPSHOT? Or, just something we've been missing all along?
Comment 22 David Williams CLA 2013-05-18 15:15:01 EDT
(In reply to comment #21)
> (In reply to comment #20)
> > Created attachment 231147 [details]
> > generate source bundle for org.eclipse.jdt branding bundle
> > 
> > Here is how to generate the source bundle for the org.eclipse.jdt branding
> > bundle.
> > Other branding bundles will be similar I guess.
> 
> Jan, is this fix dependent on some recent fix in 0.18.0-SNAPSHOT? Or, just
> something we've been missing all along?

More specifically, I didn't think "sourceTemplatePlugin" was supported. 
Or, am I confusing that with "sourceTemplateFeature" not supported? Just wondering if this type of fix helps us with bug 406868.
Comment 23 Jan Sievers CLA 2013-05-21 04:27:17 EDT
(In reply to comment #22)
> > Jan, is this fix dependent on some recent fix in 0.18.0-SNAPSHOT? Or, just
> > something we've been missing all along?

it depends on the fix for tycho bug 407172 done in 0.18.0-SNAPSHOT

> More specifically, I didn't think "sourceTemplatePlugin" was supported. 
> Or, am I confusing that with "sourceTemplateFeature" not supported? Just
> wondering if this type of fix helps us with bug 406868.

there is no 1:1 sourceTemplatePlugin mechanism with "magic" folder names on feature level as it used to work in PDE headless build. However the fix for bug 407172 allows to effectively get the same result as far as I understood the old PDE build behaviour.
Comment 24 Alexander Kurtakov CLA 2020-04-03 09:14:37 EDT
Is there smth still pending here or we can close this one?
Comment 25 Alexander Kurtakov CLA 2020-11-17 03:41:06 EST
Consider it done.