Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 225047

Summary: DBCS3.4: Externalize Strings halts when using DBCS characters
Product: [Eclipse Project] PDE Reporter: Tadayuki Yoshida <tadayuki>
Component: UIAssignee: Benjamin Cabé <contact>
Status: RESOLVED FIXED QA Contact:
Severity: major    
Priority: P3 CC: camle, caniszczyk, contact, kitlo, natta
Version: 3.4   
Target Milestone: 3.4 M7   
Hardware: All   
OS: All   
Whiteboard:
Attachments:
Description Flags
Patch
none
Archive of files to merge into org.eclipse.pde.ui.tests
none
Remove UTF-8 settings
none
Testcase modified to have no more UTF-8 encoded chars in it
none
pde.ui.tests plugin in eclipse-junit-tests-I20080501-1352.zip none

Description Tadayuki Yoshida CLA 2008-04-01 02:38:15 EDT
Build ID: I20080305-1100
OS: Mac OS X 10.5.2 (intel)
Locale: ja_JP.UTF-8

Steps To Reproduce:
1. Create a plugin project with "Hello World" template
2. Put DBCS values for plugin name or provider in plugin.xml
    otherwise, using NL packs could simplify this step since DBCS values are filled in menu and action name
3. Select the project and start Externalize Strings wizard by PDE Tools > Externalize Strings...
4. Make sure plugin.xml and MANIFEST.MF of the selected plugin are checked and click Finish
5. An error dialog appears saying "An exception has been caught while processing the refactoring 'Extenalize Strings'." followed by the detailed reason as described below:
 Some characters cannot be mapped using "ISO-8859-1" character encoding. Either change the encoding or remove the characters which are not supported by the "ISO-8859-1" character encoding.

This is right, however we can generate property files from Java class including DBCS values for fields and method body by using "Externalize Strings" wizard in JDT.

Could PDE support converting DBCS values to Unicode escaped characters as JDT does before generating plugin.properties file?


More information:
also found on Win32 (ja) and Solaris (ja)
Comment 1 Benjamin Cabé CLA 2008-04-29 05:40:34 EDT
Created attachment 97940 [details]
Patch

Proposed patch.
I've fixed bad behaviour regarding the Java properties file spec (i.e. spaces and line continuation character)

I'll attach Unit testcases as a separate zip (problems of UTF-8 encoding when creating the patch :-/ )
Comment 2 Benjamin Cabé CLA 2008-04-29 05:41:50 EDT
*JUnit* testcases ...
Comment 3 Benjamin Cabé CLA 2008-04-29 05:53:58 EDT
Created attachment 97942 [details]
Archive of files to merge into org.eclipse.pde.ui.tests

I forgot to mention that I've changed the StringHelper#preparePropertiesString() method visibility to public for test purposes.

I just hope the UTF-8 encoded TestCase file will be fine ...
Comment 4 Chris Aniszczyk CLA 2008-04-29 09:59:32 EDT
Thanks Ben!
Comment 5 Chris Aniszczyk CLA 2008-04-29 09:59:46 EDT
done!
Comment 6 Benjamin Cabé CLA 2008-04-29 10:05:31 EDT

*** This bug has been marked as a duplicate of bug 228854 ***
Comment 7 Benjamin Cabé CLA 2008-04-30 15:07:05 EDT
Oops, a JUnit failed in I20080430-0100 build :'(

http://download.eclipse.org/eclipse/downloads/drops/I20080430-0100/testresults/html/org.eclipse.pde.ui.tests_linux.gtk.x86.html#AllPDETests

Chris, will a new bug be automatically filed or something ?
Comment 8 Chris Aniszczyk CLA 2008-04-30 15:38:33 EDT
Do you have a fix?
Comment 9 Benjamin Cabé CLA 2008-04-30 15:48:00 EDT
I think the only problem is that the pde.ui.tests isn't built using "javacDefaultEncoding=UTF-8" in its build.properties, thus the JUnit test case just contains badly encoded strings... Do you agree with that?
It's the only reason I see since the tests run well "locally" :/
However, I can't see projects (I have platform, p2, ecf, etc. in my workspace) using "javacDefaultEncoding" so I don't know if it's a "good practice" to use it... 
Comment 10 Chris Aniszczyk CLA 2008-04-30 15:52:26 EDT
yap, that could cause it

added...

javacDefaultEncoding.tests.jar = UTF-8

That should take care of it.

tagged. > 20080430

Thanks for the quick response.
Comment 11 Benjamin Cabé CLA 2008-04-30 15:58:00 EDT
Phew, cool. Hope it will be OK now.
Comment 12 Benjamin Cabé CLA 2008-05-02 05:55:48 EDT
Created attachment 98399 [details]
Remove UTF-8 settings

CRAP!
PDE UI tests doesn't seem to compile well anymore :( javacDefaultEncoding may not be loved by the headless build? Because exporting o.e.pde.ui.tests plug-in from my workspace works well.......

Here is a patch to remove the UTF-8 settings ; I'll attach a modified StringHelperTestCase in a few seconds
Comment 13 Benjamin Cabé CLA 2008-05-02 05:58:50 EDT
Created attachment 98400 [details]
Testcase modified to have no more UTF-8 encoded chars in it
Comment 14 Benjamin Cabé CLA 2008-05-02 06:00:20 EDT
Reopening... and feeling very guilty about that :(
Comment 15 Benjamin Cabé CLA 2008-05-02 06:10:54 EDT
Created attachment 98401 [details]
pde.ui.tests plugin in eclipse-junit-tests-I20080501-1352.zip

FYI, here is how the pde.ui.tests plug-in looks like in the I20080501-1352 build... Note how tests.jar doesn't contain any .class files... Quite strange... :-(

Of course, PDEAllTests fails with this error:
--------------
Class not found "org.eclipse.pde.ui.tests.AllPDETests"

org.eclipse.test.EclipseTestRunner$TestFailedException: Class not found "org.eclipse.pde.ui.tests.AllPDETests"
at org.eclipse.test.EclipseTestRunner.runFailed(EclipseTestRunner.java:283)
at org.eclipse.test.EclipseTestRunner.getTest(EclipseTestRunner.java:245)
at org.eclipse.test.EclipseTestRunner.<init>(EclipseTestRunner.java:220)
at org.eclipse.test.EclipseTestRunner.run(EclipseTestRunner.java:204)
at org.eclipse.test.UITestApplication$3.run(UITestApplication.java:195)
at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35)
at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:132)
at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:3323)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3053)
at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:2394)
at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2358)
at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2210)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:494)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:288)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:489)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:112)
at org.eclipse.test.UITestApplication.runApplication(UITestApplication.java:138)
at org.eclipse.test.UITestApplication.run(UITestApplication.java:60)
at org.eclipse.test.UITestApplication.start(UITestApplication.java:210)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:193)
at org.eclipse.equinox.internal.app.MainApplicationLauncher.run(MainApplicationLauncher.java:32)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:379)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:549)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
at org.eclipse.core.launcher.Main.main(Main.java:30)
-----------
Comment 16 Chris Aniszczyk CLA 2008-05-02 09:58:53 EDT
done.

> 20080502

I left out the other test cases ,just got rid of the unicode one
Comment 17 Benjamin Cabé CLA 2008-05-02 10:35:42 EDT
Thanks, Chris.
Do you have any idea of what have caused this error in the build process? PDE Build bug? :/
Comment 18 Chris Aniszczyk CLA 2008-05-02 10:38:58 EDT
we can look at this again when M7 is over, I just want it to end
Comment 19 Benjamin Cabé CLA 2008-12-15 17:10:59 EST
*** Bug 228854 has been marked as a duplicate of this bug. ***