Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 344066

Summary:

DBCS3.7 DBCS (shift JIS) characters are corrupted in Outline view of ANT

Product:

[Eclipse Project] Platform

Reporter:

Kentaroh Noji <kennoji>

Component:

Ant

Assignee:

Michael Rennie <Michael_Rennie>

Status:

RESOLVED FIXED

QA Contact:

Severity:

normal

Priority:

CC:

camle, curtis.windatt.public, kitlo, Michael_Rennie, pwebster

Version:

4.1

Flags:

curtis.windatt.public: review+

Target Milestone:

3.7 RC1

Hardware:

OS:

Windows 7

Whiteboard:

Attachments:

Description	Flags
Screen capture	none
Ant source file in Shift JIS	none
Ant source file in UTF-8	none
proposed fix	none
better	none

Description Kentaroh Noji

2011-04-27 23:06:43 EDT

Build Identifier: I20110412-2200

If the encoding of ANT source files is Shift_JIS, DBCS characters in outline view are corrupted. If the encoding is UTF-8, the DBCS characters display correctly. I will attach the ANT files in Shift JIS and UTF-8.  

JDK: java full version "JRE 1.6.0 IBM Windows AMD 64 build pwa6460sr9fp1-20110208_03 (SR9 FP1)"
OS: Windows 7 SP7 Japanese edition. 

Reproducible: Always

Steps to Reproduce:
1.Copy ANT files attached. 
2.Open the ANT files.  
3.Open Outline view for the ANT files. Window > Show View > Outline

Comment 1 Kentaroh Noji

2011-04-27 23:15:02 EDT

Created attachment 194225 [details]
Screen capture

Comment 2 Kentaroh Noji

2011-04-27 23:15:59 EDT

Created attachment 194226 [details]
Ant source file in Shift JIS

Comment 3 Kentaroh Noji

2011-04-27 23:16:46 EDT

Created attachment 194227 [details]
Ant source file in UTF-8

Comment 4 Michael Rennie

2011-05-06 10:56:30 EDT

I reproduced this on 3.7 as well as 4.1.

The problem comes from this code in ProjectHelper:

if (source instanceof String) {
  stream = new ByteArrayInputStream(((String)source).getBytes("UTF-8")); //$NON-NLS-1$
  inputSource = new InputSource(stream);
}

so when we parse the XML file the ant model has the strings with the wrong encoding as well.

Comment 5 Michael Rennie

2011-05-06 16:54:32 EDT

Created attachment 194990 [details]
proposed fix

Created a new utility method to try and compute the encoding for the build file prior to it being parsed into the Ant model, if detection fails it will fall back to UTF-8.

smoke tested against workspace and remote (out of workspace) build files.

Steps to test:
1. open the shift JIS file from any project in your workspace - the outline should match the editor
2. open the shift JIS file from any location outside of your workspace - the outline should match the editor

Comment 6 Michael Rennie

2011-05-06 16:55:47 EDT

Curtis please try the patch

Comment 7 Curtis Windatt

2011-05-06 17:15:24 EDT

+1 approach is good and problem is fixed for me with both workspace and external files.

Comment 8 Michael Rennie

2011-05-09 11:49:18 EDT

Created attachment 195093 [details]
better

Thinking about the other patch, I found that we end up creating / disconnecting a text file buffer each time we want either the backing IDocument for a build file or the encoding (in the external from workspace case). So I moved the fix into AntModel so its encoding can be cached, which saves a lot of work and the creation of unnecessary text file buffers.

Comment 9 Michael Rennie

2011-05-09 11:50:24 EDT

Curtis please re-verify the new patch. The logic is the same, just in AntModel + cached.

Comment 10 Curtis Windatt

2011-05-09 17:11:08 EDT

+1 New fix operates the same and the caching should be beneficial.

Comment 11 Michael Rennie

2011-05-09 18:53:03 EDT

applied patch to HEAD