Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 558463

Summary: [console] Console redirection changes content
Product: [Eclipse Project] Platform Reporter: Alexandre Bartel <eclipse>
Component: DebugAssignee: Paul Pazderski <paul-eclipse>
Status: VERIFIED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: paul-eclipse, sarika.sinha
Version: 3.1   
Target Milestone: 4.17 M1   
Hardware: All   
OS: All   
See Also: https://git.eclipse.org/r/163012
https://git.eclipse.org/c/platform/eclipse.platform.debug.git/commit/?id=00d93c9a6e6af649b84604baa0c0866c4d3511e5
Whiteboard:
Bug Depends on: 558489    
Bug Blocks: 560229    

Description Alexandre Bartel CLA 2019-12-19 07:54:00 EST
When the output of the Java program is a byte stream, redirecting the output from the Eclipse console to a file changes the value and the number of bytes.

Example:

With the following program, the output should only be 0xac

public class Test {
	public static void main(String args[]) throws Exception {
		byte[] bytes = new byte[1];
		bytes[0] = (byte) 0xac;
		OutputStream os = System.out;
		os.write(bytes);
	}

}

When the program is run outside Eclipse and the output redirected to a file, the file contains the expected byte stream:

$ java Test > test.out
$ hexdump -C test.out
00000000  ac

However, when the program is run with Eclipse and the output redirected (run configuration -> common -> output file), the file contains additional bytes and the wrong byte stream:

$ hexdump -C test.out
00000000  ef bf bd
00000003

I expect the redirection within Eclipse to have the same semantic as the redirection in the console (>). The encoding is set to UTF-8. Note that since the output is redirected the encoding *should not* have any impact on the byte stream. What is going on?
Comment 1 Paul Pazderski CLA 2019-12-19 08:06:34 EST
(In reply to Alexandre Bartel from comment #0)
> What is going on?

The output is encoded regardless of the redirection.

I'll work on this but due to the way existing stream handling is implemented it is not entirely trivial.
Comment 2 Paul Pazderski CLA 2019-12-19 11:31:12 EST
Consistent but still wrong: the same behaviour can be found when reading input from file.

The following example
--------------
public class TestInput {
	public static void main(String args[]) throws Exception {
		byte[] bytes = new byte[10];
		InputStream is = System.in;
		System.out.println("Read " + is.read(bytes) + " bytes.");
	}
}
--------------
will print "Read 3 bytes." when input file just contains the byte 0xac.
Comment 3 Sarika Sinha CLA 2020-01-06 00:04:11 EST
@Paul, do you have plans to fix this for M1?
Comment 4 Paul Pazderski CLA 2020-01-06 03:24:25 EST
No.
Comment 5 Sarika Sinha CLA 2020-02-14 00:01:38 EST
@Paul, 
Do you have plans to fix this for M3?
Comment 6 Paul Pazderski CLA 2020-02-14 03:39:59 EST
No, sorry. Is mostly done but had not found the time to fix the remaining bugs. And since it contains API change I want to rush it for M3.
I remove the milestone because there is no 4.16 yet but plan to fix it next release.
Comment 7 Sarika Sinha CLA 2020-05-13 05:22:14 EDT
(In reply to Paul Pazderski from comment #6)
> No, sorry. Is mostly done but had not found the time to fix the remaining
> bugs. And since it contains API change I want to rush it for M3.
> I remove the milestone because there is no 4.16 yet but plan to fix it next
> release.

I hope you are planning for 4.16 M3.
Comment 8 Paul Pazderski CLA 2020-05-14 04:25:59 EDT
Sorry, I hadn't much time for this release. I fixed the last known bugs in my change but it is still rather complex. 
Here is why: the way the existing implementation works make it very complicate to do this change.
We have a ProcessConsole which is able to show the process output in console and redirect the output to file. The process could be anything because everyone can implement IProcess. The process output is provided through IStreamsProxy or more precise from IStreamMonitor. All of those are string based.
When a RuntimeProcess is created it immediately starts capturing the output. All before the ProcessConsole is created and know if it as to redirect to file.


Most approaches I tried failed on some point on the existing API. (I really hope I had not missed a obvious solution)
Comment 9 Paul Pazderski CLA 2020-05-14 04:46:54 EDT
Gerrit bot seem to be sleeping.
Here is the proposed patch: https://git.eclipse.org/r/163012
Comment 10 Sarika Sinha CLA 2020-05-21 06:58:29 EDT
(In reply to Paul Pazderski from comment #9)
> Gerrit bot seem to be sleeping.
> Here is the proposed patch: https://git.eclipse.org/r/163012

@Paul, are you planning this for RC1 or 4.17?
Comment 12 Paul Pazderski CLA 2020-06-27 09:25:24 EDT
Finally merged a fix.
Comment 13 Sarika Sinha CLA 2020-06-29 00:43:43 EDT
(In reply to Paul Pazderski from comment #12)
> Finally merged a fix.

^ Looks like a big change!
Comment 15 Paul Pazderski CLA 2020-07-03 14:58:57 EDT
Probably the first issue I described in bug 564849. I start to worry if the next run after I merged that change still fails. ;)
Comment 16 Paul Pazderski CLA 2020-07-08 11:34:14 EDT
Verified in I20200708-0600 with the initial example. Whether "Allocate console" is enabled or disabled the file with redirected output has the expected/unchanged content

hexdump -C bug558463.out                                                                                                                                          
00000000  ac