Bugzilla – Full Text Bug Listing
| Summary: | [console] Console output limit cannot be set higher than 1 million characters | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Eclipse Project] Platform | Reporter: | Joe Whitney <jwhitney+eclipsebugs> | ||||
| Component: | Debug | Assignee: | Sarika Sinha <sarika.sinha> | ||||
| Status: | VERIFIED FIXED | QA Contact: | |||||
| Severity: | normal | ||||||
| Priority: | P3 | CC: | daniel_megert, eclipse.org, flexx, francis.andre.kampbell, jwhitney+eclipsebugs, k0610, markus.kell.r, Michael_Rennie, pawel.1.piech, piotr.findeisen, remy.suen, sarika.sinha | ||||
| Version: | 3.5.2 | ||||||
| Target Milestone: | 4.5 M4 | ||||||
| Hardware: | All | ||||||
| OS: | All | ||||||
| Whiteboard: | |||||||
| Attachments: |
|
||||||
|
Description
Joe Whitney
Any opinions on this? We should either fix or wontfix. (In reply to comment #1) > Any opinions on this? We should either fix or wontfix. I'd like to see this fixed. I frequently find I need more than a million lines of history so my only option is to set it to be unlimited. All too often I then forget that I've done that, leave a process running for a few hours, and Eclipse will run out of heap (typically then needing to be forcibly killed). Being able to set the console size to, say, 10 million would be ideal. The 1 million line restriction seems completely arbitrary. I can't understand why there is any need to restrict it at all. Hi Me too I do not understand this strong limitation... I am using JavaCC which produces a lot a lines when tracing the lookahead decision and I cannot understand the bug in ly grammar if I do not have a global view of the lookahead traces... Please remove the control of the buffer size... it is up to the Eclipse's developper to put its own limitation, whatever it should be. FA I absolutely want this fixed. First of all: the limit is a million _characters_, not lines. I'd be (kinda) fine if it _were_ lines. At, say, 132 characters per line, the maximum setting allows 7575 lines and a few characters to spare. That quickly becomes too little in a typical debugging session. The real world case that made me come here and see if a bug was listed for this, has me at 4456 lines of backlog. The SQL statement dump I really need to see is gone. It's conveniently on the same TRACE logger as the stuff that pushed it out, so changing my logger config wouldn't help, actually. Now turning the limit off temporarily is not a huge problem, if I don't mind restarting my tomcat with the changed limit, renavigating to the same spot in the appliocation and retesting the operation I am debugging. But will I forget to turn it back on, and then start a job or large unit test overnight, in the very same tomcat I used to debug said job a couple hours earlier? Might forget to, and end up with an eclipse instance hogging loads of memory, or in fact running out of heap. So, I do want the option to limit the buffer, and I'd even give it some hundreds of MB's, or even GB's really. At the current maximum we'd be limited to giving 2MB for a limited console at maximum setting if we have 2-byte UTF-16 characters (and no surrogates and that). In terms of productivity, reconfiguring developer logging confguration constantly to only see things we need right now can be annoying and limiting either. So, I think I made the point that 1 million characters is really not a lot. Besides all those real reasons why this should be fixed, theres a simple one, too: It's just a rather dull bug, lets face it. And it sould be easy to fix, if you know where to look (I don't, unfortunately). That limit is completely arbitrary. There cannot actually be any reason whatsoever why it has to be that number, technically. Integer.MAX_VALUE is 2.147.483.647. Therefore, even if the buffer was just a simple array of chars we should have 2 billion characters, an order of magniture more. That would be 16 million lines (of 132 characters each). I could probably do with that. ;) So here someone made a rather uneducated guess as what a good "magic number" would be, and unsurprizingly the guess was wrong. So I plead to give the option no limit at all, or the actual technical limit, which I would assume can only be a multiple of Integer.MAX_VALUE or Long.MAX_VALUE. [And if I actually think of how console buffer limiting would in fact work great, then it's giving me a limit I can control on the fly at any time, by letting me enter a number directly, and while the console is allocated, in its menu bar. Also the (initial) limit should not be a global, but rather set on the launch configuration.] So I hope I could explain why this one got my vote. ;) Cheers, Flexx Oh, could someone change the summary of this bug to "Console output limit cannot be set higher than 1 million characters" please? As is, the summary misrepresents the actual problem. The upper 1M limit should just be removed. (In reply to Stuart Sharpe from comment #2) > [..] leave a process running for a few hours, and Eclipse will run out > of heap (typically then needing to be forcibly killed). That's a separate issue that could (and should) be resolved with a few additional heuristics in IOConsolePartitioner.checkBufferSize() to help avoid OutOfMemoryErrors: - Set MIN_FREE to max(50M, Runtime#maxMemory() / 20). On systems without a limit use maxMemory=10G. - If Runtime#freeMemory() < MIN_FREE and System.gc() doesn't make it > MIN_FREE, then remove characters until MIN_FREE is free again (but always keep the last 1M characters). These limits are again a bit arbitrary, but they are better than no solution for OOMEs. People who want full freedom can just turn the limit off. (In reply to Markus Keller -- away 2014-11-15 to -28 from comment #6) > MIN_FREE to max(50M, Runtime#maxMemory() / 20) And why would this be a good idea? > These limits are again a bit arbitrary, but they are better than no solution > for OOMEs. People who want full freedom can just turn the limit off. I agree that a solution that prevents (or I should day allows the end user to prevent) OOME is preferable, and I agree that those limits are arbitrary, and that is why the parameters for the heuristic should simply be _configurable_. I, and only I know what I what I need and what I wanna spend. There should not be _any_ magic numbers or limits in any part of an IDE, since we run it on all kinds of hardware. So yes, allow me to set a a) memory limit in bytes b) line limit c) character limit d) rollover segment size e) rollover segment unit (bytes, characters or lines) When rollover unit bytes is selected, must verify that maximum character size <= rollover segment size, and block saving of the configuration page (or simply implicitly delete character-wise instead of byte-wise until at least rollover segment size bytes are freed). On reaching a buffer size of a, b or c, block on the input pipe, and truncate the buffer by d (bytes, chars, lines), unblock input pipe and read data growing the buffer again. Repeat until pipe is empty. It's debatable wether the buffer should be truncated (i. e. actually shrink in memory), or kept at the maximum size and "rotated" once the maximum is reached. The former is memory-efficient, the latter probably faster. A good implementation would allow both stragegies, and let me select one in the configuration, since when using large rollover segment sizes, a longer but rarely occuring pause might be perferable, while at a small rollover segment size, a short pause is preferable to "keep things scrolling". Besides for small rollover segments, the freed heap space is neglible. Which in turn might call for a heuristic choice of the better strategy (which should be just another option: optimize for memory usage / output lag / choose best approach automatically). (In reply to Alexander Wessel from comment #7) > > MIN_FREE to max(50M, Runtime#maxMemory() / 20) > > And why would this be a good idea? The intention is to keep a minimum amount of memory free to keep Eclipse from crashing. If the process would use up more than MIN_FREE, then we better drop parts of the console buffer than risking an OOME. With less than 50M or less than 5% of allocated memory, the IDE is almost dead due to the GC overhead. Note that Eclipse is not a simple single-threaded program, so there's nothing that globally controls "free memory". Other threads and third-party plug-ins can allocate memory at will. That's why we need some heuristics here to make sure we catch the problem *before* an OOME is thrown. We already have too many preferences, and we don't intend to add more for this. Sarika: As a first step, could you please just replace the arbitrary console buffer size limit with the maximum document size? "Integer.MAX_VALUE - 100000" should be a safe value that allows room for the GapTextStore's "gap". (In reply to Markus Keller -- away till Nov 28 from comment #8) +1. Created attachment 248850 [details]
Upper limit set to Integer.MAX_VALUE - 100000
Upper limit set to Integer.MAX_VALUE - 100000
Committed patch with http://git.eclipse.org/c/platform/eclipse.platform.debug.git/commit/?id=7c47c0f6697451e6c24d45d78376e3cb5acb33d3 Verified using Eclipse SDK Version: Mars (4.5) Build id: N20141126-2000 |