Bug 313923

Comment 1 Krum Tsvetkov 2010-05-27 03:24:31 EDT

I had a look at the copy and played with copying large amount of data. I couldn't reproduce the slow operation, but I ran quickly into an OutOfMemoryError (for copying ~20.000 rows with a String value, heap size and retained size columns). I also got into similar problems with fewer rows. May be in your case it was slow because of high memory usage?

I would suggest the following: 
1) you can use the "export" functionality and export your data to a csv file. This worked fine and quickly for me and the 20.000 rows. Do not use export to TXT as it reuses the copy methods and you'll probably run into the same problems

2) I'll have a look at the copy implementation and see if I can improve something there. 
Right now for example the copy will "expand" all columns automatically, meaning that if there is one very long string in the whole table it will fill spaces to all rows to keep the table-like formatting. This is OK and works fine for smaller pieces of data, however for large pieces of data it caused OOM error. May be we should at least show up a warning and hint the user to use the "Export to ..." for larger amounts of data.

3) We need a specific implementation for "Export to TXT" - when writing the data to a file we don't have to keep the whole buffer in memory.

Thanks for giving us feedback! I hope the workaround with "Export to CSV" will help you continue your analysis smoother.

Comment 2 Krum Tsvetkov 2010-06-29 08:40:55 EDT

I submitted some changes, so that the data is streamed into a file when "Export to TXT" is executed, instead of building a full copy in memory first.

There are still some TODOs:
- I figured out that in the OQL result long strings are not limited (e.g in the object reference tree the long strings are cut to something like 1000 characters, which is not true for the OQL results).
- There are a lot of empty spaces added to achieve some nice formatting and this is OK. However, at least in the case of a left-aligned last column there is no need to add the spaces at the end of the line (but this is currently done)

Comment 3 Krum Tsvetkov 2010-07-01 05:21:55 EDT

I submitted some more changes - now there will be no trailing white spaces added if there is just one column.

While trying to improve the Copy I found also another problem - the alignment for the columns is hardcoded - the first column is left justified, and all the rest right justified. This works fine with all "standard" views in MAT, but if I use OQL to select  the address of an object and its value (say for Strings) then the strings are right justified. This however would need a bit more thinking and therefore I opened a separate message for it - Bug 318559.

Some of the most frequently called method during the copy were using StringBuffers internally, although there is no synchronization needed. I changed them to StringBuilder and hope for a bit better performance.

With this we have:
- "Export to TXT" doesn't copy all data into memory first - no OutOfMemoryError
- in case of just one column the copy will not add unnecessary blank spaces
- StringBuilder used instead of StringBuffer to increase performance
- a separate message for the alignment problem is opened

Back to the initial reporter: Unfortunately I wasn't able to reproduce the long-running copy and assumed the whole time it was slow because of high memory usage and possible high GC activity. In general the copy / paste was intended to use on smaller amount of data and the "Save to ..." for exporting larger amounts of data. 
I hope with this few changes the functionality improved and would close the message now.
If you feel the fixes do not solve the problem in an acceptable way, please reopen the bug and give us your feedback.
PS: the changes will be made available with our next service release, tentatively within the next month