Comment 1 Martin JANDA 2014-01-09 10:12:11 EST

Maybe it's related to already reported issues. It's known for long time that batch fetching is throwing exceptions. (off by one errors, memory leaks)

Comment 2 Joachim Kanbach 2015-08-20 09:30:15 EDT

I've run into some variation of this bug with EL 2.5.2.

I have a somewhat complex object hierarchy that I query from the database with a single query and several query batch hints pointing to all the nested entities. There are @OneToOne, @OneToMany, @ManyToMany, bidirectional relationships between the different levels, there is a @JoinTable, and there is also an @OrderBy - like I said, it's complex.

When I was using BatchFetchType.JOIN, I had the problem that some of the intermediate entities in the hierarchy are unexpectedly duplicated (I didn't investigate that further, this would probably deserve another bug report).

But with BatchFetchType.IN, I got the problem instead that a few of my @OneToOne relationships fields were set to null. The circumstances are basically the same as for Alexandre, i.e.

" ...
- When selecting a particular set of 1500 ... it can fails (but if it fails it always fails, this is not a random bug)
- Some other loading of set of 1500 ... work well ..."

There is nothing particular about those entities (which are all of the same type/on the same hierarchy level) at all that have this relationship field set to null (in comparison to the other entities on that hierarchy level). They seem to be completely random, but it were always the same 3 out of 1838 entities on that level that were concerned, as long as the data in the database didn't change. 

Since I've already run into other bugs with the @OrderBy annotation, I tried to remove it just for testing. In that case, the set of concerned entities changes, but the general error is the same.

The next thing I did was to specify a batch fetch size (before, I just left it at the default). And there I noticed a correlation with the number/set of entities that have this concerned relationship field set to null:

If I use a batch fetch size that exceeds the potential number of entities of the concerned type, the error goes completely away. As I continuously lowered the batch fetch size, there was at first only 1 broken entity, then there were 2 and finally 3, but eventually the number decreased again. One particular instance was always contained in this set, the others varied slightly.

So I'd assume that somehow those different batch sets aren't assembled correctly, especially because using a batch fetch size larger than the number of results prevents the error.

And finally, when I set the batch fetch size really low (10), this was when I ran into Alexandre's bug, i.e.

Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
	at java.util.ArrayList.elementData(ArrayList.java:371)
	at java.util.ArrayList.get(ArrayList.java:384)
	at org.eclipse.persistence.mappings.ForeignReferenceMapping.extractResultFromBatchQuery(ForeignReferenceMapping.java:583)
	at org.eclipse.persistence.mappings.CollectionMapping.extractResultFromBatchQuery(CollectionMapping.java:955)
        (...)

(different lines in ArrayList due to Java 7 instead of Java 8?)


Using a conditional breakpoint, I could identify some of the variable values at ForeignReferenceMapping, line 583. What strikes me most is that "startIndex" is -1, which must be a result of

startIndex = parentRows.indexOf(sourceRow);

Indeed, "parentRows" doesn't contain a row with the primary key of "sourceRow". So, "offset" is also initialized with -1 in line 574, and "index" still being 0 when line 583 is reached, the access with

parentRows.get(offset + index);

fails since the argument evaluates to -1.

Comment 3 Patrick Haller 2016-02-09 13:49:14 EST

Created attachment 259670 [details]
0001-Bug-424907-ArrayIndexOutOfBoundsException-in-Foreign

Patch against eclipselink.runtime master.

Comment 4 Patrick Haller 2016-02-09 13:56:41 EST

I've hit similar issues, likely originating from the same location:
- missing entries in the query result
- extractResultFromBatchQuery causing OutOfMemoryErrors due to remainingParentRows containing actually more rows than original parentRows. The observed effects depend on the fetch page size and overall query resultset size.

After walking through the moving window algorithm used resolve the query's rows in pages, I think that the logic used to derive remainingParentRows from parentRows sometimes misses a row or creates duplicates. Attached patch solved the situations in which I hit the problem.

Review and feedback appreciated.

Thanks, Patrick


[ForeignReferenceMapping - extractResultFromBatchQuery]
                     // Need to compute remaining rows, this is tricky because a page in the middle could have been processed.
                     List<AbstractRecord> remainingParentRows = null;
+                    // 424907 : precisely subtract processed rows from parentRows
                     if (startIndex == 0) {
                         // Tail
                         remainingParentRows = new ArrayList(parentRows.subList(index, rowsSize));
                     } else if (startIndex == offset) {
                         // Head and tail.
-                        remainingParentRows = new ArrayList(parentRows.subList(0, startIndex - 1));
-                        remainingParentRows.addAll(parentRows.subList(index, rowsSize));
+                        remainingParentRows = new ArrayList(parentRows.subList(0, startIndex));
+                        remainingParentRows.addAll(parentRows.subList(offset + index, rowsSize));
                     } else {
                         // Middle
                         // Check if empty,
                         if ((offset + index) >= (startIndex - 1)) {
                             remainingParentRows = new ArrayList(0);
                         } else {
-                            remainingParentRows = new ArrayList(parentRows.subList(offset + index, startIndex - 1));
+                            remainingParentRows = new ArrayList(parentRows.subList(offset + index, startIndex));
                         }
                     }

Comment 5 Erik Agterdenbos 2016-02-17 08:25:10 EST

Created attachment 259789 [details]
Revised patch

I experience similar issues when BatchFetchType.IN is used instead of EXISTS or JOIN:
- missing objects in the result set;
- ArrayIndexOutOfBoundsExceptions thrown

However, I doubt whether the patch proposed by Patrick Haller completely solves the bug. Suppose there is one remaining element in the 'Middle', then the 'empty check' near line 634 still contains an off-by-one error:
* the element at (offset + index) might not have been processed, it would have been the next element. The statement 'index++;' is at the *end* of the for loop that processes the rows.
* startIndex is the element where we started
* suppose (offset + index) == (startIndex - 1), then the expression ((offset + index) >= (startIndex - 1)) evaluates to true. 
* the single element at (offset + index) is still remaining, but not included in remainingParentRows.

I suggest to remove the 'empty check' from the code. If there are no remaining rows, then ArrayList(parentRows.subList(offset + index, startIndex)) would also return an empty list since (offset + index == startIndex). Moreover, when no remaining rows are expected (size == rowsSize), then (startIndex == 0), and the 'Tail' case would be executed instead. In that case ArrayList(parentRows.subList(index, rowsSize)) would give the empty list.

To reproduce bug 424907, I uploaded a small Eclipse project:
https://github.com/agterdenbos/EclipseLinkTester/tree/master/BatchLoadingTester

Comment 6 Patrick Haller 2016-02-22 12:19:47 EST

Hi Erik, nice catch.

I locally modified your EclipseLinkTester to exhaustively test through all batch sizes from 1005 down to 1 - and indeed, my first patch leaves holes.

Simulating the different batch sizes: 
- current 2.6.2 code:  999 of 1005 cases fail
- my patch:              6 of 1005 cases fail
- Erik's revised patch: all looks good!

Comment 7 Petros Splinakis 2016-06-21 07:26:38 EDT

*** This bug has been marked as a duplicate of bug 412056 ***

Comment 8 Eclipse Webmaster 2022-06-09 10:15:53 EDT

The Eclipselink project has moved to Github: https://github.com/eclipse-ee4j/eclipselink

Comment 9 Eclipse Webmaster 2022-06-09 10:23:22 EDT

The Eclipselink project has moved to Github: https://github.com/eclipse-ee4j/eclipselink