Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 552879

Summary: OQL enhancements for sub-selects, maps, context providers, DISTINCT
Product: [Tools] MAT Reporter: Andrew Johnson <andrew_johnson>
Component: CoreAssignee: Andrew Johnson <andrew_johnson>
Status: RESOLVED FIXED QA Contact:
Severity: enhancement    
Priority: P3 CC: kevin.grigorenko
Version: 1.9   
Target Milestone: 1.10.0   
Hardware: PC   
OS: All   
See Also: https://git.eclipse.org/r/152384
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=f4f97ad2226c0e8f7e714f786b99a422e52ac79c
https://git.eclipse.org/r/152396
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=52b0ccfceacc7f5171da606ac928fc54bda8f86d
https://git.eclipse.org/r/152397
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=4f9209c19cc5f8ed3d09b4854fd5146b72d53857
https://git.eclipse.org/r/152494
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=ef25b8ce2971dfc7dbb9ea934193345ff49fb1ad
https://git.eclipse.org/r/152587
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=295c89c1ef6c6b22e714977bcdbe1debe5f92dec
https://git.eclipse.org/r/152591
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=4d93284448475171f903be4892cb5f40bdfd0768
https://git.eclipse.org/r/152701
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=aab9340edd819746ee4ad5506d4a8ebb1e1f0082
https://git.eclipse.org/r/152739
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=15db89cfa4a1813b08291cf234a225ef443df324
https://git.eclipse.org/r/152775
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=cca4cae44ab5e6d6a6b4ca1ba8b2bbfa968b12d0
https://git.eclipse.org/r/152880
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=05fb6aa5ca9af3f8e7d25fe4517ef5bf76d75941
https://git.eclipse.org/r/153089
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=08558dd811931ebbdc8ffc850632e2d19177c575
https://git.eclipse.org/r/153193
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=3dc2a0f1a56b9651c261e049b8d7dc8bd1b546e3
https://git.eclipse.org/r/153527
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=6281f921952bbc2b7b2e250dace8271e7767c7b1
https://git.eclipse.org/r/157391
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=6716288dd95e72c045e70744c840db97617f735d
https://git.eclipse.org/r/157393
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=bf0348f2b5efa878957cc1166b69830298a04b2d
https://git.eclipse.org/r/157913
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=8f6fe8fb079eadaf502234aa30dcfcace1d085e4
Whiteboard:
Bug Depends on:    
Bug Blocks: 241154    

Description Andrew Johnson CLA 2019-11-09 16:10:30 EST
I have various enhancements for OQL:

selects returning tables (rather than object lists) can be processed as lists.

SELECT s AS HashMap, eval((SELECT t, t.getKey(), t.getValue()
                        FROM OBJECTS ( s[0:-1] ) t
                        WHERE (toString(t.getKey()) = "META-INF")))[0][2] AS "Value for META-INF"
                        FROM java.util.HashMap s
                        WHERE
                        ((SELECT t FROM OBJECTS ( s[0:-1] ) t
                        WHERE (toString(t.getKey()) = "META-INF")) != null)
                        
This selects HashMaps, then processes the HashMap as a collection so a list of entries, finds the hash maps
containing the key META-INF, then does the select showing the hash map and the corresponding entry
for the key extracted using another select, with the result converted using eval() then the row and column
extracted using array operations.

The tables now have meta data context providers for tables with columns holding objects so that multiple
choices for each column (and the whole row with the underlying object) is available from context menus.

DISTINCT processing now operates on values as well as objects.
Comment 1 Andrew Johnson CLA 2019-11-09 16:37:33 EST
Also, better progressing monitoring, key/values extraction for maps, making these enhancements also
work for UNION queries, simplification of complex OQL UNION copy query, query context menu with no objects
should still allow copy selection.
Comment 2 Eclipse Genie CLA 2019-11-09 16:38:39 EST
New Gerrit change created: https://git.eclipse.org/r/152384
Comment 4 Eclipse Genie CLA 2019-11-10 17:23:07 EST
New Gerrit change created: https://git.eclipse.org/r/152396
Comment 6 Eclipse Genie CLA 2019-11-10 17:49:36 EST
New Gerrit change created: https://git.eclipse.org/r/152397
Comment 8 Kevin Grigorenko CLA 2019-11-11 11:26:29 EST
Does the example take into account that a HashMap is a tree data structure? Each entry may have a field that points to another entry, and on, recursively.

Reformatting for my own clarity:

SELECT s AS HashMap,
       eval(
         (
           SELECT t, t.getKey(), t.getValue() FROM OBJECTS ( s[0:-1] ) t
           WHERE (toString(t.getKey()) = "META-INF"))
         )[0][2] AS "Value for META-INF"
FROM java.util.HashMap s
WHERE (
        (
          SELECT t FROM OBJECTS ( s[0:-1] ) t
          WHERE (toString(t.getKey()) = "META-INF")
        ) != null
      )
Comment 9 Andrew Johnson CLA 2019-11-11 12:20:26 EST
The HashMap extraction is handled by the collections extraction framework, so OQL is relying on that. If the collections queries work, then OQL should too.

One problem is that HashSets also appear in the collections framework as maps so OQL array access of a HashSet returns key/value pairs, not the values.

My first go had sub-selects returning select items as lists.

SELECT t FROM OBJECTS ( SELECT s, s.size, s.table FROM java.util.HashMap s  ) t 

t
-------------------------------------------------------------------------------------------
[java.util.HashMap [id=0x22e573c8], 1, java.util.HashMap$Entry[] [id=0x22e60be8;length=16]]
[java.util.HashMap [id=0x22e574c8], 8, java.util.HashMap$Entry[] [id=0x22e60c40;length=16]]
-------------------------------------------------------------------------------------------

Reading about SQL and OQL shows that they allow select items from the inner select to be referred to by name in the outer select.
I think that is best done by converting each row of the inner select items to a map, and adjusting the attribute parsing to also recognise maps as well
as IObject fields and Java bean get methods. It would require a grammar change to have string constant attributes (for complex select items), but using get("my.select.item") is an alternative.

It could then work like this:

SELECT t.s, t.get("s.size"), t.tab, t.entries FROM OBJECTS ( SELECT s, s.size, s.table AS tab, s[0:-1] AS entries FROM java.util.HashMap s  ) t

t.s                              | t.get("s.size") |t.tab                                              |t.entries
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
java.util.HashMap [id=0x17c1efa0]|               1 |java.util.HashMap$Entry[] [id=0x17c1f2c0;length=16]|[java.security.ProtectionDomain [id=0x17c18c80]=java.lang.Class [id=0x12a0bf68;name=java.util.HashSet$PRESENT]]
java.util.HashMap [id=0x17c1eed8]|               1 |java.util.HashMap$Entry[] [id=0x17c1f130;length=16]|[java.security.CodeSource [id=0x17c1e6a0]=java.security.ProtectionDomain [id=0x17c18c80]]
java.util.HashMap [id=0x17c1f240]|               1 |java.util.HashMap$Entry[] [id=0x17c1f4c0;length=16]|[java.net.URL [id=0x17c1e760]=sun.misc.URLClassPath$FileLoader [id=0x17c1f630]]
java.util.HashMap [id=0x17c19298]|               3 |java.util.HashMap$Entry[] [id=0x17c194d0;length=16]|[java.net.URL [id=0x17c199a8]=sun.misc.URLClassPath$JarLoader [id=0x17c19830], java.net.URL [id=0x17c19940]=sun.misc.URLClassPath$JarLoader [id=0x17c197e8], java.net.URL [id=0x17c198d8]=sun.misc.URLClassPath$JarLoader [id=0x17c197a0]]
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Comment 10 Kevin Grigorenko CLA 2019-11-11 12:28:49 EST
Ok, I wasn't aware that there was collection extract going on. I'm guessing the `s[0:-1]` in your example is performing that extraction. This would be nice to document at https://help.eclipse.org/2019-09/topic/org.eclipse.mat.ui.help/reference/propertyaccessors.html
Comment 11 Eclipse Genie CLA 2019-11-12 08:32:59 EST
New Gerrit change created: https://git.eclipse.org/r/152494
Comment 13 Eclipse Genie CLA 2019-11-13 10:30:50 EST
New Gerrit change created: https://git.eclipse.org/r/152587
Comment 15 Eclipse Genie CLA 2019-11-13 11:33:57 EST
New Gerrit change created: https://git.eclipse.org/r/152591
Comment 17 Eclipse Genie CLA 2019-11-14 17:03:06 EST
New Gerrit change created: https://git.eclipse.org/r/152701
Comment 19 Eclipse Genie CLA 2019-11-15 11:04:32 EST
New Gerrit change created: https://git.eclipse.org/r/152739
Comment 21 Eclipse Genie CLA 2019-11-16 06:56:50 EST
New Gerrit change created: https://git.eclipse.org/r/152775
Comment 23 Eclipse Genie CLA 2019-11-18 11:47:05 EST
New Gerrit change created: https://git.eclipse.org/r/152880
Comment 25 Eclipse Genie CLA 2019-11-20 16:25:25 EST
New Gerrit change created: https://git.eclipse.org/r/153089
Comment 27 Eclipse Genie CLA 2019-11-22 09:24:51 EST
New Gerrit change created: https://git.eclipse.org/r/153193
Comment 29 Eclipse Genie CLA 2019-11-28 06:28:54 EST
New Gerrit change created: https://git.eclipse.org/r/153527
Comment 31 Eclipse Genie CLA 2020-02-09 22:30:14 EST
New Gerrit change created: https://git.eclipse.org/r/157391
Comment 33 Eclipse Genie CLA 2020-02-09 22:30:35 EST
New Gerrit change created: https://git.eclipse.org/r/157393
Comment 35 Eclipse Genie CLA 2020-02-18 10:46:26 EST
New Gerrit change created: https://git.eclipse.org/r/157913
Comment 37 Andrew Johnson CLA 2020-02-23 08:57:42 EST
No comments from https://www.eclipse.org/forums/index.php/t/1101512/ so marking this as done.