Community
Participate
Working Groups
Move of these are related to hard to find hardcoded assumptions about build names. Namely, that builds were named with underscores, instead of hyphens, that we now use. Plus, apparently, it was assume R builds were named with two dates, such as R3.5_200510101000_200510101000. I can find no instances of builds named with underscores on archives site, and none that have two dates. Perhaps for performance tests, the "Reference build" was copied and given a special name? I do not find any mention of that, and would prefer to avoid. My hope is there is not something that "changes format" upon putting into the database? Plus, there were some hard coded values that "forced" the reference build to be "4.4", no matter what we specified on command line. My initial fix will fix many of those assumptions, but, admit, I have fixed, for now, by still using hard-coded values. = = = = = = Plus, there were lots of places where exceptions were ignored or invalid "input parameters" were being ignored. I have increased the error checking. = = = = = I will say, though, that after all that "clean up" and digging around in the code for hours (in debug mode, from IDE), I could see that it appears we have not been collecting data for "4.5 reference builds". Not sure yet if that is an error in the scripts we use to invoke the tests (mostly likely) or if the correct build was using during the test, but then "miscoded" in the database. = = = = = My strategy is to fix some of these obvious problems, and "reset" the data base to empty, which will make it easier to figure out if "wrong build" is used when running a "reference test" or if it is being miscoded. I addition, may find other cases of hard-coding assumptions?
This is commit of fixing the hard-coded bugs ... plus some minor changes. http://git.eclipse.org/c/platform/eclipse.platform.releng.buildtools.git/commit/?id=c823741f78978ca94400be94da37d2b798759193 = = = = = = = = = = From inspecting log and "set-ups", it sure seems like the R-4.5-* reference build should be being used, and passed into framework as the 'build' for the database, for example, eclipse.perf.config=build\=R-4.5-201506032000;config\=linux.gtk.x86_64;jvm\=8.0 whereas its non-baseline counterpart is: eclipse.perf.config=build\=I20151029-1100;config\=linux.gtk.x86_64;jvm\=8.0 So ... something is pretty complicated that I can't "see" R-4.5-201506032000 in the data.
(In reply to David Williams from comment #1) > > From inspecting log and "set-ups", it sure seems like the R-4.5-* reference > build should be being used, and passed into framework as the 'build' for the > database, for example, > > eclipse.perf.config=build\=R-4.5-201506032000;config\=linux.gtk.x86_64; > jvm\=8.0 > > whereas its non-baseline counterpart is: > > eclipse.perf.config=build\=I20151029-1100;config\=linux.gtk.x86_64;jvm\=8.0 > > So ... something is pretty complicated that I can't "see" R-4.5-201506032000 > in the data. This is either merely a another symptom of the same issue that part of this bug is about or else it might be a clue as to what is going wrong. We run the reference build first. Then during the regular performance build (being ran on the current build) we also say "assert against" The reference build. So, besides eclipse.perf.config=build\=I20151102-2015;config\=linux.gtk.x86_64;jvm\=8.0 we also specify eclipse.perf.assertAgainst=build\=R-4.5-201506032000;config\=linux.gtk.x86_64;jvm\=8.0; But, there is an error message in log that says: !MESSAGE no reference data named '{config=linux.gtk.x86_64, build=R-4.5-201506032000, jvm=8.0}' found I can't help but notice the difference in order. 99% of the time I'd say that could not possibly be related to this issue ... but, after some of the other coding I have seen, would to be too surprise if someone didn't code some hard order-assumptions. Unlikely ... but ... possible, I think.
(In reply to David Williams from comment #2) > > But, there is an error message in log that says: > !MESSAGE no reference data named '{config=linux.gtk.x86_64, > build=R-4.5-201506032000, jvm=8.0}' found > Just so I don't forget, when I come back to this ... this could also simply be the result that a test "passes" in current build, but failed in reference build.
To give some "blow by blow" status, I do think the previous "system" did change hyphens to underscores (for some reason) and more important, added a suffix of "_<timestamp>" to the "reference build id" at the time it ran (not the buildId itself, but the way it was coded for the database). This was the "systems" way of matching an I- or M- build to an R-build -- I think. This is compounded by the fact that if no R-build is found that "matches" in some assumed way (more complicated than a mere 'timestamp', I think, then "the next closest I-build" is taken as the "reference build". Since I am still angry about it, I will say only: what an incredibly poor way to make use of a database! Not to mention, such a strong assumption never being documented. There is, also, an very strong assumption in the current "system", I think, that "all the performance tests" are "ran at the same time". (That is, all those from a reference build, and then all those from an I-build). The fact that the system was built with this "monolithic" assumption is another sign of how poor the current system is. Smiley faces aside, I have a few experiments I can try to yet again "hack things to work for the immediate moment" but if they do not work ...
To give a brief status update, there is no "quick fix" I could see. I think a large part of the code should be re-written, so it is understandable and behaves in a predictable manner. To give one example, there is a pervasive assumption throughout the code that the build "just previous" to the current build (the one being checked for regressions) it the one that should be used as the reference build. I'm not sure how this ever made sense but might have if the performance tests ran "every milestone", or similar. And, it appears to do this "override" even if you specify the baseline on the command line -- which really makes no sense. (At least to me). In essence, the current program has one array, of "all builds", sorts them, and expects them to be correctly ordered regardless if it is an I-build, R-build, M-build, or N-build. To build on this case, there are three classes, each with their own "allBuilds" method. at times, they "refer to each other", and at other time, they use their own. I do not understand what the intent/purpose of that was, but suspect it was a result of years of "adding on" changes, and perhaps under the conditions it was used some of those methods were "dead code". Continuing that theme, it is hard to tell what is "dead" and what is not. Many variables have "package protection" ... and they may or may not have getters and setters too. The getters and setters often have side effects that -- as far as I can tell -- do not make conceptual sense, and were simply quick fixes for some specific issue that came up. I short, I would like to re-write all that stuff, remove any dead code, and rework the logic so the results are completely determined by the input parameters. They would never be changed "under the covers". Plus, I would like to remove (nearly) all hard coded assumptions that exist in the code. Such as "reference build prefix" ... which is already an input parameter. Completely unclear why there as a hard coded version as well? This code that generates the reports was at one time a true "UI bundle" with "actions" and "views", etc. That certainly does not work now. Did it ever? In either case, I would like to remove that functionality, and if that is ever desired to have in the future, that that "app" be either just plan re-written, or better yet, re-written as a "web app", so people could do "live queries" on the data. Another huge change I would like to make is how the data is "transformed" from one database to some other "custom" database (mostly just serialized Java objects, I believe -- but with at least some of the serialization code written as functional code, instead of true Java (Object Oriented) serialization). According to the comments, this was done because "remote access to the database was so slow". There are LOTS of better ways to solve that, if it is a problem at all, in these days of faster networks. (For example, you can just copy Derby databases, as you can files (such as with rsync or zip it up. No need for a "custom made database" that is impossible for anyone to understand or re-use. So, these are big changes I am recommending. Will discuss with the team to see if I am missing anything important from the history of this code or if anyone has any other suggestions.
Since this work will not be "complete" until after M4, am moving milestone out.
*** Bug 290163 has been marked as a duplicate of this bug. ***
Stumbled upon another bad case of deeply hidden hard coding, that might case bug http://git.eclipse.org/c/platform/eclipse.platform.releng.git/commit/?id=d173031455240f144616c97b3cf55c7f0d7b26f4
Doing a mass "reset to default assignee" of 52 bugs to help make clear it will (very likely) not be me working on things I had previously planned to work on. I hope this will help prevent the bugs from "getting lost" in other people's queries. Feel free to "take" a bug if appropriate.
We will investigate this in 4.8
I will look into this in 4.9
There are plans for this. Will revisit in 4.10
Moving out of 4.10. Please re triage appropriately
This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're closing this bug. If you have further information on the current state of the bug, please add it and reopen this bug. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. -- The automated Eclipse Genie.
