Community
Participate
Working Groups
Build Completed mail no longer reports errors and test failures. Here's how it should look like: http://dev.eclipse.org/mhonarc/lists/platform-releng-dev/msg18533.html
Also note, that the summary reflected whether the build was good: Build is complete. or had issues: Build is complete. Test failures/errors occurred.
We used to have two mails: 1) Build completed. // still available 2) Tests completed. // missing The subject of mail (2) used to be "Build is complete. ...", which didn't make much sense.
David, can we get this back. It also helps the teams to reply to this e-mail and explain the failures.
I don't see this happening in time to do much good for Kepler ... too many other priorities ... too much has changed since this last worked so would basically be "re-doing" it (AFAIK). And I can't help wonder if there's not something in Hudson (e.g. RSS Feeds) that would accomplish similar?). For example, I subscribe, and with each platoform completion, I get a message similar to Subject: [Hudson] Hudson build is still unstable: ep4-unit-lin64 #577 See <https://hudson.eclipse.org/hudson/job/ep4-unit-lin64/577/> I understand its not the same thing ... but ... just wondering if it helps until our more "custom" solution can be re-instated?
(In reply to comment #4) > I don't see this happening in time to do much good for Kepler ... too many > other priorities ... too much has changed since this last worked so would > basically be "re-doing" it (AFAIK). > > And I can't help wonder if there's not something in Hudson (e.g. RSS Feeds) > that would accomplish similar?). > > For example, I subscribe, and with each platoform completion, I get a > message similar to > > Subject: [Hudson] Hudson build is still unstable: ep4-unit-lin64 #577 > See <https://hudson.eclipse.org/hudson/job/ep4-unit-lin64/577/> Are the messages restricted to our builds, or will there be lots of noise? > I understand its not the same thing ... but ... just wondering if it helps > until our more "custom" solution can be re-instated? Problem is: every committer who is supposed to track releng would have to do that. Maybe there's a way to tell Hudson to forward the message to a specific address? How about subscribing the releng list to Hudson?
I really miss this! Currently I have to check several times unless I see all results.
I'll likely just send the summary table, rather than the list as mentioned in comment 0 -- if I can get HTML mail messages to work well and if that meets with everyone's agreement. I think you (we) mostly just want a mail signal that "it's done". And, if you like I could send a twitter message, some SMS messages, :) ... just kidding ... we'll stick to mailing list for now.
(In reply to David Williams from comment #7) > I think you (we) mostly just want a mail signal that "it's done". I'd like to see either the failed tests suites or the components that have failures. The summary table would do that.
(In reply to Dani Megert from comment #8) > (In reply to David Williams from comment #7) > > I think you (we) mostly just want a mail signal that "it's done". > > I'd like to see either the failed tests suites or the components that have > failures. The summary table would do that. Oh, well, I meant the "small" summary tables (that's currently at top of DL page). I hadn't thought of send "whole" (large) summary table ... might make for a lot of scrolling ... but, I'll investigate/prototype. I understand this can be a time-saver for committers who are "mail driven", and appreciate you bringing it back up.
(In reply to David Williams from comment #9) > I understand this can be a time-saver for committers who are "mail driven", Is there a better way?
(In reply to Dani Megert from comment #10) > (In reply to David Williams from comment #9) > > I understand this can be a time-saver for committers who are "mail driven", > > Is there a better way? At this point I would even be fine to get at least a note that the tests are done and a link that points me to the test results page.
I've committed initial fix: https://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/commit/?id=917052e45c2c7295da0e8085499b52fa3a3b8720 A lot of refactoring (to make this fix easier) was done as part of bug 467982. Implemented so that there will be a separate message for each of the three platforms. I'm not sure how it used to be, but that seems to make sense, since there can be such a long time difference between "the first to complete" and "the last". (Not to mention other complications, such "what if the last one does not complete!) Also, implemented so that these are sent for N, I, and M build unit tests. I am currently excluding the "performance tests" notifications, but easy to add when ready. (for cross-reference, see an old bug 181947 to get notification of performance tests). I am going to leave open, because I tried to get "total errors" to be part of the message but I do not think that is working. (I think that is important, so if zero errors will know there is no need to look -- though, there is still the problem of "DNFs" do not get counted as "Errors or Failures", yet).
I think the rest is fixed up. https://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/commit/?id=b4064fbc0be269527a9de9b44f61039d32d096f7
The bug mail subject can be improved: - start with the build ID, so that it will be sorted right after the "Build finished" mail - make it shorter and put the interesting info (failure count) earlier. Old: Test Results available from ep46N-unit-win32 for N20151213-2000 Failures: 2 New: 4.6.0 N-Build: N20151213-2000: 2 failures from ep46N-unit-win32 Fixed with https://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/commit/?id=e0f367452ce5713235c378469ac4019b7ac703d6 In the old days, the "Tests completed" mail ... ... contained a list of components that produced test failures. => Component committers could see at one glance whether they need to go to the test results or whether they can just delete the mail. ... was only sent when all platforms were done. => Shows right away whether failures are platform-specific or not. => Saves quite a few messages that are rarely interesting. When I'm eager to see test results, then I poll https://hudson.eclipse.org/shared/view/Eclipse%20and%20Equinox/ , where I can even guesstimate how long tests will take. We could add that link to the "Build completed" mail. For me, the small advantage of seeing test results a bit earlier doesn't outweigh the disadvantage of getting too many mails.
(In reply to Markus Keller from comment #14) > The bug mail subject can be improved: > - start with the build ID, so that it will be sorted right after the "Build > finished" mail > - make it shorter and put the interesting info (failure count) earlier. > Thanks for the improvements, Markus. I'm glad I could simplify things enough for you to be able to improve. I think there are lots of ways to improve, but thought any beyond the ones you have made would be a separate enhancement request. The reason I worked on this at all was Dani's urgency to "have something" as expressed during the last milestone rampdown: https://dev.eclipse.org/mhonarc/lists/platform-releng-dev/msg23695.html I figured the rest could sort of evolve, as we gained experience. (For example, I question if we need them for "nightlies" at all? ... but realize sometimes "mail" is the only way to get people's attention). I think I will, though, add the hudson link, via this bug. Not sure if I'll have time before 8 PM.
I have added the "Hudson URLs" to standard message. The "first one" was wrong, https://dev.eclipse.org/mhonarc/lists/platform-releng-dev/msg23724.html because I was still in the middle of fixing it before I remembered some variables had to be better defined for "production eclipse". Should be fixed, in time, when the next one gets sent out. I have also opened one enhancement request to include more detail about what failed. Bug 484429. (I consider it a lower priority than this one). = = = = Sort of related, but not directly, I have opened another bug to make it easier when others update these sort of scripts. Scripts in the "SDK Directory" have to be updated "by hand" on the build machine, as described in bug 484431.
I opened another enhancement, bug 484436, to cover doing this for performance tests. And, as predicted, the next notification did include the correct Hudson URLs: https://dev.eclipse.org/mhonarc/lists/platform-releng-dev/msg23725.html But, I am "re-opening" this one since there is a what I consider a true "bug" in current notifications. The "Time" that is displayed is not "Elapsed Time". This can be seen by comparing the time reported, in a mail message, with the time reported on the Hudson build page. For the above link, for a Hudson test, the time reported is Tests Elapsed Time: 3 h 34 m but on the Hudson page, it correctly says the time Took 4 hr 38 min on hudson-slave4 I'm fairly sure the "mail time" is literally the sum of the time the unit tests take to run. This does not include the time to "unzip and setup" the Eclipse SDK over and over again, for the various suites, and other overhead while the tests are running. I propose to fix by changing "Tests Elapsed Time:" to say "Tests Total Time:" Not sure which Hudson API to call to get that "Elapsed time", but no particular reason so, other than it is interesting how much overhead there is.
(In reply to David Williams from comment #17) > Not sure which Hudson API to call to get that "Elapsed time", but no > particular reason so, other than it is interesting how much overhead there > is. It so happens, luckily enough, the "duration" is one of the examples Hudson gives. :) For that Linux job number 59, the duration time can be obtained with https://hudson.eclipse.org/shared/view/Eclipse%20and%20Equinox/job/ep46I-unit-lin64/54/api/xml?xpath=/*/duration/text%28%29 So, I think I will add "Elapsed time" along with the renamed "total test time".
In the test result mails, the failing components would be more interesting than some elapsed time or total number of tests. The goals of these mails are: a) inform committers that the tests are done b) tell committers whether they have to go to the test results page The old mail format served these goals pretty well. The current mails per platform are too frequent and don't contain enough information. It would also be good to group the mails for the same build in a thread, e.g. by adding these two headers to all mails for the same $buildId: In-Reply-To: <I20151215-0800@build.eclipse.org/eclipse/builds/> References: <I20151215-0800@build.eclipse.org/eclipse/builds/>
(In reply to Markus Keller from comment #19) > The current mails per platform are too frequent I agree on that. One mail when all tests are finished fits my needs.
(In reply to Dani Megert from comment #20) > (In reply to Markus Keller from comment #19) > > The current mails per platform are too frequent > > I agree on that. One mail when all tests are finished fits my needs. See comment 0 how it used to look like.
(In reply to Markus Keller from comment #19) > It would also be good to group the mails for the same build in a thread, > e.g. by adding these two headers to all mails for the same $buildId: > > In-Reply-To: <I20151215-0800@build.eclipse.org/eclipse/builds/> > References: <I20151215-0800@build.eclipse.org/eclipse/builds/> Thanks for this tip. Seems to work relatively well on thunderbird, as long as the messages arrive in the right order. Should show up on dev list in tonight's run. I'm closing this bug as 'fixed' and will continue the suggested improvements in bug 484429. The "fixed" part is my interpretation of https://dev.eclipse.org/mhonarc/lists/platform-releng-dev/msg23695.html where is was said ... "It is really not efficient that everyone has to poll the results page several times until one gets all results. Usually, this is not that a big deal, but during milestone week one is eager to see the results as soon as they are there." [And, I'll add, this said as a result of "polling the results page" many hours before they could possibly be done, which is why I thought "elapsed time" would be useful, so people know how long it takes, roughly.]
