Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 542581

Summary: Bugzilla internal error sending change mails - no notifications sent
Product: Community Reporter: Michael Keppler <michael.keppler>
Component: BugzillaAssignee: Eclipse Webmaster <webmaster>
Status: RESOLVED FIXED QA Contact:
Severity: blocker    
Priority: P3 CC: christian.dietrich.opensource, daniel_megert, denis.roy, dev, frederic.gurr, jonah, loskutov, lshanmug, mikael.barbero, pierre-charles.david, walther
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Windows 10   
Whiteboard:

Description Michael Keppler CLA 2018-12-10 02:52:22 EST
I've just closed 3 bugs as duplicates. Each time bugzilla showed an internal error, when trying to send mails for the change.

There was an error sending mail from 'bugzilla-daemon@eclipse.org' to 'Lars.Vogel@vogella.com': 4.1.8 <bugzilla-daemon@eclipse.org>: Sender address rejected: Domain not found 

There was an error sending mail from 'bugzilla-daemon@eclipse.org' to 'Platform-UI-Inbox@eclipse.org': 4.1.8 <bugzilla-daemon@eclipse.org>: Sender address rejected: Domain not found

There was an error sending mail from 'bugzilla-daemon@eclipse.org' to 'tom.schindl@bestsolution.at': 4.1.8 <bugzilla-daemon@eclipse.org>: Sender address rejected: Domain not found

Related bug numbers are 542575 to 542577.
Comment 1 Michael Keppler CLA 2018-12-10 02:55:40 EST
In fact, even creating this bug led to the same issue:

 There was an error sending mail from 'bugzilla-daemon@eclipse.org' to 'denis.roy@eclipse-foundation.org': 4.1.8 <bugzilla-daemon@eclipse.org>: Sender address rejected: Domain not found

Traceback:

 at /localsite/Bugzilla/Mailer.pm line 179.
	Bugzilla::Mailer::MessageToMTA(...) called at /localsite/Bugzilla/BugMail.pm line 381
	Bugzilla::BugMail::sendMail(...) called at /localsite/Bugzilla/BugMail.pm line 269
	Bugzilla::BugMail::Send(...) called at /localsite/post_bug.cgi line 208
	ModPerl::ROOT::Bugzilla::ModPerl::ResponseHandler::localsite_post_bug_2ecgi::handler(...) called at /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm line 206
	eval {...} called at /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm line 206
	ModPerl::RegistryCooker::run(...) called at /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm line 172
	ModPerl::RegistryCooker::default_handler(...) called at /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/ModPerl/Registry.pm line 31
	ModPerl::Registry::handler(...) called at /localsite/mod_perl.pl line 137
	Bugzilla::ModPerl::ResponseHandler::handler(...) called at -e line 0
	eval {...} called at -e line 0
Comment 2 Dani Megert CLA 2018-12-10 04:56:51 EST
Notifications are no longer sent!
Comment 3 Jonah Graham CLA 2018-12-10 07:02:23 EST
*** Bug 542613 has been marked as a duplicate of this bug. ***
Comment 4 Christian Walther CLA 2018-12-10 08:15:31 EST
Possibly related, I don’t seem to be getting any emails from Gerrit either.

Another duplicate: bug 542608.
Comment 5 Denis Roy CLA 2018-12-10 09:16:37 EST
I see some bugzilla notifications flowing now...
Comment 6 Hans Müller CLA 2018-12-10 10:07:21 EST
ok thx
Comment 7 Dani Megert CLA 2018-12-10 10:20:55 EST
(In reply to Denis Roy from comment #5)
> I see some bugzilla notifications flowing now...

I see some too, but only new ones. What about the "lost" ones?
Comment 8 Dani Megert CLA 2018-12-10 10:21:22 EST
*** Bug 542608 has been marked as a duplicate of this bug. ***
Comment 9 Denis Roy CLA 2018-12-10 10:22:25 EST
If bugzilla ended with a traceback, then the old ones never got sent  :/
Comment 10 Dani Megert CLA 2018-12-10 10:41:01 EST
(In reply to Denis Roy from comment #9)
> If bugzilla ended with a traceback, then the old ones never got sent  :/

Do you know when it started to fail? That would allow us to do a search for bugs that changed since then.
Comment 11 Dani Megert CLA 2018-12-10 10:43:43 EST
*** NOTE ***

While changes to bug reports no longer give an error, I did not get any e-mail about the recent changes in this bug, e.g. for comment 9, made 20 minutes ago!
Comment 12 Jonah Graham CLA 2018-12-10 10:48:04 EST
(In reply to Dani Megert from comment #11)
> *** NOTE ***
> 
> While changes to bug reports no longer give an error, I did not get any
> e-mail about the recent changes in this bug, e.g. for comment 9, made 20
> minutes ago!

I did get Comment 9. I do see that emails going through Eclipse are very delayed. e.g. a gerrit comment had a 10 minute delay:

Received: from mail.eclipse.org (mail.eclipse.org [198.41.30.200]) by pb-mx23.pobox.com (Postfix) with SMTP for <jonah@kichwacoders.com>; Mon, 10 Dec 2018 10:44:15 -0500 (EST) (envelope-from gerrit@eclipse.org)
Received: from gerrit-vm1.eclipse.org (unknown [172.30.3.253]) by mail.eclipse.org (Postfix) with ESMTP id C1D202DD054; Mon, 10 Dec 2018 10:44:14 -0500 (EST)
Received: from localhost (localhost [127.0.0.1]) by gerrit-vm1.eclipse.org (Postfix) with ESMTP id 7227718E8828; Mon, 10 Dec 2018 10:36:20 -0500 (EST)


But the log for Comment 9 email shows no delay:

Received: from mail.eclipse.org (mail.eclipse.org [198.41.30.200]) by pb-mx9.pobox.com (Postfix) with SMTP for <jonah@kichwacoders.com>; Mon, 10 Dec 2018 10:22:26 -0500 (EST)
Received: from bugs-vm1.eclipse.org (unknown [172.30.3.253]) by mail.eclipse.org (Postfix) with ESMTP id 191C22DD134 for <jonah@kichwacoders.com>; Mon, 10 Dec 2018 10:22:26 -0500 (EST)
Received: from bugs.eclipse.org (localhost [127.0.0.1]) by bugs-vm1.eclipse.org (Postfix) with ESMTP id 40E6661EFD for <jonah@kichwacoders.com>; Mon, 10 Dec 2018 10:22:26 -0500 (EST)
Comment 13 Dani Megert CLA 2018-12-10 11:45:51 EST
(In reply to Jonah Graham from comment #12)

So far I did not get any e-mail for your comment made at 2018-12-10 16:48:04 CET.
Comment 14 Denis Roy CLA 2018-12-10 11:46:03 EST
Things are flushing out. We're making changes to our core gateways, and this is some of the dust.

Apologies for the inconvenience. All Mail should be stabilizing and flushing itself out.

These changes started Friday, around noon Ottawa time.
Comment 15 Dani Megert CLA 2018-12-10 11:50:47 EST
(In reply to Denis Roy from comment #14)
> Things are flushing out. We're making changes to our core gateways, and this
> is some of the dust.
> 
> Apologies for the inconvenience. All Mail should be stabilizing and flushing
> itself out.
> 
> These changes started Friday, around noon Ottawa time.

I got this just now, but no notification for comment 12 so far.
Comment 16 Jonah Graham CLA 2018-12-14 09:03:08 EST
(In response to request on cross-project to resend lost emails)

Hi Mikaël/webmasters,

I think the emails may actually not be permanently lost. I think bugzilla knows what it hasn't sent out. However I don't know how to get bugzilla to send them. 

For example, I just made a comment on Bug 542613 - one of the bugs that emails didn't get sent out the other day for - and the one email contained all the comments/edits made. As this email was cc'ed to webmaster@eclipse.org perhaps you have just received that too.

Thanks,
Jonah
Comment 17 Frederic Gurr CLA 2018-12-14 09:06:08 EST
Yeah, the problem is that it only does that if another comment is added to the bug (or something else is changed?) after it works again.
Comment 18 Denis Roy CLA 2018-12-14 09:08:56 EST
> I think the emails may actually not be permanently lost. I think bugzilla
> knows what it hasn't sent out. However I don't know how to get bugzilla to
> send them. 

That is correct. I just discovered the "flush bugmail queue" tool.
Comment 19 Denis Roy CLA 2018-12-14 09:10:01 EST
586 bugs found with possibly unsent mail.

Sending mail for bug 33184...
9 mails sent.
Took 1 seconds.

Sending mail for bug 68836...
13 mails sent.
Took 2 seconds.


Uh oh....
Comment 20 Pierre-Charles David CLA 2018-12-14 09:15:04 EST
(In reply to Denis Roy from comment #19)
> 586 bugs found with possibly unsent mail.
> 
> Sending mail for bug 33184...
> 9 mails sent.
> Took 1 seconds.
> 
> Sending mail for bug 68836...
> 13 mails sent.
> Took 2 seconds.
> 
> 
> Uh oh....

Probably related to this, but since a few minutes ago I've started receiving bugzilla notification mail for changes dating back from 2013. 26 mails for now. Oops, make that 28 now :)
Comment 21 Denis Roy CLA 2018-12-14 09:16:17 EST
> Uh oh....

*sigh*


I think there were a bunch of silently queued emails over the last, oh, decade.
Comment 22 Denis Roy CLA 2018-12-14 09:20:03 EST
I've stopped the process since the webmaster inbox is filling up with old, stale email notifications.
Comment 23 Dani Megert CLA 2018-12-14 10:19:29 EST
(In reply to Denis Roy from comment #22)
> I've stopped the process since the webmaster inbox is filling up with old,
> stale email notifications.

Thanks.

This was really a bad move! I now have to sort out all those old miles out of over 50 notifications. And the bad thing is, that the messages does not contain the change(d) date, hence I have to look at every single one.
Comment 24 Dani Megert CLA 2018-12-21 08:40:17 EST
Marking as FIXED, because I don't think more is going to happen here.