The following is the GNU All-permissive License as recommended in https://www.gnu.org/licenses/license-recommendations.en.html
Copyright (C) 2024 Free Software Foundation sysadmin@fsf.org
Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. This file is offered as-is, without any warranty.
Contributions are welcome. See https://savannah.gnu.org/maintenance/fsf/.
Relevant documentation: https://docs.civicrm.org/sysadmin/en/latest/setup/civimail/inbound
Bounce processing was disabled from nov 1 2016 to 2020-05-06 because it marked too many false positives, RT#1087723
There are regexes in the db used to classify bounces. We disabled some categories and a few bad regexes to fix the problem. Also, needed to update the db to ignore bounces in its log from before the date when we started using new rules.
An even better solution might be to use https://github.com/sisimai/rb-Sisimai and do our own bounce processing.
bounce addresses look like this: crmmailer+b but there are other letters: civicrm-core/CRM/Utils/Mail/EmailProcessor.php: b|bounce|c|confirm|o|optOut|r|reply|re|e|resubscribe|u|unsubscribe
select * from civicrm_mailing_bounce_type;
+----+----------+-----------------------------------------------+----------------+
| id | name | description | hold_threshold |
+----+----------+-----------------------------------------------+----------------+
| 1 | AOL | AOL Terms of Service complaint | 1 |
| 2 | Away | Recipient is on vacation | 99 |
| 3 | DNS | Unable to resolve recipient domain | 3 |
| 4 | Host | Unable to deliver to destintation mail server | 3 |
| 5 | Inactive | User account is no longer active | 1 |
| 6 | Invalid | Email address is not valid | 1 |
| 7 | Loop | Mail routing error | 3 |
| 8 | Quota | User inbox is full | 3 |
| 9 | Relay | Unable to reach destination mail server | 3 |
| 10 | Spam | Message caught by a content filter | 1 |
| 11 | Syntax | Error in SMTP transaction | 3 |
+----+----------+-----------------------------------------------+----------------+
# These seem good to stop mailing.
| 1 | AOL | AOL Terms of Service complaint | 1 |
| 6 | Invalid | Email address is not valid | 1 |
# For smaller providers, I expect there could ocasionally be a temporary bounce with this
# so change to 5 or so once we start processing all bounces
| 5 | Inactive | User account is no longer active | 1 |
# Avoid this for now until we upgrade our mail servers and maybe
# we stop being seen as spam.
| 10 | Spam | Message caught by a content filter | 1 |
# These have no effect on deliverability or some within them do, but need to be separated and have some retries
| 3 | DNS | Unable to resolve recipient domain | 3 |
| 4 | Host | Unable to deliver to destintation mail server | 3 |
| 9 | Relay | Unable to reach destination mail server | 3 |
| 2 | Away | Recipient is on vacation | 99 |
| 7 | Loop | Mail routing error | 3 |
| 8 | Quota | User inbox is full | 3 |
# All unclassified bounces also get this number. Clearly not a good idea.
| 11 | Syntax | Error in SMTP transaction | 3 |
update civicrm_mailing_bounce_type set hold_threshold=40000000 where id != 1 and id != 6 and id != 5;
# Update 2022-10-08, we got added to spamhaus sbl blocklist. So I'm lowering them
# down:
update civicrm_mailing_bounce_type set hold_threshold=99 where id = 2;
update civicrm_mailing_bounce_type set hold_threshold=10 where id = 4 or id = 8 or id = 11;
update civicrm_mailing_bounce_type set hold_threshold=6 where id = 10;
# gmail had a bug where it returned 500 for 6k contacts and they immediately went on hold.
# Civicrm was dumb to be treating any permanent error as permanent, so increate to 3.
update civicrm_mailing_bounce_type set hold_threshold = 3 where hold_threshold = 1;
To see the regexes these will match,
select * from civicrm_mailing_bounce_pattern where bounce_type_id = 1 or bounce_type_id = 6 or bounce_type_id = 5;
+-----+----------------+-------------------------------------------------------------------------------------------------+ | id | bounce_type_id | pattern | +-----+----------------+-------------------------------------------------------------------------------------------------+ | 1 | 1 | Client TOS Notification | | 21 | 5 | (my )?e-?mail( address)? has changed | | 22 | 5 | account (inactive|expired|deactivated) | | 23 | 5 | account is locked | | 24 | 5 | changed w+( e-?mail)? address | | 25 | 5 | deactivated mailbox | | 26 | 5 | disabled or discontinued | | 27 | 5 | inactive user | | 28 | 5 | is inactive on this domain | | 29 | 5 | mail receiving disabled | | 30 | 5 | mail( ?)address is administrative?ly disabled | | 32 | 5 | no longer (accepting mail|on server|in use|with|employed|on staff|works for|using this account) | | 33 | 5 | not accepting (mail|messages) | | 34 | 5 | please use my new e-?mail address | | 35 | 5 | this address no longer accepts mail | | 36 | 5 | user account suspended | | 37 | 6 | (user|recipient( name)?) is not recognized | | 38 | 6 | 554 delivery error | | 39 | 6 | address does not exist | | 40 | 6 | address(es)?( you (entered|specified))? (could|was)( not|n.t)( be)? found | | 41 | 6 | address(ee)? (unknown|invalid) | | 42 | 6 | bad destination | | 43 | 6 | badly formatted address | | 44 | 6 | can't open mailbox for | | 45 | 6 | cannot deliver | | 46 | 6 | delivery to the following recipient(s)? failed | | 47 | 6 | destination addresses were unknown | | 48 | 6 | did not reach the following recipient | | 49 | 6 | does not exist | | 50 | 6 | does not like recipient | | 51 | 6 | does not specify a valid notes mail file | | 52 | 6 | illegal alias | | 53 | 6 | invalid (mailbox|(e-?mail )?address|recipient|final delivery) | | 54 | 6 | invalid( or unknown)?( virtual)? user | | 55 | 6 | (mail )?delivery (to this user )?is not allowed | | 56 | 6 | mailbox (not found|unavailable|name not allowed) | | 57 | 6 | message could not be forwarded | | 58 | 6 | missing or malformed local(-| )part | | 59 | 6 | no e-?mail address registered | | 60 | 6 | no such (mail drop|mailbox( \w+)?|(e-?mail )?address|recipient|(local )?user|person)( here)? | | 61 | 6 | no mailbox (here )?by that name | | 62 | 6 | not (listed in|found in directory|known at this site|our customer) | | 63 | 6 | not a valid( (user|mailbox))? | | 64 | 6 | not present in directory entry | | 65 | 6 | recipient (does not exist|(is )?unknown|rejected|denied|not found) | | 66 | 6 | this user doesn't have a yahoo.com address | | 67 | 6 | unavailable to take delivery of the message | | 68 | 6 | unavailable mailbox | | 69 | 6 | unknown (local( |-)part|recipient|address error) | | 70 | 6 | unknown( or illegal)? user( account)? | | 71 | 6 | unrecognized recipient | | 72 | 6 | unregistered address | | 73 | 6 | user (unknown|(does not|doesn't) exist) | | 74 | 6 | user doesn't have an? w+ account | | 75 | 6 | user('s e-?mail name is)? not found | | 124 | 6 | ^Validation failed for: | | 141 | 5 | account that you tried to reach is disabled | | 142 | 5 | User banned | | 150 | 6 | 5.1.0 Address rejected | | 151 | 6 | no valid recipients? | | 152 | 6 | RecipNotFound | | 153 | 6 | no one at this address | | 155 | 6 | account is not allowed | | 156 | 6 | Address .<[^>]*>. not known here | | 157 | 6 | Recipient address rejected: ([a-zA-Z0-9-]+.)+[a-zA-Z]{2,} | | 158 | 6 | Non sono riuscito a trovare l.indirizzo e-mail | | 159 | 6 | nadie con esta direcci..?n | | 160 | 6 | ni bilo mogo..?e najti prejemnikovega e-po..?tnega naslova | | 161 | 6 | Elektronski naslov (je ukinjen|ne obstaja) | | 162 | 6 | nepravilno nastavljen predal | +-----+----------------+-------------------------------------------------------------------------------------------------+
these ones could be a temporary, remove them.
delete from civicrm_mailing_bounce_pattern where pattern = 'misconfigured forwarding address';
delete from civicrm_mailing_bounce_pattern where pattern = 'mailbox (temporarily disabled|currently suspended)';
in mail account settings
https://my.fsf.org/civicrm/admin/mailSettings?reset=1
I changed protocol from imap to localdir, and added a source path.
# to see on_hold logic,
# based on query in CRM/Core/BAO/Email.php
SELECT count(civicrm_mailing_event_bounce.id) as bounces,
civicrm_mailing_bounce_type.hold_threshold as threshold
FROM civicrm_mailing_event_bounce
INNER JOIN civicrm_mailing_bounce_type
ON civicrm_mailing_event_bounce.bounce_type_id = civicrm_mailing_bounce_type.id
INNER JOIN civicrm_mailing_event_queue
ON civicrm_mailing_event_bounce.event_queue_id = civicrm_mailing_event_queue.id
INNER JOIN civicrm_email
ON civicrm_mailing_event_queue.email_id = civicrm_email.id
WHERE civicrm_email.id = (select id from civicrm_email where email = 'info@zlug.ly')
AND (civicrm_email.reset_date IS NULL
OR civicrm_mailing_event_bounce.time_stamp >= civicrm_email.reset_date)
GROUP BY civicrm_mailing_event_bounce.bounce_type_id
ORDER BY threshold, bounces desc;
# to find the past bounces: of x@x.com
SELECT civicrm_mailing_event_bounce.bounce_type_id, civicrm_mailing_event_bounce.bounce_reason, civicrm_mailing_event_bounce.time_stamp
FROM civicrm_mailing_event_bounce
INNER JOIN civicrm_mailing_bounce_type
ON civicrm_mailing_event_bounce.bounce_type_id = civicrm_mailing_bounce_type.id
INNER JOIN civicrm_mailing_event_queue
ON civicrm_mailing_event_bounce.event_queue_id = civicrm_mailing_event_queue.id
INNER JOIN civicrm_email
ON civicrm_mailing_event_queue.email_id = civicrm_email.id
WHERE civicrm_email.id = (select id from civicrm_email where email = 'x@x.com')
AND (civicrm_email.reset_date IS NULL
OR civicrm_mailing_event_bounce.time_stamp >= civicrm_email.reset_date);
## setting a reset_date for all emails that have a bounce, but are not on_hold. essentially like discarding old bounces we didnt act on, when at some point in the past, we set on_hold to 0 for most contacts.
# count of records to update: emails that have a past bounce and are not on hold.
select count(*) from (select distinct(civicrm_email.id) from civicrm_email inner join civicrm_mailing_event_queue on civicrm_email.id = civicrm_mailing_event_queue.email_id INNER JOIN civicrm_mailing_event_bounce ON civicrm_mailing_event_bounce.event_queue_id = civicrm_mailing_event_queue.id where civicrm_email.on_hold = 0) as c;
# doing the update, it is so different from the counting, that I wrapped in a transaction in case it updated a different number:
start transaction;
update civicrm_email as cmail inner join civicrm_mailing_event_queue on cmail.id = civicrm_mailing_event_queue.email_id INNER JOIN civicrm_mailing_event_bounce ON civicrm_mailing_event_bounce.event_queue_id = civicrm_mailing_event_queue.id set cmail.reset_date = '2020-05-01 12:00:00' where cmail.on_hold = 0;
commit;
# if it had updated an unexpected number, then run: rollback;