Sending email to some domains I get a failure message from Axigen saying ‘Unable to perform STARTTLS’
I have searched high and low for a solution, and have followed various Axigen KB articles best I can, but the issue remains.
Testing my domain using ‘SMTP TLS Checker For Inbound Email Servers | LuxSci’ reports all ‘Yes’ except for ‘Certificate Verifiable? = No’
Axigen log files report …
SSL_connect error: error:0A000086:SSL routines::certificate verify failed
I’m using a LetsEncrypt certificate generated from with Axigen. The certificate is valid (i.e. not expired). Looking at ‘View Usage’ for the certificate it is assigned to all services.
OS is RHEL v9.3. Axigen is v10.5.15
So it seems my certificate cannot be verified but I’m at a loss as to where to look and how to fix the issue.
You mentioned that you could not send messages to external domains, so this means Axigen is acting as a client and not a server (where your certificate is used, being ‘attached’ to one or more listeners).
Just to be sure that we understand your problem correctly please set the log level to Protocol Communication for SMTP Sending service and share here the SMTP-OUT corresponding log lines for a message that could not be sent.
Also please let us know your Axigen version and the OS you are using.
I’m experiencing the same behavior. For me, it’s occurring when there are problems with the receivers’ TLS configuration. Some examples currently in my logs are domains with expired SSL certs (an SMTP-OUT attempt to oglethorpefirerescue.org appears below).
Any ideas for a custom acceptance & routing advanced rule that would send without STARTTLS when a handshake wouldn’t allow for performing STARTTLS? One that wouldn’t require me to conditionally define every misconfigured domain (see the attached screenshot)?
2024-02-04 00:01:03 +0000 08 mail PROCESSING:0026F64D: Start mail delivery
2024-02-04 00:01:03 +0000 08 mail PROCESSING:0026F64D: Set mail state to SENDING
2024-02-04 00:01:03 +0000 08 mail PROCESSING:0026F64D: Start remote delivery for 1 recipients in domain oglethorpefirerescue.org
2024-02-04 00:01:03 +0000 08 mail PROCESSING:0026F64D: Use plain connection
2024-02-04 00:01:03 +0000 08 mail PROCESSING:0026F64D: STARTTLS extension allowed
2024-02-04 00:01:03 +0000 08 mail PROCESSING:0026F64D: Allowed ssl versions set to: tls1 tls11 tls12 tls13
2024-02-04 00:01:03 +0000 08 mail PROCESSING:0026F64D: SSL certificate file set to: letsencrypt/mail.neatoware.com/cert.pem
2024-02-04 00:01:03 +0000 08 mail PROCESSING:0026F64D: SSL CA file set to: letsencrypt/mail.neatoware.com/cert_auth.pem
2024-02-04 00:01:03 +0000 08 mail PROCESSING:0026F64D: SSL DH param file set to: axigen_dh.pem
2024-02-04 00:01:03 +0000 08 mail PROCESSING:0026F64D: SSL cipher suite set to: TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:DHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH
2024-02-04 00:01:03 +0000 08 mail PROCESSING:0026F64D: Set EHLO host to <mail.neatoware.com>
2024-02-04 00:01:03 +0000 08 mail PROCESSING:0026F64D: Use plain connection
2024-02-04 00:01:03 +0000 08 mail PROCESSING:0026F64D: Use plain connection
2024-02-04 00:01:03 +0000 08 mail PROCESSING:0026F64D: Relay mail using default host: oglethorpefirerescue.org:25
2024-02-04 00:01:03 +0000 08 mail DNR:0026F64D: Search MX for ‘oglethorpefirerescue.org’ (EDNS is on)
2024-02-04 00:01:03 +0000 08 mail DNR:0026F64D: Sending query (1/1) to 9.9.9.9:53
2024-02-04 00:01:03 +0000 08 mail DNR:0026F64D: Sending query (1/1) to 9.9.9.9:53
2024-02-04 00:01:04 +0000 08 mail DNR:0026F64D: ‘mx.oglethorpefirerescue.org’ found MX ‘66.96.140.159’ with priority 30
2024-02-04 00:01:04 +0000 08 mail DNR:0026F64D: ‘mx.oglethorpefirerescue.org’ found MX ‘66.96.140.158’ with priority 30
2024-02-04 00:01:04 +0000 08 mail DNR:0026F64D: Sending query (1/1) to 9.9.9.9:53
2024-02-04 00:01:04 +0000 08 mail PROCESSING:0026F64D: Relay mail: found IPv4 MX entry 66.96.140.158 for domain oglethorpefirerescue.org with priority 30
2024-02-04 00:01:04 +0000 08 mail PROCESSING:0026F64D: Relay mail: found IPv4 MX entry 66.96.140.159 for domain oglethorpefirerescue.org with priority 30
2024-02-04 00:01:04 +0000 08 mail PROCESSING:0026F64D: Use 66.96.140.159 to relay mail 26F64D for domain oglethorpefirerescue.org
2024-02-04 00:01:04 +0000 08 mail SMTP-OUT:00000005: Relay mail 26F64D: connecting to 66.96.140.159:25
2024-02-04 00:01:04 +0000 08 mail SMTP-OUT:00000005: Relay mail 26F64D: connected to 66.96.140.159:25
2024-02-04 00:01:38 +0000 02 mail SMTP-OUT:00000005: SSL alert remote 66.96.140.159:25, undefined:fatal:unknown CA
2024-02-04 00:01:38 +0000 02 mail SMTP-OUT:00000005: SSL error remote 66.96.140.159:25, SSL_connect:failed in error
2024-02-04 00:01:38 +0000 02 mail SMTP-OUT:00000005: 66.96.140.159:25 SSL_connect error: error:0A000086:SSL routines::certificate verify failed
2024-02-04 00:01:38 +0000 02 mail SMTP-OUT:00000005: Unable to perform STARTTLS
2024-02-04 00:01:38 +0000 08 mail SMTP-OUT:00000005: Disconnected from 66.96.140.159
2024-02-04 00:01:38 +0000 08 mail SMTP-OUT:00000005: Use 66.96.140.158 to relay mail 26F64D for domain oglethorpefirerescue.org
2024-02-04 00:01:38 +0000 08 mail SMTP-OUT:00000006: Relay mail 26F64D: connecting to 66.96.140.158:25
2024-02-04 00:01:38 +0000 08 mail SMTP-OUT:00000006: Relay mail 26F64D: connected to 66.96.140.158:25
2024-02-04 00:01:50 +0000 02 mail SMTP-OUT:00000006: SSL alert remote 66.96.140.158:25, undefined:fatal:unknown CA
2024-02-04 00:01:50 +0000 02 mail SMTP-OUT:00000006: SSL error remote 66.96.140.158:25, SSL_connect:failed in error
2024-02-04 00:01:50 +0000 02 mail SMTP-OUT:00000006: 66.96.140.158:25 SSL_connect error: error:0A000086:SSL routines::certificate verify failed
2024-02-04 00:01:50 +0000 02 mail SMTP-OUT:00000006: Unable to perform STARTTLS
2024-02-04 00:01:50 +0000 08 mail SMTP-OUT:00000006: Disconnected from 66.96.140.158
2024-02-04 00:01:50 +0000 08 mail SMTP-OUT:00000006: Relay mail 26F64D: no more relays for oglethorpefirerescue.org
2024-02-04 00:01:50 +0000 04 mail SMTP-OUT:00000006: Delivery attempt completed for mail 26F64D; 1 recipients remaining; reschedule for delivery
2024-02-04 00:01:50 +0000 08 mail SMTP-OUT:00000006: Set mail state to SEND FAILURE
2024-02-04 00:01:51 +0000 04 mail PROCESSING:0026F64D: 1 temporary failed recipients; compose temporary NDR
Starting with Axigen version 10.5.12 we have switched to OpenSSL 3.0 (more details here) and we have a high level of certainty* that the reported issue is a consequence of using more restrictive default parameters (like not accepting old TLS versions or other internal options considered deprecated/obsolete).
*especially seeing log lines like undefined:fatal:unknown CA reported for SMTP-OUT
To configure the new OpenSSL library so when sending messages the SMTP-OUT client will ignore the validation results for the remote server SSL certificate (during StartTLS negociation) the Axigen admin may set the following environment variable
For Linux: in the service configuration file (e.g. /etc/sysconfig/axigen for RPM based systems or /etc/default/axigen for DEB ones) add a new line, for example at the end of the file, like:
export AXI_SMTPOUT_OPT_X_TLS_IGNORE_CERT=yes
For Windows: access the Environment Variables > System variables section specific to your Windows version and set it there
For Docker: use the dedicated --env option when running the Axigen container
Please note that Axigen service restart is required which is why we recommend you to perform these changes during a scheduled maintenance window (or anytime a restart event is suitable for your installation).
Also, when dealing with very old remote servers you may like also to see if the following settings will help as well:
Hello, Ioan! That solves my SMTP-OUT problems. Thank you!
I have another “unknown CA” issue that may be OpenSSL 3.0 related. My Axigen server was failing CA checks occasionaly when other SMTP servers attempted to connect to it. The Axigen Let’s Encrypt implementation stores the cert_auth.pem in /var/opt/axigen/letsencrypt/[domain]/, but cert_auth.pem contains only the intermediate certificate. I resolved those failures by building and utlizing a fullchain.pem for Let’s Encrypt that includes the isrgrootx1.pem and pointing “Certificate authorities file” to it. Is that a known issue? Do you know of a better way to deal with those failures?
@AdrianW , I was able to make my Axigen generated Let’s Encrypt certificates verifiable by building a fullchain.pem and pointing “Certificate authorities file” to the fullchain.pem. Let’s Encrypt’s chain of trust is available at Chain of Trust - Let’s Encrypt (letsencrypt.org)
I mentioned it to Ioan. Maybe he will have a more elegant solution?
Axigen sends and receives to most external domains successfully. It is only some external domains that fail.
My OS is RedHat (RHEL) v9.3 and Axigen is version 10.5.15.
Will do.
Will try this and advise.
I have this problem too. Only way I could resolve the issue was to not use Axigen to generate a new LetsEncrypt certificate. I used Certbot to manually generate a new certificate and then dragged and dropped the fullchain.pem and privkey.pem files from /etc/letsencrypt/archive/[domain]/ into the “Upload an SSL certificate you’ve acquired …” box in WebAdmin > Security & Filtering > SSLCertifcates > +Add. Problem I now have is that I’ll need to do this again every 90-days.
From my knowledge (and explained here as well) there are only 2 configurations needed to install a Let’s Encrypt certificate requested and renewed by Axigen server:
1/ set Certificate file with letsencrypt/[domain]/cert.pem
2/ set Certificate authorities file with letsencrypt/[domain]/cert_auth.pem
And that’s should be all.
If any of you have done this on a listener or for a WebMail virtual host that could be checked from Internet and it is not working fine please let us know so we could check and come back with our findings.
A very useful online tool to see exactly which certificate(s) is/are received from your server’s listener is:
We’re seeing TLS Certificate Verifiable errors when we set Certificate authorities file with letsencrypt/[domain]/cert_auth.pem because it’s not a fullchain.pem. To pass some SMTP servers’ tests, Axigen needs to supply Let’s Encrypt’s R3 and ISRG Root X1 certs in a fullchain.pem.
I’m finding the same as you. The Axigen built-in LE certificate management doesn’t seem to work and I too need to provide a fullchain.pem externally.
I also used the LuxSci SMTP TLS checker site and is how I discovered my TLS Certificate was not verifiable. After manually creating and installing the fullchain.pem the TLS cert now verifies OK …
MX Priority:
0
SMTP TLS Enabled?
Yes
TLS Certificate for:
[mail.mydomain]
TLS Certificate Verifiable?
Yes
SSL v3 Off?
Yes
TLS v1.0 Supported?
No
TLS v1.2 Supported?
Yes
TLS v1.2 + Good Ciphers?
No
TLS v1.3 Supported?
Yes
Weak Ciphers Off?
Yes
NIST 800-52r2 Cipher Support?
Yes
LuxSci SecureLine Compatability?
No
SMTP MTA STS
No
The “TLS v1.2+Good Ciphers = No” section has led me down the NIST path (referenced in the LuxSci test page) looking for an alternate set of ciphers for Axigen. It’s not a red-hot issue so for now I’m just tinkering with this bit.
I also used the //email/testTo: (checktls.com) site for debugging this issue. Prior to my manual intervention with installing the LE cert the test returned errors. After manually installing my fullchain.pem the test returns: CheckTLS ConfidenceFactor for “[mydomain]”: 114 of 114 (100%, 121 max)
… which is kind of nice.
Although I now have a working system (at least for the next 90-days ), I’ll have another play with Axigen’s automated LE solution, taking @indreias comments & suggestions on board.
There is another “fly in the ointment” I haven’t mentioned (and maybe I should have), my RedHat server runs Axigen (obviously) and also Apache for various websites. Ports 80 and 443 are normally used by Apache. When I needed to let Axigen automatically run the LE cert renewal (during this debugging exercise) I stopped my Apache server and let Axigen have ports 80 and 443. This worked fine during the debugging.
My ultimate solution here would be to let Certbot look after all the LE certs, as it already does for my websites. Then simply point Axigen to /etc/letsencrypt/live/[mail.mydomain] directory and the .pem soft links located there.
These ciphers seem be working well, but I’m no cipher expert. I’ve enabled TLS 1.0, 1.1, 1.2. 1.3 with the rationale that some 1.0 or 1.1 encryption is better than no encryption.
This seems to have done the trick with the STARTTLS issue. Thanks.
If not done already, maybe suggest Axigen add this to the README notes for the next release.
I still can’t figure out what cipher suite(s) to add for “TLS v1.2+Good Ciphers = Yes”, but for now, I don’t think it is a show stopper.
LuxSci SecureLine seems to be a proprietary email encryption thing, so I can live without that I think.
And MTA-STS might be something I need to add in the future, but for now, I’ll leave this on the back-burner.
Going back over @indreias comments and having a re-think, it might be that we can also resolve the Axigen automated LE Cert renewal issue by manually adding the LE CA certificate in the “Certificate authorities file:” section in WebAdmin > Services > SMTP Receiving > Listener(s) > Edit > SSL Settings > Show SSL Configuration > Certificate authorities file: and upload/point to the LE CA .PEM file (in my case /etc/letsencrypt/live/[mydomain]/fullchain.pem - see note below). Of course, we would need to do the same for all the other listeners and services.
Might be worth the experiment.
Note: this is a soft link so not sure exactly how this will work out.
Let’s Encrypt’s intermediate certificate expires in September of 2025. Here’s my temporary “10-year solution” until Axigen delivers a fix.
wget https://letsencrypt.org/certs/isrgrootx1.pem into /var/opt/axigen/letsencrypt/[domain]/ (it expires in 2035);
create a bash script that concatenates cert_auth.pem and isrgrootx1.pem;
#!/bin/bash
# Paths to the input files
cert_auth_pem="/var/opt/axigen/letsencrypt/[domain]/cert_auth.pem"
isrgrootx1_pem="/var/opt/axigen/letsencrypt/[domain]/isrgrootx1.pem"
# Path to the output fullchain.pem
fullchain_pem="/var/opt/axigen/letsencrypt/[domain]/fullchain.pem"
# Check if fullchain.pem expires in the next 2 days
if openssl x509 -checkend 172800 -noout -in "$fullchain_pem"; then
echo "Certificate $fullchain_pem is not expired. Not concatenating."
else
# Certificate will expire, concatenate the files
rm "$fullchain_pem"
cat "$cert_auth_pem" "$isrgrootx1_pem" > "$fullchain_pem"
echo "Concatenated $cert_auth_pem and $isrgrootx1_pem into $fullchain_pem"
fi
# Set proper permissions (adjust as needed)
chmod 644 "$fullchain_pem"
set Certificate authorities file to letsencrypt/[domain]/fullchain.pem ;
set a cron job to run the script every day;
let Axigen manage the domain renewal and hope that the root certificate isn’t revoked, changed, etc.
It’s an ugly hack prone to trouble. I’m certain there could be a more elegant and fault tolerant solution, but this is what I could cobble together in short order.
Greetings, Ioan! My SMTP-OUT problems have been solved. Thank you so much!
There is another “unknown CA” issue I’m experiencing, and it may have to do with OpenSSL 3.0. When trying to connect to my Axigen server from another SMTP server, CA checks would fail occasionally. In the Axigen Let’s Encrypt implementation, cert_auth.pem stores only the intermediate certificate; the cert_auth.pem contains the primary certificate. By constructing and using a fullchain.pem for Let’s Encrypt and pointing “Certificate authorities file” to it, those failures were resolved. Could you please let me know if that issue is known? If those failures continue, how can they be handled better?
), I’ll have another play with Axigen’s automated LE solution, taking 