Getting hacked and turned into a spambot

Hello - I have the free version of Axigen for Linux 10.0.0 and I only use it to support our neighborhood association’s mail list. There are only 3 accounts set up on it, one for the maillist itself, mine and admin’s The passwords for all 3 accounts are fairly complex so I doubt that the server is being broken into via a compromised password. (and besides I have changed them several times now to no joy) Somehow the server is getting turned into a spambot and when I notice my system has become slow to respond I will check the queue on the server and find hundreds if not thousands of emails being queued to send. This is resulting in my domain being blacklisted. I have tried everything I can think of, filters, blocks all with no joy. It usually appears as if the spammer simply changes the from address and comes back shortly after I delete all the spam from the queue and add the new filters and blocks. Log files don’t give me a lot of clue though I may not know/understand how to interpret them fully. I have checked the server at mxtoolbox.com and it does not report anything wrong with the server either. What do I need to do to stop this and track down how the breakin is occurring? For now I have turned off the SMTP sending process and am only activating it when there is a need to post something to the mail list but that is not a user friendly solution if one of my neighbors wants to post something themselves.

P.S. I will only be around for the next week, after that I am on vacation in Europe for a month so if this is going to be complicated or takes longer than a week to resolve, I will have to stop the server while I am gone and get back to this issue when I return.

Hello,

In order to find how the messages were injected in your Axigen server please share with us the log files that cover the time period when this happen.

Also if is possible send us the SMTP Policy file available by default at ${AXIGEN_WORK_DIR}/filters/smtpFilters.script

Regards,
Florin

I can do that! And thanks for replying Florin… I think this log file captures one of the periods during which my Axigen email was running amuck, HTHs you in tracking down what has gone wrong and why…

Marc..

P.S. I think it worked, not sure… Uploaded
everything.txt.2019-06-19-13_20_37
smtpFilters.script

Hello,

It seems nothing was uploaded - could you recheck? Or post here a link from where we could download the files.

HTH,
Ioan

Hi Loan, Yeah I thought the upload was acting funny, so not surprised it didn’t work. Dunno why, perhaps something to do with running Firefox on a Linux platform?

Anywise, I put the files in a zipped tar file and you can get it from my webserver at

http://www.marcchamberlin.com/forAxigen.tar.gz

HTHs Marc…

Hello,

In the provided log we did not find any SMTP Receiving session. If you configure Axigen log service to redirect record for SMTP Receiving record to a specific file provide us also that file.

If not, use grep command to identify the log files that contain records of SMTP Receiving service, for example:

$ grep SMTP-IN /var/opt/axigen/log/everything.txt.2019-06-*

Regards,
Florin

Thanks Florin for your help… I think it is interesting that you did not find any log entries for the SMTP Receiving session! No I have not done anything to redirect to log records to a specific file. Don’t think I even know how, being as I have never done it! I have created a file using the grep command that you suggested, you can find it at

https://www.marcchamberlin.com/axigen_grep_smtp-in.txt.gz

HTHs Marc…

Hello,

In the provided file I notice 7 SMTP session and 6 from have only one recipient and all 6 looks legit message and one with 6 recipient that also looks legit message.

However I notice that all 6 messages are from 192.168.10.100 IP address (possible NAT) that according to the SMTP Policy file is allowed to send messages without authentication. In fact the rule will allow all the IP addresses from 192.168.10.0/24 class to send messages without authentication.

Note that in this case if a computer from your local network is infected with a malware will be able to send messages without authentication.

method wizard_generated_relay {
        if (
                anyOf (
                        ipRange (remoteSmtpIp, "192.168.10.100/255.255.255.0"),
                        ipRange (remoteSmtpIp, "127.0.0.1/255.0.0.0")
                )
        ) {
                set (remoteDelivery, "all");
        }
}

My recommendation is to remove this relay rule from Webadmin -> Security & Filtering -> Routing Basic Settings ->
Allow / Disallow relaying section and configure all your email clients to authenticate for SMTP.

Regards,
Florin

1 Like

(Dunno why this is getting flagged as spam, sorry if this is a repeated posting, I appear to be having some sort of troubles posting… Marc…)

Hi Florin, Thanks for your reply. :slightly_smiling_face: And yes I think you make a good point. Let me explain my circumstances a bit more and what I have done in response to your suggestion. Correct me if you think I did this wrong…

I run a SOHO network at my home and I have a Linux system acting as my main firewall, router and interface to the internet. The external facing NIC on this system as a number of static IP addresses assigned to it and the internal facing NIC has the static local IP address of 192.168.10.100 that you noticed. One of these external IP addresses is assigned to a domain name for my neighborhood association and is routed to another computer (where Axigen is running). This computer handles the web and email services for my neighborhood association. So the external facing system has a local IP address of 192.168.10.100, the internal system, on which the Axigen mail server is running has a local IP address of 192.168.10.50. I want anyone who is on the mail list to be able to send an email to all the members of the mail list, to include my wife and I who will use various computers, also on my local net.

So I turned off the Allow relaying from any IP as you suggested. Once that was turned off I could not check the box for Require authentication which seems both logical to do and what you were asking? All my email clients are configured to do SMTP authentication so I hope I am not breaking their ability to send emails to the mail list. I then took a guess and added an Exception rule for just 192.168.10.100 and set the “Allow Delivery” and “Require Auth” to yes.

I also have turned the SMTP sending service back on and will let you know if I encounter any more problems. (and please let me know if I configured this wrong)

Also, I realized, after re-reading your last email that you were not just asking for the results of the grep operation, but were asking me to identify the files with such messages in them. So I have updated the file on my web server at -

Hmmm it appears that some sort of spam detection software is not letting me post with a link to my server. (It is the same link as I previously posted, so weird!) Anywise I will show the link without it being an actual link to see if that will get past this…

www dot marcchamberlin dot com slash forAxigen.tar.gz

to include a number of the saved log files that contain records for the SMTP Receiving service. Dunno if this will be useful but it is there if you want to look at more of the logfiles.

HTHs Marc…

Hi again, well today it appears Axigen was again broken into, despite my lasted changes to tighten up security. I will try to upload today’s log file everything.txt, look for the address essexlss2@gmail.com which seems to be the from address this spammer is using to send emails from. If the upload fails, then download the log from www dot marcchamberlin dot com slash everything.txt

I removed these emails from the queue.everything.txt (141.1 KB)

Marc…

Hello Marc,

The log you have shared contain the needed information only for one message:

SMTP-IN:00000178: New mail <23CD41C8-6229-4170-B912-0B6472CE6AC4@gmail.com> received from mail-io1-f43.google.com (192.168.10.100) with envelope from <shelley********@gmail.com>, recipients=1 (********-subscribe@ammeterheights.org), size=2653, enqueued with id 023AC3

Could you please share more logs from that period of time?

My suggestion is to use a command like the one mentioned below (issued on the Axigen server):

grep "New mail" /var/opt/axigen/log/everything.txt.2019-06-2[67]*. | grep essexlss2@gmail.com

Note: what we are trying to do is to find the logs that contains SMTP-IN sessions from 26 and 27 June that have any relations with the mentioned email address.

Looking forward for your logs.

Best regards,
Ioan

Hi Loan - Tell you what, I put the entire log directory in a gzipped tarball and will let you poke through it at you please! :smile: It is not tooooo large and there isn’t anything in it that I need to worry about AFAIK. So it is sitting on my server and the URL (in a format to get past the spam detectors on Axigen’s forum’s server) is www dot marcchamberlin dot com slash AxigenLogs.tar.gz

I have a fast 100MBs internet connection so shouldn’t take you too long to download it… HTHs Marc

BTW Loan - I just discovered two more emails in the queue that this spambot is trying to use Axigen to send. I will upload one of them using the Axigen’s forum uploader so you can inspect it, and I suspect it will be in the logfile tarball I just sent you, also.

This does have the look and feel of an infected computer but I have no idea which one on my network it might be nor how it is breaking in to the serverqueue-message.tar (10 KB) …

 Marc..

Hello Marc,

I’m sorry to let you know but it seems that your changes related to “remote delivery” only for authenticated sessions seems to not be the right ones. For example, I have found the following session (see below)

Pay attention to Set remote delivery to all immediately before 250-bigbang Axigen ESMTP hello). This is overwriting previous “remote delivery” instructions and will let non authenticated users to send messages to external recipients.

Please reshare one more time your smtpFilters.script file so we could evaluate one more time where is your problem now and correct it for you.

BR,
Ioan

$ grep SMTP-IN:00000203 everything.txt.2019-06-27-09_56_35
2019-06-28 08:37:08 -0700 08 bigbang SMTP-IN:00000203: [192.168.10.51:25] connection accepted from [192.168.10.100:65480]
2019-06-28 08:37:08 -0700 08 bigbang SMTP-IN:00000203: Set smtp greeting to [mail.ammeterheights.org]
2019-06-28 08:37:08 -0700 16 bigbang SMTP-IN:00000203: >> 220 mail.ammeterheights.org
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: << ehlo [103.99.1.144]
2019-06-28 08:37:09 -0700 08 bigbang SMTP-IN:00000203: Set remote delivery to none
2019-06-28 08:37:09 -0700 08 bigbang SMTP-IN:00000203: Set remote delivery to auth
2019-06-28 08:37:09 -0700 08 bigbang SMTP-IN:00000203: Greylist disabled
2019-06-28 08:37:09 -0700 08 bigbang SMTP-IN:00000203: Set max data size to 10240 KB
2019-06-28 08:37:09 -0700 08 bigbang SMTP-IN:00000203: Set max received headers to 30
2019-06-28 08:37:09 -0700 08 bigbang SMTP-IN:00000203: Maximum recipient count set to 1000
2019-06-28 08:37:09 -0700 08 bigbang SMTP-IN:00000203: Wait for processing response at least 10 seconds
2019-06-28 08:37:09 -0700 08 bigbang SMTP-IN:00000203: STARTTLS extension allowed
2019-06-28 08:37:09 -0700 08 bigbang SMTP-IN:00000203: 8BIT MIME accepted
2019-06-28 08:37:09 -0700 08 bigbang SMTP-IN:00000203: BINARY DATA extension allowed
2019-06-28 08:37:09 -0700 08 bigbang SMTP-IN:00000203: PIPELINING extension allowed
2019-06-28 08:37:09 -0700 08 bigbang SMTP-IN:00000203: Set local delivery to all
2019-06-28 08:37:09 -0700 02 bigbang SMTP-IN:00000203: SPF ERROR: Invalid domain name: [103.99.1.144]
2019-06-28 08:37:09 -0700 08 bigbang SMTP-IN:00000203: SPF result for EHLO domain <[103.99.1.144]> connected from <192.168.10.100>: None (spfHeader = ''; spfExplanation = 'null')
2019-06-28 08:37:09 -0700 08 bigbang SMTP-IN:00000203: Set remote delivery to all
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: >> 250-bigbang Axigen ESMTP hello
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: >> 250-PIPELINING
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: >> 250-AUTH PLAIN LOGIN CRAM-MD5 DIGEST-MD5 GSSAPI
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: >> 250-AUTH=PLAIN LOGIN CRAM-MD5 DIGEST-MD5 GSSAPI
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: >> 250-8BITMIME
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: >> 250-BINARYMIME
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: >> 250-CHUNKING
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: >> 250-SIZE 10485760
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: >> 250-STARTTLS
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: >> 250-HELP
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: >> 250 OK
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: << mail FROM:<reservatio@goldeninnapartments.com> size=2006
2019-06-28 08:37:09 -0700 08 bigbang SMTP-IN:00000203: SPF result for MAIL FROM <reservatio@goldeninnapartments.com@goldeninnapartments.com> issued from EHLO domain <[103.99.1.144]> connected from <192.168.10.100>: SoftFail (spfHeader = 'softfail (goldeninnapartments.com: domain of transitioning reservatio@goldeninnapartments.com does not designate 192.168.10.100 as permitted sender) client-ip=192.168.10.100; envelope-from=reservatio@goldeninnapartments.com; mechanism=default; identity=mailfrom; receiver=bigbang;'; spfExplanation = 'null')
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: >> 250 Sender accepted
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: << rcpt TO:<essexlss2@gmail.com>
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: >> 250 Recipient accepted
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: << rcpt TO:<essexlss2@hotmail.com>
2019-06-28 08:37:09 -0700 16 bigbang SMTP-IN:00000203: >> 250 Recipient accepted
2019-06-28 08:37:10 -0700 16 bigbang SMTP-IN:00000203: << rcpt TO:<mrkeendyuzok@outlook.com>
2019-06-28 08:37:10 -0700 16 bigbang SMTP-IN:00000203: >> 250 Recipient accepted
2019-06-28 08:37:10 -0700 16 bigbang SMTP-IN:00000203: << rcpt TO:<godswillochei@yahoo.com>
2019-06-28 08:37:10 -0700 16 bigbang SMTP-IN:00000203: >> 250 Recipient accepted
2019-06-28 08:37:10 -0700 16 bigbang SMTP-IN:00000203: << rcpt TO:<allinall112@aol.com>
2019-06-28 08:37:10 -0700 16 bigbang SMTP-IN:00000203: >> 250 Recipient accepted
2019-06-28 08:37:10 -0700 16 bigbang SMTP-IN:00000203: << data
2019-06-28 08:37:10 -0700 16 bigbang SMTP-IN:00000203: >> 354 Ready to receive data; remember <CRLF>.<CRLF>
2019-06-28 08:37:11 -0700 16 bigbang SMTP-IN:00000203: << 2069 bytes and final dot read
2019-06-28 08:37:12 -0700 08 bigbang SMTP-IN:00000203: New mail <1561736230805026387@bigbang> received from [103.99.1.144] (192.168.10.100) with envelope from <reservatio@goldeninnapartments.com>, recipients=5 (essexlss2@gmail.com, essexlss2@hotmail.com, mrkeendyuzok@outlook.com, godswillochei@yahoo.com, allinall112@aol.com), size=2066, enqueued with id 290D7F
2019-06-28 08:37:12 -0700 16 bigbang SMTP-IN:00000203: >> 250 Mail queued for delivery
2019-06-28 08:37:12 -0700 16 bigbang SMTP-IN:00000203: << quit
2019-06-28 08:37:12 -0700 16 bigbang SMTP-IN:00000203: >> 221-bigbang Axigen ESMTP is closing connection
2019-06-28 08:37:12 -0700 16 bigbang SMTP-IN:00000203: >> 221 Good bye
2019-06-28 08:37:12 -0700 08 bigbang SMTP-IN:00000203: closing session from [192.168.10.100:65480]

Here you go Loan, I am uploading the script here.smtpFilters.script.txt (4.7 KB)

Thanks, Marc…

Hello Marc,

Most probably the culprit is the following configuration (from smtpFilters.script)

method wizard_generated_relay {
	if (
		anyOf (
			ipRange (remoteSmtpIp, "192.168.10.100/255.255.255.0"),
			ipRange (remoteSmtpIp, "127.0.0.1/255.0.0.0")
		)
	) {
		set (remoteDelivery, "all");
	}
}

My suggestion is to disable this rule (named wizard_generated_relay from WebAdmin > Security & Filtering > Acceptance & Routing > Advance Settings - it should be under On EHLO event) or to edit it and remove 192.168.100.10 from the “conditions” of allowing remoteDelivery without authentication (leaving only 127.0.0.1 as the only condition).

HTH,
Ioan

Hi Loan - I followed your suggestion and disabled the wizard_generated_relay rule. So far I am not seeing the breakin so maybe that will at least stop this. I hope my neighbors can still use the mail list while I am gone for the next month, I will try to monitor it from afar but I won’t be able to do much until I get back in Aug unless I can find an internet cafe and VNC in to the server if I have to… Will do my best to keep you informed and thanks so much for your help! Marc…