Updated: April 22, 2024
Effective log management is important for maintaining a healthy and secure email system. In this guide, we'll walk you through the process of setting up Fluent Bit to collect logs from the Axigen Mail Server and forward them to Elasticsearch. This setup enables sysadmins to efficiently analyze log data, enhancing the monitoring and troubleshooting capabilities of their email infrastructure.
Step 1: Install Fluent Bit
We'll begin with the installation of Fluent Bit using its Linux Packages. The simplest method is to execute the Fluent Bit install script with the following command:
curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
For other environments or installation options, refer to the Fluent Bit Installation Guide.
Step 2: Configure Fluent Bit for Axigen Logs
Create the Log Parser Configurations
Create /etc/fluent-bit/axigen_parser.conf and define parsers for Axigen's everything.txt and security.txt log files.
vim /etc/fluent-bit/axigen_parser.conf
Name axi.everything_parser
Format regex
# Default axigen install
Regex ^(?<logTime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \+\d{4}) (?<logLevel>[^ ]*) (?<host>[^ ]*) (?<service>[^:]*):(?<jobID>[^ :]*): (?<log>.*)$
Time_Format %Y-%m-%d %H:%M:%S %z
# Axigen with AXI_LOG_TIMESTAMP_PRECISION enabled
# Regex ^(?<logTime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d+ \+\d{4}) (?<logLevel>[^ ]*) (?<host>[^ ]*) (?<service>[^:]*):(?<jobID>[^ :]*): (?<log>.*)$
# Time_Format %Y-%m-%d %H:%M:%S.%L %z
Time_Key logTime
# Uncoment the below line to also include the logTime field that is used as a source for @timestamp
# Time_Keep On
[PARSER]
Name axi.security_parser
Format regex
# Default axigen install
Regex ^(?<logTime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \+\d{4}) (?<logLevel>[^ ]*) [^ ]* SECURITY:(?<service>[^;]*);(?<jobID>[^;]*);(?<remoteIP>[^;]*);(?<remotePort>[^;]*);(?<result>[^;]*);(?<account>[^;]*);(?<userAgent>.*);(?<explanation>[^;]*);(?<details>.*)$
Time_Format %Y-%m-%d %H:%M:%S %z
# Axigen with AXI_LOG_TIMESTAMP_PRECISION enabled
# Regex ^(?<logTime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d+ \+\d{4}) (?<logLevel>[^ ]*) [^ ]* SECURITY:(?<service>[^;]*);(?<jobID>[^;]*);(?<remoteIP>[^;]*);(?<remotePort>[^;]*);(?<result>[^;]*);(?<account>[^;]*);(?<userAgent>.*);(?<explanation>[^;]*);(?<details>.*)$
# Time_Format %Y-%m-%d %H:%M:%S.%L %z
Time_Key logTime
# Uncoment the below line to include also logTime filed that is used as source for @timestamp
# Time_Keep On
Adjust the Regex and Time_format if nanoseconds precision is enabled in your Axigen logs. Please note that this is applicable to both axi.everything_parser and axi.security_parser. For a deeper dive into Axigen's logging mechanisms, see the Axigen Logging Documentation.
Tailor the Fluent Bit Configuration to Include the New Parsers
Rename the existing Fluent Bit configuration file for backup purposes and create a new one tailored for Axigen log processing:
vim /etc/fluent-bit/fluent-bit.conf
Incorporate the following settings, adjusting paths, and Elasticsearch details as necessary:
Parsers_File /etc/fluent-bit/parsers.conf
Parsers_File /etc/fluent-bit/axigen_parser.conf
Flush 10
[INPUT]
Name tail
Path /var/opt/axigen/log/everything.txt
Tag axi.everything
Mem_Buf_Limit 50M
DB /var/opt/axigen/log/fluent-bit.db
Refresh_Interval 10
[FILTER]
Name parser
Match axi.everything
Key_Name log
Parser axi.everything_parser
[FILTER]
Name modify
Match axi.everything
Add tag axi.everything
[INPUT]
Name tail
Path /var/opt/axigen/log/security.txt
Tag axi.security
Mem_Buf_Limit 50M
DB /var/opt/axigen/log/fluent-bit.db
Refresh_Interval 10
[FILTER]
Name parser
Match axi.security
Key_Name log
Parser axi.security_parser
[FILTER]
Name modify
Match axi.security
Add tag axi.security
[OUTPUT]
Name es
Host <ES_HOSTNAME_OR_IP>
Index <INDEX_NAME>
Port 9200
# If the Elasticsearch endpoint is secure (SSL / TLS), uncomment the line below
# tls On
# If a self-signed certificate is used, certificate validation should be disabled
# tls.verify Off
HTTP_User <USER_NAME>
HTTP_Passwd <PASSWORD>
Time_Key_Nanos On
Suppress_Type_Name On
Match *
Replace <ES_HOSTNAME_OR_IP>, <INDEX_NAME>, <USER_NAME>, and <PASSWORD> with your Elasticsearch configurations. The TLS settings are essential if your Elasticsearch instance is secured with HTTPS.
For more details and options, see the Fluent Bit Configuration File.
Step 3: Start Fluent Bit
Activate Fluent Bit to begin processing and forwarding Axigen logs:
systemctl start fluent-bit
Conclusion
You've now set up Fluent Bit to collect, parse, and forward logs from Axigen to Elasticsearch. This setup enhances your ability to monitor and analyze email system performance and security. For deeper insights and log visualization, you can also integrate Kibana with your Elasticsearch setup, as described in this dedicated post.
To further advance your log management capabilities you can also integrate Fluent Bit with Kafka for real-time data streaming, allowing for more dynamic and resilient data handling.
With these tools, you can effectively monitor and analyze your systems, ensuring proactive management and improved responsiveness.