Back to Blog Home
Mar 26, 2024

How to Set Up Fluent Bit to Collect Axigen Logs and Forward Them to Elasticsearch


Updated: April 8, 2024

 


Effective log management is important for maintaining a healthy and secure email system. In this guide, we'll walk you through the process of setting up Fluent Bit to collect logs from the Axigen Mail Server and forward them to Elasticsearch. This setup enables sysadmins to efficiently analyze log data, enhancing the monitoring and troubleshooting capabilities of their email infrastructure.

axigen-fluentbit-elasticsearch

Step 1: Install Fluent Bit

We'll begin with the installation of Fluent Bit using its Linux Packages. The simplest method is to execute the Fluent Bit install script with the following command:

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

For other environments or installation options, refer to the Fluent Bit Installation Guide.

Step 2: Configure Fluent Bit for Axigen Logs

Create the Log Parser Configurations

Create /etc/fluent-bit/axigen_parser.conf and define parsers for Axigen's everything.txt and security.txt log files.

vim /etc/fluent-bit/axigen_parser.conf

[PARSER]
  Name    axi.everything_parser
  Format  regex
# Default axigen install
  Regex   ^(?<logTime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \+\d{4}) (?<logLevel>[^ ]*) (?<host>[^ ]*) (?<service>[^:]*):(?<jobID>[^ :]*): (?<log>.*)$
  Time_Format %Y-%m-%d %H:%M:%S %z
# Axigen with AXI_LOG_TIMESTAMP_PRECISION enabled
# Regex   ^(?<logTime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d+ \+\d{4}) (?<logLevel>[^ ]*) (?<host>[^ ]*) (?<service>[^:]*):(?<jobID>[^ :]*): (?<log>.*)$
# Time_Format %Y-%m-%d %H:%M:%S.%L %z
  Time_Key  logTime
# Uncoment the below line to also include the logTime field that is used as a source for @timestamp
# Time_Keep On

 

[PARSER]
  Name    axi.security_parser
  Format  regex
# Default axigen install
  Regex   ^(?<logTime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \+\d{4}) (?<logLevel>[^ ]*) [^ ]* SECURITY:(?<service>[^;]*);(?<jobID>[^;]*);(?<remoteIP>[^;]*);(?<remotePort>[^;]*);(?<result>[^;]*);(?<account>[^;]*);(?<userAgent>.*);(?<explanation>[^;]*);(?<details>.*)$
  Time_Format %Y-%m-%d %H:%M:%S %z
# Axigen with AXI_LOG_TIMESTAMP_PRECISION enabled
# Regex   ^(?<logTime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d+ \+\d{4}) (?<logLevel>[^ ]*) [^ ]* SECURITY:(?<service>[^;]*);(?<jobID>[^;]*);(?<remoteIP>[^;]*);(?<remotePort>[^;]*);(?<result>[^;]*);(?<account>[^;]*);(?<userAgent>.*);(?<explanation>[^;]*);(?<details>.*)$
# Time_Format %Y-%m-%d %H:%M:%S.%L %z
  Time_Key  logTime
# Uncoment the below line to include also logTime filed that is used as source for @timestamp
# Time_Keep On


Adjust the Regex and Time_format if nanoseconds precision is enabled in your Axigen logs. Please note that this is applicable to both axi.everything_parser and axi.security_parser. For a deeper dive into Axigen's logging mechanisms, see the Axigen Logging Documentation.

Tailor the Fluent Bit Configuration to Include the New Parsers

Rename the existing Fluent Bit configuration file for backup purposes and create a new one tailored for Axigen log processing:

mv /etc/fluent-bit/fluent-bit.conf /etc/fluent-bit/fluent-bit.orig 
vim /etc/fluent-bit/fluent-bit.conf


Incorporate the following settings, adjusting paths, and Elasticsearch details as necessary: 

[SERVICE]
    Parsers_File    /etc/fluent-bit/parsers.conf
    Parsers_File    /etc/fluent-bit/axigen_parser.conf
    Flush 10

 

[INPUT]
    Name    tail
    Path    /var/opt/axigen/log/everything.txt
    Tag     axi.everything
    Mem_Buf_Limit 50M
    DB      /var/opt/axigen/log/fluent-bit.db
    Refresh_Interval 10

 

[FILTER]
    Name      parser
    Match     axi.everything
    Key_Name  log
    Parser    axi.everything_parser 

 

[FILTER]
    Name      modify
    Match     axi.everything
    Add       tag axi.everything

 

[INPUT]
    Name    tail
    Path    /var/opt/axigen/log/security.txt
    Tag     axi.security
    Mem_Buf_Limit 50M
    DB      /var/opt/axigen/log/fluent-bit.db
    Refresh_Interval 10

 

[FILTER]
    Name      parser
    Match     axi.security
    Key_Name  log
    Parser    axi.security_parser

 

[FILTER]
    Name      modify
    Match     axi.security
    Add       tag axi.security

 

[OUTPUT]
    Name es
    Host <ES_HOSTNAME_OR_IP>
    Index <INDEX_NAME>
    Port 9200
#   If the Elasticsearch endpoint is secure (SSL / TLS), uncomment the line below
#   tls On
#   If a self-signed certificate is used, certificate validation should be disabled
#   tls.verify Off
    HTTP_User <USER_NAME>
    HTTP_Passwd <PASSWORD>
    Time_Key_Nanos On
    Suppress_Type_Name On
    Match *


Replace <ES_HOSTNAME_OR_IP>, <INDEX_NAME>, <USER_NAME>, and <PASSWORD> with your Elasticsearch configurations. The TLS settings are essential if your Elasticsearch instance is secured with HTTPS. 

For more details and options, see the Fluent Bit Configuration File.

Step 3: Start Fluent Bit

Activate Fluent Bit to begin processing and forwarding Axigen logs:

systemctl start fluent-bit

Conclusion

You've now set up Fluent Bit to collect, parse, and forward logs from Axigen to Elasticsearch. This setup enhances your ability to monitor and analyze email system performance and security. For deeper insights and log visualization, you can also integrate Kibana with your Elasticsearch setup, as described in this dedicated post

Stay tuned for further guides on leveraging logging tools and services with Axigen to maintain and optimize your email infrastructure efficiently.