Building on our previous exploration of log management with Fluent Bit and Elasticsearch, this guide focuses on enhancing your Axigen log management by integrating Fluent Bit with Kafka. Effective log management ensures that you can process and analyze your data efficiently, and pairing Fluent Bit with Kafka extends these capabilities by facilitating real-time data streaming.
What is Fluent Bit?
Fluent Bit is an open-source log processor and forwarder, optimized for high performance and flexibility, ideal for environments that require efficient log collection, processing, and forwarding.
What is Kafka?
Apache Kafka is a distributed event streaming platform designed to handle large volumes of data with minimal latency. It offers scalable and reliable stream processing, suitable for managing high-throughput data feeds.
Step 1: Install Fluent Bit
Install Fluent Bit using the provided script for a quick and easy setup. This method is suitable for most Linux distributions:
curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
For alternative installation methods, including specific package managers for different Linux distributions, see the official Fluent Bit Installation Guide.
Step 2: Configure Fluent Bit to Collect Axigen Logs
Fluent Bit will collect the content of everything.txt, the Axigen aggregated log file, and security.txt, the Axigen security log.
Create the Log Parser Configurations
Start by creating /etc/fluent-bit/axigen_parser.conf and defining parsers for the above mentioned log files:
vim /etc/fluent-bit/axigen_parser.conf
Name axi.everything_parser
Format regex
# Default axigen install
Regex ^(?<logTime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \+\d{4}) (?<logLevel>[^ ]*) (?<host>[^ ]*) (?<service>[^:]*):(?<jobID>[^ :]*): (?<log>.*)$
Time_Format %Y-%m-%d %H:%M:%S %z
# Axigen with AXI_LOG_TIMESTAMP_PRECISION enabled
# Regex ^(?<logTime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d+ \+\d{4}) (?<logLevel>[^ ]*) (?<host>[^ ]*) (?<service>[^:]*):(?<jobID>[^ :]*): (?<log>.*)$
# Time_Format %Y-%m-%d %H:%M:%S.%L %z
Time_Key logTime
# Uncomment the below line to include also logTime field that is used as source for @timestamp
# Time_Keep On
[PARSER]
Name axi.security_parser
Format regex
# Default axigen install
Regex ^(?<logTime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \+\d{4}) (?<logLevel>[^ ]*) [^ ]* SECURITY:(?<service>[^;]*);(?<jobID>[^;]*);(?<remoteIP>[^;]*);(?<remotePort>[^;]*);(?<result>[^;]*);(?<account>[^;]*);(?<userAgent>.*);(?<explanation>[^;]*);(?<details>.*)$
Time_Format %Y-%m-%d %H:%M:%S %z
# Axigen with AXI_LOG_TIMESTAMP_PRECISION enabled
# Regex ^(?<logTime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d+ \+\d{4}) (?<logLevel>[^ ]*) [^ ]* SECURITY:(?<service>[^;]*);(?<jobID>[^;]*);(?<remoteIP>[^;]*);(?<remotePort>[^;]*);(?<result>[^;]*);(?<account>[^;]*);(?<userAgent>.*);(?<explanation>[^;]*);(?<details>.*)$
# Time_Format %Y-%m-%d %H:%M:%S.%L %z
Time_Key logTime
# Uncomment the below line to include also logTime field that is used as source for @timestamp
# Time_Keep On
If nanoseconds precision is enabled in the Axigen logs, you will need to comment the Regex and Time_format lines under Default axigen install and uncomment the Regex and Time_format lines under Axigen with AXI_LOG_TIMESTAMP_PRECISION enabled. Please note that this needs to be done for both parsers.
Adapt the Fluent Bit Configuration to Use the Axigen Log Parsers
First, create a backup of the default configuration Fluent Bit configuration file, then create a new configuration file from scratch:
vim /etc/fluent-bit/fluent-bit.conf
You will need to specify, among others, the input (the Axigen logs), the use of the parsers you’ve just configured, and the output destination (Kafka).
Parsers_File /etc/fluent-bit/parsers.conf
Parsers_File /etc/fluent-bit/axigen_parser.conf
Flush 10
[INPUT]
Name tail
Path /var/opt/axigen/log/everything.txt
Tag axi.everything
Mem_Buf_Limit 50M
DB /var/opt/axigen/log/fluent-bit.db
Refresh_Interval 10
[FILTER]
Name parser
Match axi.everything
Key_Name log
Parser axi.everything_parser
[FILTER]
Name modify
Match axi.everything
Add tag axi.everything
[INPUT]
Name tail
Path /var/opt/axigen/log/security.txt
Tag axi.security
Mem_Buf_Limit 50M
DB /var/opt/axigen/log/fluent-bit.db
Refresh_Interval 10
[FILTER]
Name parser
Match axi.security
Key_Name log
Parser axi.security_parser
[FILTER]
Name modify
Match axi.security
Add tag axi.security
[OUTPUT]
Name kafka
Brokers <KAFKA_HOST>:<KAFKA_PORT>
Topics <TOPIC>
Match *
You will need to replace <KAFKA_HOST> and <KAFKA_PORT> with your specific Kafka details, as well as set a relevant <TOPIC> that the matching logs will be forwarded under. You can as well add multiple Kafka brokers on the same line, separated by comma.
Step 3: Start Fluent Bit
With the configuration file set, start the Fluent Bit service to begin processing and forwarding logs:
systemctl start fluent-bit
Conclusion
Integrating Fluent Bit with Kafka for Axigen log management enhances your capability to monitor, analyze, and react to events in real time. By following these steps, you establish a robust system that not only collects and parses detailed log data but also streams it efficiently to a Kafka cluster for further processing. This integration plays a relevant role in proactive system monitoring and incident response.