How to Install and Use Fluent Bit for Efficient Log Processing 🚀

Saturday, Dec 28, 2024 | 7 minute read

GitHub Trend

The lightweight telemetry agent excels in high-performance log processing, offering low resource usage, extensive plugin support, and robust data handling features. Its flexibility and cross-platform compatibility make it an essential tool for efficient data management! 🚀✨

“With the astounding advancements in information technology, the amount of data being generated is skyrocketing like a rocket! For businesses and developers, efficiently processing and analyzing this massive amount of data has become an extremely important task! 🔍”

In modern business operations, log processing and data monitoring are key to ensuring system stability and security. ✨ Fluent Bit, a high-performance open-source telemetry agent, is like a superhero that provides developers with a powerful and flexible log processing solution while standing out in the rapidly evolving tech landscape! So, let’s unveil the unique charm of Fluent Bit and explore how to use it! 🚀

🔥 Fluent Bit: The Lightweight Messenger of High-Performance Log Processing

Fluent Bit is a lightweight telemetry agent designed for efficiently processing logs, metrics, and traces, offering impressive performance! 🙌 It easily works across multiple operating systems (such as Linux, macOS, Windows, and BSD), demonstrating outstanding cross-platform compatibility. Additionally, as part of the Fluentd ecosystem, it belongs to a subproject under the Cloud Native Computing Foundation (CNCF)! 🛠️ Whether it’s about data processing or forwarding, Fluent Bit can seamlessly connect to various backend storage platforms like Fluentd, Elasticsearch, and Splunk, which is truly remarkable! 🌍

⚡ Exceptional: The Unique Appeal of Fluent Bit

Fluent Bit boasts extraordinary performance, rapidly processing data with low CPU and memory usage, perfectly meeting a range of modern application needs! 😍 Its data parsing capabilities are impressive, converting unstructured logs into various practical formats such as JSON, Regex, LTSV, and Logfmt, greatly enhancing the usability of the data! At the same time, Fluent Bit is compatible with Prometheus and OpenTelemetry, making monitoring and metric aggregation incredibly simple. 📊

Moreover, Fluent Bit has a robust data processing mechanism, with features like backpressure handling and data buffering, ensuring the integrity and durability of data during transmission. Built-in TLS/SSL support also provides strong security guarantees for networking. Last but not least, its pluggable architecture is indeed a highlight, allowing users to choose from over 70 built-in plugins to flexibly configure inputs, filters, and outputs to meet various needs! 🛠️

🏆 Developer’s Choice: Why Choose Fluent Bit?

Fluent Bit has earned widespread recognition in the industry for its unparalleled reliability, providing users with a worry-free experience for complex data processing tasks, making it an unmatched tool for developers! In modern cloud-native architectures, the application frequency of Fluent Bit is on the rise each year, aiding countless users and organizations in effectively managing logs and creating a series of success stories! 📈

Additionally, Fluent Bit supports numerous plugins, allowing developers to easily extend system functionality to fully respond to the ever-changing real-world needs! Since its launch, Fluent Bit has accumulated over 15 billion downloads, with more than 10 million daily deployments, undoubtedly proving its profound impact and absolute necessity in the industry! 🌟

The innovation and transformation brought by Fluent Bit to the open-source log processing field deserve the attention and practice of every IT practitioner! ✨

Install Fluent Bit 🚀

Want to start using Fluent Bit? Let’s see how to effortlessly build and install it! Please follow these steps:

git clone https://github.com/fluent/fluent-bit  # Clone Fluent Bit source code from GitHub
cd fluent-bit                                   # Navigate to the cloned directory
mkdir build                                     # Create a new build directory
cd build                                        # Enter the newly created build directory
cmake ..                                        # Run CMake to generate build files
make                                            # Use the make command to compile
sudo make install                               # Execute installation with superuser privileges

• git clone: This command downloads the Fluent Bit source code from GitHub, making it convenient for us to compile and install later. 📥
• cd fluent-bit: Enter the source code directory to ensure subsequent commands are executed in this directory. 📁
• mkdir build: Create a dedicated build directory to keep the source code directory tidy! ✨
• cmake ..: Calling CMake generates the Makefile, configuring build tasks. ⚙️
• make: Compile the Fluent Bit source code to produce the executable! 💻
• sudo make install: Installs the compiled program into the system path for global availability, truly convenient! 🎉

Before building and installing Fluent Bit, please ensure your system has the following dependencies installed:

• CMake >= 3.0
• Flex
• Bison
• YAML library/header files
• OpenSSL library/header files

Ensure these dependencies are complete for a smooth compilation! 👍

Input Plugin Example 🛠️

Fluent Bit provides various input plugins, and the following example can help you easily collect CPU usage data configurations:

[INPUT]                                         # Input plugin section
    Name cpu                                   # Specify the input plugin as CPU
    Tag cpu.local                               # Set the data tag as cpu.local

• [INPUT]: This section defines the way data is input.
• Name cpu: Selecting the “cpu” plugin to collect CPU usage data. 🖥️
• Tag cpu.local: This line sets a tag for CPU data, enabling easy matching and processing in the output configuration. 🔖

With such a configuration, Fluent Bit will start monitoring CPU usage in real-time and elegantly tag the results as cpu.local! 🌟

Output Plugin Example 📤

Want to output the collected data to the console? Check out this simple configuration example:

[OUTPUT]                                        # Output plugin section
    Name stdout                                 # Choose the standard output plugin
    Match *                                     # Output all input data

• [OUTPUT]: This section defines the way data is output.
• Name stdout: Select to use the standard output plugin (i.e., print data to the console). 🖥️
• **Match ***: This configuration means that all input data will be sent to the output plugin for processing.

With this setup, we can view all collected data in the console in real-time, aiding in debugging and verification! 🧐

Configuration File Example 📄

Let’s take a look at a simple Fluent Bit configuration file example that combines input, output plugin settings, and service parameters:

[SERVICE]                                       # Service configuration section
    Flush 5                                     # Automatically flush output every 5 seconds
    Log_Level info                              # Set log level to info

[INPUT]                                         # Input configuration section
    Name dummy                                  # Use the dummy plugin to generate test data
    Tag dummy.local                             # Set tag for dummy data

[OUTPUT]                                        # Output configuration section
    Name stdout                                 # Set output to standard output
    Match *                                     # Match all inputs

• [SERVICE]: Service configuration section used to set basic service parameters.
• Flush 5: Set to flush output data every 5 seconds! ⏱️
• Log_Level info: Set the log level to info for richer information in the console! 📋
• [INPUT]: This section uses the dummy plugin to generate test data, making it very convenient.
• [OUTPUT]: Outputs all data tagged as dummy.local to the console, allowing us to easily validate the effectiveness of the configuration! 💡

This configuration file enables us to quickly generate data and view the output in real-time during the testing phase, truly a boon for novices! 💖

HTTP Output Example 🌐

The following configuration example shows how to send collected data via HTTP to a specified endpoint:

[OUTPUT]                                        # Output plugin section
    Name http                                   # Choose HTTP output plugin
    Match *                                     # Match all input
    Host your-api-endpoint.com                  # Set target host
    Port 80                                      # Specify target port as 80
    URI /api/logs                               # Set request URI

• Name http: Using the HTTP output plugin to send data to an HTTP endpoint.
• Host your-api-endpoint.com: This needs to be replaced with the actual API server address. 💻
• Port 80: Specify the port for sending requests (the commonly used HTTP port). 🌐
• URI /api/logs: Set the specific URI for sending data, ensuring your service can handle the incoming log data.

By combining these configurations, Fluent Bit will send all data to the specified endpoint via an HTTP POST request, making subsequent processing straightforward! 📤

Monitoring Configuration 📊

Fluent Bit has built-in performance monitoring capabilities; here’s how to configure Prometheus output to scrape monitoring metrics:

[OUTPUT]                                        # Output plugin section
    Name prometheus_remote_write                 # Choose Prometheus remote write plugin
    Match *                                     # Match all input
    Host prometheus-server                       # Specify Prometheus server
    Port 9090                                    # Specify target port as 9090
    Path /api/v1/write                          # Set writing path

• Name prometheus_remote_write: Choose Prometheus plugin to write data to a remote monitoring system.
• Host prometheus-server: This needs to be replaced with the actual Prometheus server address. 🏢
• Port 9090: Specify the Prometheus server port, typically 9090. 🔌
• Path /api/v1/write: Configure the API path for Prometheus to receive data, ensuring it can process the incoming data.

With such a configuration, Fluent Bit can directly send data to a specified Prometheus server, achieving real-time monitoring and data analysis effortlessly! 🔍

Fluent Bit is an incredible tool, ideal for real-time data stream processing and log collection! Through the configuration of various input, output, and filtering plugins, we can meet diverse needs and integrations, offering unparalleled convenience for daily log management and monitoring! 🎉