Filebeat在CentOS上的自定义输出插件

Custom Output Plugin Development for Filebeat on CentOS

Filebeat, a lightweight log shipper from Elastic, supports custom output plugins to extend its data routing capabilities beyond built-in modules (e.g., Elasticsearch, Logstash). This guide walks through the process of creating and deploying a custom output plugin on CentOS, including prerequisites, code implementation, compilation, configuration, and validation.

1. Prerequisites

Before starting, ensure the following are installed on your CentOS system:

• Filebeat: Install via sudo yum install filebeat (use the latest version compatible with your plugin).
• Go Environment: Install Go (≥1.16) from the official website or EPEL repository. Configure GOPATH and add $GOPATH/bin to your PATH.
• Source Code: Download Filebeat’s source code from Elastic’s GitHub repository to access core libraries (e.g., libbeat).

2. Create the Plugin Directory

Organize your plugin code in a dedicated directory. For example:

mkdir -p ~/filebeat-custom-output/plugin
cd ~/filebeat-custom-output/plugin

This directory will hold your Go source files and compiled plugin.

3. Write the Plugin Code

Create a Go file (e.g., custom_output.go) and implement the required interfaces. Below is a simplified example of a custom output plugin that writes events to a local file:

package main

import (
	"fmt"
	"os"

	"github.com/elastic/beats/v7/libbeat/beat"
	"github.com/elastic/beats/v7/libbeat/common"
	"github.com/elastic/beats/v7/libbeat/logp"
	"github.com/elastic/beats/v7/libbeat/outputs"
)

// CustomOutput defines the plugin's configuration and state.
type CustomOutput struct {
	filePath string // Path to the output file
	file     *os.File
}

// New initializes the plugin with configuration.
func New(b *common.Config) (outputs.Output, error) {
	config := struct {
		FilePath string `config:"file_path" validate:"required"`
	}{}
	if err := b.Unpack(&config); err != nil {
		return nil, fmt.Errorf("error unpacking config: %v", err)
	}

	// Open the output file (create if it doesn't exist)
	f, err := os.OpenFile(config.FilePath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
	if err != nil {
		return nil, fmt.Errorf("error opening file: %v", err)
	}

	logp.Info("CustomOutput plugin initialized. Writing to: %s", config.FilePath)
	return &CustomOutput{
		filePath: config.FilePath,
		file:     f,
	}, nil
}

// Publish sends events to the custom output (e.g., a file).
func (c *CustomOutput) Publish(events []common.MapStr) error {
	for _, event := range events {
		line, err := event.StringToPrint()
		if err != nil {
			logp.Err("Error converting event to string: %v", err)
			continue
		}
		if _, err := c.file.WriteString(line + "\n"); err != nil {
			logp.Err("Error writing to file: %v", err)
			return err
		}
	}
	return nil
}

// Close cleans up resources (e.g., close the file).
func (c *CustomOutput) Close() error {
	if c.file != nil {
		return c.file.Close()
	}
	return nil
}

// Register the plugin with Filebeat's output framework.
func init() {
	outputs.RegisterType("custom_file", New)
}

Key Notes:

• Replace the Publish logic with your custom logic (e.g., sending events to a WebSocket, Kafka, or HTTP endpoint).
• Use config structs to define configurable parameters (e.g., file_path).
• Implement the outputs.Output interface (methods: New, Publish, Close).

4. Compile the Plugin

Compile the Go code into a shared object (.so) file using the -buildmode=plugin flag:

go build -buildmode=plugin -o custom_output.so custom_output.go

This generates custom_output.so in the current directory. The .so file is required for Filebeat to load the plugin dynamically.

5. Configure Filebeat to Use the Plugin

Edit Filebeat’s configuration file (/etc/filebeat/filebeat.yml) to register and configure the custom output:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log  # Adjust paths to your log files

output.custom_file:
  file_path: "/tmp/filebeat_custom_output.log"  # Path from the plugin config
  plugin: "~/filebeat-custom-output/plugin/custom_output.so"  # Path to the compiled .so file

Explanation:

• output.custom_file: References the custom plugin type (custom_file) defined in the init() function.
• file_path: Configures the output file path (must match the FilePath field in the plugin’s New method).
• plugin: Specifies the absolute path to the compiled .so file.

6. Restart Filebeat and Validate

Restart Filebeat to apply the configuration changes:

sudo systemctl restart filebeat

Check Filebeat’s logs for errors or confirmation that the plugin loaded successfully:

sudo tail -f /var/log/filebeat/filebeat

Verify the custom output by checking the target file (e.g., /tmp/filebeat_custom_output.log). You should see log entries from Filebeat written to this file.

Key Considerations

• Compatibility: Ensure the plugin is compiled with the same Go version and Filebeat version you’re using.
• Permissions: The Filebeat process must have read access to the .so file and write access to the output destination (e.g., /tmp/).
• Testing: Test the plugin in a non-production environment before deploying to production.
• Error Handling: Add robust error handling in the plugin code to avoid data loss.

By following these steps, you can create a custom output plugin for Filebeat on CentOS to route logs to any destination or process events in a way that suits your needs.