How to Analyze Ubuntu System Logs with Filebeat
Before configuring Filebeat, you need to install it on your Ubuntu system. Use the following commands to add Elastic’s GPG key, register the Elastic repository, and install Filebeat:
sudo apt update && sudo apt upgrade -y
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
sudo apt update && sudo apt install filebeat -y
This installs the latest version of Filebeat (adjust the version in the URL if needed).
Filebeat includes a System Module that simplifies the collection and parsing of Ubuntu system logs (e.g., /var/log/syslog, /var/log/auth.log). This module automatically configures inputs, parsers, and dashboards for system logs.
To enable it:
sudo filebeat modules enable system
Verify the module is enabled:
sudo filebeat modules list
The output should show system under “Enabled modules.”
If you prefer not to use the System Module, you can manually configure Filebeat to collect system logs by editing the filebeat.inputs section in /etc/filebeat/filebeat.yml. For example:
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/syslog
- /var/log/auth.log
ignore_older: 72h # Ignore logs older than 72 hours
This configuration tells Filebeat to collect logs from the specified paths and ignore files older than 3 days.
Filebeat needs an output destination to send logs. The most common setups are Elasticsearch (for direct indexing) or Logstash (for advanced parsing).
If you have a local Elasticsearch instance running on localhost:9200, add this to filebeat.yml:
output.elasticsearch:
hosts: ["localhost:9200"]
index: "filebeat-system-%{+yyyy.MM.dd}" # Daily indices for system logs
If you’re using Logstash for advanced log processing (e.g., grok parsing), configure Filebeat to send logs to Logstash:
output.logstash:
hosts: ["localhost:5044"] # Logstash listens on port 5044
Disable Elasticsearch output if using Logstash:
# output.elasticsearch: # Comment out or remove this section
After configuring Filebeat, start the service and enable it to run on boot:
sudo systemctl start filebeat
sudo systemctl enable filebeat
Check the service status to ensure it’s running:
sudo systemctl status filebeat
You should see “active (running)” in the output.
To confirm Filebeat is collecting system logs, check the Filebeat logs:
sudo journalctl -u filebeat -f
Look for messages like “Successfully sent log lines” or “Publishing events to Elasticsearch/Logstash.”
To verify logs in Elasticsearch, use curl to list indices:
curl -X GET "localhost:9200/_cat/indices?v"
You should see indices named filebeat-system-* (if using Elasticsearch output) or filebeat-* (if using Logstash).
If you’re using Kibana, you can visualize and analyze system logs:
http://localhost:5601).filebeat-system-*).@timestamp, host.name, process.name, and message.For more control over log parsing or enrichment, you can:
filebeat.yml. Example:processors:
- add_cloud_metadata: ~ # Add cloud provider metadata (if applicable)
By following these steps, you can effectively use Filebeat to collect, parse, and analyze Ubuntu system logs, enabling you to monitor system health, troubleshoot issues, and detect security threats.
