Linux dropped监控方法

Linux Dropped Packet Monitoring Methods

Dropped packets (dropped) in Linux refer to data packets that fail to reach their destination or are discarded by the system due to various reasons (e.g., network congestion, interface errors, firewall rules). Effective monitoring helps identify the root cause—whether it’s a local interface issue, a network path problem, or a misconfiguration. Below are structured methods to monitor and diagnose dropped packets:

1. Quick Local Interface Check with ip or ifconfig

The most straightforward way to detect dropped packets is by checking network interface statistics. Use the ip command (modern replacement for ifconfig) to view detailed metrics:

ip -s link show <interface>  # Replace <interface> with eth0, ens33, etc.

Focus on the RX (receive) and TX (transmit) sections. Key fields:

• dropped: Number of packets discarded by the interface (e.g., buffer full, CRC errors).
• errors: Total errors (e.g., CRC, framing errors).
  A consistently increasing dropped value indicates local interface issues (e.g., faulty hardware, misconfigured MTU).

2. Advanced Interface Statistics with ethtool

For deeper insights into interface-level drops, use ethtool (requires root). It shows hardware-specific counters (e.g., ring buffer overflows, missed packets):

sudo ethtool -S <interface>  # Example: sudo ethtool -S eth0

Look for counters like:

• rx_dropped: Packets dropped by the receive path (e.g., buffer overflow).
• tx_dropped: Packets dropped during transmission (e.g., queue full).
• rx_errors: Receive errors (e.g., CRC, symbol errors).
  These counters help pinpoint whether drops occur at the receive or transmit stage.

3. System-wide Dropped Packet Tracking with netstat/ss

To analyze protocol-level drops (e.g., TCP retransmissions, UDP errors), use:

• netstat -s: Displays aggregated statistics for all protocols (TCP, UDP, ICMP). Key filters:

  netstat -s | grep -E "segments retransmitted|packet receive errors|dropped"
  

  • segments retransmitted: TCP packets retransmitted due to lost acknowledgments (indicates network instability).
  • packet receive errors: Invalid or corrupted packets received (e.g., buffer overflow).
• ss -s: A faster alternative to netstat (from the iproute2 package). Shows socket state summaries (e.g., orphaned connections, TIME-WAIT queues).

  ss -s  # Check for high "orphaned" connections (may indicate application issues)
  

Both tools help identify if drops are caused by protocol errors or resource exhaustion.

4. Real-time Kernel Drop Monitoring with dropwatch

For low-level, real-time monitoring of kernel-level drops (e.g., skb drops, queue overflows), use dropwatch:

sudo dropwatch -l kas  # Load kernel address symbols
sudo dropwatch -start  # Start monitoring

Press Ctrl+C to stop. The output shows where drops occurred (e.g., icmp_rcv for ICMP packets, tcp_v4_rcv for TCP). This is ideal for diagnosing kernel-level bottlenecks.

5. Performance Analysis with perf

perf (Linux performance toolkit) tracks kernel events related to dropped packets. To monitor kfree_skb events (when the kernel frees a socket buffer due to drops):

sudo perf record -g -a -e skb:kfree_skb  # Record events
sudo perf script  # Analyze results

The output shows stack traces of where drops occurred (e.g., network driver, protocol stack). This helps correlate drops with specific code paths.

6. Automated Continuous Monitoring with Scripts

For long-term tracking, create scripts to log dropped packet metrics. Example:

#!/bin/bash
LOG_FILE="/var/log/dropped_monitor.log"
INTERFACE="eth0"
while true; do
    TIMESTAMP=$(date "+%Y-%m-%d %H:%M:%S")
    DROPPED_RX=$(ip -s link show $INTERFACE | grep "RX:" | awk '{print $3}')
    DROPPED_TX=$(ip -s link show $INTERFACE | grep "TX:" | awk '{print $7}')
    echo "$TIMESTAMP - RX Dropped: $DROPPED_RX, TX Dropped: $DROPPED_TX" >> $LOG_FILE
    sleep 60  # Log every minute
done

This script logs RX/TX drops to /var/log/dropped_monitor.log, allowing you to analyze trends over time.

7. Visualization & Alerting with Prometheus + Grafana

For enterprise-grade monitoring, use Prometheus (collects metrics) + Grafana (visualizes data). Steps:

1. Install node_exporter on Linux nodes to expose network metrics (e.g., node_network_receive_drop_total).
2. Configure Prometheus to scrape node_exporter endpoints.
3. Create a Grafana dashboard to visualize dropped packets (e.g., line charts for RX/TX drops, alerts for thresholds).
   This setup enables real-time dashboards and proactive alerts for critical drops.

By combining these methods, you can comprehensively monitor dropped packets—from quick local checks to advanced kernel-level analysis—ensuring timely identification and resolution of network issues.