For any Linux administrator, keeping track of network usage and bandwidth consumption is critical. Excessive network I/O can indicate problems like malware or misconfigured applications. Identifying which processes are responsible for heavy network activity helps troubleshoot issues and optimize resource allocation.

In this comprehensive guide, we will explore the tools and techniques to monitor network usage on a per-process level on Linux.

Overview of Monitoring Network Usage by Process

Here are some key aspects we will cover for monitoring network I/O of processes:

• Check total bandwidth used by each process transmitting or receiving data.

• Analyze traffic by IP protocol - TCP, UDP, ICMP etc.

• Break down bandwidth usage by open connections and sockets.

• Identify processes hogging bandwidth to detect anomalies.

• Monitor network usage in real-time for quick troubleshooting.

• Analyze bandwidth usage over time to identify trends.

• Filter network activity by criteria like port, protocol, top talkers etc.

• Understand the role of key Linux services like systemd-networkd in network activity.

Many standard Linux tools like top and ps can reveal CPU, memory and I/O usage by process, but not network usage specifically. For that, we need specialized utilities as explained next.

Monitoring Network Usage in Real-Time

For real-time monitoring of active network connections and bandwidth usage by process, these tools are handy:

1. ntopng

ntopng is an interactive network profiler capable of showing live usage and connections at a packet level.

To install on Debian/Ubuntu:

sudo apt install ntopng

Once launched, the ntopng interface displays network activity organized by protocol, top talkers, flows and more. It can be filtered to focus on specific processes.

2. iftop

iftop is a terminal-based bandwidth monitor that shows live transmission and reception rates by host/connection along with total usage.

Install on Ubuntu/Debian:

sudo apt install iftop

iftop's interface visualizes bandwidth usage and highlights spikes:

3. nethogs

nethogs is a Linux top-like tool specialized for bandwidth usage by process.

To install on Ubuntu/Debian:

sudo apt install nethogs

Here is sample nethogs output tracking bandwidth consumption by process in real time:

These utilities give real-time insight into network usage that can help identify any suspicious activity.

Capturing Historical Usage Data

For analyzing network usage over time, these tools provide historical data capturing capabilities:

1. ntopng

In addition to live monitoring, ntopng can save network traffic data to SQLite or MySQL databases for generating historical reports. This helps identify trends in usage.

2. NetFlow

NetFlow is a network protocol developed by Cisco for monitoring IP network traffic. NetFlow collectors can be used with nfdump tools to capture and report on historical network usage on Linux servers.

3.SNMP

The Simple Network Management Protocol (SNMP) can be used to collect network usage data over time. Tools like MRTG utilize SNMP to generate reports on historical network load.

For traffic analysis over time, ntopng, NetFlow and SNMP offer in-depth data collection capabilities.

Analyzing Traffic by Connections and Sockets

For understanding network usage at a connection and socket level, these tools come handy:

1. ss

ss is a socket statistics utility that comes as part of the iproute2 suite. It can show sockets opened by processes along with bandwidth usage:

ss -t -a -p

2. lsof

lsof lists open files (sockets) by processes including network files:

lsof -i

3. netstat

The netstat tool can show network connections correlated with processes:

netstat -tp

These tools give a breakdown of network usage at the Linux socket and file descriptor layer used by applications.

Identifying Top Talkers

To quickly identify "top talkers" i.e. processes that are heavy on network usage, these tools come in handy:

1. nload

nload shows total bandwidth consumption by process category in real-time:

2.nethogs

As seen earlier, nethogs lists processes by their actual network usage, making it easy to identify top talkers.

3. iftop

iftop can also filter by host IP/port to pinpoint chatterboxes on the network.

These utilities complement each other to help uncover processes performing excessive network I/O.

Special Case - Analyzing systemd-networkd

In systemd-based distros like RHEL 8, Ubuntu 18.04+, systemd-networkd handles network configuration. So a chunk of network traffic gets attributed to to systemd-network process.

To identify traffic originating from actual applications, filter out systemd-networkd when using tools like nethogs and nload.

Conclusion

Linux provides diverse tools to monitor network usage at a per-process level. Key takeaways:

• ntopng, iftop and nethogs help monitor real-time bandwidth consumption by process.

• ntopng, NetFlow and SNMP aid in analyzing historical network usage data.

• ss, lsof and netstat examine open sockets and connections by process.

• nload and nethogs reveal processes that are network bandwidth hogs.

• Filter out systemd-networkd traffic if needed on modern distros.

From detecting anomalous activity to identifying chatty applications, these tools provide unparalleled visibility into process network usage on Linux. They help analyze traffic patterns, troubleshoot issues, and make data-driven resource allocation decisions.

Hopefully this guide helps you gain a deeper understanding of network usage in Linux at a per-process level. Let me know if you have any other favorite network monitoring tools or techniques!