Essential Linux Troubleshooting Commands for DevOps Engineers

Disclosure: this article contains an affiliate link. DotExe may earn a commission if you purchase through it, at no extra cost to you. Any discount is subject to Hostinger's current offers (20% OFF) and eligibility terms.

When you are managing production infrastructure, the ability to quickly diagnose issues using the command line is a superpower. Whether you are investigating a sudden spike in latency or debugging a container that refuses to stay running, the right set of Linux troubleshooting commands for DevOps engineers can save hours of downtime. This guide covers the essential tools and workflows for diagnosing performance, network, and process-level issues in both bare-metal and containerized environments.

The Philosophy of Systematic Troubleshooting

Effective troubleshooting follows a structured path: observe, isolate, and verify. Before jumping into complex tools, always start with the basics. Is the system reachable? Is it under heavy load? Are there processes consuming excessive resources? By following a logical flow, you avoid the common trap of guessing at the root cause. Start by establishing a baseline. If you do not know what "normal" looks like for your infrastructure, you cannot identify what is "abnormal."

1. System Performance and Resource Monitoring

When a server feels sluggish, your first task is to determine if the issue is CPU, memory, or I/O bound. The top or htop commands are the standard starting points, but for deeper analysis, you need more granular tools.

• top / htop: These provide a real-time overview of process activity. Use htop for a more readable, color-coded interface that allows you to sort by CPU or memory usage easily. Pressing F6 inside htop allows you to sort by specific metrics, which is invaluable when hunting down rogue processes.
• vmstat: This command reports information about processes, memory, paging, block IO, traps, and CPU activity. It is excellent for identifying if your system is swapping, which is often the silent killer of performance. Running vmstat 1 provides a rolling update that helps you visualize trends over time.
• iostat: If you suspect disk bottlenecks, iostat provides detailed statistics about CPU and I/O usage for devices and partitions. It helps confirm if a high wait time is due to slow disk throughput. Use iostat -xz 1 to see extended statistics, which include the percentage of time the disk was busy.

For a deeper dive into memory, understanding how to measure real memory pressure on a Linux server is crucial, as simple free memory reporting can be misleading due to kernel caching. Many engineers mistakenly believe that "free" memory is the only metric that matters, but in Linux, unused RAM is wasted RAM. Always look at the buff/cache values to understand how the kernel is managing your data.

2. Network Connectivity and Latency Diagnostics

Network issues in distributed systems are notoriously difficult to track. Whether you are dealing with inter-service communication or external API calls, these tools are indispensable.

• ss: The modern replacement for netstat. Use ss -tulpn to see which processes are listening on which ports. This is the first command you should run when a service fails to bind to a port. The -t flag filters for TCP, -u for UDP, -l for listening sockets, -p to show the process, and -n to prevent DNS resolution, which speeds up the command.
• mtr: A combination of traceroute and ping. It provides a continuous diagnostic of the path to a destination, showing packet loss and latency at every hop. This is significantly more useful than a one-off traceroute because it reveals intermittent issues that might only show up after several seconds of testing.
• tcpdump: When you need to see exactly what is happening on the wire, tcpdump is the gold standard. It captures packets, allowing you to verify if requests are actually reaching the server or if they are being dropped by a firewall. Use tcpdump -i eth0 port 80 to filter traffic on a specific interface and port.

If you are working with containerized environments, remember that troubleshooting Docker networking issues often involves inspecting virtual interfaces that don't exist on the host OS directly. You may need to enter the container's network namespace to run these commands effectively.

3. Debugging Processes and System Calls

Sometimes a process is running but not behaving as expected. It might be stuck in a loop, waiting on a file descriptor, or failing to communicate with the kernel.

• strace: This tool traces system calls. If a service is crashing or hanging, strace -p <pid> allows you to see exactly which system call is failing. It is invaluable for debugging permission errors or missing configuration files. Be careful when running this on production systems, as it can significantly slow down the target process.
• lsof: "List Open Files." In Linux, everything is a file. If you need to know which process is holding a lock on a file or which network socket a process is using, lsof provides the answer. For example, lsof -i :8080 will tell you exactly which PID is occupying that port.

4. Troubleshooting Containerized Environments

In a Docker-heavy environment, the host-level tools above still apply, but you must account for the container abstraction. When you encounter a container that isn't behaving, start with docker inspect to view its configuration and docker logs to check stdout/stderr. If a container keeps restarting, the exit code is your best friend. A code of 137 usually indicates an OOM kill, while 127 often points to a missing executable.

If you find that your containers are exiting immediately, follow the steps in debugging a Docker container that exits immediately to identify configuration or entrypoint errors. Additionally, ensure you are managing your logs correctly, as managing Docker container logs is a frequent requirement to prevent disk space exhaustion in production. Without log rotation, a runaway container can fill your entire root partition in minutes.

Resource Constraints and The OOM Killer

One common failure mode in production is the OOM (Out of Memory) killer. If your containers are being killed unexpectedly, verify your memory limits. You can optimize Stirling-PDF resource limits as a practical example of how to balance performance with stability. Always test your application's memory footprint before deployment to avoid system-wide instability. If you set a memory limit that is too low, the kernel will kill your process without warning, leading to "ghost" crashes that are hard to replicate in development.

5. Security and Hardening

Troubleshooting is also about identifying unauthorized access or misconfigured services. Always maintain a hardened Linux server to minimize the attack surface. Use auditd to track system calls and file access if you suspect malicious activity. By auditing file access to sensitive directories like /etc/, you can catch unauthorized configuration changes before they cause a production outage.

Practical Example: Diagnosing a Slow API

Imagine your web application is reporting high latency. Here is your checklist:

1. Check System Load: Run htop. Is CPU usage pegged? If yes, identify the process. If CPU is low, check for high load averages which suggest I/O wait.
2. Check Network: Run ss -tulpn to ensure the service is listening. Use mtr to check for network congestion between the client and server. If packet loss occurs at the first hop, the issue is likely local to your rack or switch.
3. Check I/O: Run iostat -x 1. Is the disk wait time high? This often points to database contention or logging overhead. If disk utilization is near 100%, your application is likely waiting for disk writes to complete.
4. Check Logs: Use journalctl -u <service-name> -f to see real-time application errors. Look for patterns in timestamps that correlate with the latency spikes.

The Importance of Observability

While these CLI tools are essential for incident response, they are not a replacement for long-term monitoring. You should integrate these metrics into a dashboarding tool. Understanding how correlating Docker container spikes with system-level resource contention works will save you from manually running these commands during every incident. When you have historical data, you can distinguish between a "new" problem and a "known" performance bottleneck that occurs during peak traffic hours.

Understanding Process States

When using ps aux or top, you will see a "STAT" column. Understanding these codes is vital for troubleshooting. D represents uninterruptible sleep, usually waiting for I/O. R is running or runnable. S is interruptible sleep, waiting for an event. Z is a zombie process. If you see many D state processes, your disk subsystem is likely failing or severely overloaded. This is a classic indicator that you need to look at iostat or check your storage backend health.

Advanced Diagnostics with perf

For deep-dive performance tuning, perf is the gold standard. It allows you to profile CPU usage, cache misses, and branch mispredictions. While it has a steep learning curve, it is the only way to diagnose "hidden" performance issues where the code is executing correctly but inefficiently. Use perf top to see which functions are consuming the most CPU cycles in real-time. This is particularly useful for compiled languages like Go or C++ where you need to identify hot paths in your code.

Troubleshooting Kernel Panics and OOMs

When the kernel itself is the problem, you need to check dmesg. This command prints the kernel ring buffer, which contains messages from the kernel and drivers. If your system crashed or a hardware device failed, the answer is almost always in dmesg. Pipe it to grep -i error or grep -i fail to quickly surface critical issues. If you suspect an OOM event, look for "Out of memory: Kill process" in the output.

Conclusion

Troubleshooting is an iterative process. By mastering these essential Linux commands, you move from guessing to knowing. Start by building a solid foundation with htop, ss, and strace. As you gain experience, you will find that most production issues share common patterns, and your ability to diagnose them quickly will become second nature. Remember that the goal is not just to fix the immediate problem, but to understand the root cause so you can prevent it from recurring. Keep a log of your findings, automate your checks where possible, and always verify your changes in a staging environment before pushing to production.

Affiliate resource and disclosure

Review Hostinger Web Hosting and check whether it fits your requirements, pricing, and terms.

Disclosure: this article contains an affiliate link. DotExe may earn a commission if you purchase through it, at no extra cost to you. Any discount is subject to Hostinger's current offers (20% OFF) and eligibility terms.