When you type ls -la and hit Enter, the shell
(bash/zsh) reads that string. First it checks if it's a built-in
command (like cd, echo). If not, it
searches through directories listed in your
PATH environment variable — one by one — until
it finds a binary named ls.
Once found, the shell calls fork() — cloning
itself into a parent and child process. The child process then
calls exec() — which replaces the child's
entire memory with the ls program. The parent shell
calls wait() — pauses itself until the child
finishes. When ls is done, the child exits, and the parent wakes
up and shows you the prompt again.
This fork-exec pattern is the foundation of how every command runs in Linux. Even when you run a Python script — the shell forks, the child execs the Python interpreter, Python interpreter runs your script, exits, shell resumes.
- Built-in commands (cd, echo, export) run in the shell process itself — no fork needed.
-
PATH is searched left to right — first match wins.
which lsshows you which binary gets found. - If binary not found anywhere in PATH → "command not found" error.
-
&at the end (e.g.ls &) makes the shell NOT wait — runs the child in the background.
In Linux, a process is what the kernel gives
its own virtual address space, file descriptor table, and PID.
When you run python script.py, that's a process.
Each process is isolated — one crashing doesn't kill others.
A thread (called a "light-weight process" in Linux) lives inside a process. All threads share the heap, code segment, and open files. They each have their own stack and CPU registers. Creating a thread is much cheaper — no new address space needed. Communication is just reading/writing shared memory — but that needs locks or you'll have race conditions.
| Aspect | Process | Thread |
|---|---|---|
| Memory | Separate, isolated | Shared heap, own stack |
| Creation | fork() — heavy | pthread_create() — light |
| Communication | IPC: pipes, sockets | Shared variables (with mutex) |
| Crash isolation | Yes — others unaffected | No — one crash kills all |
| Linux command | ps aux |
ps -eLf or top -H |
-
Linux doesn't really distinguish processes and threads
internally — both are "tasks".
clone()syscall creates both, with different flags for what to share. -
ps auxshows processes.topshows real-time.htopis the nicer version. - A process always has at least one thread — the main thread.
When a child process exits, it doesn't disappear immediately.
The kernel keeps a tiny record of it in the process table — its
PID, exit status, resource usage — waiting for the parent to
come and collect this info via wait() or
waitpid(). Until the parent does that, the child
sits as a zombie — it's done running, uses no
CPU or memory, but its PID slot is occupied.
A handful of zombies is normal. Thousands of zombies means the parent is badly written — never calling wait(). This exhausts PIDs (Linux has a finite PID limit, typically 32768). Fix: fix the parent code, or kill the parent (which makes zombies orphans, adopted by init/systemd, which properly reaps them).
-
Zombie shows as
Zin process state inps auxoutput. - You cannot kill a zombie — it's already dead. Only killing the parent cleans it up.
- When parent dies, init (PID 1) adopts all orphan children and reaps them properly.
- Fix properly: parent installs SIGCHLD signal handler and calls wait() in it.
- People say "kill the zombie" — you literally can't, kill -9 won't work on a zombie. Kill the parent instead.
Daemons are the workhorses of Linux. They start when the system
boots (managed by systemd or older init) and
keep running silently. sshd waits for SSH
connections. crond runs scheduled jobs.
nginx or apache serves web pages.
syslogd collects logs.
To become a daemon, a process typically: forks and lets the
parent exit (so the shell thinks the job is done), calls
setsid() to detach from the terminal (no
controlling TTY), closes stdin/stdout/stderr (redirects them to
/dev/null or a log file), changes working directory to / (so it
doesn't block unmounting any filesystem).
- Daemon names conventionally end in 'd': sshd, crond, httpd, mysqld, dockerd.
-
Check running daemons:
systemctl list-units --type=service -
Start/stop daemons:
systemctl start nginx,systemctl stop nginx -
View daemon logs:
journalctl -u nginxor/var/log/ -
Running a process in background with
&is NOT a daemon — it's still tied to your terminal session. Close the terminal, process dies.
Every file in Linux has an inode — a data structure holding the file's metadata and location on disk. The filename is just a label pointing to an inode. A hard link creates another filename pointing to the exact same inode. Both names are equal — neither is "the original." The file data is only deleted when ALL hard links to it are removed (inode reference count hits 0).
A soft link (symlink) is a special file that contains a path string — like a shortcut. When you access a symlink, Linux follows the path inside it to the target. Delete the target? The symlink still exists but points to nothing — a "dangling symlink." Soft links can cross filesystems and point to directories; hard links cannot.
-
Create hard link:
ln file.txt hardlink.txt. Create symlink:ln -s file.txt symlink.txt - Hard links can't cross filesystem boundaries (different partitions). Soft links can.
- Hard links can't point to directories (prevents loops). Soft links to directories: common and useful.
-
ls -lishows inode numbers — hard links to same file have same inode number. -
Soft links are what
/usr/bin/pythonoften is — a symlink to the actual python3.x binary.
Think of a filesystem like a library. The book content is stored on shelves (data blocks). The catalog card for each book is the inode — it tells you who wrote it (owner), when it was last read (timestamps), how many pages (size), and which shelf it's on (data block pointers). The book's title is just a label in a directory that points to the catalog card number (inode number).
Every file and directory has exactly one inode. Directories are
just files containing a list of (filename → inode number)
mappings. When you access /home/user/file.txt,
Linux traverses: root inode → home directory inode → user
directory inode → file.txt's inode → reads data blocks.
Filenames live in directories, not in inodes. That's why hard
links work — two names, one inode.
-
stat filenameshows the inode of a file — all its metadata. -
ls -ishows inode numbers alongside filenames. - Inode count is fixed at filesystem creation — you can run out of inodes even with free disk space (many tiny files).
-
df -ishows inode usage. Critical for log servers and mail servers that generate millions of small files. - Inode doesn't store filename — that's stored in the parent directory's data.
At companies like Flipkart/Amazon, storage systems can hit "disk
full" errors even when df -h shows free space —
because inodes are exhausted. Knowing to run
df -i and finding which directory has millions of
tiny files (usually /tmp or a logging path) shows production
debugging maturity.
Linux file permissions have three levels: owner (the user who owns the file), group (members of the file's group), and others (everyone else). Each level gets three possible permissions: read (4), write (2), execute (1). Add them up to get the octal digit.
So 755: Owner = 4+2+1 = 7 (full control). Group = 4+0+1 = 5 (read and run, can't modify). Others = 4+0+1 = 5 (read and run, can't modify). This is the standard permission for executables and web directories — everyone can read and run, only owner can modify.
-
Symbolic form:
chmod u+x file(add execute for user),chmod go-w file(remove write from group and others). - Execute on a directory means you can enter it (cd into it). Without execute, you can't traverse the directory even if you know file names inside.
-
chown user:group filechanges ownership.chmodchanges permissions. -
Special bits: setuid (4xxx), setgid (2xxx), sticky bit (1xxx).
chmod 1777on /tmp — sticky bit means only file owner can delete their own files.
kill PID sends
SIGTERM (signal 15). This is a request, not a
demand. A well-written program catches SIGTERM, closes database
connections, flushes logs, releases locks, then exits cleanly. A
process can even ignore SIGTERM if it wants — which is why
sometimes kill doesn't work on a stubborn process.
kill -9 PID sends
SIGKILL (signal 9). This one the kernel handles
directly — the process never even sees it. No cleanup happens.
No signal handlers run. File handles may not be flushed, temp
files not deleted, database connections abruptly cut. Use it as
the last resort when kill doesn't work.
- Always try SIGTERM first. Give it 5-10 seconds. Then escalate to SIGKILL.
-
kill -llists all signals (there are 64). Common ones: 1=SIGHUP (reload config), 2=SIGINT (Ctrl+C), 15=SIGTERM, 9=SIGKILL. -
pkill processnamekills by name.killall nginxkills all processes named nginx. - SIGKILL and SIGSTOP are the only two signals a process CANNOT catch, block, or ignore.
These three tools do completely different things and people
often mix them up. grep looks through file
content — give it a pattern and files, it prints matching lines.
find walks the directory tree looking for files
matching criteria — name, permissions, size, modification time.
locate queries a database (built by
updatedb, runs nightly) of all filenames — blazing
fast but won't find files created today until updatedb runs
again.
# grep — search INSIDE files for pattern $ grep -r "ERROR" /var/log/ # recursive search $ grep -n "def main" script.py # show line numbers # find — search for FILES by properties $ find /home -name "*.log" # by name pattern $ find /tmp -mtime +7 # modified >7 days ago $ find / -size +100M # files bigger than 100MB # locate — fast filename search (pre-indexed) $ locate nginx.conf # instant, uses database $ sudo updatedb # rebuild the database
-
Combine them:
find /var/log -name "*.log" | xargs grep "ERROR"— find files, then search inside them. -
grep -i= case insensitive.grep -v= invert (exclude matching lines).grep -c= count matches. -
find -execruns a command on each result:find /tmp -name "*.tmp" -exec rm {} \;
Output redirection is how you save command output to files
instead of printing to screen. > is destructive
— it opens the file, truncates it to zero, then writes. If the
file doesn't exist, it's created. If it does exist, all previous
content is gone. >> opens the file at the end
and appends — previous content is safe.
$ echo "line 1" > file.txt # creates/overwrites $ echo "line 2" > file.txt # "line 1" is GONE $ echo "line 1" >> log.txt # appends $ echo "line 2" >> log.txt # both lines now in file # Redirect stderr (2) and stdout (1) $ command > out.txt 2> err.txt # separate $ command > all.txt 2>&1 # merge both to one file $ command 2>/dev/null # discard errors
-
2>&1means "redirect stderr (fd 2) to wherever stdout (fd 1) is going" — very common in scripts. -
/dev/nullis the black hole — anything written there is discarded. Perfect for silencing noisy commands. -
<redirects input from a file:wc -l < file.txt
The pipe is one of Linux's most elegant ideas. Instead of running a command, saving output to a file, running next command on that file — you just connect them directly. The kernel sets up an in-memory buffer between them. The first command writes to it, the second reads from it. They run concurrently — producer and consumer.
This Unix philosophy of small, focused tools that do one thing
well and chain via pipes is why Linux command line is so
powerful. cat reads, grep filters,
sort sorts, uniq deduplicates,
wc counts — combine them any way you need.
# Count ERROR lines in a log $ cat app.log | grep "ERROR" | wc -l # Find top 5 largest files $ du -sh * | sort -rh | head -5 # Who are the top memory-using processes? $ ps aux | sort -k4 -rn | head -10 # Find all unique IP addresses in access log $ awk '{print $1}' access.log | sort | uniq -c | sort -rn
- Pipe is unidirectional — data flows left to right only.
-
Named pipes (FIFOs) persist on the filesystem:
mkfifo mypipe— useful for inter-process communication. - Each command in a pipeline runs as a separate process — they execute concurrently, connected by kernel buffers.
-
teesplits the pipe: sends output to both stdout AND a file:command | tee output.txt | next_command
Think of RAM as your desk and swap as a filing cabinet. When your desk is full, you move less-used papers to the filing cabinet to make room. You can still access them — just much slower (you have to get up, walk to the cabinet, find the paper). Swap works the same way — the kernel moves cold memory pages to a swap partition or swap file on disk. This prevents out-of-memory crashes, but performance tanks heavily when you're swapping a lot (called "thrashing").
Modern SSDs make swap less painful but still nowhere near RAM speeds. A server running on swap is a server in trouble — it means you're out of RAM and need to add memory or kill memory-hungry processes.
-
free -hshows RAM and swap usage.swapon --showshows swap partitions/files. -
swappinesskernel parameter (0–100) controls how aggressively Linux uses swap. Default 60. Set lower (10-20) on servers to prefer RAM. -
High swap usage = performance problem. Check with
vmstat 1— if si/so (swap in/out) columns are non-zero, you're swapping. - Swap can be a partition or a file. Docker and some cloud VMs have no swap by default.
Linux is smart about memory. A lot of "used" RAM is actually the
page cache — disk data cached in RAM for speed.
When memory is needed, the kernel first reclaims this cache — it
can always re-read from disk. This is why
free -h often shows very little "free" memory on a
healthy system — Linux uses all available RAM for caching.
That's normal and good.
If reclaiming cache isn't enough, Linux starts swapping — moving inactive process memory pages to disk. Performance degrades. If swap is also exhausted — the OOM Killer (Out of Memory Killer) activates. It scores every process by memory usage, importance, and other factors, and kills the highest-scoring one. This is a last resort. Often it kills your app server instead of a less important process — a painful production incident.
-
OOM kill events are logged:
dmesg | grep -i "oom"orgrep -i "killed process" /var/log/syslog -
You can influence OOM killer with
oom_score_adj: set to -1000 to make a process unkillable, +1000 to make it first victim. -
free -h: "available" column is most useful — how much RAM is actually available for new processes (includes reclaimable cache). -
Memory overcommit: Linux by default allows processes to
allocate more memory than physically available (banking on
most never using what they asked for). Controlled by
vm.overcommit_memory.
Knowing that "low free memory" on Linux is NORMAL (Linux fills RAM with page cache) separates you from candidates who panic at seeing 95% memory usage. Interviewers love asking "our server shows 95% RAM used — is it a problem?" The correct answer: check "available" column and swap usage, not just "free."
TCP (Transmission Control Protocol): Before sending any data, TCP does a 3-way handshake (SYN → SYN-ACK → ACK) to establish a connection. Every packet sent must be acknowledged by the receiver. Lost packets are retransmitted. Packets arrive in order. This reliability costs time — extra round trips, more overhead. Use TCP when data integrity matters: HTTP/HTTPS, SSH, databases, file transfers.
UDP (User Datagram Protocol): Just send packets — no connection setup, no acknowledgment, no retransmission, no ordering guarantees. It's fast and has minimal overhead. If a packet is lost, it's lost — the application layer has to handle it if it cares. Use UDP when speed beats reliability: live video streaming, online gaming, DNS queries, VoIP calls (a dropped frame in video is fine — waiting for retransmission ruins the experience).
| Aspect | TCP | UDP |
|---|---|---|
| Connection | Yes (3-way handshake) | No (connectionless) |
| Reliability | Guaranteed delivery | Best effort, may drop |
| Ordering | In-order delivery | Out-of-order possible |
| Speed | Slower (overhead) | Faster (no overhead) |
| Use cases | HTTP, SSH, FTP, DB | DNS, video, gaming, VoIP |
| Header size | 20–60 bytes | 8 bytes |
- TCP 3-way handshake: SYN (client) → SYN-ACK (server) → ACK (client). Connection established.
- TCP 4-way termination: FIN → ACK → FIN → ACK. (More complex because both sides need to close independently.)
- HTTP/3 (QUIC) runs over UDP but implements its own reliability — best of both worlds.
-
netstat -anorss -anshows active TCP/UDP connections and their states.
HTTP (HyperText Transfer Protocol) is the language browsers and servers use to talk. The problem: everything is sent in plaintext. Your login credentials, credit card numbers, session cookies — all visible to anyone doing a man-in-the-middle attack on a public WiFi network. Packet capture tools like Wireshark make this trivial.
HTTPS adds a TLS (Transport Layer Security) layer. Before any HTTP data flows, TLS does a handshake — client verifies the server's certificate (issued by a trusted CA like Let's Encrypt), they negotiate an encryption algorithm and exchange keys, and from then on all data is encrypted. Even if someone captures the packets, they see gibberish.
- HTTP = port 80. HTTPS = port 443. These are defaults — can be changed.
- TLS certificate has: domain name it's valid for, expiry date, CA signature. Browser verifies all three.
- Let's Encrypt provides free SSL/TLS certs — made HTTPS accessible to everyone.
- HSTS (HTTP Strict Transport Security): server tells browser "always use HTTPS for this domain, never HTTP."
- Mixed content: HTTPS page loading HTTP resources — browser blocks it as a security risk.
Before SSH, people used Telnet to connect to remote servers — username, password, all commands sent completely in plaintext. Anyone on the network could sniff the traffic and steal your password. SSH (1995) fixed this with strong encryption. Every byte between your terminal and the server is encrypted.
SSH also supports key-based authentication —
even more secure than passwords. You generate a key pair:
private key stays on your laptop, public key goes on the
server's ~/.ssh/authorized_keys. When you connect,
SSH proves you have the private key (without sending it) using a
cryptographic challenge-response. No password transmitted, and
even if someone has your public key, they can't log in without
your private key.
# Basic SSH connection $ ssh user@192.168.1.10 # Generate SSH key pair $ ssh-keygen -t ed25519 -C "my-laptop" # Copy public key to server (enables passwordless login) $ ssh-copy-id user@server # SSH tunneling — forward local port 8080 to remote 80 $ ssh -L 8080:localhost:80 user@server # Connect with specific key and port $ ssh -i ~/.ssh/mykey -p 2222 user@server
- Default SSH port is 22. Changing it to a non-standard port (security through obscurity) reduces automated bot attacks.
-
Disable password authentication in
/etc/ssh/sshd_config— force key-based auth only. Much more secure. - SSH tunneling/port forwarding: access services on remote network through encrypted tunnel.
-
~/.ssh/configfile lets you set aliases:ssh myserverinstead of typing full address every time.
Every Linux system has the crond daemon running in
the background. Every minute it checks the crontab files to see
if any job is due. If yes — it runs the command. It's how
sysadmins automate: backup every night at 2 AM, clean temp files
every week, check disk space every hour and alert if nearly
full, rotate logs daily.
# Edit your crontab $ crontab -e # Format: MIN HOUR DAY MONTH WEEKDAY COMMAND # * * * * * # Run backup every day at 2:30 AM 30 2 * * * /home/user/backup.sh # Run every 5 minutes */5 * * * * /scripts/health_check.sh # Run every Monday at 9 AM 0 9 * * 1 /scripts/weekly_report.sh # Run at system reboot @reboot /scripts/start_app.sh # List your cron jobs $ crontab -l
- Use absolute paths in cron commands — cron runs with minimal PATH, your usual aliases/commands may not be found.
-
Cron output goes to email by default. Redirect to a log file:
command >> /var/log/myjob.log 2>&1 -
System-wide crons in
/etc/cron.d/,/etc/cron.daily/,/etc/cron.hourly/ -
Use
crontab.guruwebsite to validate cron expressions — everyone uses it, even seniors.
Every process in Linux inherits environment variables from its parent. When you start a shell, it inherits from its parent, and passes them to commands it runs. This is how you pass configuration to programs without hardcoding it — database connection strings, API keys, feature flags, Java home directory, Python paths.
PATH is the most important environment
variable. It's a colon-separated list of directories. When you
type ls, the shell searches
/usr/local/bin, then /usr/bin, then
/bin, etc. — whichever comes first in PATH wins.
Adding a directory to PATH makes any executables in it available
as commands.
# View all environment variables $ env $ printenv # View specific variable $ echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin # Set a variable (only in current shell) $ export DB_URL="postgres://localhost:5432/mydb" # Add to PATH permanently — put in ~/.bashrc or ~/.zshrc $ export PATH="$HOME/.local/bin:$PATH" # Pass env var to a single command only $ DEBUG=true node app.js
-
export VAR=valuemakes it available to child processes. Without export, it's only in the current shell. -
Permanent changes go in
~/.bashrc(interactive bash),~/.bash_profile(login bash), or/etc/environment(system-wide). - Never commit secrets (API keys, DB passwords) in code — use environment variables. Services like AWS Secrets Manager or .env files (never committed) handle this.
-
which commandtells you which PATH directory the command is found in.
fork() duplicates the calling process — same code, same memory (using copy-on-write, so it's efficient), same file descriptors, but a new PID. Fork returns 0 in the child process and the child's PID in the parent — this is how you know which process you're in. At this point, both parent and child are running the same code.
exec() (actually a family: execv, execve, execl...) replaces the current process image with a new program. The process keeps its PID, file descriptors, and environment, but its memory (code, stack, heap) is completely replaced by the new program. After exec(), the old program's code is gone — only the new program runs.
This two-step design is intentional. The gap between fork and exec gives you a chance to set up the child's environment — redirect file descriptors, set up pipes, change user permissions — before replacing it with the target program.
- Copy-on-write: fork doesn't actually copy all memory immediately — pages are shared until one process writes, then a copy is made. Makes fork fast even for large processes.
- fork() returns -1 on failure (process limit reached), 0 in child, child PID in parent.
- vfork() is like fork() but child borrows parent's address space — must exec immediately. Lighter but dangerous.
-
In Python:
os.fork()andos.exec*(). Thesubprocessmodule uses fork-exec internally.
This is the most important question for any backend/DevOps role. Interviewers want to see that you have a systematic approach — not guessing randomly. There's a clear order of operations. Don't jump to checking application logs before confirming the server is even reachable. That's wasted time in a real incident.
Think in layers: Network → OS → Process → Application → Data. Each layer needs to be ruled out before going deeper. And always communicate — in a real incident, you're not working in silence. You're updating your team every few minutes: "Server is reachable, SSH works, checking process status now."
-
Step 1 — Can I reach the server?
ping server-ip— if no response, network/DNS issue or server completely dead. -
Step 2 — Can I SSH in?
ssh user@server— if yes, OS is alive. If no, check VPN, security groups, firewall. -
Step 3 — Is the process running?
ps aux | grep nginxorsystemctl status nginx— crashed? Restart + find why. -
Step 4 — Check the logs.
tail -f /var/log/nginx/error.log,journalctl -u appname -n 100— look for errors right before crash. -
Step 5 — Check system resources.
toporhtop(CPU),free -h(RAM),df -h(disk),df -i(inodes). -
Step 6 — Check network/connections.
ss -an | grep :80(port listening?),netstat -s(connection stats), check firewall rules. -
Step 7 — Check recent changes. Any
deployments? Config changes? Cron jobs that ran?
lastcommand shows recent logins.historyshows commands run. -
Step 8 — If OOM killed.
dmesg | grep -i oom— which process was killed? Add more RAM or fix memory leak.
# ─── CONNECTIVITY ───────────────────────────────── $ ping -c 4 server-ip $ curl -I https://yoursite.com # HTTP response headers $ traceroute server-ip # where does it stop? # ─── PROCESS STATUS ─────────────────────────────── $ systemctl status nginx $ ps aux | grep "app_name" # ─── LOGS ───────────────────────────────────────── $ journalctl -u nginx -n 200 --no-pager $ tail -f /var/log/app/error.log $ dmesg | grep -i "oom\|error\|fail" # ─── RESOURCES ──────────────────────────────────── $ top # CPU + memory live $ free -h # RAM + swap $ df -h && df -i # disk space + inodes $ iostat -x 1 5 # disk I/O # ─── NETWORK ────────────────────────────────────── $ ss -tlnp # listening ports + processes $ ss -an | grep ":80" | wc -l # connection count $ iptables -L -n # firewall rules
ps aux). Check if
it's listening (ss -tlnp | grep 80). Check
application logs for errors. Check if firewall is blocking
(iptables -L). Check if disk is full (app can't
write logs, refuses requests). Restart the service, watch logs
in real time.
top — check CPU (high user%? application bug.
high sys%? kernel issue. high wa%? disk I/O wait). Check RAM
and swap usage. Check connection count. Check database query
times. Check if a cron job is running and consuming resources.
Mention runbooks — pre-written step-by-step guides for common incidents. Say "in production, we'd have a runbook for this exact scenario — I'd follow that, then update it with any new findings." Interviewers at senior companies love candidates who think about process and documentation, not just heroic one-off debugging.
Ready to test Linux?
Practice commands, permissions, processes, and system basics in one run.