Day 14 — I Built ProcWatch : A Linux Process Security Scanner for Forensics & Incident Response

Source: Dev.to

The Moment This Tool Became Necessary

If you’ve ever solved a Linux forensics CTF, you know the ritual:

1. SSH into the machine.

2. Everything looks normal.

3. Then you notice:

   • CPU usage feels weird
   • There’s network traffic
   • Logs don’t line up

So you begin the ancient ceremony:

ps aux | grep -v grep
netstat -tulnp
ls -la /tmp
cat /proc/<pid>/cmdline

You scroll. You guess. You re‑run commands. Somewhere in those 200+ processes is a reverse shell, a dropper, or a cryptominer… and you are manually playing “Where’s Waldo: Incident Response Edition.”

After one challenge where I lost 40 minutes hunting a shell hiding in /dev/shm, I realized:

I wasn’t solving a cybersecurity problem.
I was solving a visibility problem.

So I built a tool. That tool became ProcWatch.

What ProcWatch Actually Is

ProcWatch is a Python‑based process‑behavior scanner.

• Not antivirus.
• Not signature detection.

It asks a different question:

Is any process on this system behaving like an attacker would?

Instead of checking files, it watches process‑behavior patterns used in real intrusions.

The Threat Model

Before writing detections, I mapped how attackers typically behave after gaining Linux access:

Every ProcWatch detection maps directly to one of these stages.

Detection 1 — Executing from Writable Directories (CRITICAL)

SUSPICIOUS_LOCATIONS = [
    "/tmp",
    "/dev/shm",
    "/var/tmp",
    "/run/user",
    "/dev/mqueue",
]

Malware loves writable directories because:

• No permissions required
• Rarely monitored
• Easy to clean up

/dev/shm is especially interesting. It’s RAM‑backed storage; reboot the machine and evidence disappears. Legitimate software almost never runs from here.

If you see:

/dev/shm/hidden_binary

you didn’t find a program—you found the attacker.

Detection 2 — Suspicious Interpreter Usage

SUSPICIOUS_NAMES = ["bash", "sh", "nc", "python", "perl", "ruby", "socat"]

On its own this means nothing, but combined with location or network activity it becomes powerful.

• A python3 process is normal.
• A python3 process in /tmp connected to port 4444 is not.

This is called living‑off‑the‑land: attackers use legitimate tools so they don’t need to upload malware.

Detection 3 — Privilege‑Escalation Indicators

if uids.real != uids.effective:
    # SUID privilege escalation in action

Linux processes have a real UID and an effective UID. If they differ, you’ve caught a privilege‑escalation event.

Even more suspicious:

root process running from /home/user

Root processes belong in /usr/bin, not a user’s Downloads folder.

Detection 4 — Reverse‑Shell & C2 Detection

SUSPICIOUS_PORTS = [4444, 5555, 7777, 31337]

These ports are classics for reverse shells and C2 listeners.

ProcWatch distinguishes between:

• LISTEN → suspicious
• ESTABLISHED outbound connection → almost certain compromise

A process connecting outward to port 4444 is basically waving a flag saying: “someone else is controlling me.”

Detection 5 — Cryptominer Detection

Keyword detection

MINER_KEYWORDS = ["xmrig", "monero", "stratum", "pool"]

Behavioral detection

if cpu_usage > 85:   # percent
    # possible miner

Why this matters: modern attackers often don’t destroy systems; they monetize them. Many real breaches are discovered because servers suddenly run at 100 % CPU 24/7.

Detection 6 — Deleted Executable Trick

Attackers often:

1. Run malware
2. Delete the file

Linux keeps the process alive in memory, so the file disappears from disk but continues executing.

Kernel hint:

/proc/<pid>/exe -> binary (deleted)

ProcWatch detects exactly this. You can even recover the binary:

cp /proc/<pid>/exe recovered_binary

That’s a real forensic technique.

Detection 7 — LD_PRELOAD Injection (Advanced)

LD_PRELOAD=/tmp/libevil.so

Instead of dropping a full binary, attackers load a malicious library that intercepts system calls to:

• Hide files
• Steal passwords
• Fake authentication

This is how many user‑space rootkits work. Catching this is almost always a confirmed compromise.

The CLI

I designed the tool to feel like git:

procwatch scan          # one‑time scan
procwatch scan -v -j   # verbose JSON output
procwatch watch         # live monitoring
procwatch info          # details about a process
procwatch list          # list all tracked processes

The Most Useful Mode

procwatch watch

It alerts only on new suspicious processes, giving you a live feed of attacker activity. It feels surprisingly close to a real SOC monitoring console.

How This Helps in CTFs

Immediately after login:

procwatch scan -j

You now have a timestamped snapshot of the system’s process state, letting you spot the malicious process in seconds instead of minutes.

Investigating 200 Processes? You Only Need to Look at 3

That changes a 45‑minute challenge into a 5‑minute one.

Limitations

• This tool operates in user space via /proc, so it cannot detect kernel rootkits. If the kernel itself lies, ProcWatch will believe it.
• Some interpreter alerts are false positives.
• Containers hide processes via namespaces.

Future upgrades I’m considering

• YARA memory scanning
• eBPF syscall monitoring
• ptrace detection

Running It

pip install psutil
sudo python3 procwatch.py scan -v

Screenshots

(Root gives full visibility into environment variables and connections.)

What I Learned

I started this to automate a CTF workflow. Instead, I learned something important:

• Attackers don’t hide files.
• They hide behavior inside normal processes.

Behavior is much harder to fake.

ProcWatch doesn’t replace investigation—it just shines a flashlight directly where you should start looking.