The Compromise of a Production VPS

Source: Dev.to

Initial Symptoms

Security incidents often arrive quietly, without alarm bells or dramatic warnings. They sneak into production environments when we least expect them—on weekends, during moments of downtime, or when routine vigilance slips just a little.

On a calm Sunday evening I opened my production app to check something, but it didn’t load. No error page, no timeout—just a silent blank screen with only the splash screen visible. Assuming a temporary hiccup, I tried to SSH into the VPS. The connection timed out repeatedly, leading me to suspect a shutdown due to an overdue bill.

After several attempts the SSH prompt finally appeared, but the machine was unusually slow. Even simple commands like ls took 3–4 seconds.

First Investigation

Once logged in, the system felt painfully sluggish. I ran:

top

and saw a process named rysyslo (located at /usr/bin/rsyslo) consuming roughly 80 % CPU, a significant amount of RAM, and almost all disk space. Ubuntu does not ship such a binary, and a quick search confirmed it was malicious. My server had been breached.

I deleted the suspicious binary, removed unfamiliar services, and killed the rogue processes. The load dropped and the system behaved normally—yet I knew the compromise might not be gone with such simple steps.

Escalation

The following morning the app was still unavailable, and SSH behaved erratically, refusing connections for minutes at a time. After finally regaining access, top revealed:

• xmrig – a notorious crypto‑mining tool
• Multiple disguised binaries with random names
• Fake kernel threads impersonating system processes
• A mysterious program called monarx-agent consuming memory
• Node processes I had not launched

Killing these processes was ineffective; the attackers had set up persistence mechanisms.

Persistence Mechanisms

Scanning for malicious services uncovered two systemd units:

• alive.service
• lived.service

Both pointed to scripts in:

/tmp/runnv/alive.sh
/tmp/runnv/lived.sh

The unit files contained:

Restart=always
RestartSec=5s

These scripts continuously fetched, restored, and relaunched the miner—reappearing seconds after deletion. This explained why every kill or removal was short‑lived.

Impact and Data Backup

Over time the VPS became increasingly unstable:

• SSH stalled or dropped repeatedly
• System operations slowed to a crawl
• Memory usage spiked unpredictably
• Logs showed irregularities
• The fake monarx-agent reappeared despite removal

Given the root‑level foothold, the operating system could no longer be trusted. My priority shifted to extracting important data before the machine became completely unusable.

Backup Procedure

I systematically backed up:

• PostgreSQL database dumps (pg_dump)
• Application server files
• Media directories
• Configuration files
• User uploads
• Essential environment variables
• Reverse‑proxy configs
• Logs for later analysis

Because the system was barely responsive, I used rsync over SSH with custom scripts that handled intermittent connection drops. Eventually every critical asset was safely backed up and verified.

Decision to Decommission

After confirming data integrity, I discovered that the application data itself was untouched—the attackers only wanted to mine cryptocurrency. I decided to shut down the compromised VPS permanently and migrate to a fresh instance, as the system was beyond repair.

Root Cause Analysis

I reviewed forensic data, including:

• /var/log/auth.log
• SSH access attempts
• Systemd unit file timestamps
• Root‑level process creation logs
• Cron jobs
• Unknown script origins
• Unfamiliar executables
• Open ports (netstat/ss)
• Binaries in /usr/bin, /tmp, /dev/shm, /usr/share/updater

The logs showed only rejected SSH attempts—no evidence of leaked keys or direct brute‑force success. The likely entry point was the recently disclosed React2Shell vulnerability (CVE‑2025‑55182), which allows remote code execution in misconfigured environments. Nginx access logs contained forced POST requests to numerous endpoints, suggesting an attacker exploited vulnerable web routes.

Additionally, the VPS had an open network configuration (no restrictive firewall), dramatically increasing the attack surface compared to cloud platforms with strict security groups.

Lessons Learned

• Never run applications as root. Even non‑privileged services (e.g., PostgreSQL) remained safe because they ran under dedicated users.
• Apply security patches promptly, especially for newly disclosed vulnerabilities like CVE‑2025‑55182.
• Restrict network exposure with firewalls or security groups; close unused ports.
• Implement proper monitoring and alerting for abnormal CPU, memory, and disk usage.
• Use immutable infrastructure: rebuild servers from known‑good images rather than attempting deep clean‑ups.
• Regularly back up critical data and test restoration procedures.

This incident reinforced a principle I will never ignore again: Never run applications as root.