Often, Linux malware will delete itself after it starts so that file scanners and integrity checks won’t see the binary present. It also can make forensic analysis harder if you can’t get to the binary easily. However, it is easy to recover a deleted process binary on Linux as long as the process is still in memory. Let’s take a look.
Recovering a Deleted Linux Process Binary Command
The basic command to recover a deleted process binary is simple. Just do the following:
cp /proc/<PID>/exe /tmp/recovered_bin
Where <PID> is the Process ID of the deleted process you want to recover. The above command will copy the binary to /tmp/recovered_bin so you can do with it what you will.
Practicing Recovery of a Deleted Process
We’re going to use the sleep command to simulate a process that has deleted itself from the disk. You can run this series of command safely on your own Linux system so you can practice recovery of a deleted binary.
cd /tmp cp /bin/sleep x ./x 3600 & rm x
Now check that the binary is in fact gone:
ls -al /tmp/x ls: cannot access /tmp/x: No such file or directory
The above copies the sleep command as “x” under /tmp and runs for 3600 seconds and then exits. We then delete “x” so they system binary appears removed.
You can see the process still running in the ps listing below.
Forensics Examination of Linux /proc for a Deleted Process
Now let’s look at the /proc directory for this process. The /proc filesystem is a special filesystem on Linux that tracks almost all kernel and process information you would ever want.
Under /proc the system flagged the binary as deleted. The command format is:
ls -al /proc/<PID>/exe
Where again <PID> is the process ID we’re interested in like so:
ls -al /proc/12725/exe lrwxrwxrwx 1 root root 0 May 23 19:39 /proc/12725/exe -> /tmp/x (deleted)
Recovery of Deleted Linux Process
But, the Linux file system will not actually delete that file as long as a running process still has it open. The file is there, but it simply is not being shown to you any more.
This /proc/12725/exe link will easily get you the binary that started the process. Simply copy it to where you want and the binary that was deleted is now yours:
cp /proc/12725/exe /tmp/recovered_bin
The hashes of the recovered binary and the sleep command that we used to emulate the attack are identical which proves the file recovered correctly.
sha1sum /tmp/recovered_bin d76373f39b93bbc5191e5f8ca0f9c2f692e5aa71 /tmp/recovered_bin sha1sum /bin/sleep d76373f39b93bbc5191e5f8ca0f9c2f692e5aa71 /bin/sleep
You can now use this binary for further analysis as you wish such as a quick look with strings or disassembler of your choice.
Never Kill a Suspicious Process Blindly
The above is reason #4387 to never simply kill a suspicious process you find until you have thoroughly investigated what is going on. If you were to kill the process first, you’d lose the binary as it would be released by the filesystem and likely destroyed.
If a system is infected with something, isolate it on the network and then take your time and look around. Do not go in with guns blazing because you will destroy important evidence that will help you figure out how bad the situation is.
As we advise in the Linux command line forensics cheat sheet: Don’t Panic. Take your time, collect your evidence, and only then rebuild the system after you’ve figured out what is going on and how the attack happened (root cause analysis). What’s the point of rebuilding a box if you don’t know how the person entered and whether they’ll just do the same thing again immediately when you bring it back online?
Finally, did you know that Sandfly’s agentless Linux security bot automatically hunts for deleted processes and grabs forensic data about the situation for you? Why not give it a try?