Investigating Linux Process File Descriptors for Incident Response and Forensics
Let’s talk about Linux file descriptors and how to investigate a malicious process using them.
What Is a File Descriptor?
Since the inception of Unix (on which Linux is based), everything is a file. We can quibble about the specifics, but mostly this is true and for our purposes is all we need to know. A file descriptor is a handle used by the operating system to access a file or other I/O mechanism. It is the same as a phone number you dial to get a specific person, except on Linux the number will give you a pointer to a file.
We’re simplifying this for our discussion, but just know that basically all processes on Linux that are running are going to have some kind of file descriptor open. These file descriptors can be any legal file or resource on Linux such as:
Files
Network sockets
Named pipes
Devices
On Linux the most basic file descriptors you’ll see open by most processes will be stdin, stdout and stderr. These allow the process to communicate back to the terminal and take data input (stdin), output data to the terminal (stdout) and pass out errors (stderr). But more than this, the file descriptors will also show what files, sockets, etc. the process may be talking to when running. Often live malware will write data to the disk it is stealing or using for other reasons (such as data logs).
The key point here is if the process is running you can quickly see what files it has open and can use that information to determine if it is malicious and what it may be doing.
Show Me the File Descriptors Already
To find the open file descriptors of a process, we will go to our old friend the /proc file system. The data we want is here:
/proc/<PID>/fd
The basic format for listing the open file descriptors of a process is simply:
ls -al /proc/<PID>/fd
Once you do this you’ll see what files the process has open and if any are interesting.
Suspicious Linux Process Example
Below we see a suspicious process called tmpwrk and it has a particular interest in a file under the /tmp directory. We can use this information to view the contents of that file and determine what may be going on.
We look under the /proc/<PID>/fd directory and do a simple ls command to see what the process has open:
We see a file and a socket open. The file has a very strange name of /tmp/.data and we want to see what it is. We can use standard file tools on the file link itself or the file it links to directly.
Below we cat out the file and see it holds some secret data. This can help confirm that the binary is in fact malicious. We can view the file, copy it, run hashes on it or anything else. It’s a standard file.
What About the Open Socket?
The open socket can also be investigated by using the number in [brackets] which is the inode. You can run it through netstat in a variety of ways. Andrew Case contributed in a tweet this command:
netstat -eepan | grep <INODE>
Where INODE is again the number in brackets from above. This will show you the kind of socket the process is using (e.g. network).
What About lsof?
lsof is a tool that shows all open files on a system. If not installed, you will have to use the methods above. You can use lsof with the following:
lsof -p <PID>
lsof -c <command name>
The lsof command will show all open files in one easy to use way.
Why Not Use lsof?
There are two reasons why using lsof may not be the first thing to do when looking at a suspicious process:
It may not be installed and you don’t want to alter a compromised machine by loading new packages.
The lsof command is not commonly called and could alert malware or an intruder about investigators present.
The last point may be a bit academic, but the simpler ls and cat commands are very common and hard to tell why they may be used. lsof on the other hand is pretty specific and there is always the chance it is being targeted by malware to avoid detection or worse. This is a small risk, but it is there.
Let the Malware Help You
The biggest takeaway from this is a malicious process is happy to tell you what it is interested in if you take the time to look at what files it has open. You can do this without the risk of attaching debuggers and it is far safer and faster.
The second takeaway is to never go in and kill a suspicious process on Linux until live process forensics have determined what it may be doing. If you go in guns blazing you will not be able to see what the malware may be up to, what files it has open, who it is talking to and other useful pieces of data. Put away the kill -9 and take your time when investigating a Linux process. It will save you lots of effort.