Grant Emery demonstrated how to recover from a compromised Linux box. The machine was installed with Red hat Linux 5.2 from a CD, which is by default vulnerable to many exploits. Prior to the Security Camp, Grant broke into a machine with conventional script kiddie exploits. During the meeting, he pulled machine off the net and began looking around as root.
Grant first checked the log files. Some root kits are smart enough to clean up logs, but most of the time there's useful information about the level of compromise and source. The absence of logs is also a clue. If the logs are simply missing, you know there's been a root compromise. Look in /var/log/.
While looking though the "messages" log file, Grant found a suspicious entry for "mountd" that contained lots of control-Ps. The log entry claims it was a "blocked attempt." Nevertheless, the exploit caused a buffer overflow and devulged root access. Later in the log file there was an interesting login as "moof" from uu.net. Not much of interest was in the "secure" log file.
Grant noted that in a root-level compromise, one can no longer trust any of the binaries on the machine. It's common practice for crackers to install custom binaries to hide their activities. Hence, Grant brought his own disk of clean binaries on a write-protected floppy. An audience member pointed out that a "clean floppy" is not foolproof. For instance, the kernel could be hacked to ignore the floppy. Or a module could be loaded. Grant has not _yet_ seen things like hacked kernel modules that would require a separate rescue disk or anything like that. These recovery tactics rely on the crackers not being extremely sophisticated.
Grant found that the cracker replaced ls, ps, netstat, and other critical binaries to hide certain processes.
To use the clean binaries, Grant brought a disk of clean binaries and copy of libc:
mount /dev/fd0 /mnt/floppy export LD_LIBRARY_PATH=/mnt/floppy
Executing "ps aux | less" looks normal, but "/mnt/floppy/ps aux | less" reveals that sshd and /lib/doserver are now running.
An audience member asked about a trojan runtime linker. Grant responded that we haven't run into such sophisticated attacks yet. What about modules? Same answer. We haven't seen a trojaned shell yet either.
There's only one other idea: Take out the compromised hard drive and mount on clean machine. This should by-pass almost anything the cracker modified.
Angie Kelic noted that a cracker once tried to recompile a kernel on a compromised machine to enable sniffing. The cracker then deleted the kernel. Not very successful. It's hard enough to compile a kernel at the console, crackers are not yet sophisticated for perfect remote kernel changes. But eventually they will be capable.
The best solution for a compromised machine: reinstall, apply patches. But watch out for red herrings like forged source addresses or arpwatch. Such fake sources can make a perfectly fine machine look like a source of an attack.
Guess and check to look for the files that the cracker may have installed. This is typically a total shot in the dark, but there are some tricks, such as looking for file names with spaces, regular files in /dev, and files with mtimes/ctimes in the right time period for the breakin. many root kits come with linsniffer which generate a log file with a consistent name (tcp.log).
look for strange names:
find / -name tcp.log find / -name ".??*" find / -name " *"
Use tripwire.
Run "strings" to determine what files actually do. Sniffer logs can often reveal the source of the attack and accounts which need passwords force-changed. If you know approximately when the attack happened, "find -ctime -<num>" can help. Replace <num> with the number of days ago the attack happened. -ctime is usually better than -mtime because it's somewhat harder to create fake ctime entries.
Most rootkits contain install scripts (but no readme file! :-)). By reading though the scripts, you can see exactly how it replaces binaries. Rootkits sometimes install ssh. "chattr" in linux FS lets you force a file not to change. rootkits do this.
Other suggestions to find cracker files: * Parse for _good_ passwords from the sniffer log. * Look at open ports (netstat -a). * Read history files. * Rebooting can lose state (e.g., ps aux). If you want to track the cracker donw, unplug the machine from the net and examine open connections. * Run "lsof" to find out what files strange procs write to. * Log interesting network activity. Sniffers can sniff the sniffers.
Now we get some crackers who know about the Athena environment and files.
Lots of places keep phone numbers and security cameras in library computers. This can trace back source via physical means with a subpoena.
Retrieve all the interesting files from the machine for later analysis. In partiular, we take syslogs so we can do followup with the attacker's ISP, sniffer logs so we can run them through sparse (or whatever other parsing tool we have) and handle any passwords that need changing, and the root kit and exploit code so we can keep a library of such things, and because we're computer geeks and like to see how these things work. ;)
Tell the user to reinstall the machine and give them pointers to security patches for linux. we also have a one-sheet that talks about some quick ways to get some security improvement.