My answers and the winners for the March 2004 Crack the Hacker Challenge: Lord of the Ring-Zero

By Ed Skoudis, March 24, 2004

 

As usual, we had some very well-thought-out answers to our challenge this month.  I'll tell you the winners in a moment, but first, here are my own answers to this challenge, which is based on an actual incident I worked about six months ago.  The bad guy was an insider who had hacked a kernel and tried to hide on the system, but was discovered by an intrepid system administrator.

•1. What two mistakes had Gollum/Smeagol and/or his tool made in this attack?

•In actuality, Gollum/Smeagol made more than just two mistakes.  Here are some of the most prominent errors he and/or his tool made:

•A) He didn't hide the directory /temp/".. ", which tipped off Skodo in the first place.  This was an obvious "gimme", as I mentioned in the narrative itself that Gollum had made this error.

•B) He left the owner of the ".. " directory as his own account!  That's what let Skodo quickly figure out who the most likely attacker was.  However, keep in mind that a particularly treacherous attacker might set the owner of an evil file or directory to another user to divert attention from the real attacker's identity.

•C) Gollum/Smeagol shouldn't have been muttering about the nature of his hack (a kernel attack of some kind) in his cubicle within earshot of the system administrator.  That's just plain dumb for an attacker to do.  But, as we all know, Gollum has a habit of announcing his nefarious intentions aloud when he thinks no one is listening.

•D) Here's where things get particularly interesting.  If you look carefully at the output from the ls commands run by Skodo, you'll notice some significant discrepancies introduced by Smeagol's RootKit.  First, look at the total size inside the ".. " directory.  Whenever you run the ls command with the "-al" flags, the first part of the output displays the total size of the directory contents assuming 1024 byte blocks.  The ls command output shows a total count of 2020!  So, the contents of this directory could be up to 2020 * 1,024 bytes, or just over 2 Megs!  Yet, the directory appears to be empty.  For an empty directory, the total size displayed by "ls –al" should be 8 (that is because the "." and ".." links each require 4096 bytes, or one default-sized block for RedHat distros).  Folks, we've found a glitch in the Matrix.  How could an empty directory take up over 2 Megs, when it should only be 8 K?  Most likely, the ls command or the kernel itself is lying to Skodo.

•E) Beyond the directory size, look also at the link counts inside of the ".. " directory.  You'll see that the link count of /tmp/".. "/. is 5, and the link count of /tmp/".. "/.. is 7.  That's just not right.  For an empty directory, the link count of "." should be 2, which is a link to itself and its parent.  So, the link count for /tmp/".. "/. is wrong! The link count for "." of 5 actually implies that there are 3 directories within /tmp/".. " (one for /tmp/".. "/., one for /tmp/".. "/.., and three others which are apparently hidden!)  Note that the parent link count (of /tmp/".. "/..) is 7, which is correct.  This value counts the number of directories connected to the parent of ".. ", which is /tmp/., /tmp/.., /tmp/".. ", /tmp/.font-unix, /tmp/.ICE-unix, /tmp/.sawfish-root, and /tmp/.X11-unix, or a total of 7.  So, the 5 is very fishy, indicating that 3 directories are hidden, and the 7 is ok.  

 

•2. Suppose Skodo is allowed to reboot the box. How can Skodo determine what really happened? What tools should he use?

•First off, as many of you noted, we cannot trust any results given to us by any tools on the box.  Therefore, the ls command shouldn't be trusted, nor should the kernel itself.  However, rebooting the box is very dangerous at this point.  It could destroy some vital evidence.  I'd rather run some other tools first, before rebooting, as described in the answer to number 3.  I really like those of you who responded that rebooting the box is a bad idea at this point.  Excellent observation!

•However, after completing the steps of number 3, if I were to do a reboot, I'd definitely boot to a forensics CD-ROM.  Several free ones are available on the Internet.  I'd use FIRE from William Salusky at http://fire.dmzs.com/, the Knoppix Security Tools Distribution (STD) at http://www.knoppix-std.org/, or the Penguin Sleuth Bootable CD at http://www.linux-forensics.com/.  Each of these tools offers a wonderful environment for analyzing a system with a kernel and tools that you can trust more than the ones on a compromised box.

•3. Now suppose Skodo is unable to reboot the Middle Earth file server. What tools should he use to determine what is really happening without shutting the box down or rebooting it?

•The one tool I rely on heavily for analyzing a compromised box without rebooting is Chkrootkit, available for free at www.chkrootkit.org.  It'll run on a variety of UNIX and UNIX-like environments, including Linux, *BSD, Solaris, HP-UX, and Tru64.  This tool would automatically note the link count problems described in Answer 1 above, which would be a huge tip-off that something was awry.  Also, by running a file integrity checker, such as Tripwire (www.tripwire.org) or AIDE (http://www.cs.tut.fi/~rammer/aide.html), we could narrow down whether the ls binary has been changed, or whether the kernel has been altered.  If the ls binary has been altered, we are likely dealing with a user-mode RootKit.  If the ls binary hasn't been altered (or its alterations are hidden from our view), we are likely dealing with a kernel-mode RootKit.  The combo of Chkrootkit, a file integrity checker, and a bootable Linux distribution for analysis is immensely valuable in doing this kind of investigation.

•4. Short of storing it on a chain around his neck, how can Skodo protect his kernel from being seized by a user on the machine?

•Ahh…. I always try to ask an open ended question in these challenges to give you guys a chance to shine.  For this one, we had some wonderful answers, but I think the best bet here would be to keep the box thoroughly and rapidly patched.  As much as possible, keep the bad guys from getting root, and we'll all be in much better shape.  There was a massive kernel vulnerability discovered in Linux in recent months that could let an attacker take over the system (http://isec.pl/vulnerabilities/isec-0013-mremap.txt).  I hope you've all patched by now.  I wonder if Skodo has?!?

•Beyond keeping the box patched, there are a variety of other kernel modifications that can be applied, as described quite thoroughly by Raul Siles in his winning answer.

 

Speaking of winners… you are probably wondering who the winners were!  Well, without further adieu, here they are.  Actually, let's do just a little bit more adieu first.

Although several people commented on the wacky Leonard Nimoy song, no one mentioned the two files in /tmp that I thought you'd find most entertaining.  Didja see the file called "Guide to Effective Parenting"?  That gem was owned by Denethor, the father of Boromir and Faramir from the Lord of the Rings.  You remember him, the guy who was eating the tomato in the movie, dribbling it all over his chin?  He sure should read that parenting file!  Also, note that Skodo himself had a file called "There and back again, got the T-Shirt".  This is likely an augmented work based on the "There and Back Again" book written by Bilbo and Frodo in the movie.  But you guys probably all noticed that stuff and just rolled your eyes.

And now, the winners:

 

Raul Siles: As always, thoroughly researched answers, with some wonderful insights into the details of the total size, link counts, and timestamps of the evidence.  Great stuff.

Joe Matyaz: Very solid answers, and an excellent plug for "Buy_Malware_by_Ed_S.txt and Hack_Counter-Hack_Training.mov"  : )

Dion Mendel: Your answers feature excellent use of plot, and good technical analysis as well.   I loved how you brought Tom Bomadil and Deagol into your answers.  Great work!

 

Congrats to all of our winners.  Your prizes will be on the way shortly.

Now, stay tuned for our late-April challenge, which will be called "Hackers of the Lost Ark"!

 

--Ed.