Security

Tales from the Server: When a Hack Cleanup Goes Right

July 24, 2012   ·   By   ·   4 Comments   ·   Posted in Security

We clean up a lot of hacks. We actually spend most of our time cleaning up hacks, and helping customers recover and secure their websites from hackers. It’s a fun, challenging job, and we love doing it. It’s interesting – the tricks hackers will use to throw you off their trail, or trying to track down how the hacker gained access in the first place.

In the spirit of sharing interesting stories, I figured I’d write a bit about a hack cleanup we took care of late last night.

What happened?

This particular hack cleanup started similarly to many that we get – the owner of the site had been hacked, but was pretty technically savvy, and removed the symptoms of the hack. From the frontend, the site looked clean again. However, he knew better than to call it a day – so he got in touch with us to have a look at the server, to see if we could make any suggestions about how he could avoid the situation in the future.

We see this a lot. Site owners are often savvy enough to clean up the symptoms of the hack they’ve been hit with, but they’re left with a justifiably uneasy feeling, and they want a little reassurance that they got the whole problem. In the vast majority of cases, there’s at least one shell (hacker’s access point) left behind, giving the hacker access whenever he decides to come back and make trouble.

This case was no exception. We handle these types of cleanup by logging into the server, and running a custom filescanner to see what we can dig up.

Scanning time

So, I logged in and scanned the server. Not much suspicious going on, and nothing matching any of our malicious code definitions. The WordPress install looks generally vanilla, with a few uninteresting exceptions. So, we’ll need to do some manual review to see what we can find. Again, a pretty common occurrence.

Sniffing out Suspicious Code

In this case, 1 suspicious looking file showed up pretty quickly. The file was located in the web root, and was called “.stats.php”. Guessing at suspicious files just by filename is a bit of an art – and this one is a great example. First – a file called “stats” in the web root? Doesn’t seem too unreasonable, after all – lots of people have various (sometimes home brewed) stat tracking programs. Still – in the web root? Further – tracking website statistics can be a pretty complex job – here we’ve got just one file, and it’s apparently a stat tracking program? This is starting to feel suspicious.

Aside from things just not quite adding up about a lonely “stats” file in the web root, it starts with a “.”. Again, not totally suspicious in itself – occasionally other filenames start with a dot, most notably the ubiquitous .htaccess. But files starting with a dot have a special property – on most systems, files starting with a dot are “hidden” by default. They’re not hard to find, or secured, but they’re also often not visible from an ftp program with default settings. So we’ve got a file that feels a little suspicious, and now it looks like maybe the author is trying to hide it from ftp clients? This could still be a legitimate file, but it feels pretty unlikely.

So what does it do?

So lets get right into it – What is in the file? What exactly does it do? The only question we really want to answer is: Does this file give access to people who shouldnt have it? Theres only one way to find out: read the code. Oftentimes you’ll open a file like this and find the whole thing has been encrypted or obfuscated, which is a pretty good indicator that the code is malicious. When the code isn’t obfuscated, it’s more interesting, because you’re tasked with figuring out the code’s function. In this case, things started out pretty normally, but suspicious things kept coming up – things like functions to figure out what software the server was running, some sort of file archiver/unarchiver, and finally, a bit of code that appears to let you “run commands against the server”. That’s it – this is a shell. No question. Just as a second verification, I load the file up in a browser to see if I can actually see it in action. Now I’m sure – definitely a shell.

What else am I missing?

Now I’ve got a shell which didn’t get caught by our scanners, and I’ve never seen before. The first thing I do is put together a definition to catch this particular shell, and rescan the site to see if it’s anywhere else on the server. Sure enough, it is – the main akismet plugin file (which is included with WordPress by default) has been replaced with another copy of this shell. This is not an uncommon practice – hackers will sprinkle shells all around the server, hoping that even if you catch one, or most of them, you’ll miss at least one, and they’ll be able to maintain access.

Akismet is a pretty common target, specifically because it’s a plugin included with WordPress. Because of this, a hacker can set up his scripts to always drop a shell in the akismet folder, and have a very good chance that the folder exists. Further – this file won’t get overwritten when WordPress is updated. Lastly, the site owner won’t get bugged about updating akismet (should an update be released), which would overwrite the shell, because wordpress can’t recognize this folder as akismet anymore. The plugin gets silently deactivated (I say silently, but there *will* be a warning the next time the owner visits the plugin page), and the shell goes on living.

So, how did it get there?

This is where things tend to get messy. The vast majority of sites do not have access logging turned on, or if they do, they’re only logging that day’s server activity. In this case, the modification to the akismet file had been made nearly a month prior to my arriving at the server. Fortunately, this was a pretty savvy site owner, who had access logging turned on and humming – so there’s a fairly good chance I’ll be able to track down the initial entry point (or “attack vector”, if you’re into jargon).

So, first I start looking for what ip addresses have accessed either the .stats.php file in the web root, or the akismet.php file. This lets me compile a list of ip addresses that are likely involved in the hack. Once I’ve got that, I start checking the access logs to see what else these ip addresses have accessed – hopefully pointing me at

  • other malicious code
  • other entry points
  • where and how the hackers first made contact

Unfortunately, you can’t always find all of these things. Hackers switch ip addresses and use other tricks to throw me off their trail. In this case, however, it was clear as day. After filtering out the more mundane access records (they were posting lots and lots of data to these shells in this particular case), I found this:

So: The hackers didn’t drop this shell on the server through some obscure vulnerability, or through a hole in the server OS – they just logged into WordPress. Not only did they log into WordPress, they did so in one try. Clearly, they had login credentials. (Note: How they got login credentials is another mystery, obviously. I’ll get to that shortly).

After logging in, they headed over to the plugin editor and replaced akismet with a web shell. Immediately after that, they used the new akismet shell to create the .stats.php file – and at that point, they were off to the races.

But how did they get login credentials?

I don’t know. Unfortunately, the trail goes cold at this point – I cant find evidence of access from this set of ip addresses before the login. On top of that, they could have got these credentials from a plethora of places: Accessing the blog from an insecure network, malware on a personal computer, a compromised email address where lost passwords are sent – there are many, many possibilities.

So, how could this have been prevented?

In this particular case, a couple of things might have stopped the attackers. All come with their own drawbacks.

Preventing the shell drop

The real mess started not when the user logged in, but when they replaced the akismet file with a shell. This gave the hackers full access to the server. This was allowed by the WordPress plugin editor, and the same could have been achieved with the theme editor. This can be stopped in a couple of different ways. First, WordPress provides a method of disabling file modification with a constant you put in your wp-config.php file:

   define('DISALLOW_FILE_EDIT',true);

However, the hacker could just upload a plugin of his own, using the plugin installation feature of wordpress. That too can be disabled, along with plugin updates, and theme update/installation, with this constant:

   define('DISALLOW_FILE_MODS',true);

What are the drawbacks here? You’re forced to use FTP to make file changes. I prefer to work that way anyway. However – because of this, you can’t use the WordPress automatic updater, making it more likely that you end up with out of date plugins, which is a more likely source of vulnerabilities and attacks. Feels like a catch-22. Personally, I’d prefer to leave the WP automatic updater in place, and avoid neglecting plugin updates.

Read more on those constants here

The other way to prevent the shell drop would be to ensure that the web user (this is the filesystem user that apache/php use to run) doesn’t have write access to the server. This is really a scorched earth technique though – because now you can’t modify files, update plugins, or even update wordpress. If you need to make absolutely sure hackers can’t drop shells from the web, this might be a good technique. Be warned: It will make for a massive headache when updating plugins, WP core, etc.

Preventing Admin Access

Clearly, the hackers had the site login credentials – but because we have no idea how they got them, we’re going to ignore that and move on to how the site owner could have protected the site anyway.

The hackers had to log into WordPress to gain access in teh first place – so what if we could have stopped them there? Using the site .htaccess file, we can actually block either specific ip addresses (blacklisting) from accessing the admin section, or blocking all ip addresses except those you trust (whitelisting). Because the effort required to use a different ip is minimal, in this situation, blacklisting is almost useless. The far better choice would be to whitelist your ip address, meaning that in order to access wp-admin, you have to be doing so from your specific internet connection.

The drawbacks here are pretty obvious – what if you want to edit something from your phone on the road? You’re out of luck. Work from a coffee shop? No dice. Have your brother writing a guest post from indiana? He can’t log in either. Of course, you can add all of these other ip addresses to your whitelist, you just have to recognize the hassle involved.

Overall, your site’s security is a constant balance between security and convenience/functionality. The decision about how to handle it has to be made with your specific site’s needs and risks in mind.

Wrapping up

So: We got the admin password changed, checked for any unexpected users in the wordpress admin, removed .stats.php, and uploaded a fresh copy of akismet, and we’re good to go. Obviously, since this was just last night, we’ll need to keep an eye on the site (assuming you’ve totally locked out the hackers is the best way to look and feel really, really dumb), but at this point, I’m relatively confident that we’re all set. Huzzah!

4 Comments
  1. Peter – what a GREAT post! It reads like a non-fiction novel (I guess that is a contraction in terms, but you get the idea!).

    Loved the line, “Overall, your site’s security is a constant balance between security and convenience/functionality. The decision about how to handle it has to be made with your specific site’s needs and risks in mind.” This clearly defines how security should be handled.

    I will be back to read more!

    Paul.

    • Glad you enjoyed it Paul. It’s nice to know that somebody finds this sort of thing as entertaining as we do:)

      • Entertaining AND informative (probably the later more then the prior!)

        Thanks again!

        Paul.

  2. Great post!

    I always force clients to login to WordPress via ssl (https://) – even if they do not have a signed certificate, the annoyance of the little popup is worth the added security of that login info being sent with encryption.

    Also, if you have ssh access to the host, try running the command, ‘last username’ where username is their ftp username. You would be surprised on how many times the vulnerability is actually on the user’s computer and some malware stole the FTP details from their personal computer (Filezilla and a lot of FTP clients store user/pass as plain text). This command will show a list of all the ftp logins from the user with IP address. You might be surprised how many show IPs from countries you’ve never heard of before haha.

Submit a Comment