Another title might be “A Website’s For Life, Not Just For Christmas”. It’s also technically a Lesson From ’12, but hey, it’s still January, so near enough.

I made the mistake of not regularly updating one of the applications installed on this server for a few months. Zen Photo, which runs the gallery where I host my photographs, had a pair of lethal security holes thanks to some insecure versions of 3rd-party code (TinyMCE and ajaxFilemanager) which resulted in a rather ingeniously stealthy hack being deployed without me noticing.

UPDATE: acrylian from ZenPhoto mentions that TinyMCE itself was not involved in the breach, just the ajaxFilemanager plugin.

Plus, I’m fairly sure I wouldn’t have detected it, if it hadn’t been for the security breach at my hosting provider, Dreamhost, a few days ago.

Some “suspicious activity” resulted in a full password reset on all FTP/shell accounts, but I decided I would refresh the passwords on my blog, Zen, and MySQL too, just to err on the side of caution. And hey, with my current regime in place, changing passwords is a painless triviality. But while doing so, I typed an incorrect URL while trying log into WordPress.

Instead of a 404 error, I was immediately bounced to a server in Russia. What the sh..?

Taking a look at my .htaccess files via SSH revealed an alarming change – a set of cunning redirects had been wrapped around the existing code. These redirects trigger only if

• The USER_AGENT is that of a search engine (thus pushing all crawler traffic away from my site)
• You type in an incorrect URL

Very neat. The site owner will only notice a problem if they happen to incorrectly enter a known URL – had they left out the ErrorDocument changes, I wouldn’t have noticed this at all.

This all shows my web programming naivety somewhat – this hack has been around in various forms for quite a while. The Zen Photo holes had allowed someone to drop a web shell onto my server – effectively a PHP file that gives full file-system access to anyone, allowing them to monkey with whatever they wished.

More alarming still was that after cleaning the .htaccess damage, in 30 minutes the redirects were back. I would presume automated vulnerability-scanning tools are in play to find those insecure Zen Photo components but clearly they are also used to keep the damage installed, should anything try and restore the system without closing their way in.

Repair

The first step was to lock out and further damage – reinstalling Zen Photo with an up-to-date version. I also deleted the TinyMCE and ajaxFilemanager plugins, despite assurances that they had been made-safe. Once bitten. I will likely go and trim off all the other plug-ins I don’t use, whittling down the total amount of executable code to the bare minimum.

Step two – remove any dropped PHP web shells. The web shell – whose source similar to this one – started in ‘FilesMan‘ mode, so I decided to scan for that. After a bit of research, I discovered it would be wise to also search for “eval(base64_decode“, used frequently to deploy ‘encrypted’ (base64 encoded, rather) malware scripts.

find . -type f -exec grep -l "FilesMan" {} +

Once those are gone, it’s time to make sure all the .htaccess files are restored, either through backup restore or manually. To make sure I got them all, I also recursively searched for the string “.ru/” (as all the redirects had been to a site in Russia)

find . -name ".htaccess" -exec grep -l ".ru/" {} +

As an extra quick check, I also did a search to see if anything had any rogue permissions set in the aftermath.

find . -perm +002

This checks to see if there is anything which has the public-writable permission bit set.

Putting the site through http://sitecheck.sucuri.net/scanner/ showed that the infection had been removed. For now.

Like the time you first lose something important due to not having a back-up strategy, this has taught me to be much more aware of the integrity of my site – if anything, just occasionally typing in an invalid URL, just to check…