Document Actions
Post-mortem on site downtime of 20100614
Up to Support
Posted by
This is just an FYI on what happened with the server yesterday...
I'm fairly certain that a brute force ssh attack succeeded in breaching a shell account on the system. The sshd logs show a successful login after multiple failed attempts. The account had no home directory, so once inside the attacker would only have had /tmp available to them with write permissions. I believe (but can't confirm) that they sftp'd one or more tools into /tmp and proceeded to run them in an attempt to escalate the breech. In particular, when I detected the attack there were dozens of active processes named 'slowdssh'. There is nothing on the system that provides 'slowdssh' and in fact (amazingly) google knows *nothing* about any such process.
Sadly, in my haste to recover from the attack, I didn't think to archive what was in /tmp, and when I rebooted the system, anything in there was wiped.
As I was telling AJ over chat when it happened, I had temporarily lost my ability to su to root, but that occasionally happens 'normally' when nscd crashes (which it has been prone to do). After rebooting, I didn't detect anything amiss about the system, including all of the account and credential information (I did a full dump and diff of my ldap db, and there were no changes dating back over a week).
So my hope is that the attack succeeded only in gaining a temporary beachhead, but that the attacker was subsequently unable to do anything permanent.
My evening was then spent setting up fail2ban, which I stupidly had never done.
fail2ban for the win. It's now happily swinging a heavy ban cudgel on any IP that even looks like it hasn't showered and shaved in the morning.
I also buttoned up my sshd config even tighter than it already was (namely restricting ssh access to a short manual list of usernames). And I deleted some old shell accounts that I know are no longer used. The way the ssh brute force attacks work is by trying common usernames and then poking away at any that are discovered to exist on a given system in a (slow) attempt to get lucky with a dictionary password attack.
AHH now i understand thats exactly what i thought had happened. I was going to repair the server from my cell phone myself but I too lost my ability to su to root and on second thought ,,better let Hoss handel this since I have no idea what so ever what the hell he just said. Hoss I continue to be amaized thanks for everything you do for us..sammy
I have my external ssh going to a VM that just hosts my simple web server. Hopefully, there is little obvious incentive to make a second jump. Of the hundreds of ssh requests a day I get, they are almost exclusively for root or without name. I haven't seen a username sweep in a long time. I block root and also filter on known places to ssh from. Also, by using openDNS, I effectively get blocks from their known list of bad places:
host name/name mismatch: 189-72-11-242.sance300.ipd.brasiltelecom.net.br != hit-nxdomain.opendns.com