500

Over the past four or so months, I've been having a hell of a time with seeing 500 errors on my site.  And while I'm on my site a few hours a day, the amount of 500s my visitors must have seem is probably huge.  And that's a problem.  A big, annoying problem.  The following details the situation from start to solution, so read on if you'd like to know how it all came and went.

The Symptoms

How did I know there was a problem and that it wouldn't be an easy fix?  Well...

• The 500s were being seen on davidwalsh.name and dwf.tw (my short URL / redirector)
• While sometimes a page would come up, images and other assets would 500
• There was seemingly no pattern -- sometimes I was fast, others slow, others would just 500
• I ran a Google Page Insights report to see if the problem was just something I was seeing, and Google reported a 500 error.  Not good.

So all of this started happening more and more frequently.  And I was confused as hell.

The Variables

It was really hard to point to one specific issue that could have introduced the 500s.

• Around that time I had added more advanced gzipping to the site.  I kind of knew what I was doing, and I followed my host's (Media Temple) instructions on how to do it.
• I had upgraded to WordPress 4
• Media Temple had just upgraded my server
• Blog traffic was up about 10%
• WordPress plugins (need I say more?)

All of these things were happening so what could I easily point to?  Nothing.

What I Did

As always I tried to fix the issue myself using process of elimination.

• I disabled any fringe WordPress plugins
• I removed the gzip directive I had added to the server
• I crunched all of my assets even more
• I removed non-mission-critical directives from my .htaccess file

And still none of this made a dent.  Still seeing 500 errors left and right.  WTF.

Media Temple (Mostly) to the Rescue

At this point I needed to contact Media Temple:  I'm not a server admin and the issue was clearly beyond me.  So the volley of communications with Media Temple went as follows.

• I was first told that they could not reproduce the error, which started me down a path of fury
• I was then told to look at the error_log, and this annoyed me.  The log had numerous warnings, notices, and a few errors, but nothing that should take down my site the way it had been.
• They then ran a site diagnostic for me...but on the wrong (old) server.  Ugh.  Not pleased with that but we were getting somewhere so I was fine with them restarting on the newer server.
• Media Temple did further digging, at my tearful pleading, and finally found the issue:  I was being DDOS'd from China.  On a Sunday afternoon, when traffic is always low, there were 2500 (!) open connections to my site.  I was also being hotlinked by a very popular site, apparently.  Damn.

So there we go -- being DDOS'd will most definitely kill your site.  So now what?

500 Prevention and Moving Forward

Here's the gameplan for moving on:

• For $79 Media Temple installed fail2ban, a utility which sniffs out malicious attempts and bans offenders.  I receive an email for every ban and thus far I've received 2 dozen.  Most are malicious attempts but a few of them have been bans from failed WordPress logins...from Turkey.  Best $79 I've ever spent (probably not true).
• Media Temple also installed (D)DOS Deflate.  "This program will ban connection based DDoS attacks when the connection limit of {x} connections from an IP address is exceeded. This ban will remain in place for 30 minutes after each occurrence."
• Media Temple also moved my SSH/SFTP port -- a good move I wish I'd thought of
• I'm going to start using a CDN again.  I stopped using one because it took too much time for the CDN to refresh when I was in heavy theme development, but using a CDN will prevent hotlinking problems.
• For giggles I added hotlink prevention within my .htaccess file as well:

	# F*** hotlinkers
	<IfModule mod_rewrite.c>
	RewriteEngine on
	RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?davidwalsh.name [NC]
	RewriteRule \.(js|css)$ - [NC,F,L]
	</IfModule>

Notice I didn't add image files because I want Twitter, Facebook, and other social networks to pick them up in Twitter cards, etc

Closing Thoughts

A few opinions and ideas to close my post out:

• I need to stop thinking problems will work themselves out -- I should have attacked this earlier. My apologies to all of you.
• Take the time to secure your shit -- it only takes a few hours and can save everyone loads of pain.
• This is going to make me sound like a world class diva but I'm going to say it anyway:  when you sponsor a blogger, and you know everyone knows the blogger uses you, make a big effort to ensure everything is right.  Configure a fast server, give good support.  Media Temple and I got there in the end but the tediousness of the situation was frustrating.
• I still love Media Temple
• Please don't DDOS me

Thanks for reading and good luck in your own fights!