500

By  on  

Over the past four or so months, I've been having a hell of a time with seeing 500 errors on my site.  And while I'm on my site a few hours a day, the amount of 500s my visitors must have seem is probably huge.  And that's a problem.  A big, annoying problem.  The following details the situation from start to solution, so read on if you'd like to know how it all came and went.

The Symptoms

How did I know there was a problem and that it wouldn't be an easy fix?  Well...

  • The 500s were being seen on davidwalsh.name and dwf.tw (my short URL / redirector)
  • While sometimes a page would come up, images and other assets would 500
  • There was seemingly no pattern -- sometimes I was fast, others slow, others would just 500
  • I ran a Google Page Insights report to see if the problem was just something I was seeing, and Google reported a 500 error.  Not good.

So all of this started happening more and more frequently.  And I was confused as hell.

The Variables

It was really hard to point to one specific issue that could have introduced the 500s.

  • Around that time I had added more advanced gzipping to the site.  I kind of knew what I was doing, and I followed my host's (Media Temple) instructions on how to do it.
  • I had upgraded to WordPress 4
  • Media Temple had just upgraded my server
  • Blog traffic was up about 10%
  • WordPress plugins (need I say more?)

All of these things were happening so what could I easily point to?  Nothing.

What I Did

As always I tried to fix the issue myself using process of elimination.

  • I disabled any fringe WordPress plugins
  • I removed the gzip directive I had added to the server
  • I crunched all of my assets even more
  • I removed non-mission-critical directives from my .htaccess file

And still none of this made a dent.  Still seeing 500 errors left and right.  WTF.

Media Temple (Mostly) to the Rescue

At this point I needed to contact Media Temple:  I'm not a server admin and the issue was clearly beyond me.  So the volley of communications with Media Temple went as follows.

  • I was first told that they could not reproduce the error, which started me down a path of fury
  • I was then told to look at the error_log, and this annoyed me.  The log had numerous warnings, notices, and a few errors, but nothing that should take down my site the way it had been.
  • They then ran a site diagnostic for me...but on the wrong (old) server.  Ugh.  Not pleased with that but we were getting somewhere so I was fine with them restarting on the newer server.
  • Media Temple did further digging, at my tearful pleading, and finally found the issue:  I was being DDOS'd from China.  On a Sunday afternoon, when traffic is always low, there were 2500 (!) open connections to my site.  I was also being hotlinked by a very popular site, apparently.  Damn.

So there we go -- being DDOS'd will most definitely kill your site.  So now what?

500 Prevention and Moving Forward

Here's the gameplan for moving on:

  • For $79 Media Temple installed fail2ban, a utility which sniffs out malicious attempts and bans offenders.  I receive an email for every ban and thus far I've received 2 dozen.  Most are malicious attempts but a few of them have been bans from failed WordPress logins...from Turkey.  Best $79 I've ever spent (probably not true).
  • Media Temple also installed (D)DOS Deflate.  "This program will ban connection based DDoS attacks when the connection limit of {x} connections from an IP address is exceeded. This ban will remain in place for 30 minutes after each occurrence."
  • Media Temple also moved my SSH/SFTP port -- a good move I wish I'd thought of
  • I'm going to start using a CDN again.  I stopped using one because it took too much time for the CDN to refresh when I was in heavy theme development, but using a CDN will prevent hotlinking problems.
  • For giggles I added hotlink prevention within my .htaccess file as well:
  • 	# F*** hotlinkers
    	<IfModule mod_rewrite.c>
    	RewriteEngine on
    	RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?davidwalsh.name [NC]
    	RewriteRule \.(js|css)$ - [NC,F,L]
    	</IfModule>
    

    Notice I didn't add image files because I want Twitter, Facebook, and other social networks to pick them up in Twitter cards, etc

Closing Thoughts

A few opinions and ideas to close my post out:

  • I need to stop thinking problems will work themselves out -- I should have attacked this earlier. My apologies to all of you.
  • Take the time to secure your shit -- it only takes a few hours and can save everyone loads of pain.
  • This is going to make me sound like a world class diva but I'm going to say it anyway:  when you sponsor a blogger, and you know everyone knows the blogger uses you, make a big effort to ensure everything is right.  Configure a fast server, give good support.  Media Temple and I got there in the end but the tediousness of the situation was frustrating.
  • I still love Media Temple
  • Please don't DDOS me

Thanks for reading and good luck in your own fights!

Recent Features

  • By
    Regular Expressions for the Rest of Us

    Sooner or later you'll run across a regular expression. With their cryptic syntax, confusing documentation and massive learning curve, most developers settle for copying and pasting them from StackOverflow and hoping they work. But what if you could decode regular expressions and harness their power? In...

  • By
    Facebook Open Graph META Tags

    It's no secret that Facebook has become a major traffic driver for all types of websites.  Nowadays even large corporations steer consumers toward their Facebook pages instead of the corporate websites directly.  And of course there are Facebook "Like" and "Recommend" widgets on every website.  One...

Incredible Demos

  • By
    PHP / MooTools 1.2 Accordion Helper

    The MooTools Accordion plugin seems to be the plugin that people seem to have the most problems with. It's an awesome plugin, so I can see why so many people want to use it, but I think that may be part of the problem.

  • By
    Degradable SELECT onChange

    Whenever I go to Google Analytics I notice a slight flicker in the dropdown list area. I see a button appear for the shortest amount of time and the poof! Gone. What that tells me is that Google is making their site function...

Discussion

  1. Blimey – that sounds so very frustrating! I sympathize enormously… these kinds of problems are just so darn hard to diagnose – luckily MT came through for you! Incidentally, are you on one of their VPS setups or a dedicated server? Just curious…

  2. CDN-wise, use site links during prep and the switch to CDN links when you are happy (assuming you are doing source pull)

  3. OZ

    i never saw so noobish hosters, who can’t determine ddos attack, lol.
    Sorry for bad news, but 2500 connections is a VERY small number. You should be able serve 10-20 thousands. But it’s not so easy with wordpress, I know. But it means can be ddosed again by any script-kiddy.
    Good idea to use CDN, also consider move to google app engine ir heroku, for scaling easy.

  4. I had a nightmare with 500 errors too. In my case every night from 2:00 to 3:00 AM I had hundreds or thousands of 500 errors (I could see in my log). I contacted with my provider (1and1, pray if you have problems) and they can’t reproduce the error like you wrote :(, but if the errors was in the same hour it could be due some maintanance process, wasn’t it?. Well, some day they disappeared and I still don’t know why.
    Since then I have panic of the 500 page (I see 500 errors….)

  5. Use Cloudflare man .

  6. Binyamin

    David, you can’t block (“hotlink”) your website CSS and JS from search engines like Google, it would affect your page rank or even penalize. Use other ways to faith with various website attacks.

  7. I do agree that specialized managed hosting services like Media Temple makes your website secure and lesser the chances of DDoS attacks but everyone can’t afford them.

    I would suggest you to try a CDN e.g MaxCDN or CloudFlare to do away with 500 errors.

  8. I agree 100% with the first item in the list: disabled any fringe WordPress plugins. How many times I’ve seen a rogue plugin cause unforeseen problems.

Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!