Suggested musical pairing: Judas Priest – Breaking The Law
At approximately 19:00 yesterday evening, our automated alert systems for ListenOnRepeat starting pinging off alerts to the tech team that our CPU utilization was rising rapidly on our frontend server clusters. Fortunately, since we utilize Elastic Load Balancers, our frontend servers were seamlessly scaled up (absorbing the attack, albeit at marginally increased cost) while I assessed the situation.
Dumping the frontend access logs, it became immediately apparent that we were under DDoS attack via a reflective attack, utilizing the (default: ON) Pingback functionality in the popular WordPress CMS. In short, by spoofing requests to thousands of WordPress blogs, the attacker was causing these WordPress blogs to query our servers repeatedly, driving huge amounts of traffic at our frontend and tying up resources.
WordPress Pingback Reflection DDoS attacks are easy to identify (and, fortunately, relatively easy to mitigate once identified). Since this reflection attack leverages WordPress CMS installs in order to function, the attacker has absolutely no control over the useragent sent in the pingback request, and can therefore be easily identified by the following format requests in your httpd access logs
[X] - - [24/Nov/2017:19:14:46 +0000] "GET / HTTP/1.1" 200 4110 "[X]" "WordPress/[VERSION]; [REFLECTION URL]; verifying pingback from [ATTACKER IP]"
In case you’ve come here seeking a mitigation, I’ll outline that immediately. Since, as outlined above, the attacker has no control over the reflection HTTP headers, simple useragent dropping of requests in your frontend server (in our case, Apache) handles the issue.
For us, adding the following to our configuration mitigated the problem:
BrowserMatchNoCase WordPress wordpress_ping
BrowserMatchNoCase WordPress wordpress_ping
Deny from env=wordpress_ping
Resulting in 403 Forbidden errors serving to all WordPress queries to our frontend servers.
During the rapid rush to identify and mitigate this DDoS attack, I only managed to grab log fragments for the first 8 minutes of the attack (this was due to Elastic Beanstalk subsequently down-scaling our front-end environment after the attack). However, during just the first 8 minutes of the attack, we received ~173,000 server requests, averaging ~22,000 per minute. All of these requests were via SSL, and were tying up significant resources handling SSL handshakes for every single request.
Unique WordPress Hosts
During the recorded period, our servers were ‘attacked’ by 13,713 unique WordPress hosts. The majority (88%) of these hosts sent multiple (more than one) pingback request in the 8 minute interval, with a pretty shocking 20% of hosts sending >20 requests during the 8 minute period.
The most frequent pingback host sent a grand total of 127 separate pingback requests (one every four seconds) during the logged period.
Since the request helpfully includes the WordPress version, analyzing the associated WordPress version of the hosts was relatively simple. The most common (by far) WordPress version was WordPress/4.8.3 (released October 31st 2017) although it was a little terrifying to see a fair few WordPress/3.8.X versions (some not updated since 2013…)
Since WordPress/3.9, WordPress has helpfully appended the origin IP of the pingback request to the request (I’d prefer you just, y’know, stopped this behaviour by default, but who am I to dictate…) which means we are able to analyse the IP addresses of the originating attackers.
Interestingly, despite clocking in a total of 911 unique originating IP addresses, if we look at the frequency of attacks from the top ten IP addresses…
We can see that a whopping 162,156 requests (94% total) came from just five originating IP addresses. The remaining top 10 originating requests were non-real (eg loopback, internal network, internal Docker) addresses, although, interestingly, a significant volume of the remaining 6% did come from a large spread of seemingly ‘real’ IP addresses.
My guess here would be that their DDoS platform attempted to use unreliable open proxies before falling back just to sending from the host they were running from.
Unsurprisingly, these IP addresses have been frequently reported for doing a ton of sketchy shit and appear to be owned by “safevpn.io”, an equally sketchy looking self-toting “logless” VPN. Seems highly likely that the attackers were running the reflection-DDoS script through SafeVPN in order to mask their real origins.
All in all, reflective WordPress pingback DDoS attacks are easy to mitigate, but can be quite a pain if you’re (A) not familiar with them and (B) don’t have adequate alert systems to warn you of an attack before damage is already being done.
I’m pretty adamant that WordPress should disable pingback functionality by default (what possible case can be made for not doing so?) or at absolute minimum modify the pingback functionality so it has some type of rate-limiting on external server connections (once per four seconds…really?).
If you’re not running WordPress, I’d suggest hardening up your server configuration by blacklisting the WordPress pingback useragent pre-emptively.