RTCW Reflected DDoS

So about mid afternoon I got a nagios alert stating that rtcw was down on one of our servers, this is highly unusual and I thought maybe someone had restarted it.

Further investigation however proved otherwise, the server was up and running but had full input an output network buffers.

I wondered what was going on so I jumped into the screen session the console runs under and saw loads of “NET_SendPacket ERROR: Resource temporarily unavailable to X.X.X.X:xxxxx” entries, mostly for the same IP but not all, this piqued my interest and so I started running tcpdump on the most prolific address, after firewalling it off, this is what I got:

HH:MM:SS.976507 IP SRC-IP.PORT > DEST-IP.PORT: UDP, length 14
E..*W.@.3.q…!z.)..mLm8..oN….getstatus…..

Not worrying on it’s own but when they’re flying past on the screen so quick it makes me wonder, there was roughly 0.0001 seconds between each request so that’s about 1000 request/s each requiring a response.

At first I though we were seeing a DoS, then saw more addresses have a go for a few minutes, so now it’s a DDoS. Then someone said something which got me thinking, they’d made the link between the reverse DNS of the offending IP and a situation pbbans.com had a while back. This wasn’t a DDoS meant for us, this is meant for the IP that seems to be the abuser.

Wikipedia calls this a DRDoS (Distributed Reflected Denial of Service) but I remember using DR DOS and don’t want to go anywhere near that.

The principle behind this is that UDP traffic is stateless, each packet is sent on the network and forgotten about, this makes it far much easier to spoof the source IP as there’s no data required to be passed back to the true originator of the IP packet.

So now we have a situation where our game server is taking part in a DDoS against a ggc-stream.com server. We’ve got customer’s trying to use the server so I can’t just turn it off for a bit, and this query is important to the game but I’ve played with this before using iptables packet matching, this is the perfect opportunity to expand upon that and add rate limiting.

Like before I used the packet matching however I wasn’t concerned with the size of the packet, but I did want to limit the rate they’re accepted.

This meant removing the “-m length –length 64:65535” and creating 2 rules for each match, the base rule ended up as

iptables -A INPUT -p udp -m udp --dport 27960 -m string \
    --hex-string "|ffffffff676574696e666f|" --algo bm --to 65535

The details for these are in the above link, however this time I need 2 rules, the first one will accept the packet if it’s within the rate limit threshold, the 2nd will drop the rest.

rate limiting was hard to pick, but I chose 3 packets in a minute I can alter this easily enough if the need arises, this was done with adding

-m limit --limit 3/min

Now these packets need accepting so the usual “-j ACCEPT” was added to that line, the 2nd line without the limiting in is set to DROP with “-j DROP”.

The complete rules end up as:

iptables -A INPUT -p udp -m udp --dport 27960 -m string \
    --hex-string "|ffffffff676574696e666f|" --algo bm   \
    --to 65535 -m limit --limit 3/min -j ACCEPT
iptables -A INPUT -p udp -m udp --dport 27960 -m string \
    --hex-string "|ffffffff676574696e666f|" --algo bm   \
    --to 65535 -j DROP
iptables -A INPUT -p udp -m udp --dport 27960 -m string   \
    --hex-string "|ffffffff676574737461747573|" --algo bm \
    --to 65535 -m limit --limit 3/min -j ACCEPT
iptables -A INPUT -p udp -m udp --dport 27960 -m string   \
    --hex-string "|ffffffff676574737461747573|" --algo bm \
    --to 65535 -j DROP

This is to cover the “getstatus” and “getinfo” requests which are both public.

It occurs to me at this point that you may be asking why I don’t reject the packets rather than drop them and the reason is simple, setting to reject will send an ICMP “port unreachable” packet to the source IP and it looks like we’re potentially taking part in a DDoS so the less data we send back the better.

Looking at our bandwidth graphs after was quite worrying and show just how effective this can be if not caught. We were receiving requests at about 2mbit/s and sending to the possibly faked source at over 10mbit/s maxing our ethernet link which we lock to 10mbit as the machine should never reaching that limit in normal circumstances and limits impact if something like this happens. Yes it’s nice to have a 100mbit link to the internet but if you don’t need it limit it.

Update (16th April: 00:15)

It’s looking more like this is actually a DDoS against us, or reflected against us now, a 60 second tcp dump came up with this:

9969 76.233.108.158.27005 > D.E.S.T:P: UDP, length 14
3220 88.184.4.2.27005 > D.E.S.T:P: UDP, length 14
25414 91.192.111.122.27035 > D.E.S.T:P: UDP, length 14
76402 99.239.70.147.27005 > D.E.S.T:P: UDP, length 14
25408 99.99.207.236.27015 > D.E.S.T:P: UDP, length 14

I’m now past the point of censoring the offending IP addresses, as 140,000 packets is clear abuse.

Update (17th April: 02:11)

I’ve changed my firewall rules round, I’m now passing all the rtcw/et traffic via it’s own chain to do the filtering which allows me to return the packet back into the rest of the firewall. This keeps it tidy for me.

iptables -N RTCW_ET_CHECK
iptables -A RTCW_ET_CHECK -p udp -m udp --dport 27960 -m string  \
    --hex-string "|ffffffff676574696e666f|" --algo bm --to 65535 \
    -m length --length 64:65535 -j DROP
iptables -A RTCW_ET_CHECK -p udp -m udp --dport 27960 -m string      \
    --hex-string "|ffffffff676574737461747573|" --algo bm --to 65535 \
    -m length --length 64:65535 -j DROP
iptables -A RTCW_ET_CHECK -p udp -m udp --dport 27960 -m string  \
    --hex-string "|ffffffff676574696e666f|" --algo bm --to 65535 \
    -m limit --limit 20/min -j RETURN
iptables -A RTCW_ET_CHECK -p udp -m udp --dport 27960 -m string      \
    --hex-string "|ffffffff676574737461747573|" --algo bm --to 65535 \
    -m limit --limit 20/min -j RETURN
iptables -A RTCW_ET_CHECK -p udp -m udp --dport 27960 -m string  \
    --hex-string "|ffffffff676574696e666f|" --algo bm --to 65535 \
    -j DROP
iptables -A RTCW_ET_CHECK -p udp -m udp --dport 27960 -m string      \
    --hex-string "|ffffffff676574737461747573|" --algo bm --to 65535 \
    -j DROP
iptables -A RTCW_ET_CHECK -j RETURN

Then when I want to pass the traffic into the chain, I use

iptables -A INPUT -p udp -m udp --dport 27960 -j RTCW_ET_CHECK