How did this Memcache thing happen?

Published: 2018-02-28. Last Updated: 2018-02-28 03:18:55 UTC
by Kevin Liston (Version: 1)
5 comment(s)

As pointed out earlier (https://isc.sans.edu/forums/diary/Why+we+Dont+Deserve+the+Internet+Memcached+Reflected+DDoS+Attacks/23389/) this memcached reflected DDoS thing is pretty bad.  How bad?  Well, US-CERT updated its UDP-Based Amplification Attacks advistory (https://www.us-cert.gov/ncas/alerts/TA14-017A) to add Memcache to the list of potential attack vectors.  The really telling bit is the chart that shows the Bandwidth Amplification Factor.  Before memcache was added the largest factor was 556.9 from NTP where each byte sent in to a vulnerable server would return about 557 bytes in attack traffic.  Memecache is listed as 10,000 to 51,000.  That's remarkably large.

How common is an internet-facing memcache daemon?  I did a little poking with Shodan (https://www.shodan.io/) and it had identified 100k or so systems.  Mostly in cloud provider spaces as you would expect.

So how did this happen?  I don't know much about memcache myself, so I think that makes me particulary qualified to take a stab at the answer.  Since I don't know what it is, or how to install it, I turn to google to help me out.  Buried in the wiki for memcached (https://github.com/memcached/memcached/wiki/ConfiguringServer) under the networking section one might read:

"By default memcached listens on TCP and UDP ports, both 11211. -l allows you to bind to specific interfaces or IP addresses. Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users. Using SASL authentication here helps, but should not be totally trusted."
 

But who has time for instructions, right?  What's Google tell me to do?  The top answer gives us:

Change the memcached configuration setting for CACHESIZE and -l :

  1. Open /etc/memcached.conf in a text editor.
  2. Locate the -m parameter.
  3. Change its value to at least 1GB.
  4. Locate the -l parameter.
  5. Change its value to 127.0.0.1 or localhost.
  6. Save your changes to memcached.conf and exit the text editor.
  7. Restart memcached. 

Source: http://devdocs.magento.com/guides/v2.1/config-guide/memcache/memcache_ubuntu.html 

Hey, that looks right.  What about Digital Ocean's instructions (they have a lot of these open)?  https://www.digitalocean.com/community/tutorials/how-to-install-and-use-memcache-on-ubuntu-14-04

It has this as how it's started:

/usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1

That looks fine too.  So it's not a rash of bad instructions on the Internet.

Okay, how about stackoverflow/serverfault?  In the top question about memcache (https://serverfault.com/questions/347621/memcache-basic-configuration) we find an illuminating answer:

"The -l option regulates to which interfaces memcached will listen to, not which hosts will be served."
 

I wonder how many web-admins thought they needed to open up access to all of their clients. 

Keywords: memcached
5 comment(s)

Comments

a) https://github.com/memcached/memcached/wiki/ReleaseNotes156
This is a bugfix release, but it primarily disables the UDP protocol by default.

b) https://twitter.com/dormando/status/968579781729009664
quote: if you're getting attacked by memcached's, it's pretty easy to disable them since the source won't be spoofed. They may accept "shutdown\r\n", but also running "flush_all\r\n" in a loop will prevent amplification.
/quote
Same old, same old. memcached servers are hung directly on the Internet for the same reason that SMB servers and RDP servers and Telnet servers and __________ are. Because that's how someone got it to work. No doubt their management congratulated them for getting it available so fast and at such a low cost as well.

As long as companies have their top three priorities a Availability, Availability and Availability this nonsense will keep right on happening.

An IT person at work heard me say the above and said "Availability is how the company makes its money that pays you." I agreed and replied "And IT Security is what lets the company keep the money they worked so hard for."

He's still working on his retort, I guess.
It's Teardrop/Ping of Death all over again.... Amplification attack and misuse of UDP (lack of built in control might be a better description) is nothing new. It is what sent me in to Information Security (and the hack was the first mentioned on national evening news). Oh! nostalgia: this just sends me back 20 years. (or is that "sets us back 20 years"?). Sorry not being technical, but sometimes we need a history lesson....
I think you're misunderstanding how people come to install and run memcached. There's no need to go looking for instructions or opening config files. They read instructions for the likes of nextcloud or mediawiki, decide they need memcached and then they do `pkg-yum-apt-whatever install memcached` and then `rc-systemd-thing enable memcached` and that's that, they edit the config file for the software that prompted them to install memcached, which can involve uncommenting a single line, it just works and they forget about it. At no point does memcached need manual configuration, and at no point of this process are they given any indication that it's accessible from the internet, not to mention that it can be turned into a ddos ballista.

This is why software should have sane defaults.
1,35 terabit per second

https://www.heise.de/security/meldung/Rekord-DDoS-Attacke-mit-1-35-Terabit-pro-Sekunde-gegen-Github-com-3985411.html

Diary Archives