This is a guest diary submitted by Remco Verhoef.

The cloud is bringing a lot of interesting opportunities, enabling you to scale your server farm up and down depending on the load. Everything is being taken care of automatically by auto scale groups.There is nothing to worry about anymore.

But this brings me to the following point, in particular, because IPv4 addresses are harder and harder to come by: How quickly are public IP addresses being reused and what if I can collect requests, specifically HTTP(s) requests, intended for a prior user of the IP.

Easily said. I’ve created a server which will just store all requests in an Elasticsearch cluster, a client who does nothing more than just listening for http and https (with an expired self signed certificate) and a userdata script that automatically starts the client, waits for about 10 minutes and then shuts it down again. I’ve configured spot instances in multiple regions and kept it running for a while. When spot instances are being shut down (because of the price, or because the instance shuts itself down), it will automatically start a new instances if it is still within the given pricing range.

About the results: I’ve loaded the Elasticsearch dataset into R for easy visualisation and querying and together with good old awk, sed, wc and grep I’ve been able to draw some interesting results.

Within the dataset we found several occurrences with Amazon in the User-Agent. This leads to the following Amazon related user-agents:

• Amazon Route 53 Health Check Service
• Amazon Simple Notification Service Agent
• Amazon CloudFront

The health check service checks if the host is online and if it returns the expected result (statuscode 200), it will include this server to the dns based load balancing configuration.

Simple Notification Service Agent is being used to send notifications to the server. We’ve received several notifications, like cache keys expired, varnish flush caches, but also a load of messages containing userdata.

The Amazon CloudFront traffic is traffic originated to the original webservers, of data that needs to be cached for a longer period of time. This is interesting as well, because you’ll be able to poison the cache for a long amount of time.

An analysis on the other useragents gave some interesting results as well. We’ve seen the following useragents:

• Stripe (the url being called was a webhook and contained information about a updated subscription.
• GitHub-Hookshot (a webhook being called)
• com.google.* (the user-agents are being used by Google applications) for example docs, gmail, ios
  • com.google.chrome.ios
  • com.google.Classroom
  • com.google.Docs
  • com.google.Gmail
  • com.google.GoogleMobile
  • com.google.hangouts
  • com.google.hangouts_iOS
  • com.google.ios.youtube
  • com.google.Slides
• Gmail
• GmailHybrid
• Google
• Google.Docs
• Google.Drive
• Google.DriveExtension
• Google.Slides
• GoogleAnalytics
• GoogleCsi
• Hangouts
• SLGoogleAuthService
• com.apple.appstored is the useragent being used by the apple store application)
• FBiOSSDK is facebook app
• FitbitMobile is the fitbit app
• Haiku%20Learning
• Instagram the instagram app
• Spotify

In total we’ve found 159 different user agents (with the version number stripped off). Which is a lot. The complete dataset contained ~80k requests.

Those useragents are strange, not the variety I had expected. This seems just normal traffic passing a proxy or a gateway. Let’s look at the hostnames, in total we have encountered 578 different hostnames, containing a lot of interesting hosts . This is just a subset.

• Minecraftprod.rtep.msgamestudios.com:443
• www.googleapis.com:443
• apis.google.com:443
• inbox.google.com:443
• global.bing.com
• api-global.netflix.com
• m.baidu.com
• www.fitbit.com:443
• audio-sv5-t1-1-v4v6.pandora.com
• s3.amazonaws.com:443
• pagead2.googlesyndication.com
• maps.googleapis.com:443
• www.google.com.tw:443
• d1e5xacotudrwg.cloudfront.net
• www.google.pl
• www.wikipedia.org
• admin.brightcove.com
• crt.usertrust.com
• www.twitter.com:443
• oauth.vk.com:443
• en-US.appex-rf.msn.com
• 4-edge-chat.facebook.com:443
• tados-s.westeurope.cloudapp.azure.com:15002
• google.com:443
• m.google.com:443
• www.baidu.com:443
• www.bing.com
• s3-us-std-102-prod-contentmover.prod-digitalhub.com.akadns.net
• i.instagram.com:443
• localhost
• mail.google.com:443
• itunes.apple.com:443

So this is really strange. What about the X-Forwarded-For headers, if it should have been destined to a proxy server, we should be able to find those. Looking at this traffic we didn’t see really awkward things, most of them were from left over dns entries for other hosts. One super weird thing though, is this traffic destined for www.datapool.vn.

X-Forwarded-For: 25.34.94.111, 197.191.79.83, 162.63.120.125, 149.133.46.21, 21.186.158.233, 214.148.185.253, 249.166.69.42, 20.90.2.104

Host: www.datapool.vn

Remote Address: 103.9.158.76

Where the remote address is from somewhere in Vietnam, but the X-Forwarded-For explains the request has been forwarded by 8! different proxy servers. If you look up the block owners of these proxy servers you’ll find these organizations:

• UBS AG (UBSAG)
• DoD Network Information Center (DNIC)
• DoD Network Information Center (DNIC)
• Computer Sciences Corporation (CSC-68)
• UK Ministry of Defence
• Airtel Ghana
• The Swatch Group Ltd

So traffic from a vietnamese client address, via my Amazon server, with in the X-Forwarded-For the UK Defense as the Department of Defense with a vietnamese website as destination. Interesting. Don’t know how to position this, any ideas are welcome.

Now I’m looking at the requesting addresses, retrieving all the netblock owners of the requesting addresses. This is not 100% solid, because netblock could change ownership in time. But lets see what it brings us. To have some focus, I’ve filtered out all the traffic with the com.google.* user-agents. This is traffic I wouldn’t expect to arrive at my systems. The netblock owners are the following:

• AT&T Internet Services (SIS-80)
• Cellco Partnership DBA Verizon Wireless (CLLC)
• Gestion de direccionamiento UniNet
• Hawaiian Telcom Services Company, Inc. (HAWAI-3)
• McAllen Independent School District (MISD-15)
• PSINet, Inc. (PSI)
• San Diego County Office of Education (SDCS)
• Time Warner Cable Internet LLC (RCSW)
• Time Warner Cable Internet LLC (RRMA)
• Time Warner Cable Internet LLC (RRMA)
• Time Warner Cable Internet LLC (RRSW)
• Time Warner Cable Internet LLC (RRSW)
• Time Warner Cable Internet LLC (RRWE)
• Time Warner Cable Internet LLC (RRWE)
• T-Mobile USA, Inc. (TMOBI)
• WideOpenWest Finance LLC (WOPW)

In total there were 95 different originating ip addresses, with 1446 requests (with useragent com.google.*), which is just a small sample of the complete set. AT&T and Time Warner have done the most requests.

So with all the data we’ve crunched, we’ve found the following cases:

• we receive data from client addresses mapped to telecom and internet providers, with http method CONNECT and with traffic destined for Google, Microsoft and much more.
• we receive data for Amazon Route 53 health checks, Amazon Simple Notification Service Agent and Cloudfront. Many of the SNS messages contain privacy related information.
• we receive many more requests from left over ip addresses, these consist of GET requests, but also POST requests containing data as tokens, keys and logs.

This leaves us with the following attack scenarios:

• proxy the traffic on our server with the real destination, with for example a falsely generated certificate. Many apps will reject the certificate, but we all know the troubles CAs have with incorrect issued certificates. So at least in theory it will be possible to man in the middle traffic with accepted certificates.

• For the left over hosts it will be even possible to request a real lets https certificate with a bit of timing and luck, because the challenge response works via http.

• Another important issue to worry about is that it is possible as well to get accepted as a server into the load balancing configuration of Route 53, by just returning valid response status codes. As long as we’ll return valid status codes, it will be as long as someone notices part of the configuration.

• And last but not least, the amount of data we were getting on our http endpoints, like the sns notifications, webhooks and all. This could be circumvented easily by just enforcing valid https certificates.

Disclosure Timeline

We’ve disclosed our findings with Amazon and Amazon has implemented an IP cooldown feature, which should fix those issues.

2016/10/13: issue reported to Amazon. AWS confirms receipt of the report, and reaches out requesting additional information
2016/10/27: AWS Security requests additional details to further the investigation, and sets up time for an initial telephone sync
2016/11/08: Conference call confirmed for 2016/11/11
2016/11/11: Telephone sync takes place to provide additional technical information. AWS confirmed that it is possible for traffic configured with a destination IP to be received by any instance with that public IP regardless of attachment. Shared that AWS would be adding a cooldown period before an IP is reassigned to reduce the likelihood of an IP being reused while traffic destined for it was in flight. This would be delivered in early 2017
2017/02/24: IP cooldown feature released

Disclaimer: it is possible that the dataset has been manipulated by forged requests, but as no-one knows about this research and the launch of spot instances where to random I don’t expect that to be the case.

If you have any more questions just let me know. I’ll be happy to answer them. You can contact me using:

@remco_verhoef
github.com/dutchcoders/
remco@dutchsec.nl