Incident response and the false sense of security

Incident response and the false sense of security
This is a guest diary submitted by Tomasz Miklas. Interested in providing a guest diary yourself? Please send a proposal (title/outline) using our contact form. Interested in becoming a hanlder and regular contributor? See you Handler Roadmap. Some time ago I was asked to help with incident response for a small company. While the incident itself was not very exciting, the lessons learned were a bit more than a surprise. The victim was shocked how spectacularly they failed even though they considered themselves to follow good security practices or at least to be above the “low hanging fruit” category. This is classic example of false sense of security. Key lessons learned: Running a hardened web server as reverse proxy to protect the actual application is a great idea, however if the actual web application also listens on publicly available TCP port, there is nothing to stop the attacker from going after the application directly, bypassing the proxy. (If possible always bind the applications to localhost only or at least use the firewall to limit access to the application. This is how the attacker got a foothold on the system - known vulnerable web application and bypassing simple but efficient virtual patching done by the reverse proxy.) Hard-coded passwords and password reuse - as it turned out, all of the IT systems and components used the same administrator password. The original password could be found in a publicly readable backup script on a compromised server located in the DMZ. (Backup process is one of the most sensitive elements of the system - should everything else fail, backup is all you have. If privilege separation was implemented and properly used the attacker wouldn’t get the administrator’s password. Finally there is no excuse for password reuse - password management applications are widely available and really easy to use. ) Centralised logging can be very useful, especially if it’s used with some kind of log monitoring solution. Unfortunately it can also create extra work if you try to review logs from the incident and notice large portion of the systems having their clocks off by minutes or hours. (Keeping all your system clocks in sync is really important. NTP clients do the job very well and are already built into most if not all of the network equipment and general purpose operating systems. Another thing to keep in mind are time zones - make sure all systems use the same time zone and if possible pick one that doesn’t observe Daylight Saving Time (DST) as this has great potential to create additional issues or delays if the incident spans systems located in more than one country, especially if it happened around DST time change. Remember - simplicity is your friend.) Some interesting DST facts: Different countries observe DST on different dates - for example in US, Mexico and most of Canada DST begins about two weeks earlier than European countries. China which spans five time zones uses only one time zone (GMT+8) and doesn’t observe DST. In Southern Hemisphere where seasons of the year are in opposite to the Northern Hemisphere, so is the DST - starting in late October and ending in late March. Many countries don’t observe DST at all. -- Tomasz Miklas Twitter: @tomaszmiklas	Handlers 76 Posts
Reply Subscribe	Jan 6th 2014 5 years ago
I'm not sure "they considered themselves to follow good security practices" can be followed by "all of the IT systems and components used the same administrator password [which] could be found in a publicly readable backup script", without a facepalm or two during or after that statement.	oleksiy 34 Posts
Reply Quote	Jan 6th 2014 5 years ago
RE: DST That is why everything should be on UTC/GMT/ZULU time. It will never matter what state DST is in under these circumstances. In addition, any central logging / SEIM worth it's salt will support localization of time zones during log indexing. Finally, using the same time servers across an environment is critical. Even if that time is wrong, every device will be collectively wrong together. This goes a long way to stitching an event together using logs across many different systems and devices. It is also a requirement in PCI DSS 3.0 and ISO 27001:2013, if you need something more than "best practice" to make your case ;)	oleksiy 14 Posts
Reply Quote	Jan 6th 2014 5 years ago
Yes, time mis-synchronization is my principle pet peeve. You know how they say that ex-smokers are the most intolerant of smokers? In a former scientific career I actually caused a loss of days of research and tens of thousands of dollars because of a time synch problem for which I was responsible. It literally can add a sizable chunk to to the cost of an incident response engagement in terms of analytic hours, and contribute to mass uncertainty. It is amazing that large commercial enterprises can spend tens of millions on IT and pointedly neglect something that is trivial in terms of deployment cost, and zero hardware cost.	Anonymous
Reply Quote	Jan 6th 2014 5 years ago
@oleksiy: this is the difference between rather optimistic self-assessment by the organization and the hard reality check. False sense of security, often failing at total basics.	Tomasz 3 Posts
Reply Quote	Jan 6th 2014 5 years ago
"Different countries observe DST on different dates - for example in US, Mexico and most of Canada DST begins about two weeks earlier than European countries." Yes... some countries go forward instead of backwards. Some countries occassionally cancel or reschedule DST. Timezones are a real pain: http://www.youtube.com/watch?v=-5wpm-gesOY	Mysid 146 Posts
Reply Quote	Jan 6th 2014 5 years ago
IMHO Homeland Security should request that Congress abolish DST.	PhilBAR 24 Posts
Reply Quote	Jan 7th 2014 5 years ago
IMO, the U.S. should not only abolish DST but also our timezones (following China's example). We shouldn't be messing with time but instead, adjusting our human schedules to where we geographically live. Using one timezone, if you live on the east coast, your workday runs from 8am to 5pm. If you live on the west coast, your workday runs from 11am to 8pm. If you live in Europe, your workday runs from 2pm to 11pm. Why we, as humans, decided to modify time to fit our needs is baffling. I would vote for a world-wide standardization on Zulu time if I had the chance. Timezones and DST are outdated concepts based on outdated reasons and cause nothing but confusion. In today's modern world, we should all be on the same time. Want to have a conference call between your office in Denver and your supplier in Copenhagen tomorrow at 6:00pm? No problem, 6:00pm in Denver is 6:00pm in Copenhagen and 6:00pm everywhere. I remember once I had a red-eye flight during a DST change and I thought I had 1hr and 30mins for my connection when I really only had 30mins. Timezones and DST are terrible concepts in real life and even worse in cyber life.	da1212 69 Posts
Reply Quote	Jan 9th 2014 5 years ago

This is a guest diary submitted by Tomasz Miklas. Interested in providing a guest diary yourself? Please send a proposal (title/outline) using our contact form. Interested in becoming a hanlder and regular contributor? See you Handler Roadmap.

Some time ago I was asked to help with incident response for a small company. While the incident itself was not very exciting, the lessons learned were a bit more than a surprise. The victim was shocked how spectacularly they failed even though they considered themselves to follow good security practices or at least to be above the “low hanging fruit” category. This is classic example of false sense of security.

Key lessons learned:

Running a hardened web server as reverse proxy to protect the actual application is a great idea, however if the actual web application also listens on publicly available TCP port, there is nothing to stop the attacker from going after the application directly, bypassing the proxy.
(If possible always bind the applications to localhost only or at least use the firewall to limit access to the application. This is how the attacker got a foothold on the system - known vulnerable web application and bypassing simple but efficient virtual patching done by the reverse proxy.)
Hard-coded passwords and password reuse - as it turned out, all of the IT systems and components used the same administrator password. The original password could be found in a publicly readable backup script on a compromised server located in the DMZ.
(Backup process is one of the most sensitive elements of the system - should everything else fail, backup is all you have. If privilege separation was implemented and properly used the attacker wouldn’t get the administrator’s password. Finally there is no excuse for password reuse - password management applications are widely available and really easy to use. )
Centralised logging can be very useful, especially if it’s used with some kind of log monitoring solution. Unfortunately it can also create extra work if you try to review logs from the incident and notice large portion of the systems having their clocks off by minutes or hours.
(Keeping all your system clocks in sync is really important. NTP clients do the job very well and are already built into most if not all of the network equipment and general purpose operating systems. Another thing to keep in mind are time zones - make sure all systems use the same time zone and if possible pick one that doesn’t observe Daylight Saving Time (DST) as this has great potential to create additional issues or delays if the incident spans systems located in more than one country, especially if it happened around DST time change. Remember - simplicity is your friend.)

Some interesting DST facts:

Different countries observe DST on different dates - for example in US, Mexico and most of Canada DST begins about two weeks earlier than European countries.
China which spans five time zones uses only one time zone (GMT+8) and doesn’t observe DST.
In Southern Hemisphere where seasons of the year are in opposite to the Northern Hemisphere, so is the DST - starting in late October and ending in late March.
Many countries don’t observe DST at all.

--

Tomasz Miklas

Twitter: @tomaszmiklas

Handlers

76 Posts

I'm not sure "they considered themselves to follow good security practices" can be followed by "all of the IT systems and components used the same administrator password [which] could be found in a publicly readable backup script", without a facepalm or two during or after that statement.

oleksiy

34 Posts

RE: DST

That is why everything should be on UTC/GMT/ZULU time. It will never matter what state DST is in under these circumstances. In addition, any central logging / SEIM worth it's salt will support localization of time zones during log indexing.

Finally, using the same time servers across an environment is critical. Even if that time is wrong, every device will be collectively wrong together. This goes a long way to stitching an event together using logs across many different systems and devices. It is also a requirement in PCI DSS 3.0 and ISO 27001:2013, if you need something more than "best practice" to make your case ;)

oleksiy
14 Posts

Yes, time mis-synchronization is my principle pet peeve. You know how they say that ex-smokers are the most intolerant of smokers? In a former scientific career I actually caused a loss of days of research and tens of thousands of dollars because of a time synch problem for which I was responsible. It literally can add a sizable chunk to to the cost of an incident response engagement in terms of analytic hours, and contribute to mass uncertainty. It is amazing that large commercial enterprises can spend tens of millions on IT and pointedly neglect something that is trivial in terms of deployment cost, and zero hardware cost.

Anonymous

@oleksiy: this is the difference between rather optimistic self-assessment by the organization and the hard reality check. False sense of security, often failing at total basics.

Tomasz

3 Posts

"Different countries observe DST on different dates - for example in US, Mexico and most of Canada DST begins about two weeks earlier than European countries."

Yes... some countries go forward instead of backwards. Some countries occassionally cancel or reschedule DST.
Timezones are a real pain:
http://www.youtube.com/watch?v=-5wpm-gesOY

Mysid

146 Posts

IMHO Homeland Security should request that Congress abolish DST.

PhilBAR

24 Posts

IMO, the U.S. should not only abolish DST but also our timezones (following China's example). We shouldn't be messing with time but instead, adjusting our human schedules to where we geographically live. Using one timezone, if you live on the east coast, your workday runs from 8am to 5pm. If you live on the west coast, your workday runs from 11am to 8pm. If you live in Europe, your workday runs from 2pm to 11pm. Why we, as humans, decided to modify time to fit our needs is baffling. I would vote for a world-wide standardization on Zulu time if I had the chance. Timezones and DST are outdated concepts based on outdated reasons and cause nothing but confusion. In today's modern world, we should all be on the same time. Want to have a conference call between your office in Denver and your supplier in Copenhagen tomorrow at 6:00pm? No problem, 6:00pm in Denver is 6:00pm in Copenhagen and 6:00pm everywhere.

I remember once I had a red-eye flight during a DST change and I thought I had 1hr and 30mins for my connection when I really only had 30mins.

Timezones and DST are terrible concepts in real life and even worse in cyber life.

da1212

69 Posts