Why patch management is ALSO REQUIRED in ICS infrastructure

Published: 2015-01-07
Last Updated: 2015-01-08 05:32:28 UTC
by Manuel Humberto Santander Pelaez (Version: 1)
1 comment(s)

Security patch management is a delicate issue in critical infrastructure. This is caused for the specific configuration, operating system version and related software required by the ICS platform. Most support contracts states that any modification outside the parameters stated by the manufacturer will void the relation and release manufacturer and seller from any responsibility about malfunction and any consequence on the industrial process.

Unfortunately, when we talk about ICS software running on windows the restriction is often applied to domain controllers as well. The case I will cover on the present diary is about an incident on a Water ICS system controlled by Emerson Open Enterprise installation and how all the servers located in the domain got modified their attributes on the domain controller without being able to tell what changes were executed.

Before this incident happened, all domain controllers and servers were configured to log the required security events to Arcsight. Now we need to find out which of the billions of logs we need to search. For this installation, we are talking about 3000 events per second. The incident began when the Open Enterprise HMI System began to loose readings from all the RTUs and after a couple of minutes it was unable to send commands to any RTU.

Since we are talking here about Windows Server 2012R2, There is an interesting repository were all event IDs for Windows 8 and Windows 2012 can be located. After looking inside the repository, the following events are interesting for searching inside the arcsight database:

Category Subcategory Event ID Message Summary
DS Access Directory Service Changes 5136 A directory service object was modified.
DS Access Directory Service Changes 5137 A directory service object was created.
DS Access Directory Service Changes 5138 A directory service object was undeleted.
DS Access Directory Service Changes 5139 A directory service object was moved.
DS Access Directory Service Changes 5141 A directory service object was deleted.

Event ID 5136 states that the attribute field must be filled with the information changed for the specific object. Check the following evidence collected:

Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          23/11/2014 1:30:42 PM
Event ID:      5136
Task Category: Directory Service Changes
Level:         Information
Keywords:      Audit Success
User:          N/A
Computer:      DC.Domain.com
A directory service object was modified.
 Security ID:  Domain\Administrator
 Account Name:  Administrator
 Account Domain:  Domain
 Logon ID:  0x5f8a3

Directory Service:
 Name: Domain.local
 Type: Active Directory Domain Services
 Class: XXXXX
 Syntax (OID): X.X.X.X
 Type: Value Deleted
 Correlation ID: {ba5fa2fe-9a61-12fa-1b95-3bf03643b4e2}
 Application Correlation ID: -

For some reason, the attribute field was missing and we could not know what attributes were deleted. After researching on this issue, we found it's a bug documented as KB2458125. Six months after, this patch was not installed and because of that we were unable to know what attributes were changed on the server.

Even if we are dealing with ICS systems, the ICS admin must evaluate what patches can be applied without affecting any feature of the system like this patch. If ICS admins does not correctly control risks, incident response tasks could be useless and it will not be possible to control impact and execute containment and eradication successfully.

Want to implement patch management in your organization? Check the specific NIST guide for more information.

Manuel Humberto Santander Peláez
SANS Internet Storm Center - Handler
e-mail: msantand at isc dot sans dot org

1 comment(s)


An interesting case, but it doesn't provide any solutions to the above mentioned problem of not being able to patch without voiding the manufacturer warranty and releasing them from responsibility. I've found that doing file integrity checks on ICSs and reverting to a known good at the first sign of any kind of change is the best way to go between vendor releases (honestly, how often should ICSs change between releases?) Not always the best option for investigation purposes admittedly, but it at least it offers a slightly better posture in terms of recovery.

NIST also has publications specific for ICS, NIST SP 800-82(http://csrc.nist.gov/publications/nistpubs/800-82/SP800-82-final.pdf)

Diary Archives