Cleaning house

Published: 2011-03-02
Last Updated: 2011-03-02 12:25:50 UTC
by Chris Mohan (Version: 1)
12 comment(s)

There are times as a security professional you have to roll up the sleeves and get your hands dirty to make sure some of the basics are applied to the environment we’re looking after. As a common example, most of us have had to patch the odd Windows machine, or three, to help out a friend to make sure they’re safe and up to date from the various nasties out there.
 

What happens when you’re presented with forty seven Windows XP computers: all networked, in a Windows workgroup, have varying levels of patches installed, hardly any internet connectivity and a limited time frame to get them to a current patch level? Now throw in every machine is infected and the infections is causing embarrassing and crippling problems to the users.


Here’s my solution; if you have a better one, or helpful pointers, feel free to comment.


- Assess the situation, explain the discovered risks to the business and come up with a plan of attack.

Uncovered background on the problem

  • Tech support for the network is one poor soul that “is good with computers” but it’s not their primary job
  • The same antivirus software (AV) was on all machines, but the definitions were totally out of date
  • The admin password for all of the machines is the same
  • Switched fast Ethernet network linked the machines
  • The internet link was very expensive, very limited and only used for email
  • The machines were riddled with malware -  Conficker being the most obvious
  • The file server was another XP machine used to store all the data
  • This is a favour to a friend so no budget and had to be done over an evening
  • Business owner had signed off on the risk of patching everything in one go
  • These machine were on four different floors (lots of running around), but only one network
  • Random application software installed
  • Did I mention no budget for anything IT, including support or training

Leaving aside the bigger picture of no security policies or procedures and a total lack of fundamental IT management, it was important to get to a measured and consistence baseline where the users could actually work.

 

The aim was to bring a standard, baseline patch level to the Windows machines, avoid full rebuilds and purge the main malware problems.

Here's my quick five steps :

  1. Back up the file server, verifying data copy is malware-free and valid
  2. Get the current service pack installed
  3. Get all the current hotfixes installed
  4. Check that all the machines are patched
  5. Get all machines to the current AV definition level, scan and clean any malicious activity on the machines

Faced with a very hostile network, trusting one of the existing machines was not any option.

Thankfully virtualisation provides a great option to plug in a machine and dispose/revert it if this type of situation arises.  I also happen to have a Windows server virtual machine (VM) with Windows Server Update Service (WSUS) [5] installed on my laptop, that had been recently synced with the latest updates.*
 

Step 1: Backup the critical data off the XP file server is a copy an external USB drive

The drive was then plugged in to secured machine with a current AV. The autorun nasties were removed from the drive, the data was scanned and cleaned from all know problems. Then someone from the company confirmed the data was good.
 

Step 2: Create a share on the virtual machine for XP SP3 and deploy it to all machines

The wonderful PSEXEC [1] comes to the rescue as it can be used to deploy and execute the SP3 patch from the VM's share. Smarter scripting techniques [2] with PSEXEC mean you can automate this process for deployment.
As an example, this command copies SP3 to the target machine, then silently starts the installation and forces a reboot once the SP3 has been installed.
psexec computer  -c -f -s servershareWindowsXP-KB936929-SP3-x86-ENU.exe / quiet /forcerestart

Step 3: Deploy all current patches

Having a WSUS server as a virtual machine means fast, portable patch management with reporting. Using PSEXEC to deploy registry keys [3] to point all the XP machines to my WSUS VM, this forced all the machines to register and download the current updates. This proved a log of all the machines that connected, and what Windows patch level they were at.
 

Step 4: Check that all the machines are patched

Microsoft’s free tool Microsoft Baseline Security Analyzer (MBSA) [5] is a quick and effective way to verify that all the machines are up to the correct patch level, as it can reference the portable WSUS server at the patch baseline for each machine.
 

Step 5: Update AV and force a full scan

Copy the current AV definition to the same share as SP3. PSEXEC strikes again to copy to each machine and a simple batch script to kick of a full scan and confirm the scan has run successfully by dumping the results to the VM's share. This allows for a quick and easy way to check every machine has run AV, plus see what the AV detected on the machines.

End Result

This got the all the machines to the same consistent state and removed the problematic malware - all over one long evening.


Notes:

  • If you do use the WSUS trick, remember to remove the registry keys afterward; using PSEXEC with a batch script will to do this nicely .
  • WSUS could have been used to deploy SP3, but I find forcing patch deployment with PSEXEC is a lot faster. Had there only been a few patches missing, a WSUS server is somewhat over kill. A simple batch file utilizing hotfix chaining [4] and PSEXEC is a very fast way to deploy patches remotely.


[1] http://technet.microsoft.com/en-us/sysinternals/bb897553
[2] http://ss64.com/nt/psexec.html
[3] http://support.microsoft.com/kb/328010
[4] http://support.microsoft.com/default.aspx?scid=kb;en-us;296861
[5] http://technet.microsoft.com/en-us/security/cc184924
[6] http://technet.microsoft.com/en-us/windowsserver/bb332157
 

*Doesn’t everyone? Well if you’re building machines and travelling to places with poor internet access all the time, it makes patching a darn sight easier!
 

Chris Mohan --- Internet Storm Center Handler on Duty

Keywords: WSUS
12 comment(s)

Comments

I have used autopatcher (http://www.autopatcher.com/) on either a network share or a cd to install patches where there is little to no internet as well.
I once implemented a caching proxy server which saved entire Windows Update softwares in its cache. As a result, every updated only had to be downloaded once since subsequent requests could be served from the cache.
In addition to the M$ patches and AV, lest we not forget Adobe Reader, Flash, and Java. I use LANsweeper (free version) to audit multiple computers for software, have the most current downloads of each in a folder along with a txt file with the silent install switches, then leverage PDQDeploy (free) to push the updates accordingly.
If you do not have a WSUS handy you could use the "poor man's wsus":
http://www.wsusoffline.net/
Very convenient, I always use it to keep up to date machines without internet access.
A good evening of work, and while I realize your time was limited and there may have been business issues for management in doing so, I would also have recommended that you configure and validate automatic updates and av definition updates schedules to help prevent a recurrence. Hopefully the entire episode has highlighted to business management that they are going to have to implement some controls and budget a little money to IT spend.
Hopefully, Mark, although the cynic in me says, "why would they budget? They just found out they can get it for free."
The WSUS server requires the installation on a legal install of the server OS and each client must have a valid server CAL. I wish that msft would at least allow for patching of all legally licensed products and not require the extra server cals. It's to patch issues with their OS, it should be free. I used to use qchain after a string of patch installs and reboot and then sus was born.
http://www.wsuswiki.com/WSUSFAQ

"Q. Is WSUS free?

A. Yes, WSUS is a no-cost download from Microsoft. However, you must have a valid Windows Server 2003 license for the WSUS server itself, as well as Windows Client Access Licenses (CALs) for each machine updated by WSUS. This is an important consideration for organizations that run Windows desktop operating systems, but non-Windows server operating systems such as Novell or Linux. Additionally, if you decide against the free MSDE/WMSDE database and instead use SQL Server 2000, you also need SQL licenses (either per-processor, or CALs for each WSUS-managed computer). Be sure to discuss your unique licensing needs with a Microsoft Partner or your Microsoft Account Representative to ensure compliance.".

This should be free, OS included.
I have Secunia PSI installed on my machines and it auto updates 3rd party software. It saves a lot of time.
I've had a lot of luck using Offline Update (now WSUS Offline Update (don't worry, it's GPL)) to download patches en masse. I'd then burn them onto DVD or CD (the scenario above stipulates a number of infected machines, so no writable devices) and patch a few machines at a time. It's pretty easy to write a script that will walk through a directory of hotfixes and install each one in unattended mode. I've found that the sweet spot (for one admin, anyway) is to patch six machines at a time this way, working from one side of the room to the other.

Walk up, drop the disk in, run the script, move on to the next machine.
I would apologize, say that whats required can't be done in the time or budget alloted, offer suggestions on what to look for when they put out bids for it, and run screaming from the building.

Barring that, I would probably harvest the license keys from the infected hosts, software inventories, get good backups of user data, then exercise the nuke and rebuild option.

Diary Archives