Intel Network Card (82574L) Packet of Death

Published: 2013-02-06
Last Updated: 2013-02-08 21:07:10 UTC
by Johannes Ullrich (Version: 3)
13 comment(s)

[Update]  Intel released a statement about this issue. and Kristian updated his blog

An interesting blog post by Kristian Kielhofer describes how a specific SPI packet can "kill" an Intel Gigabit ethernet card [1]. If a card is exposed to this traffic, the system has to be physically power cycled. A reboot will not recover the system. 

The network card crashed whenever the value 0x32 or 0x33 was found at offset 0x47f. Kristian first noticed this happening for specific SIP packets, but in the end, it turned out that any packet with 0x32 at 0x47f caused the crash. Intel traced the problem to an EEPROM used in this specific card (82574L). There are some links in the comment to the blog suggesting that others have run into this problem before. For example, the commend: "ping -p 32 -s 1110 x.x.x.x" can crash an affected card remotely.

[Update] A few asked why this doesn't happen just randomly every 128th packet: Once the card receives the value "0x34" in this position, it appears to be no longer vulnerable. There are also a number of earlier bug reports about this card that sound very similar, and appear to be related to ASPM, a PCI power safe feature. Kristian claims he eliminated this issue. if you try to reproduce this issue, power up the system and then issue the "ping" command shown above quickly after reboot in order to avoid the "inoculation" wiht 0x34. We would like to hear any reports of being able to reproduce (or not) this issue. 

There are also some reports about similar issues in certain 3G USB modems.



Johannes B. Ullrich, Ph.D.
SANS Technology Institute

Keywords: ethernet card intel
13 comment(s)

Are you losing system logging information (and don't know it)?

Published: 2013-02-06
Last Updated: 2013-02-06 20:30:17 UTC
by Johannes Ullrich (Version: 1)
1 comment(s)

(This is a guest diary submitted by Bill Parker)

How many administrators review log files in /var/log/*, but don't realize they may be losing possibly important (or even critical) information?
In working with a commonly used IDS (Snort 2.9.x) on one of my test platforms (CentOS 6.3 64-bit inside of VirtualBox 4.2.6), I happened to notice a unusual line in /var/log/messages when snort initialized via startup script in /etc/init.d:
Feb  5 13:07:52 plugh snort[12105]:       Non-RFC Compliant Characters: 0x00 0x01 0x02 0x03 0x04 0x05 0x06 0x07 
Feb  5 13:07:52 plugh snort[12105]:       Whitespace Characters: 0x09 0x0b 0x0c 0x0d 
Feb  5 13:07:52 plugh snort[12105]: rpc_decode arguments:
Feb  5 13:07:52 plugh snort[12105]:     Ports to decode RPC on: 111 32770 32771 32772 32773 32774 32775 32776 32777 32778 32779 
Feb  5 13:07:52 plugh snort[12105]:     alert_fragments: INACTIVE
Feb  5 13:07:52 plugh snort[12105]:     alert_large_fragments: INACTIVE
Feb  5 13:07:52 plugh snort[12105]:     alert_incomplete: INACTIVE
Feb  5 13:07:52 plugh snort[12105]:     alert_multiple_requests: INACTIVE
Feb  5 13:07:52 plugh snort[12105]: FTPTelnet Config:
Feb  5 13:07:52 plugh snort[12105]:     GLOBAL CONFIG
Feb  5 13:07:52 plugh rsyslogd-2177: imuxsock begins to drop messages from pid 12105 due to rate-limiting <- LOOK HERE
Feb  5 13:07:57 plugh snort[12106]: Daemon initialized, signaled parent pid: 12105
Feb  5 13:07:57 plugh snort[12106]: Reload thread starting...
Feb  5 13:07:57 plugh snort[12106]: Reload thread started, thread 0x7f4039a37700 (12106)
Feb  5 13:07:57 plugh kernel: device eth0 entered promiscuous mode
Feb  5 13:07:57 plugh snort[12106]: Decoding Ethernet
Feb  5 13:07:57 plugh snort[12106]: Checking PID path...
Feb  5 13:07:57 plugh snort[12106]: PID path stat checked out ok, PID path set to /var/run/
Feb  5 13:07:57 plugh snort[12106]: Writing PID "12106" to file "/var/run//"
Feb  5 13:07:57 plugh snort[12106]: Set gid to 40000
Feb  5 13:07:57 plugh snort[12106]: Set uid to 40000
Feb  5 13:07:57 plugh snort[12106]: 
Feb  5 13:07:57 plugh snort[12106]:         --== Initialization Complete ==--
Feb  5 13:07:57 plugh snort[12106]: Commencing packet processing (pid=12106)
It turns out that many modern Linux distributions come with 'rsyslog', which is a replacement for 'syslogd' or 'sysklogd', but starting with version 5.7.1 of rsyslog, a feature known as rate-limiting was added to the utility, and if a given process ID (PID) were to send more than 200 messages to /var/log/messages in a 5 second interval (the default setting in rsyslog), it will start to drop messages and place the following warning inside of /var/log/messages:
Feb  5 13:07:52 plugh rsyslogd-2177: imuxsock begins to drop messages
from pid 12105 due to rate-limiting
In the case of daemons or processes logging to /var/log/messages (or any other directory/file which rsyslog happens to be handling  logging for), a great deal of important and/or critical logging  data could be lost to security or system administrators.
While rate-limiting on routers/firewalls/web servers is a useful method of containing certain types of network based attacks, in the case of system and/or application logging, this may create a logistical nightmare for SIEM's or applications which collect and analyze large amounts of system and/or application logs for event information/messages/warnings.
In doing some research on rsyslog, I found two solutions which can be used to solve this condition on systems where rsyslog is the default system logging method.
Note - Back up any file(s) listed below before proceeding!
The first solution is to simply increase the messages allowed and the time interval before rate-limiting occurs in rsyslog.  To do this, locate the rsyslog.conf and/or rsyslog.early.conf (usually in /etc) and add the following lines:
$SystemLogRateLimitInterval 10
$SystemLogRateLimitBurst 500
after any ModLoad commands in rsyslog.conf and/or rsyslog.early.conf
This will tell rsyslog to start rate-limiting (discarding messages) when more than 500 messages from a single PID are received within a 10 second interval (these numbers are not absolutes, they can be tailored to any given system, btw).
The second solution is to simply turn off rate-limiting for rsyslog, and to do this, add the following line to rsyslog.early.conf and/or rsyslog.conf using your favorite editor (Iím a vi/vim/gvim hound):
$SystemLogRateLimitInterval 0
after any ModLoad commands in rsyslog.conf and/or rsyslog.early.conf
This will disable any rate-limiting in effect for the rsyslog process running on this system.  Note that by doing this, an out of control process ID on your system can fill up /var/log/messages with a lot of useless messages (which is why rate-limiting is enabled by default in rsyslog).
Remember to stop/start or restart the rsyslog daemon in order to make the changes to rsyslog.conf and/or rsyslog.early.conf take effect.
The following Linux systems use 'rsyslog' as the default system logger (these are distributions which I am actively using, btw):
CentOS 6.x
Debian 5.0 or greater
Fedora 13 or greater
OpenSuSE 11.x/12.x
Ubuntu 10.0 or greater
BSD based systems (FreeBSD 8.x/9.0, OpenBSD 5.x, and NetBSD 5.x/6.0) use traditional syslogd as the default system logging utility.
If you need more information about rsyslog, you can visit the following URL:
Bill Parker (wp02855 at gmail dot com)


1 comment(s)

HTTP Range Header and Partial Downloads

Published: 2013-02-06
Last Updated: 2013-02-06 19:07:01 UTC
by Johannes Ullrich (Version: 1)
1 comment(s)

Last week, I was debugging the podcast access script, I came across some interesting behaviour regarding the "Range" header in HTTP requests. The purpose of the "Range" header is to allow for resumable downloads via HTTP. The client may ask the server to only sent a certain part of the page, instead of the entire response. Not all servers (or browsers) necessarily support this feature. The feature is very different from "Chunked encoding", another feature that can be used to break up a page, but not to break it up as demanded by the client.

Client Side / Request

A request may include a range header, asking only for a part of the file. For example:

Range: bytes=0-100

would request the first 100 bytes from the response. The server may ignore this header, and the browser should accept whatever comes back, even if it is more or less then the requested range

Server Side / Response

A partial response always uses the status code 206 instead of 200. In addition, a header indicating the range delivered, and the total length of the file will be included:

From the RFC:


HTTP/1.1 206 Partial content
Date: Wed, 15 Nov 1995 06:25:24 GMT
Last-Modified: Wed, 15 Nov 1995 04:58:08 GMT
Content-Range: bytes 21010-47021/47022
Content-Length: 26012
Content-Type: image/gif

The Content-Range header indicates the range delivered, and the number following the / is the size of the file. In addition, you should still see a content-length header.

So what could possibly go wrong? I played with various invalid combinations, and so far, what I found is that the browser will ignore them. I haven't gotten around to test them all with respect to an IDS, but assume that a properly configured HTTP preprocessor will reassemble these ranges. Of course, without preprocessor, there will be a wide range of evasion/insertion attacks.

An issue I found is that some podcast clients will first try to download byte range 0-1, then they will download the file. Most of the time in one attempt, but frequently in multiple ranges. This can confuse web log analysis software as it will register them as multiple "hits" to the same file. You need at least to look at the status code (200 vs. 206). Also, the clients did not access the complete file if the server returned the entire file instead of just bytes 0-1. 

It is also possible to specify multiple byte ranges in one request, and older versions of Apache had a denial of service vulnerability if an excessive number of byte ranges was specified.

Let us know if you find anythingelse interesting when it comes to processing the Range header.

References: RFC 2616

Johannes B. Ullrich, Ph.D.
SANS Technology Institute

Keywords: http range
1 comment(s)
Sysinternals in particular Process Explorer update
ISC StormCast for Wednesday, February 6th 2013


Diary Archives