Overview

You need to write a program that will submit your log lines in one of the formats that we support. You also must ensure that you only submit new log lines that you haven't already submitted. You must format the subject line correctly.

The DShield format is real close to the internal format of our database. So logs submitted in the DShield format need minimal processing on our server before being entered in the database. (Fat client, thin server, for this operation.) But if you submit your log lines "as is" then we run a parser to strip the lines apart into individual fields, edit as is needed, and then write them into DShield format. (Thin client, fat server.)

If we have conversion parsers defined for the format your log writes, then it is easier for you to write a client. But there are many different kinds of log formats and it is difficult for us to write conversion parsers for all of them. This is where you can help us out: by writing your own client software that converts your log files into one of the formats we already support. Ideally into the DShield format because that requires less processing on our server.

The idea is that your client program will break each of your log lines into fields, edit any fields as needed, and then write a new log file in the desired format. Here is the regular expression (in Perl) we use to parse Linux KERNEL style logs before converting them to DShield format before inserting into the database. You can use it as an example of one method of parsing your own log lines into separate fields.

If you don't care to use Perl style regular expressions, then use any method you want to reformat your log lines. All that matters is that they are submitted as one of the formats that we support.

You also must ensure that you don't send in log lines that you have previously submitted. The method we use for the clients we have written is to save the timestamp of the last log line that is processed. Then, on subsequent runs of the client, compare the timestamp of each log line and only submit log lines that are later than this comparison date. Then save the comparison timestamp to disk so it can be used for the next conversion.

You might find it helpful to examine the client programs that have already been written to see how they work. Most come with source code.

Submitting Logs

Report submissions are accepted via e-mail. The email has to be sent to 'reports@dshield.org'. Please submit at least once a day, if possible, but no more often than once an hour.

The subject line of the email should identify the format and the UserID (if known.) Please use one of the following formats:

Subject: FORMAT IPCHAINS USERID 12345  TZ -05:00 [Clientname Version #]    Linux 2.2.x Kernel format (ipchains)
Subject: FORMAT IPTABLES USERID 12345  TZ -05:00 [Clientname Version #]    Linux 2.4.x Kernel format (iptables)
Subject: FORMAT SONICWALL USERID 12345  TZ -05:00 [Clientname Version #]    SonicWall format as created by the SonicWall firewall
Subject: FORMAT RAPTOR USERID 12345  TZ -05:00 [Clientname Version #]    Raptor format as created by the Raptor firewall
Subject: FORMAT DSHIELD USERID 12345  TZ -05:00 [Clientname Version #]    DShield format

DShield is the preferred format.

(12345 is the UserID in this case. Please substitute your own UserID.)
TZ -05:00 would be for EST (Eastern Standard Time) Please put in your own time zone.
Determine your time zone from this list:

(Optional) ClientName Version # should be the name of the client program and its version number. This is optional, but, if included, this helps us to debug any problems. Do not include the [] brace characters--they are to indicate that this is optional.

Time Zone Offset

  • The default time zone is 'GMT' (Greenwich Mean Time.)
  • If you use a different time zone, indicate it's offset from GMT in the subject line after the letters 'TZ'. Eg.: TZ +01:00 for Central Europe, or TZ -05:00 for Eastern Standard Time (US). Don't forget to account for Daylight Savings Time. e.g., TZ -04:00 for Eastern Daylight Time (US).
  • ZoneAlarm logs will add the time zone information to each line of the log. This is ok, you don't need the 'TZ' part of the subject line.
  • For the 'DSHIELD' format, you may add time and time zone to the 'date' field, separated by a space. e.g. right now:
    2000-12-23 20:58:01 -05:00

DSHIELD Format

The DSHIELD format is a tab delimited format containing the following items (in this order):

  1. Date (in YYYY-MM-DD HH24:MI:SS Z format. Z is time zone. )
    i.e., 11:10:05 pm (Eastern Standard Time) on January 24, 2002 would be
    2002-01-24 23:10:05 -05:00
  2. Author (UserID)
  3. Count (number, used to summarize identical records, default=1)
  4. Source IP address (in 1.2.3.4 format or IPv6 without any abbreviations)
    Important for IPv6: Do not abbreviate. Instead of 2001:db8::1 use 2001:0db8:0000:0000:0000:0000:0000:0001
  5. Source port
  6. Target IP address (just like source IP)
  7. Target port
  8. Protocol (either number or text like 'TCP','UDP'...). If the protocol can't be determined, put "???" in this field.
  9. TCP flags. (Optional field.) Either a single letter:
    S - SYN
    A - ACK
    F - FIN
    U - URG
    R - RES
    P - PSH
    etc. e.g., SA if the SYN and ACK flags are set. Or you can use the common three letter abbreviations separated by a comma. e.g., SYN,ACK. Use whichever format that the firewall log that you are converting uses. If it uses single letter flags, then pass single letter flags. If it uses 3 (or 4) letter codes, then pass them. Pass on whatever flags are in the firewall log, even if they aren't specifically listed, above.

Each field must be delimited by a tab (ASCII value 09) character.

ICMP: use the source port field to log the ICMP type and the target port field to log the ICMP code (subtype).

IGMP: only the IP header is logged. Keep all other fields at 0 or use '???'.

Obsfuscation If you feel the need to obsfuscate (hide) the Target IP address, please choose from one of these methods:

  • Partial Change the first quad of the address to "10." Example. If the target IP you want to obsfuscate is 123.456.789.012, then obsfuscate it by changeing it to 10.456.789.012
  • Complete Change the Target IP to '10.0.0.1'.

The partial method is preferred, because is still provides more information on the extent of accesses.

Note that if you use either method of obsfuscating the target IP, then we will not send any FightBack abuse reports on your behalf, because ISPs require valid target IPs in abuse reports.

Examples

An example of a log line in the DShield format:

        2001-03-03 17:27:01 -05:00 <TAB> 12345 <TAB> 2 <TAB>
        204.xxx.xxx.xxx <TAB> 4312 <TAB> 24.xxx.xxx.xxx <TAB> 27374
        <TAB> TCP <TAB> SYN
        

( <TAB> is ASCII char 9. The tab character does not need to have spaces before and after it.) This assumes your timezone is GMT - 5 hrs (EST) and your USERID is 12345.

The subject line of your e-mail would look like:

        FORMAT DSHIELD USERID 12345 TZ -05:00 Your_Client_Name Version #
        

Here is a more "tokenized" example. (Shown word-wrapped for clarity. Each log line must be on a single line!):

        Date Time TZ <TAB> USERID <TAB> Count <TAB> Source IP
        <TAB> Source Port <TAB> Target IP <TAB> Target Port
        <TAB> Protocol <TAB> Flags
        

RegEx to parse KERNEL style logs

        sub parse_line {
            my $line;
            my ($month,$day,$date,$action,$proto,$source,$sourceport,$target,$targetport);
            ($month,$day,$action,$proto,$source,$sourceport,$target,$targetport) =
                /(\w+) +(\d+) \d\d:\d\d:\d\d \w+ kernel: Packet log: \w+ (\w+) \w+ PROTO=(\d+) (\d+\.\d+\.\d+.\d+):(\d+) (\d+\.\d+\.\d+.\d+):(\d+)/i;
        #        Month  day      HH:MM:SS hostname                chain act. if      protocol     sourceip     s.port     targetip       t.port
            $month=$Months{uc($month)};
            $proto=getprotobyname($proto) unless $proto=~/\d/;
            if ( $month && $day && $source && $sourceport && $target && $targetport && $proto && ($action ne 'ACCEPT')  ) {
                $line= "$year-$month-$day\t$author\t%%COUNT%%\t$source\t$sourceport\t$target\t$targetport\t$proto\n";
            }
            return $line;
        };
        

Still have questions? Contact Us!