Collecting IOCs from IMAP Folder

Published: 2020-04-30. Last Updated: 2020-04-30 05:41:48 UTC
by Xavier Mertens (Version: 1)

I've plenty of subscriptions to "cyber security" mailing lists that generate a lot of traffic. Even if we try to get rid of emails, that's a fact: email remains a key communication channel. Some mailing lists posts contain interesting indicators of compromize. So, I searched for a nice way to extract them in an automated way (and to correlate them with other data). I did not find a solution ready to use that matched my requirements:

Connect to any mailbox (preferably via IMAP)
Produce data easy to process (JSON)
Be easy to deploy (Docker)

So, I built my own Docker image... It is based on the following components:

procmail
getmail
some Python libraries
The project es_mail_intel[1]

The last tool is an old project that achieves exactly why I expect: It extracts IOCs from emails and stores them in ElasticSearch. But, if you don't want ElasticSearch, it can also produce a JSON file! Parsing emails is a pain! So, I did not want to write my own parser.

Data are processed in this way: Emails are fetched via IMAP at regular intervals by getmail and pushed to procmail. It pushes them to the Python script that extracts interesting data.

IMAP data >> getmail >> procmail >> mail_parser2json_extract.py >> JSON data

Here is my Dockerfile:

FROM ubuntu:18.04
MAINTAINER Xavier Mertens <xavier@rootshell.be>
RUN apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \
        ca-certificates \
        getmail \
        procmail \
        git \
        python \
        python-ipaddress \
        python-pdfminer \
        python-elasticsearch \
        python-xlrd \
        && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*
RUN mkdir -p /root/.getmail
RUN git clone https://github.com/clverhack/es_email_intel.git /opt/es_email_intel
COPY getmail.conf /
RUN echo ":0" >>/procmailrc
RUN echo "|/opt/es_email_intel/mail_parser2json_extract.py 2 >>/log/ioc.json" >>/procmailrc
COPY run.sh /
RUN chmod u+x /run.sh
RUN touch /tmp/firstboot
CMD ["/run.sh"]

It needs a getmail.conf with the parameters of the mailbox you'd like to monitor:

[retriever]
type = SimpleIMAPSSLRetriever
server = CONF_SERVER
username = CONF_LOGIN
password = CONF_PASSWORD

[destination]
type = MDA_external
path = /usr/bin/procmail
user = getmail
group = getmail
arguments = ('/procmailrc', )

[options]
verbose=1
read_all=false
delete=false
delete_after=0
delete_bigger_than=0
max_bytes_per_session=0
max_message_size=0
max_messages_per_session=0
delivered_to=false
received=false
message_log=/log/getmail.log
message_log_syslog=false
message_log_verbose=true

getmail is a very powerful tool with plenty of options. Just have a look at the documentation[2] to find your best way to interact with your mailboxes. The script 'run.sh' will be executed by the container and, at first boot, configure your credentials:

#!/bin/bash
if [ -r /tmp/firstboot ]; then
        sed -i "s|CONF_SERVER|$IMAP_SERVER|g" /getmail.conf
        sed -i "s|CONF_LOGIN|$IMAP_USER|g" /getmail.conf
        sed -i "s|CONF_PASSWORD|$IMAP_PASS|g" /getmail.conf

        groupadd getmail
        useradd -u $UID -g getmail -d /home/getmail getmail
        mkdir /home/getmail && chown getmail:getmail /home/getmail
        test -d /log || mkdir /log
        touch /log/getmail.log /log/ioc.json
        chown -R root:getmail /log
        chmod -R g+w /log
        rm /tmp/firstboot
fi
while true
do
        /usr/bin/getmail -r /getmail.conf
        sleep $IMAP_WAIT
done

And, finally, my docker-compose.yml file:

version: '3'
services:
    iocollector:
        build: .
        image: "xme/iocollector"
        restart: always
        hostname: iocollector
        container_name: iocollector
        volumes:
            - /etc/localtime:/etc/localtime:ro
            - /data/iocollector/log:/log
        environment:
            - UID=1000
            - IMAP_SERVER=<server_ip_or_fqdn>
            - IMAP_USER=<username>
            - IMAP_PASS=<password>
            - IMAP_WAIT=30
        network_mode: bridge

Start your docker and it will populate the mapped /log directory with an 'ioc.json' file:

{
    "bitcoin_wallet": [
        "1ncgvmeqno77x5pqbp7yi4779si",
        "3k04tl9upoapvqrcbntcsnglvcr5yn",
        "366vraic2k6ocpol7q9ck8vs2o5by0dp",
        "1fgtytuwedagp1krbukpd1dkkpfs",
        "3pmusoxmgn1qmrzedtxsqbua6dua6t",
        "34pg4dq162op34bd1aospshvsyxwhsxyo",
        "38wvzcnz49gvuwo29iuf8vxyssvmfi66w",
        "1nvgzipjvq5hdd16iszjdx9jzvf3",
        "1igrmo3w477lnb4e3tsz3gfdrciwo",
        "1yenq1jfsx54ptvvdtbplcqumu2dz3we",
        "3wusjglqxkkro3nfzd1mkwx1qqf",
        "1tjkoizmzrof4kgkok3jciwysmrxprj7"
    ],
    "ctime": "Thu Mar  5 17:54:23 2020",
    "domain": [
        "xpressvpngoodforpsdgo.duckdns.org",
        "windowsfirewallsecurityauthorise.duckdns.org",
        "cartonal.net",
        "asco.az",
        "duckdns.org",
        "dnverificationfailedbymicrosoftthanks.duckdns.org",
        "saranadinamika.co.id",
        ...
        "windowsdefenderserversecuresofficeiq.duckdns.org",
        "tecnol2.com",
        "synergiespakistan.com",
        "chxsndy3manufacturingandinsurancebusines.duckdns.org",
        "ecomglobalpurchasecentercom.duckdns.org",
        "globalsharesecurefilesgood.duckdns.org",
        "ewrt55.com",
        "matriximportsusa.com",
        "pstv-energy.com",
        "msofficecloudtransferfileprotocolsys.duckdns.org",
        "workshop002.duckdns.org",
        "globalfbdnsaddressgoogle.duckdns.org",
        "www.virustotal.com",
        "marroiq.com",
        "challengerevertprocessupdate.duckdns.org",
        "windowsdefenderserversecureserver.duckdns.org",
        "windowsdefenderserversecuresofficew.duckdns.org"
    ],
    "email": [
        "1042@home7.solarlog-web.de",
        ...
    ],
    "epoch": "1583427263",
    "filename": [
        "64443.exe",
        "soa.xlsx",
        "copy.xlsx",
        "p.o.xlsx",
        "agenttesla.xlsx",
        "vbc.exe",
        "products.xlsx",
        "quote.pdf"
    ],
    "ipv4": [
        ...
    ],
    "md5": [],
    "message_text": "...",
    "mutex": [],
    "sha1": [],
    "sha256": [
        "3a0965bd363e46e8c30590d7ddfd22bfc353728c6311c3ca61479da6f874bd5f",
        "f3a98da7e0a2c7cfe0907f5b0b769a4ff3bad45d7dce7ed905d6c3c2bf835201",
        "cb01d834237a37d3825f4075961cf6eda5f856af13526d699e4f5c93edf00c15",
        "2ef88915bb08f6d0d4c5fb2a90777930ad4e0d0b757e4b2b51988813338470f9",
        "df852235358fe361ba0577ad4c064b642148b2ac52d5857316d20bca33ea3d04",
        "172542fe3f7aae11deaf6960652d0074a1a7d83ae5596951ad69ba151a9f9630"
    ],
    "ssdeep": [],
    "url": [
       "windowsdefenderserversecureserver.duckdns.org/mich/vbc.exe",
        "globalsharesecurefilesgood.duckdns.org/love/vbc.exe",
        "windowsfirewallsecurityauthorise.duckdns.org/big/vbc.exe",
        "workshop002.duckdns.org/vbc.exe",
        "windowsdefenderserversecuresofficew.duckdns.org/windows/vbc.exe",
        "dnverificationfailedbymicrosoftthanks.duckdns.org/lee/vbc.exe",
        "dnverificationfailedbymicrosoftthanks.duckdns.org/lavin/vbc.exe",
        "ecomglobalpurchasecentercom.duckdns.org/love/vbc.exe",
        "windowsdefenderserversecuresofficew.duckdns.org/reg/vbc.exe",
        ...
        "globalfbdnsaddressgoogle.duckdns.org/lvc/vbc.exe"
    ]
}

Note: The complete is email is parsed. You will find in the JSON file all SMTP headers, the email body, etc. Less relevant for IOC's but still interesting in some cases (by example, to analyze spam).

Here is a recap of the data flow: