I hope this finds you all safe, healthy, and sheltered to the best of your ability.
SpectX architecture differs from other log analyzers in that it queries raw data without indexing directly from storage. SpectX runs on Windows, Linux or OSX, in the cloud, or an offline on-prem server. The Desktop (community) version is limited to four cores max, and 300 queries a day via the SpectX API. That said, this is more than enough juice to get a lot of work done. The team, in Tallinn, Estonia, including Liisa and Raido, has been deeply engaging and helpful offering lots of insight and use case examples that I’ll share here.
Again, note the file protocol, you can call logs from so many sources, as referred to in Input Data. You can also create a local datastore that contains all the related log files, I do so as seen in Figure 1. Figure 1: HolisticInfoSec Logs Datastore You will assuredly want to familiarize yourself with pattern development, the Pattern Development Guide is there to help you. I built a simple pattern to parse my logs for 2019 as follows
and saved it as apacheLog.sxp in user/patterns/apache. You can save patterns and queries to the default SpectX home directory. On a Windows system the default path is pretty brutal, mine is C:\Users\rmcree\AppData\Local\SpectX\SpectXDesktop\data\user\rmcree. Save there, and files are readily available in the menu tree under user. Again, I chose to store all my patterns and queries in subdirectories of my C:\logs directory as called by the HolisticInfoSecLogs datastore referred to above, resulting in much file path ease of use with a simple datastore call. I initially followed the first example provided in Analyzing Activity From Blocklisted IP addresses and TOR Exit Nodes, and received a couple of hits against my logs but wanted to experiment further. While this is a good example, and you should play with it to become accustomed with SpectX, the case study is dated as the US Cert Grizzly Steppe advisory and provided CSV data are circa 2016. IP addresses are a questionable indicator even in real-time, definitely not from three and a half years ago. As such I chose to call real-time data from Darklist.de, an IP blocklist that uses multiple sensors to identify network attacks and spam incidents. They provide a raw IP list for which I built a view as follows
and saved it as Darklist_View.sx. This view is then incorporated into a query that cross-references all twelve of my monthly logs with the real-time raw IP blocklist from Darklist, then match only records with blocklist IP addresses in those logs.
A quick walkthrough of the three query phases. Figure 2: Darklist blocklist results For a free desktop client built on Java, I have to say, I’m pretty impressed. Query times are quite snappy under those otherwise adverse operating conditions. ;-) My simple query to parse the entire year of logs for my website (not high traffic) returned 1,605,666 rows in 1.575 seconds. Matching all those rows against the real-time Darklist blocklist took only 3.513 seconds as seen in Figure 2. Also, hello Russia! Figure 3: Clustermap of blocklist traffic origins Two modes are available in the Map: Cluster and Heatmap. You see a cluster map in Figure 3. The most blocklist traffic originates from AS60729 in Germany. This is furher illuminated courtesy of the heatmap view, as seen in Figure 4. Figure 4: Heatmap of blocklist traffic origins I’ve posted my queries and pattern, in a GitHub repo (SpectX4DFIR) so you may follow along at home. The logs are stored for you on my OneDrive. Read as much of the SpectX documentation as you can consume, then experiment at will. Let me know how it goes, I’d love to hear from you regarding successful DFIR analyses and hunts with SpectX. This is another case of a tip-of-the-iceberg review, there is a LOT of horsepower in this unassuming desktop client version of SpectX. Thanks to the SpectX team for reaching out, and for building what appears to be a really solid log parser and query engine. Cheers…until next time. |
Russ McRee 198 Posts ISC Handler Apr 21st 2020 |
Thread locked Subscribe |
Apr 21st 2020 9 months ago |
Sign Up for Free or Log In to start participating in the conversation!