Understanding how Splunk handles data/logs

[Skatman88]

Hi all,

First time posting here and it's the first time I've been playing with Splunk. Downloaded and installed on Windows 10 (which is already seeming like a mistake - Splunk's handling of Syslog isn't great from what I've seen so far - one source on UDP/514 is bonkers!). I've used a lot of different SIEM tools in my time and have been working in IT Security for a number of years now; so I have a strong understanding of the usual process for log ingestion and it can usually be split into the following categories (this isn't a 'standard', it's just how I work things out in my head):

1. Collection - How the logs are transported towards a SIEM (Syslog, Agent, API, etc.)
2. Ingestion - Accepting the log sources into a SIEM.
3. Parsing - Figuring out what logs belong to what log source types (i.e. CISCO ASA log belongs to CISCO ASA Version 'x')
4. Indexing - Putting the correct Sections of logs into the correct columns (authentication event ID into that column)
5. Association/Rules - Understanding which Events are successful logins and creating rules to link what you want to be alerted on.
6. Searching and alerting - How alerts are displayed and searching through historical data using correlation rules etc.

I understand Splunk isn't a 'SIEM' as such, so I'm not expecting to do the correlation bit just yet (probably an advanced way of achieving this, but I'm currently struggling with what I think is 3, 4, and 5.

I've managed to get some of my logs into Splunk (3 x Windows devices and 1 x , so I'm pretty happy with the Collection and Ingestion side of things, but I've downloaded and installed two separate Security 'Apps' (InfoSec App for Splunk and Splunk Security Essentials) and neither appear to be understanding the logs that are being ingested.

For instance, if I navigate to the "InfoSec App for Splunk", and just go Continuous Monitoring -> Firewalls or Network Traffic, I get absolutely nothing. See below:

However, I know that the logs are arriving because if I go to "Search & Reporting", type the hostname in, I'm getting results back:

I'm using a Sophos NGFW as my Core Firewall which has all sorts of features enabled on it (IPS, URL Filtering, DNS Alerting, QoS, etc.) Issue is that the apps don't appear to be seeing the logs which makes me thing that it's something to do with categories 3 to 5. I just don't know which one.

I've downloaded, installed, and accelerated CIM, I've installed this add-on (which I thought covered the parsing and indexing stages); which leads me to believe it could be an association/rules issue. (Apparently I can't post links. Add on is "Sophos XG Technical Add-on")

My major problem here is that I simply do not understand Splunk well enough to figure this out. So I was hoping some of you lovely people could help!

Best,

[richgalloway]

This is a big topic and I probably won't cover it completely so let's hope others contribute as well.

First, you're right about it being a mistake to run Splunk on Windows. It's fine for dev, but not for production. See https://answers.splunk.com/answers/516059/splunk-architecture-on-windows-os-why-should-i-avo.html?pa...

Second, don't send syslog data directly to Splunk. The preferred method is to send to a dedicated syslog server (Linux box running rsyslog or syslog-ng) and have a Splunk Universal Forwarder send the data to Splunk.

In Splunk, your steps 3-4 are combined (sort of) and performed by the indexer. Steps 5-6 typically are done at search time.

The InfoSec app makes extensive use of Splunk's accelerated data model feature. Data models (DM) add structure to unstructured data. Accelerated DMs search your indexes at regular intervals and store what it finds in indexes for faster access. The CIM app will make it easier to work with DMs. Don't try to accelerate all of the data models as a Windows box will suffer greatly.

A common mistake is to install an app and think you're done. It's rare for an app to work perfectly right out of the box. You will need to customize it, at least by providing index names that match where you store the data.

[gwcon]

Wasn't the splunk universal forwarder only available for the paid version ?

[Skatman88]

Hey,

Thanks for the information.

I'm only running Splunk at home, so I didn't really mind too much about it being implemented on Windows at first. I always expected it to run better on Linux, but I always have no end of grief with Linux-based OS upgrades. I find them so flaky it's unreal. I've run all sorts of Servers and Distro's at home and I've never found one to not have a problem eventually after upgrades/updates.

However the thing that's most likely to force my hand is the incredibly poor handling of Syslog within Splunk. I've read about using a Syslog server being best-practice, so started having a look for one to run on Windows and the only "free" one I could really find was Kiwi, which is both limited to 5 Syslog log sources (I have 6) and can only segregate files based on date; rather than source IP as is required. So I find myself being forced down the Linux route, despite my initial issues. Never mind. I'm sure it'll all be fine... until I update/upgrade.

For the life of me, however, I cannot understand why Splunk's handling of Syslog is so poor. Literally every single SIEM on the market can ingest Syslog log sources with ease; only being limited by the hardware capabilities. Perhaps I'm missing the point of something as there must be a justification somewhere.

Anyway, onto the main question. How do I provide the index names to the data models? That "Sophos XG Technical Add-On" appears to have created data models for the data, but I've no idea how to make use of it.

[richgalloway]

I'm not familiar with the Sophos XG add-on, but the data model should have a way to specify index name(s). Often, there is a macro that must be updated.

[Skatman88]

OK, How do I find out about these Macros?

[richgalloway]

Go to Settings->Advanced Search->Search macros