Adaptive filtering/Stateful cross-event logic to filter unwanted events

[fatsug]

When collecting Linux logs using a Universal Forwarder we are collecting a lot of unnecessary audit log from cronjobs collecting server status and other information. In particular the I/O from a little script.

To build a grep based logic on a Heavy Forwarder there would have to be a long list of very particular "grep" strings not to loose ALL grep attempts. In a similar manner, commands like 'uname' and 'id' are even harder to filter out.

The logic needed o reliably filter out only I/O generated by the script would be to find events with comm="script-name", get the pid value from that initial event and drop all events for the next say 10 seconds with a ppid that matches the pid.

To make things complicated there is no control over the log/files on the endpoints, only what the universal forwarder is able to do then the heavy forwarder before the log in indexed.

Is there any way to accomplish this kind of adaptive filtering/stateful cross-event logic in transit and under these conditions? Is this something that may be possible using the new and shiny Splunk Edge Processor once it is generally available?

[bowesmana · Accepted solution]

EP will not do that cross event analysis, at least not in its current form, but I would imagine that aggregations, and hence the ability to handle event relationships, is something that will come.

Your comment 'a lot of unnecessary audit logs from a little script' makes me wonder if your little script could be pruned to be even smaller 😀

[fatsug]

That's a shame, but reasonable I suppose. This feature should be resource heavy as it would need to keep a bunch of data in memory running "searches" with a high frequency, but it would have been a nice fix for my problem.

'pruned' is an understatement. Not to give it all away (in case someone happens to google their way here) but I've repeatedly pointed out that "you do not need to check if your running on a virtual machine/VPS once every hour, every day, forever". Especially if you are the service provider and know that you will never ever ever ever run on "Alibaba Cloud" and most likely never use QEMU. But that's a problem related to "drop-in solutions" for metric collection which no one has any interest of optimizing and over which I have no control.

Hence, insanely detailed grep solutions and allowing some completely pointles audit log to trickle in is just the way it has to be for now.

Thank you for your feedback, much appreciated.

[bowesmana]

Splunk got rid of DSP which did that and I'm aware that the aggregation features of DSP is something that EP is hopefully going to support at some point - note that Cribl could do this if you really wanted to go that route although it would entail another tech stack if you don't already use it.

[fatsug]

Ah, yes. Crible. Looked at their product earlier for other issues. Seems like a good product and solution but as you point out, another product to manage.

Supposedly a logstash in front of our heavy forwarders could also do the trick. But that also includes another product and I assume that log shipping would then also become HEC traffic and in my (very limited but still) experience also mean having to deal with a new log format and sourcetype for a standard Linux audit log.

[PickleRick]

Dunno about Edge Processor but the general answer to any "inter-event" question regarding forwarding/filtering is "no". Splunk processes each event independently and doesn't carry any state from one event to another (which makes sense when you realize that two subsequent events from the same source can be forwarded to two different receivers and end up on two completely different indexers).