Getting Data In

Filter Log Data on Matches

kwl33181
New Member

Is it possible to filter log data by matching values when filtering with a heavy forwarder. I have multiple universal forwarders sending modsecurity, and apache error logs to a heavy forwarder and then onto an indexer. Is it possible to filter the apache access logs at the heavy forwarder so that only the access log events that match and error or modsecurity log entry would be forwarded on to the indexer. Sadly The log volume of the access logs them self are too large to forward and index so I need a way to filter to a subset of useful information.

0 Karma
1 Solution

jbsplunk
Splunk Employee
Splunk Employee

Yes, it is possible. You can route data to what is called the 'nullQueue'

http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad

start with this:

You can eliminate unwanted data by routing it to nullQueue, Splunk's /dev/null equivalent. When you filter out data in this way, the filtered data is not forwarded or added to the Splunk index at all, and doesn't count toward your indexing volume.

View solution in original post

yongly
Path Finder

If you want to just filter the access logs from your apache server, then you can do so by filtering on a term or regular expression just like jbsplunk has specified.

I have a similar set up with a webserver sending ssl_access_log to an intermediary heavy forwarder which then filters and only forwards events with a string of /WPNotification to the indexer.

Not sure what other kind of filtering you might be wanting to do??

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

Yes, it is possible. You can route data to what is called the 'nullQueue'

http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad

start with this:

You can eliminate unwanted data by routing it to nullQueue, Splunk's /dev/null equivalent. When you filter out data in this way, the filtered data is not forwarded or added to the Splunk index at all, and doesn't count toward your indexing volume.

View solution in original post

jbsplunk
Splunk Employee
Splunk Employee

No, you can't make the regex based on some already indexed data. I don't think you can selectively filter in the manner you are describing.

0 Karma

kwl33181
New Member

Correct, but can you make a regex based on other incoming data? Thought process is when a modsecurity event trips for value X then I want the accesslogs associated with X. But X changes dynamically based on the various errorlog or modsecurity log events. I think you have me going the right direction, but I am missing the triggering or matching(to dynamic value) piece.

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

As long as you can come up with a regex to match on, you should be able to use this method. If you want to tell Splunk to only look at particular files, you can use blacklist/whitelist in the input stanza.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Whitelistorblacklistspecificincomingdata

0 Karma

kwl33181
New Member

This is great for limiting what gets forwarded, but how do I use the errorlog or modseclog events as the matching parameter before null routing? Even a method to forward a subset of events based on uri or time stamp would be useful.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!