Getting Data In

Is it possible to install a Splunk forwarder on centralized rsyslog server to filter out unwanted events before forwarding data to our indexer?

jwalzerpitt
Motivator

We use the nxlog agent on out Windows domain controllers/Exchange servers/IIS servers and forward to a centralized rsyslog server.

Would it be possible to install the Splunk Forwarder on the centralized rsyslog server and filter the syslogs that would be forwarded from the centralized rsyslog server to our Splunk indexer to help filter out the unwanted events?

Thx

1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Yeah, provided you use a heavy forwarder instead of a universal forwarder.

Note, you can have your indexer(s) filter data regardless of where it came from.

View solution in original post

Raghav2384
Motivator

Absolutely! We have a similar setup for the HA with heart beat setup. Like martin said, Heavy Forwarder only.

jwalzerpitt
Motivator

Thx for the reply

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Yeah, provided you use a heavy forwarder instead of a universal forwarder.

Note, you can have your indexer(s) filter data regardless of where it came from.

martin_mueller
SplunkTrust
SplunkTrust

For filtering events based on field values you should take a look at the TRANSFORMS-foo entries in props.conf together with transforms.conf. Here's an example: http://docs.splunk.com/Documentation/Splunk/6.2.0/Forwarding/Routeandfilterdatad

That can be done on a Heavy Forwarder xor an Indexer.

jwalzerpitt
Motivator

Awesome - thx Martin!

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

I don't quite grasp that sentence, could you elaborate on what you're trying to do?

There's white/blacklisting on files in inputs.conf, filtering events on heavy forwarders, filtering events on indexers, routing entire files on either, ...

0 Karma

jwalzerpitt
Motivator

I believe you have answered my poorly worded question. What I mean by blacklisting is say for example I want to prevent Windows Event ID 545 from being indexed, can I create a monitor:// stanza that excludes or prevents Windows Event ID 545 from being indexed?

My fear was that I'd have to create monitor:// stanza for events I wanted to be indexed, but you stated there is blacklisting via filtering events so it appears I can filter out unneeded events so they don't get indexed.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Yup, having stuff written to a log file and then read by a forwarder with a regular monitor:// stanza is good practice.

jwalzerpitt
Motivator

Another one more question 🙂

Is it possible to set the forwarder to be a black list and exclude events/logs that I don't want to be indexed, or does it only function as a white list?

Thx again

0 Karma

Raghav2384
Motivator

Yes,HF,filter using props & transform,send the ones you do not want to nullqueue. Again, like Martin mentioned below..

0 Karma

jwalzerpitt
Motivator

Thx for the reply and info.

At some point I hope they migrate the filtering function onto the Indexer so one can start by indexing everything at first, and then allow the user the ability to go in and start selecting events to be excluded. Not sure how this would effect scalability/performance of the indexer though if it was implemented as such.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

That's there already, it's called "searching" 😛

0 Karma

jwalzerpitt
Motivator

Martin,

I wanted to expand on this question to pick your brain a little more and run through a scenario. We are planning to consume firewall syslog and we log at debugging level, which as you know, is very chatty. With that, if we have the indexer set as a listener, can we filter using transforms on the indexer, or would the firewall syslog need to go to a forwarder first to have events filtered before the data is indexed?

Thx

0 Karma

jwalzerpitt
Motivator

How true! My fears are two-fold: 1) I'm indexing data that's not needed to shine light upon connections/patterns/etc and 2) I'd be excluding events/field values that I think aren't important, but actually are

Considering using Hadoop on the backend to send all data to and then Hunk to search against the entire data set, but int he meantime re: using Splunk for real-time, proactive alerting I'd like to index only those relevant events/field values

0 Karma

jwalzerpitt
Motivator

Thx again

0 Karma

jwalzerpitt
Motivator

Martin,

I assume for the rsyslog configuration, it's ok to log to a file, in which I would then configure the forwarded to forward the rsyslog log files to the Indexer, correct?

Thx again

0 Karma

jwalzerpitt
Motivator

Thx for the reply Martin

0 Karma