topic Re: How do I exclude log from sending to Splunk to save quota? in Getting Data In

How do I exclude log from sending to Splunk to save quota?

hakusama1024 — Thu, 30 Aug 2018 18:56:31 GMT

Hi guys.

I have daily quota for 3G. but the log is too much.
So, I'm trying to exclude some logs, like heart beat, to send to Splunk to save some usage.
I'm trying to use Splunk Filter Rules:
-> Exclude Patterns

Some keywords I clicked exclude.
But, i still am able to see these words when i search on Splunk.
Can anyone help? Thanks.

Re: How do I exclude log from sending to Splunk to save quota?

sudosplunk — Thu, 30 Aug 2018 19:25:41 GMT

Hello,

The heart beat messages - Are you referring to messages from one splunk component to other? Because, splunk licensing doesn't count towards _internal logs. If not, please provide some sample events.

Re: How do I exclude log from sending to Splunk to save quota?

horsefez — Thu, 30 Aug 2018 19:43:03 GMT

Hi @hakusama1024,

I'm not really sure what you mean by "Exclude Patterns", but I can tell you about two ways to filter data before it gets indexed.

Either you filter data at the source, which is the best option, because it doesn't generate additional log traffic:
If you have a Universal Forwarder installed on a Linux System for example and you want to monitor all the files in /var/log/messages/ you could try to specify what particular files out of this file system you are interested in, by splitting up your single monitoring stanza into multiple stanzas.

Or, if granular filtering at the source doesn't work you can filter at Indexer level (also at HF level).
So you could actually filter out and throw away data that is matched via regular expression and avoid it getting indexed.

I have a very good splunk answer from @lguinn here:
https://answers.splunk.com/answers/59370/filtering-events-using-nullqueue-1.html

This should help you understand how it can be done.

If you give us additional information about the logfiles you want to filter out, we can assist you further.

Re: How do I exclude log from sending to Splunk to save quota?

mstjohn_splunk — Fri, 31 Aug 2018 23:00:06 GMT

hi @hakusama1024 ,

Did @pyro_wood 's answer solve your problem? If so, please resolve this post by approving one of them. If not, keep us updated so that someone else can help solve your problem.

Also, if you're feeling generous, give out an upvote to the user that helped ya. 🙂